Leaflets are Ineffective, Tell Your Friends

Somehow the meme got established in Effective Altruism communities that convincing people to go vegetarian or vegan is cheap and easy, and the only question is whether doing so as a substitute for reducing consumption yourself was ethical.  Me, John, Jai, and one shy friend dug deep into the research and discovered so many problems with the design of the studies showing this that we wrote them off entirely.  I wrote up a whole blog post explaining how Animal Charity Evaluator’s analysis was wrong, and leafleting was not effective.  In an act of thoroughness I would soon be very grateful for, I went to ACE’s website to make sure I was representing the studies exactly right.  Turns out ACE made essentially the same criticisms we did, and also concluded the studies were insufficient to show any net effect from leafleting.  If you go to the slate star codex link and follow the link to his sources, one has since renounced his math and the other says that his numbers are not meant so much to be “true” as to be “motivating in their concreteness”.

I did eventually find some organizations claiming leafleting was genuinely effective.  Vegan Outreach cites Farm Sanctuary, which uses the exact study Animal Charity Evaluators criticized.  ACE doesn’t go quite as far as I would: they note the 95% confidence interval of the effect and then the systemic biases of the study, whereas I would say “if you can’t get an effect size bigger than .001 in a study so egregiously biased towards your view, there is no effect.”  But the criticisms have always been there.

It’s not like anyone’s default belief is “lots of humans can be convinced to make enormous  permanent changes by one glossy 8.5×11 piece of paper,” so how this belief become established with so little data?  How did I dig into ACE’s data deeply enough to understand the design flaws myself without noticing they saw the flaws too? (partial answer: their calculator still shows leafleting having an effect ).  Unless there’s data I don’t know about, there doesn’t appear to be any support for the idea that leafleting reduces animal suffering.  We really need to figure out how this spread in a movement dedicated to quantification so we can fix the systemic issue.

EDIT: I’ve had a couple of requests to include the specific criticisms.  I originally didn’t because it felt mean to rehash ACE’s criticisms, but since the whole point is you can read their documentation without realizing them, that reasoning seems dumb now.

Everyone knows there’s a social desirability bias (reporting converting to veg*nism when you haven’t).  This is especially an issue for the people who report reducing but not eliminating animal products- it’s easy to lie about that, to yourself or others.  But the denominator (how many people received flyers total) is also unreliable, because there are a lot of reasons people who received a pamphlet will report they didn’t when asked two months later.  Maybe they threw it out without looking at it.  Maybe they read it and forgot.  Maybe they totally remember it and realize that if they say yes to the surveyor there will be a long conversation that implies they torture animals, and they would like to not do that.

We don’t have to assume this is a problem: one of ACE’s studies attempted to use a control group, and flat out couldn’t, because no one reported receiving the control flyer.  The lack of control group is a big problem, because it means you will give flyers credit for people that would have gone veg*n anyway.

Then you have to predict how long they stay veg*n.  ACE’s numbers are outlined here, and there’s several problems.  There’s social desirability bias again, and the samples are representative neither of the population at large or the population being leafleted.  I have a strong prior that people who make changes based on a flyer are less likely to stick with them than the general population

This is minor, but ACE doesn’t count the value of the leafleters time when calculating effectiveness.  Even if they’re volunteers, you need to consider the value of what else they could be doing with that time.