Leaflets are Ineffective, Tell Your Friends

7679534638_59865527ef_oSomehow the meme got established in Effective Altruism communities that convincing people to go vegetarian or vegan is cheap and easy, and the only question is whether doing so as a substitute for reducing consumption yourself was ethical.  Me, John, Jai, and one shy friend dug deep into the research and discovered so many problems with the design of the studies showing this that we wrote them off entirely.  I wrote up a whole blog post explaining how Animal Charity Evaluator’s analysis was wrong, and leafleting was not effective.  In an act of thoroughness I would soon be very grateful for, I went to ACE’s website to make sure I was representing the studies exactly right.  Turns out ACE made essentially the same criticisms we did, and also concluded the studies were insufficient to show any net effect from leafleting.  If you go to the slate star codex link and follow the link to his sources, one has since renounced his math and the other says that his numbers are not meant so much to be “true” as to be “motivating in their concreteness”.

I did eventually find some organizations claiming leafleting was genuinely effective.  Vegan Outreach cites Farm Sanctuary, which uses the exact study Animal Charity Evaluators criticized.  ACE doesn’t go quite as far as I would: they note the 95% confidence interval of the effect and then the systemic biases of the study, whereas I would say “if you can’t get an effect size bigger than .001 in a study so egregiously biased towards your view, there is no effect.”  But the criticisms have always been there.

It’s not like anyone’s default belief is “lots of humans can be convinced to make enormous  permanent changes by one glossy 8.5×11 piece of paper,” so how this belief become established with so little data?  How did I dig into ACE’s data deeply enough to understand the design flaws myself without noticing they saw the flaws too? (partial answer: their calculator still shows leafleting having an effect ).  Unless there’s data I don’t know about, there doesn’t appear to be any support for the idea that leafleting reduces animal suffering.  We really need to figure out how this spread in a movement dedicated to quantification so we can fix the systemic issue.

EDIT: I’ve had a couple of requests to include the specific criticisms.  I originally didn’t because it felt mean to rehash ACE’s criticisms, but since the whole point is you can read their documentation without realizing them, that reasoning seems dumb now.

Everyone knows there’s a social desirability bias (reporting converting to veg*nism when you haven’t).  This is especially an issue for the people who report reducing but not eliminating animal products- it’s easy to lie about that, to yourself or others.  But the denominator (how many people received flyers total) is also unreliable, because there are a lot of reasons people who received a pamphlet will report they didn’t when asked two months later.  Maybe they threw it out without looking at it.  Maybe they read it and forgot.  Maybe they totally remember it and realize that if they say yes to the surveyor there will be a long conversation that implies they torture animals, and they would like to not do that.

We don’t have to assume this is a problem: one of ACE’s studies attempted to use a control group, and flat out couldn’t, because no one reported receiving the control flyer.  The lack of control group is a big problem, because it means you will give flyers credit for people that would have gone veg*n anyway.

Then you have to predict how long they stay veg*n.  ACE’s numbers are outlined here, and there’s several problems.  There’s social desirability bias again, and the samples are representative neither of the population at large or the population being leafleted.  I have a strong prior that people who make changes based on a flyer are less likely to stick with them than the general population

This is minor, but ACE doesn’t count the value of the leafleters time when calculating effectiveness.  Even if they’re volunteers, you need to consider the value of what else they could be doing with that time.

4 thoughts on “Leaflets are Ineffective, Tell Your Friends

  1. “I wrote up a whole blog post explaining how Animal Charity Evaluator’s analysis was wrong, and leafletting was not effective. In an act of thoroughness I would soon be very grateful for, I went to ACE’s website to make sure I was representing the studies exactly right. Turns out ACE made essentially the same criticisms we did, and also concluded the studies were insufficient to show any net effect from leafleting.”

    I can’t tell from the links exactly which criticisms your group found that were also made by ACE, and what quantitative weight you gave them.

    “How did I dig into ACE’s data deeply enough to understand the design flaws myself without noticing they saw the flaws too? (partial answer: their calculator still shows leafletting having an effect ). ”

    Their actual charity recommendations gives particular numbers (with vague disclaimers) about leaflet and facebook ad effectiveness:

    http://www.animalcharityevaluators.org/research/organizations/the-humane-league-review/#2

    “To combine these estimates into one overall cost-effectiveness estimate, we need to translate them into comparable units. This will introduce several sources for errors and imprecision, so the resulting estimate should not be taken literally.45 However, it will allow us to judge whether THL’s efforts are comparable in efficiency to other groups’.46 We use our Online Ad Impact Calculator and Leafleting Impact Calculator47 to find that THL spares about 5.5 animals from life on a factory farm per dollar spent on online ads,48 and about 2 animals per dollar spent on leafleting (using the average costs of 25 cents per view and 33 cents per leaflet49 ).”

    What does “leaflets are ineffective” mean in terms of credences in the inputs of that calculation, and you own estimates/credible intervals?

    I also find the optimistic estimates not very credible (considering outside view for advertising, that the reported effects could be primarily via increasing social desirability bias, etc), but I had a hard time identifying the specific reasoning or conclusions of your team from this post.

    Maybe you could post the draft?

Comments are closed.