The Limits of Metrics

For a long time now I’ve been trying to describe a hesitation I’ve had around EA.  Outcome metrics are great.  Outcome metrics are a huge improvement over “but look how much money we spent.” and “have you seen how sad this child is?“.  And yet.  My original stated concern was that over-reliance on metrics would drive us to focus on easy-to-measure outcomes over equally more* important hard-to-measure outcomes, or on known outcomes over more important unknown outcomes.*

Now I have a better analogy.  Metrics are like nutritional labeling.  Nutritional labeling is great when you want to decide between cheetos and soylent, or between soylent, mealsquares, and any one of their homebrew competitors.**  But suppose I set a fiber quota for myself.  The ideal way to do that would be to eat a variety of fruit, vegetables, beans, and nuts throughout the day, but that is super hard to keep track of.   I either have to eat in exact serving sizes (forcing the continuous variable of hunger to the granular treatment of serving size) or calculate exactly how much I ate after the fact (a pain in the ass and/or impossible), and then look up how much fiber is in the food (ignoring any natural variation), write it down, total it up… and if it’s midnight and I’m short, eat a ton more food I may not want.  Or I can pour a bunch of psyllium husks in a glass in the morning, check “eat fiber” off my todo list, and eat HoHos for the rest of the day.

Obviously the first choice is better overall, even if I ultimately end up with less fiber. But it is much harder to measure, in part because the benefits accrue over a wide variety of nutrients, whereas the psyllium and HoHos diet produces one big shiny number to trumpet in brochures.  I think this is a problem in charity too.  The Ugandan girls-club study I looked at last week had some outcomes that were both easy to measure and to value (spending), easy to measure but hard to estimate the value of (delayed marriage and childbirth), and kind of fuzzy to measure and of unclear value (age at which they do marry, as measured by proxy “when would you like to get married”).  Luckily for that project the increase in girls’ income per unit NGO spending was almost as high as it was for pure vocational training, plus it had these social benefits, but suppose it had been 75% as good?  Half as good?  10% as good?  What is the cut off for being better than pure vocational training.

I’m solving this problem in my nutritional life by drinking a full serving of vitaminized protein powder*** mixed with chia seeds every day, plus whatever the hell I feel like eating.   The almost-food frees up my stomach and brain to figure out what I especially need and seek that out, without fear I’m letting some other deficiency fester.  This is startlingly similar to Holden Karnofsky’s (co-founder of GiveWell) suggestion that westerners focus on the problems of the 3rd world they are in a good position to fix (e.g. malaria), and let the locals do the rest.   So I guess Effective Altruism has addressed this problem, it’s just that it addressed it by limiting itself, which is not the most emotionally satisfying answer but is something the world could do with more of.

BONUS FACT: EA and soylent have both found their home primarily with the rationalist community, and my rationalist friends (all of whom I met through EA) are simultaneously the most likely of anyone I know to drink soylent and to host communal dinners with secular grace.

*E.g. Food aid to the third world looks great measured by “people who stop starving in the short term.”  We know now that this destroyed the local farming economy and left entire regions either starving or in ongoing dependence on 1st world aid.

**Of these, mealsquares have been the clear winner among my friends.

***Not quite the same as soylent because it lacks the fat, carbs, and fiber to be a meal replacement.  This presents two slightly different problems.  The lack of fat and sugar I feel fully prepared to make up for in the rest of my diet.  But nutrients are digested differently depending on what other nutrients they are in proximity to.  The chia seeds are attempt to get the benefits of protein x fiber.

2 thoughts on “The Limits of Metrics

Comments are closed.