Epistemic Spot Check: The Tapping Solution (Nick Ortner)

This is part of a series called epistemic spot checks, in which I investigate claims a book makes to see if it’s worth paying attention to, without attempting to be comprehensive about it.

Introduction

This is a weird review to write.  I went into reading The Tapping Solution with two beliefs:

  1. The scientific claims would be far less supported than the author implies.  The best case scenario was “as terrible as your average therapy research.”
  2. The book’s prescriptions work for me anyway, in the sense that they make me calmer and happier and enable me to take better actions.

This book is about EFT, which stands for emotional freedom technique. I write that in a very small font in the hopes you won’t notice how stupid it sounds.  EFT is also known as tapping, because the primary action is tapping your fingers against your face.

I originally learned about EFT in a book that went full blown magic about it: you tap your fingers on your face, it changes energy currents in your body, and the universe magically gives you what you want.  There’s no point evaluating the science in books like that; they are what they are.   The Tapping Solution markets itself as the more studious cousin of that book.  It keeps the energy channels but backs off the magic gifts claim, offering the much more defensible explanation that tapping changes something in you that lets you create better outcomes.

The basic idea of EFT is you tap out a pattern on your body, mostly your face, while repeating a statement about something with a lot of negative emotional affect for you, especially ones that activate the sympathetic nervous system (fight/flight/freeze).  Repeat until you feel better.

[There’s a lot of different techniques claiming to be The Best EFT Script and, while I suspect there are individual variations in what works best for each person, I can’t possibly care about the intra-EFT wars.  Any script you use should just be a starting point for making your own anyway.]

Why would tapping improve your mood?  I have some guesses:

  • It makes anxiety et al. boring.  There are a lot of activities where people deliberately activate their SNS (sky diving, horror movies, drugs), so there must be something fun rewarding about being activated.    Plus, lots of the things that happen to you in response to anxiety are quite pleasant.  People cuddle you and bring you ice cream.  You put off doing the stressful thing.  I don’t think many people deliberately push themselves into hysterics for the attention, but I do think these benefits bias how people handle their stress.  Tapping does not offer those kinds of rewards; after two or three rounds of tapping, you are bored.  There are times I have gone and done the stressful thing because I would rather deal with it than have to do another round of tapping.  It’s nice to have my intolerance for boredom harnessed for good.
    • I suspect this is some of how cognitive behavioral therapy works as well.  Having taught myself both, EFT is less work and yet harder to develop an immunity too, although hybrid systems do better still.
  • A sense of control lowers stress.  Having A Thing You Can Do While Stressed that you think lowers your stress level is already lowering your stress level.  You can dismiss this as a self-fulfilling prophecy, but that’s only the point if you’re actually evaluating the concept of energy meridians.  If what you want is to calm down so you can respond to comments on your code review, it doesn’t matter if it’s a placebo.
  • Something something vagus nerve.  The vagus nerve is this weird nerve that skips the spinal cord and runs all over your body, including most major organs and a lot of your face.
    • Its tasks include:
      • Parasympathetic (relaxing) stimulation of all major organs except the adrenal glands.
      • Parasympathetic stimulation of muscles around the mouth and larynx.
      • Possibly reduces systemic inflammation
      • Sympathetic (fight/flight/freeze) stimulation of blood vessels.
      • A bunch of sensory stuff around the face.
    • Activity on your face is already known to affect your body via the vagus nerve.
      • Cold water on the face slows down your heart, and this is attributed mainly to the vagus nerve.
      •  Direct electrical stimulation of the nerve is touted as a cure for all kinds of stuff.  My sense is the science on that is… optimistic, but there is a reason it is being done to the vagus nerve and not something else.
    • There’s an alternate EFT script that involves tapping only on the hands.  I have fond this to be a calming distraction at best.  Hands are also pretty innervated, so this points to the effects being due to something specifically in the face, as opposed to sensitivity in general.
    • I yawn *a lot* while tapping.  Heart problems can cause yawning via the vagus nerve. I’m obviously not damaging my heart by tapping, I mention this just to show that the vagus nerve and yawning are related.
    • So I don’t know what’s going on, but I suspect the effect of tapping is mediated via the vagus nerve.
  • It’s a framework for breaking your problem into bite sized chunks, which is the ideal size for problems to be.  EFT practices vary in how much you work off a verbal script you’re given, vs introspect on your own issues and tap on what comes up.  I predict script-style work to be at best competitive with relaxation exercises, and only introspective EFT leads to actual improvements.
  • Who knows, maybe energetic meridians are a real thing, or at least a workable metaphor for a real thing.  Lots of things sound stupid until you know how they work.
    • In particular, if you mixed up the explanations for EFT and the much more legitimate EMDR (deliberate eye movements rewiring your brain), I’m not convinced anyone could tell which one was the Officially Sanctioned Therapy and which was the crackpot treatment.
    • Mark at meditationstuff.com argues that what gets sensed as energy flow is severe awareness of your own nervous system.  He provides no compelling evidence for this, but it is interesting.
    • Many Properly Credentialed Authorities believe things that are no weirder.

 

How I evaluated this book: usually when doing these checks I evaluate any statement I find interesting.  In this case, I’m sticking to the ones for which the author explicitly claims scientific backing.  For stuff that is essentially running on placebos and metaphors, I find a calm, confident, made up explanation is better than a hedged, hesitant, literally true one, so I’m not going to investigate the obviously exaggerated claims.  But if you’re going to claim scientific validity, I am going to check.

Claim: “The amygdala is the source of emotions and long term memories, and it’s where negative experiences are encoded (p4)”.

True.  Simplified, but obviously trying to explain how the amygdala was relevant to a particular concept, not give a comprehensive overview of our friend the amygdala.  The amygdala is in fact so good at emotional memory that it can be invoked by visual cues even in people blinded by brain damage.  This confused me at first, so let me note that the amygdala is not involved in fight/flight/freeze, but the longer, cortisol-driven chronic kind of stress.

Claim: Stimulating acupoints calms down the amygdala, and this is observable in fMRI and PET machines (p5).

Misleading, either bad faith or credulous.  Both studies cited were done with acupuncture, not acupressure or tapping.  I consider that relevant evidence for EFT, but dislike that he tried to make it even stronger evidence by hiding that both studies involved needles.  The effectiveness of acupuncture appears to have large if weak support; I very quickly pulled up many more studies demonstrating the exact same thing, all of which were tiny (the largest was 18), and used fMRIs, which are suspect.

In general, studies of acupuncture have shown that it kind of works, but Official Legitimate Chinese Medicine Points don’t do any better than a random spot, so this adds more legitimacy to randomly stabbing yourself than it does to meridian points.

Claim: Other studies show that pressure works just as well for stabbing, maybe even better for anxiety (p5).

Seems legit.  I didn’t find any citation for this but I’m willing to spot him that touching works better than stabbing for anxiety.

Claim: A study demonstrated that EFT reduces cortisol levels in the saliva (p5).

True, evidence weak but better than I guessed.  The study cited is real, and with some effort I even found a full PDF.  EFT did better than both a support group and no treatment on both a symptoms assessment and cortisol levels (24% decrease vs 14%).  The differences in symptoms between EFT and the other groups are small, and some were not statistically significant.  OTOH, every one of them goes in the same direction.  I find this pretty compelling, assuming they published every trait they recorded.  As usual, small study, vulnerable to p-hacking, etc.

Claim: This John Hopkins approved doctor agrees with us (p7).

Misleading, possibly very.  The named person (David Friedman) does exist, but he’s a doctor of psychology, not psychiatry.  The level that JHU approves of him is unclear.  On his CV (PDF) he lists himself as “research associate”, “instructor”, and “faculty.”  None of these words are “professor”, which makes me think he was an adjunct and certainly didn’t have tenure.

Claim: Competing systems telling you to never think about the negative are idiotic.  True things are true (p8).  In particular The Secret is bullshit.

Seems legit.  “Make bad things approachable”  just seems like a better tactic than wishing really hard. I also enjoy watching different alt modalities fight with each other.

Claim: Meridians have been scientifically validated, they’re called Bonghan channels (p10).

False.  The official name of Bonghan channels is the primo-vascular system, and there’s minimal evidence it exists.  Given that it’s pretty hard to prove that there’s a link between them and meridians in any scientific sense.  But it’s established fact within the meridian community, so it’s at least well sourced bullshit.

 

A few more notes on The Tapping Solution.

As expected, Tapping Solution has failed the RCT test.  What about the model test?

Well, it’s a fairly vague model, and energy meridians can be used to power anything.  On the other hand it avoids my biggest complaint about heal-yourself-with-the-placebo-effect books, and also religion, certain parts of medicine, and psychology, which is that the solution to failure is often do the same thing harder.  Tapping by and large avoids that trap.  For actual physical problems you’re encouraged to see a doctor first, then tap, and if that doesn’t work see a doctor again.  If a particular tap isn’t working you’re given alternate prompts to try.  Additionally, tapping claims that often it will work so well you’ll forget you will ever upset about something, and the solution is not to hand over money to the nice man to keep the good vibes flowing, it’s to keep track of how upset you are at the beginning of the session.  That level of empiricism shouldn’t make a book stand out, but it does.  Tapping Solution, although not every book on EFT, is also pretty clear that you’re not imposing your will on the universe, you’re calming down so you can take better actions.

I don’t want to write out instructions for tapping because I believe the process of reading a book adds a lot of value over a quick run through (the same way doing yoga is better for you than waving a magic wand and becoming more flexible).  But to help you decide if even starting the book is worth your time, here are some genres of problems I think tapping is most appropriate for:

  • Somaticizations, especially back pain.
  • Emotions you find too overwhelming to deal with, especially anxiety.
  • Legit life problems that are just too big to deal with all at once and need to be broken into bite size pieces.

Model

Simplicity: very low.  “Magical energy currents” sounds simple in that you can explain it quickly, but it takes a very long time to explain what things it can’t do and why.

Explanation quality: poor.  Merdians can power anything.

Explicit predictions: okay. You have to make your own explicit predictions, but the book very much encourages you to do so.

Acknowledging limitations: mixed.

Relative to other heal-yourself-with-the-placebo-effect systems, The Tapping Solution is modest in its claims about what your mind can do.  It goes out of its way to establish that the mind-body connection is in fact a connection, it doesn’t mean your body is a hallucination you can will into whatever form you want.

GemasAugust2015.png
Lesbian space rocks whose bodies are solid holograms  are not representative study subjects.

And then on the next page there’s a story of how a woman cured her lung cancer with EFT.  So it’s not amazing on this axis.

Measurability: extremely good.  This is where EFT really shines.  They claim it’s such a good technique you will forget you ever had a problem, and encourage you to keep track so you won’t forget.

 

Empirics

I’m deliberately not giving a lot of details on how to do it yourself, because I think there might be value to going through the book beyond the technique.

I taught this technique to five people, one of whom had a good response to it. Counting myself, that’s 1/3 successes, which is not great. But it’s cheap enough and has high enough potential I still recommend trying it.

 

Product Endorsement: Secret Stuff Chalk Cream

This is strictly for the climbers and other friction-ey athletes in the audience.

Chalk Cream is a lotion you rub on your hands, then air dry so that the alcohol evaporates and the chalk remains. I did not realize how amazing it was until I tried regular chalk. Regular chalk comes off in maybe three holds. Chalk Cream is still present at the top of very tall belaying routes. Yes, it is harder to reapply, but you need to do so so much less often.

I have no idea why this stuff hasn’t gone to fixation because it’s eons better and not that much more expensive than traditional chalk- possibly cheaper, if you go by dollar per time with chalk covered hands.

Epistemic Spot Check: Full Catastrophe Living (Jon Kabat-Zinn)

Full Catastrophe Living is a little weird, because between the first edition and the second a lot of science came out testing the thesis.  For this blog post, I’m reviewing the new, scienced-up edition of FCL.  However I have ordered the older edition of the book (thanks, Patreon supporters and half.com) and have dreams of reviewing that separately, with an eye towards identifying what could have predicted the experimental outcome.  E.g. if the experimental outcome is positive, was there something special about the model that we could recognize in other self-help books before rigorous science comes in?

I originally planned on fact checking two chapters, the scientific introduction and one of the explanatory chapters.  Doing the intro was exhausting and demonstrated a consistent pattern of “basically correct, from a small sample size, finding exaggerated”, so I skipped the second chapter of fact checking. I also skipped the latter two thirds of the book.

Overview

You’ve probably heard about mindfulness, but just in case: mindfulness is a meditation practice that involves being present and not holding on to thoughts, originally created within Buddhism.  Mindfulness Based Stress Reduction is a specific class created by the author of this book, Jon Kabat-Zinn.  The class has since spread across the country; he cites 720 programs in the introduction.   Full Catastrophe Living contains both a playbook for teaching the class to yourself, the science of why it works (I’m guessing this is new?), a section on stress, and followup information on how to integrate meditation into your life.

Introduction

Claim: Humans are happier when they focus on what they are doing than when they let their mind wander, which is 50% of the time.

Accurately cited, large effect size, possible confounding effects. (PDF).  The slope of the regression between mind wandering and mind not-wandering was 8.79 out of a 100 point scale, and the difference between unpleasant mind wandering and any mind not-wandering task was ~30 points.  Pleasant mind wandering was exactly as pleasant as focusing on the task at hand.  Focusing accounting for 17.7% of the between-person variation in happiness, compared to 3.2% from choice of task.

Some caveats:

  • People’s minds are more likely to wander when they’re doing something unpleasant, and when they are having trouble coping with that unpleasantness.   The study could be identifying a symptom rather than a cause.
  • The study population was extremely unrepresentative, consisting of people who chose to download an iPhone app.

Claim: Loss of telomeres is associated with stress and aging; meditation lengthens telomeres by reducing stress (location 404).

Research slightly more theoretical than is represented, but theoretical case is strong. (Source). First, let’s talk about telomeres.  Telomeres are caps on the ends of all of your chromosomes.  Because of the way DNA is copied, they will shorten a bit on every division.  There’s a special enzyme to re-lengthen them (telomerase), but leading thought right now is that stress inhibits it.  Short telomeres are associated with the diseases of aging (heart issues, type two diabetes) independent of chronological age.  This is hard to study because telomere length is a function of your entire life, not the last week, but is pretty established science at this point.

Mindfulness reduces stress, so it’s not implausible that it could lengthen telomeres and thus reduce aging.  The authors also present some evidence that negative mood reduces the activity of telomerase.  This is a very strong theoretical case, but is not quite proven.

Claim: Happiness research Dan Gilbert claims meditation is one of the keys to happiness, up there with sleep and exercise (location 461).

Confirmed that Gilbert is a happiness researcher and said the quote cited, although I can’t find where he personally researched this.

Claim: “Researchers at Massachusetts General Hospital and Harvard University have shown, using fMRI brain scanning technology, that eight weeks of MBSR training leads to thickening of a number of different regions of the brain associated with learning and memory, emotion regulation, the sense of self, and perspective taking. They also found that the amygdala, a region deep in the brain that is responsible for appraising and reacting to perceived threats, was thinner after MBSR, and that the degree of thinning was related to the degree of improvement on a perceived stress scale.” (location 502)

Accurate citation, but: small sample size (16/26), and for the first study the effect size was quite small (1%) for regions of a priori interest, and the second had quite wide error bands (source 1) (source 2).  However the book does refer to these findings as preliminary.

Claim: “They also show that functions vital to our well-being and quality of life, such as perspective taking, attention regulation, learning and memory, emotion regulation, and threat appraisal, can be positively influenced by training in MBSR.” (location 508).

Misleading.  These are really broad claims and no specific study is cited.  However, source 2 above has the following quote: “The results suggest that participation in MBSR is associated with changes in gray matter concentration in brain regions involved in learning and memory processes, emotion regulation, self-referential processing, and perspective taking.”  This is a very carefully phrased statement indicating that mindfulness is in the right ballpark for affecting these things, but is not the same as demonstrating actual change.

Claim: “Researchers at the University of Toronto, also using fMRI, found that people who had completed an MBSR program showed increases in neuronal activity in a brain network associated with embodied present-moment experience, and decreases in another brain network associated with the self as experienced across time. […]  This study also showed that MBSR could unlink these two forms of self-referencing, which usually function in tandem.” (location 508).

Accurate citation, small sample size (36) that they made particularly hard to find (source).  I can’t decipher the true size of the effect.

Claim: Relative to another health class, MSBR participants had smaller blisters in response to a lab procedure, indicating lower inflammation (location 529).

True, but only because the other class *raised* inflammation (source). Also leaves out the fact that both groups had the same cortisol levels and self-reported stress.  So this looks less like MBSR helped, and more like the control program was actively counterproductive.

For the record, this is where I got frustrated.

Claim: “people who were meditating while receiving ultraviolet light therapy for their psoriasis healed at four times the rate of those receiving the light treatment by itself without meditating.” (location 534)

Accurate citation (of his own work), small sample size (pdf).

Claim: “we found that the electrical activity in certain areas of the brain known to be involved in the expression of emotions (within the prefrontal cerebral cortex) shifted in the MBSR participants in a direction (right-sided to left-sided) that suggested that the meditators were handling emotions such as anxiety and frustration more effectively. […]

This study also found that when the people in the study in both groups were given a flu vaccine at the end of the eight weeks of training, the MBSR group mounted a significantly stronger antibody response in their immune system”

Accurate citation (of his own work), slightly misleading, small sample size.  Once again, he’s strongly implying a behavioral effect when the only evidence is that MSBR touches an area of the brain. On the other hand, the original paper gets into why they make that assumption, so either it’s correct or we just learned something cool about the brain.

Claim: MSBR reduced loneliness and a particular inflammatory protein among the elderly (location 551).

Not statistically significant. (source)  More specifically; the loneliness finding was significant but uninteresting, since the treatment was “8 weeks with a regular social activity” and the control was “not.”  The inflammation finding had p = .075.  There’s nothing magic about p < .05 and I don’t want to worship it, but it’s not a strong result.

I also researched MBSR in general, and found it to have a surprisingly large effect on depression and anxiety.

The Model

To the extent Full Catastrophe Living has a model, it’s been integrated so fully into the cultural zeitgeist that I have a hard time articulating it. It could be summarized as “do these practices and some amount of good things from this list will happen to you.” Which kills my hypothesis that having a good model is necessary to getting good results.

 

You Might Like This Book If…

I don’t know. I found it a slog and only read the first third, but the empirical evidence is very much on mindfulness’s side and I don’t know what better thing to suggest.

 

 

 

Thanks to the internet for making it possible for me to do these kinds of investigations.

Thanks to Patreon supporters for giving me money.

 

 

The Parable of Ignac Semmelweiss

When I was a kid, my dad told me the parable of the first physician to realize maybe mothers would not suffer quite so many horrifying deaths if doctors washed their hands between autopsies and childbirth. Unfortunately this doctor was an asshole, so everyone ignored him. He eventually went crazy from the stress of knowing so many women were being killed by their doctors, and died in a mental hospital. And that is why we don’t dismiss ideas just because they come from crazy assholes, no matter how much we want to.

I really like this story, and tell it to myself sometimes when I want to dismiss someone for being crazy and/or an asshole. Recently I got curious how true it actually was, so I pulled a couple of books on the topic, of which I finished one, The Doctor’s Plague by Sherwin B. Nuland.

First: the story as told by my dad is way more accurate than a story half remembered 25 years later has any right to be. The doctor in question is Ignac Semmelweis. Like most such discoverers, Semmelweis’s genius was not an entirely unique idea, other people had noticed autopsies and childbed fever seemed to go together, but he was the one to invent handwashing. He got a little more support than my dad mentioned, but managed to alienate them by, as I was told, being an asshole. He refused to write up his results because he had already proven them to his satisfaction. He wrote angry letters attacking the most prominent doctors in Europe. He did not play well with the other children. And he did indeed die in a mental hospital. The only thing my dad got wrong was the cause of the insanity: it was probably Alzheimer’s, not frustrated genius.

But there was another part of the story I knew but hadn’t considered; the autopsies that were contaminating doctors were being done in pursuit of curing childbed fever. The infection was spread by examinations meant to teach students. The very things doctors were doing to cure women were hurting them. Over the medium term, everyone would have been better of if they’d stopped trying. I find this terrifying.

 

Open Opportunities in Longevity

My hobby at the Foresight Institute Vision Weekend was asking people involved in longevity research what their talent gaps were. Here are the answers:

  1. “Do whatever you’re most interested in”
  2. A groundbreaking creative work that brings immortality into the mainstream.
  3. Translating the research into something that laymen can get excited about.
    1. Did I mention I write for geroscience.com ? Relive the days when I explained basic medical science a lot.
  4. There is no talent gap, they just need money.
  5. Communications, both connecting scientists to each other and spreading their research, and increasing the knowledge of smart laymen.
  6. Young people.  And then communication.
  7. Money.

 

I just published three documents supporting my big Mental Health Shallow Review, published at the effective altruism forums.  Check it out here.

Thanks to Peter Hurford for funding this research. If you would like to hire me for a research project, please reach out at elizabeth – at – this – domain .

Cost Effectiveness of Mindfulness Based Stress Reduction

The Problem

The WHO estimates that depression and anxiety together account for 75,000,000 DALYs annually, making up ~5% of total DALYs. In “Measuring the Impact of Mental Illness on Quality of Life”, I argue that there is good reason to think that the system used to generate these estimates severely underestimates the impact of mental illness, and thus the true damage may be much higher. To try to get an estimate on the harms of mental health and the benefits of alieviating mental health problems, I did a preliminary cost-effectiveness analysis of Mindfulness Based Stress Reduction (MBSR).

The Intervention

MBSR is an eight week class that uses a combination of mindfulness, body awareness, and yoga to improve quality of life and perhaps physical health for a variety of conditions.

MBSR was created by Jon Kabat-Zinn at the University of Massachusetts in the 1970s, but has spread widely since then. The exact extent of this spread is hard to measure because no official registration is required to teach mindfulness and many classes and books claim to be mindfulness inspired. For the purpose of this evaluation I looked only at things that were officially MBSR or adhered very closely to the description.

Cost of MBSR

Herman, et al. (2017) estimated the marginal cost of an MBSR class participant at $150. The first three hits on google (run in an incognito browser but suspiciously near the location from which I ran the search) for MBSR listed a cost of $395-$595, $275-$425, and $350. The difference between the top of the range and the marginal cost indicates that the high end of that range probably covers all of the costs involved with MBSR (space rental and instructor time for eight weeks of classes plus one eight hour retreat) and then some, so I will use $600 as the ceiling on costs and $150 as the floor.

MBSR has an unusually high time ongoing cost (one hour per day). To model this, I included a range of DALYs as a cost, ranging from 0 (indicating no cost) to 1/24 (as if the participant were dead for that hour). It is unclear how the one hour duration was chosen and I could not find any studies on the comparative impact of different lengths of meditation; it’s quite plausible one could get the same results in less time. For the purpose of this document I used the official program, because it was the most consistently studied.

Cost Effectiveness Analysis of MBSR

Both depression and anxiety are measured with a variety of clinical surveys. To estimate impact, I assumed that the top score on each survey caused a DALY loss equal to severe depression/anxiety, as estimated by the World Health Organization, and that a drop of N percentage points led to a drop of disability weight * N. For example, a drop of 8 points on an 80 point scale of anxiety (disability weight of severe anxiety: 0.523) causes a gain of .0523 DALYs.

For a survey of papers showing potential impact, see this spreadsheet. The estimates range from 2% to 11%, clustered around 7%.

I have created a Guesstimate model to estimate the impact of MBSR. Results were quite promising. On a randomly selected guesstimate run, the average cost was $290/DALY, with a range from $43/DALY to $930/DALY. This is close to but better than Strong Mind’s $650/DALY and overlaps with estimates of antimalarial treatment ($8.15-$150/DALY). Note that the MBSR estimate may understate the impact due to systemic biases in how DALYs are calculated. However it may also overstate the impact, as medical studies tend to overstate intervention impacts for a variety of reasons.

The model makes no attempt to account for co-morbid disorders. Individuals with depression and anxiety would likely see higher benefits, since the same hour of meditation would impact both.

This model makes the rather optimistic assumption the benefits persist for life. This assumes that the participant would have been counterfactually depressed forever without treatment.  In reality the average depressive episode lasts six months, and of people who have suffered at least one episode, the average lifetime number of episodes is four. If we assume the participant gets two years of benefit out of treatment the cost becomes $1200 to $14,000/DALY, with an average of $5200/DALY.

Caveats

All of the effectiveness studies cited were done on developed world citizens with only mild to moderate mental illnesses. Most were middle aged, and access to MBSR implies a minimum SES bar. It is possible that more severe depression is not amenable to MBSR, or that it is amenable and shows a larger absolute change because there is farther to improve.

I could find no studies on MBSR in the developing world, although since mindfulness meditation was originally created before there was such a thing as the developed world, there is a higher than typical chance that its usefulness will survive cultural translation.

All of the studies referenced had small sample sizes. They all show a consistent effect, but it’s possible publication bias is keeping negative studies out of view.

Official MBSR has an unusually high time cost compared to medication and therapy. The costs are high both upfront (eight weeks of classes and an all day retreat) and ongoing (one hour of meditation/day). Some patients may be able to get the benefits of MBSR with less time; others may not be able to practice at all due to the time demands.

 

For more on this see my shallow review of mental health .

Measuring the Impact of Mental Illness on Quality of Life

Introduction

I am currently evaluating multiple interventions aimed at mental illness. In order to compare these to each other and interventions in other areas, it is important to make an estimate of severity of the problem and of the impact of interventions. Several standard systems for evaluating health interventions exist, each of which has strengths and weaknesses. How accurate/useful are these systems for mental illness?

Death Rate

Mental illness has a death toll (primarily from suicide and overdoses) that can be compared to deaths from physical ailments. Death has the advantage of being a binary state subject to very little measurement error or differing definitions across culture. However it is an imperfect proxy for suffering inflicted by mental illness. Depending on culture one country may have a higher depression rate but lower suicide rate. A country with better medical services may have a worse drug problem but fewer deaths from overdoses. Cause of death is subject to manipulation. Mortality is also a very poor measure of anxiety, since anxiety is almost never the immediate cause of death (although it may shorten lifespan).

Disability Adjusted Life Years

Disability adjusted life years (DALYs) are an attempt to use a single number to express the health of a population. The calculation method can vary from study to study; for purposes of this post I will be referring only to the methods used in the Global Burden of Disease 2010 (hereafter GBD 2010) study.

Aggregated DALYs for a population are calculated by multiplying the [disability prevalence] x [disability weight] x [years until remission or death]. Some surveys (but not all) include further discounts for age, assuming that a year lived as a 70 year old is less valuable than a year lived as a 25 year old. This is known as age-weighting. Disability weight is calculated by asking individuals to compare two scenarios and rate which person seems “healthier.” GBD 2010 surveyed approximately 14,000 individuals from five countries (Bangladesh, Indonesia, Peru, the United Republic of Tanzania and the United States of America) and offered a web based survey as well, which was eventually taken by approximately 16,000 people. Previous versions of the GBD exclusively used the evaluations of health care practitioners.

Because they are only are a measure of health, DALYs are not a good measure of suffering. For example, a loved one dying is an obvious cause of suffering via grief, but has no impact on the DALY metric of the survivors. DALYs also deliberately exclude the availability of mitigations: vision impairment has the same DALY cost regardless of the availability of corrective lenses (Voight & King, 2010). These choices make DALYs highly legible and comparable, at the cost of excluding many things one might care about. Additionally, “Healthy” is a highly ambiguous term, which many cultures consider to refer only to physical health. This suggests that if one cares about suffering, or includes mental health in their definition of health, DALYs are likely to severely underrate the impact of mental illness.
Quality Adjusted Life Years

QALYs are explicitly designed to evaluate quality of life, not just health. Instead of choosing which of two individuals is healthier, survey participants may choose which situation they would rather live in (e.g., five years of blindness or four years of deafness), what risk of death they would accept in order to cure an ailment (e.g. 10% risk of death for surgery to restore function to your leg), or “how bad does this sound to you on a scale of 1-100?”

QALYs are noticeably better than DALYs for measuring the impact of mental illness, in that everyone agrees mental illnesses lower quality of life. However there is still concern that they underestimate the impact because people are bad at imagining themselves in different situations, and bad at imagining mental illness in particular. Dolan (2008) argues that any rating based on trade-offs is inherently weak, because humans are so bad at remembering the past and anticipating the future. He favors using ratings of subjective well being from people currently suffering from a condition. Brazier, et al. (2008) cites data that the general public rates mental health issues as less important than physical health, less so than those who suffer from mental illness (Brazier (2008), which if true would lead to an underestimate of the cost of mental illness. Meanwhile De Wit, Busschbach, and De Charro (2000) argue that people underestimate their ability to adapt to situations, and thus all QALY cost estimates are overestimates. Michael Plant argues that this applies only to physical ailments, and that this leads people to underestimate the severity of mental illness relative to physical illness.

Issues Comparing DALYs/QALYs for Mental Illness with Other Illnesses

The cost-effectiveness estimates for malaria nets are based solely on the averted physical suffering. In order to truly compare malaria QALYs with depression QALYs, we must take into consideration the mental health toll of malaria. This turns out to be a very complicated question that can’t be answered without getting into moral ontology, which is beyond the scope of this document.

For a very, very crude idea of the effect on bednets on suffering, see this guesstimate model, which lets you estimate the mental illness cost of malaria from mourning and mental-health related side effects. Ultimately the DALY/$ (guesstimated in the range of 10^-3 and 10 ^-5) are insignificant next to the DALY/$ gain from deaths averted (in the range of 10^-1).

Financial Cost

Illness (mental or physical) can exact an enormous physical toll on sufferers, in both cost of treatment and lost productivity. Productivity loss is more difficult to measure than death and thus not as precise a metric, but it is significantly more objective and comparable across ailments than DALYs or QALYs. For more information on the productivity costs of mental illness, see this post.

A second issue is that using productivity loss as a metric will bias interventions towards people with higher potential incomes, which is the opposite of most people’s instincts.

Conclusions

None of these measurements met my goals of being easy to measure and capturing the entire impact of mental illness. This is not surprising, since even the impacts of physical ailments are hard to measure. The only clear conclusion is that QALYs are better than DALYs for any purpose I can think of. Of the options available, death and financial cost are the most objective, easiest to measure, and easiest to compare to other ailments, but lose a lot of data around suffering. QALYs capture that data, but are still of questionable suitability for comparing to other ailments.

Impact of Depression and its Treatment on Productivity

Introduction

One argument for prioritizing treatment of mental illness is that the secondary effects (such as higher productivity and improved health-related behavior) may be especially impactful. Illnesses like depression and addiction are incredible drains on productivity, which can be reversed with treatment. In this essay I investigate the productivity cost of untreated (or unsuccessfully treated) mental illness and the impact of treatment on productivity.

How Bad is it?

World Health Organization Data

Alonso, et al. (2011) surveyed workers to determine how many days they missed work due to a variety of chronic illnesses, including depression and anxiety. Their sample included 63,000 people spread across 24 countries, with a range of cultures and income levels. Across all countries, the following disorders caused the average person with that disorder to lose the following days of work. Note that comorbidity is common and days-missed are additive- e.g. a person with depression and generalized anxiety in a lower income country would miss 26.6 days of work.
Days of Productivity Lost to Illness

Lower income countries Medium income countries Higher income countries All countries
Additional days Additional days Additional days Additional days
Mean s.e Mean s.e. Mean s.e. Mean s.e.
Depression 13.1 5 14.7 4.1 4.1 3.2 9 2.5
Bipolar disorder 36.5 15 23.2 9.6 9.6 5.8 17.3 4.9
Panic disorder 24.3 12.9 17.7 5.5 11.7 4.1 14.3 3.5
Specific phobia −6.6 5.2 4.2 4.7 6.7 3.3 3.9 2.5
Social phobia 5.7 10 9 8.4 7.5 2.9 7.3 2.8
GAD 13.5 9.1 24.6 8.4 7.6 4.9 7.7 3.6
Alcohol abuse −2.8 7.2 8.2 5 −0.3 4.5 1.9 3.2
Drug abuse 14.7 13.9 3.9 12.2 1.2 5.5 2.5 4
PTSD 15.3 11.3 −1.1 9.5 16.2 4 15.2 3.5
Insomnia 5.7 5.3 4.6 5.4 9.4 3.2 7.9 2.7
Headache or migraine 10 3.6 6.5 3.3 4.5 2.1 7.1 1.5
Arthritis 6.1 4.4 0.8 5 1.8 2.4 2.7 1.8
Pain 0.9 3.1 11 2.4 19.6 2.1 14.3 1.5
Cardiovascular 2.7 6.7 1 3.6 7.2 2.7 5.7 2.1
Respiratory 10.7 3 −1.1 2.6 0.9 1.4 2.6 1.3
Diabetes 4 6.4 0.5 5.6 9.6 3.8 8.6 2.8
Digestive −4.3 4.8 −0.4 4 16.6 4.8 7.6 3
Neurological 33.7 23 18.6 7 15.3 7.4 17.4 5.8
Cancer 19.4 17.9 −4.2 12.9 6.9 3.6 5.5 3.5

 

[Note that negative numbers mean the condition is associated with an increase in number of days worked.]

Alonso, et al (2011) did not attempt to measure workers who attended work but were less productive due to illness (presenteeism), or control for average number of days of work for a given country.

Chrisholm, et al. (2016) attempted to estimate the economic impact of depression and anxiety, including the cost of lost productivity, using primarily the data above. They estimate that treatment for depression leads to a 5% increase in attendance (in any country) and 5% increase in productivity while present. This implies a normal worker has 180 working days in high income countries and 260 in low income countries, which is low (see OECD data), meaning the 5% estimate for absenteeism is too high. However I believe their estimate for presenteeism is much too low. Just the diagnostic criteria of depression suggests more than a 5% drop in productivity.

 

Comparison to Sleep Deprivation

The effects of depression can be similar to sleep deprivation, in part because depression can cause either insomnia or a need for excess sleep, and in part because both produce a “brain fog” (weirdly, sleep deprivation may also treat depression). Given the paucity of information on the relationship between depression and productivity and the abundance of information on the relationship between sleep and productivity, I turned to sleep deprivation as a model for the effects of depression on productivity, contingent on a given a worker making it to their job. The following are mostly small studies but unsurprisingly all show sleep deprivation having a large negative impact on productivity.

 

Kessler, et al. (2011) estimate that insomnia causes presenteeism equivalent to 7.8 days of missed work per year, an estimated financial loss of $2,280 per person. This used the WHO Health and Work Performance Questionnaire, which relies entirely on workers self-reports of their productivity relative to co-workers. It is also designed only to measure whether someone is more or less productive than average, not the magnitude of the difference.

 

Gibson & Shrader (2014) estimated that a one hour increase in average nightly sleep led to a 16% increase in wages (on average, $6,000). I will use that as my lower bound for the benefits of treating depression. I assume the actual increase productivity is larger than the increase in wages, because some of the benefit is captured by the employer. If we assume the employer and employee capture equal value, this implies an actual productivity increase of 32%. And if we assume depression is equivalent to 2-3x the cost of missing one hour of sleep, that is almost a halving of productivity (note that for actual sleep, the costs of missed sleep probably increase exponentially). This study is especially promising because it is rather large and used a natural experiment (distance from timezone line) to establish study conditions.

 

What Does Treatment Accomplish?

Strong Minds

[When not otherwise stated, data comes from Strong Mind’s 2015 report.]

Strong Minds is an NGO in Africa that runs 12 week group therapy classes in Uganda. Their three month month program produces a noticeable drop in depression.

Strong Minds monitors its effect on depression using a modified version of the PHQ-9 (Patient Health Questionnaire- 9). The scale of this test is unknown, making it hard to evaluate the absolute improvement, but lower scores are relatively better (less depression) than higher scores. This questionnaire is an accepted tool for monitoring severity of depression.

Of women participating in Strong Mind’s 12 week pilot program, 92% had reduced scores on the PHQ-9; 11% of the control group had reduced scores. Most of the other effects reported in Strong Mind’s report are given in absolute terms, with no reference to the control group. Based on the reduction in PHQ-9 scores, I will assume 88% of any result is due to participation in the program. Key results:

  • 15 percentage point increase in participation in primary occupation (79% -> 94%).
  • 40 percentage point reduction in families going 24 hours without a meal (53% -> 13%).
  • 17 percentage point reduction in medical care visits (58% -> 41%). This is likely to understate the improvement in health, as some participants probably had physical problems they had previously been too depressed to treat.
  • 18 percentage point increase in families sleeping in protected shelters (65% -> 83%).
  • 10 percentage point increase in school attendance (33% -> 43%).

Income is not reported in this study. The authors do not say this explicitly, but it is common in developing world studies to examine consumption, because income is so variable.

Qualms about data: the study recorded 46 variables, of which less than 10 were reported in their report (not all of which made it into this report). The report included different metrics from phase one studies (eating 3 meals/day, ability to save any amount of income).  Given that it appears this data was still collected in phase two, the absence of results in the report raises concerns about cherry picking. I included this study despite my qualms because so little data was available about the effect of treatment of depression in developing countries.

Cost: $240/12 women in the program = $20/person. This is almost certainly an underestimate of even the marginal cost of the program.

Schoenbaum, et al.

In The Effects of Primary Care Depression Treatment on Patients’ Clinical Status and Employment, researchers reported that six months after their intervention (treatment for depression by a primary care physician, in the USA), 24% (vs 70% in control group) were depressed, and 72% (vs 54%) were employed.

Summary

Translating these productivity impacts into dollars is difficult because we can’t assume they hit all incomes equally, however the WHO estimates that in aggregate depression and anxiety together cost one trillion dollars US/year in lost productivity worldwide, slightly more than 1% of total GDP. On an individual level, there is no satisfying answer here. Depression has a very broad definition: the worst cases can destroy all productivity. The typical case destroys somewhere between 5% and 50% of productivity. Treatment of depression can restore that lost productivity in some but not 100% of participants.  

 

Areas for Further Investigation

I used sleep deprivation to generate heuristics for how damaging depression might be, with the answer being “quite bad”. Those numbers are even more accurate for estimating the effect of sleep deprivation. Because the scope of this paper was limited to economic effects stemming from workplace productivity, I have left out many other costs of sleep deprivation, including health costs and developmental damage to children. Given the costs and prevalence of sleep deprivation, sleep-promoting interventions, especially in children and adolescents, may be a promising area for intervention.