Epistemic Spot Check: The Fate of Rome (Kyle Harper)

Introduction

Epistemic spot checks are a series in which I select claims from the first few chapters of a book and investigate them for accuracy, to determine if a book is worth my time. This month’s subject is The Fate of Rome, by Kyle Harper, which advocates for the view that Rome was done in by climate change and infectious diseases (which were exacerbated by climate change).

This check is a little different than the others, because it arose from a collaboration with some folks in the forecasting space. Instead of just reading and evaluating claims myself, I took claims from the book and made them into questions on a prediction market, for which several people made predictions of what my answer would be before I gave it. In some but not all cases I read their justifications (although not numeric estimates) before making my final judgement.

I expect we’ll publish a post-mortem on that entire process at some point, but for now I just want to publish the actual spot check. Because of the forecasting crossover, this spot check will differ from those that came before in the following ways:

  1. Claims are formatted as questions answerable with a probability. If a claim lacks a question mark, the implicit question is “what is the probability this is true?”.
  2. Questions have a range of specificity, to allow us to test what kind of ambiguities we can get away with (answer: less than I used).
  3. Some of my answers include research from the forecasters, not just my own.
  4. Due to timing issues, I finished the book and a second on the topic before I did the research for spot check.
  5. Due to our procedure for choosing questions, I didn’t investigate all the claims I would have liked to.

 

Claims

Original Claim: “Very little of Roman wealth was due to new technological discoveries, as opposed to diffusion of existing tech to new places, capital accumulation, and trade.”
Question: What percentage of Rome’s gains came from technological gains, as opposed to diffusion of technical advantages, capital accumulation, and trade?

1%-30% log distribution

Data:

  • The Fall of Rome talks extensively about how trade degraded when the Romans left and how that lowered the standard of living.
  • https://brilliantmaps.com/roman-empire-gdp/ shows huge differences in GDP by region, implying there was a big opportunity to grow GDP through trade and diffusion of existing tech. That means potential growth just from catch up growth was > 50%.
  • Wikipedia doesn’t even show growth in GDP per capita (with extremely wide error bars) from 14AD to 150AD.
  • Rome did have construction and military tech (https://en.wikipedia.org/wiki/Roman_technology)
  • It also seems likely that expansion created a kind of Dutch disease, in which capable, ambitious people were drawn to fighting and/or politics, and not discovering new tech.
  • One potential place where Roman technology could have contributed greatly to the economy was lowering disease via sanitation infrastructure. According to Fate of Rome and my own research, this didn’t happen; sanitation was not end to end and therefor you had all the problems inherent in city living.

Original Claim: “The blunt force of infectious disease was, by far, the overwhelming determinant of a mortality regime that weighed heavily on Roman demography”
Question: Even during the Republic and successful periods of the empire, disease burden was very high in cities.

60%-90% normal distribution

The wide spread and lack of inclusion of 100% in the confidence interval stem from the lack of precision in the question. What distinguishes “high” from “very high”, and are we counting diseases of malnutrition or just infectious ones? I expected to knock this one out in two minutes, but ended up feeling the current estimates of disease mortality lack the necessary precision to answer it.

Data:

 

Original Claim: “The main source of population growth in the Roman Empire was not a decline in mortality but, rather, elevated levels of fertility”
Question: When Imperial Rome’s population was growing, it was due to a decline in death rates, rather than elevated fertility.

80-100%, c – log distribution

“Elizabeth, that rephrase doesn’t look much like that original claim” you might be saying quietly to yourself. You are correct- I misread the claim in the book, at least twice, and didn’t catch it until this write-up. This isn’t as bad as it seems. The claims are not quite opposite, because my rephrase was trying to explain variation in growth within Rome, and the book was trying to explain absolute levels, or possibly the difference relative to today.

Back when he was doing biology, Richard Dawkins had a great answer to the common question “how much is X due to genetics, as opposed to environment?”. He said asking that is like asking how much of a rectangle’s area is due to its length, as opposed to its width. It’s a nonsensical question. But you assign proportionate responsibility for the change in area between two rectangles.

Fate‘s original claim was much like asking how much of a trait is due to genetics. This is bad and it should feel bad, but it’s a very common mistake, and I give Fate a lot of credit for providing the underlying facts such that I could translate it into the “what causes differences between things” question without even noticing.

Since weak framing wasn’t a systemic problem in the book and it presented the underlying facts well enough for me to form my own, correct, model, I’m not docking Fate very harshly on this one.

Original Claim: “The size of Roman merchant ships was not exceeded until the 15th century, and the grain ships were not surpassed until the 19th.”
Question: “The size of Roman merchant ships was not exceeded until the 15th century, and the grain ships were not surpassed until the 19th.”

0-10% log distribution.

This is true within the Mediterranean, but if  you check Chinese ships it’s obvious it’s off by at least 100 years, possibly more.

Original Claim: too diffuse to quote.
Question: The Roman Empire suffered greatly from intense epidemics, more so than did the Republic or 700-1000 AD Europe.

90-100% c – log distribution

https://en.wikipedia.org/wiki/List_of_epidemics shows a pretty clear presence of epidemics in the relevant period and absence in the others.

 

Original Claim: too diffuse to quote.
Question: Starvation was not a big concern in Imperial Rome’s prime.

80-100% c – log distribution

https://en.wikipedia.org/wiki/List_of_famines shows Roman famine in 441 BC (the Republic) and isolated famines from 370 on, but pretty much validates that during the prime empire, mass starvation was not a threat.

Conclusion:

My fact checking found two flaws:

  1. An inaccuracy in when ships that exceeded the size of Roman trade ships were built, and/or forgetting China was a thing. The inaccuracy does not invalidate the author’s point, which is that the Romans had better shipping technology than the cultures that followed them.
  2. Bad but extremely common framing for the relative effects of disease mortality vs. birth rates.

These is well within tolerances for things a book might get wrong. I’m happy I read this book, and would read another by the same author (with perhaps more care when it refers to happenings outside of Europe), but they are not jumping to the of my list.

Is The Fate of Rome correct in its thesis that Rome was brought down by climate change and disease? I don’t know. It certainly seems plausible, but is clearly advocating for a position rather than trying to present all the relevant facts. There are obvious political implications to Fate even if it doesn’t spell them out, so I would want to read at least one of the 80 million other books on the Fall of Rome before I developed an opinion. I’m told some people think it had to do with something military, which Fate barely deigns to mention. In the future I hope to be a good enough prediction-maker to put a range on this anyways, however wide it must be, but for now I’m succumbing to the siren song of “but you could just get more data”.

[Many thanks to my Patreon patrons and Parallel Forecast for financial support for this post]

PS. This book is the first step of an ongoing experiment with epistemic spot checks and prediction markets. If you would like to participate in or support these experiments, please e-mail me at elizabeth-at-this-domain-name. The next round is planned to start Saturday August 24th.

Epistemic Spot Check Scaling Experiment

Tl;dr: would you like to exchange money for an extra-rigorous epistemic spot check or automating me out of a hobby? I have an opportunity for you.

Most of you reading this probably know the epistemic spot check series on this blog, in which I somewhat-arbitrarily check claims early in a book to calibrate my trust level in said book.

I’ve been approached by a pre-public prediction market org to see if we can scale ESCs using a forecasting tournament. As conceived of right now, I would extract claims from a book and put them in the tournament, where anyone could bet on how I would eventually rule on the claim. I then check a subset of the claims (the others resolve as “ambiguous”) and money is distributed to the winners. In order to standardize things, this will be done with more rigor and consistency than is usually seen in epistemic spot checks.

We currently have prize money to distribute to the winners, but not to cover my time. We’re looking for $1,000-$2,000 depending on the book and any particular requests you have. If you’re feeling generous, more prize money would not hurt either.

If you’re at all interested, e-mail me at elizabeth – at – this- domain and we can chat.

Epistemic Spot Check: The Role of Deliberate Practice in the Acquisition of Expert Performance

Epistemic spot checks typically consist of references from a book, selected by my interest level, checked against either the book’s source or my own research. This one is a little different that I’m focusing on a single paragraph in a single paper. Specifically as part of a larger review I read Ericsson, Krampe, and Tesch-Römer’s 1993 paper, The Role of Deliberate Practice in the Acquisition of Expert Performance (PDF), in an attempt to gain information about how long human beings can productivity do thought work over a time period.

This paper is important because if you ask people how much thought work can be done in a day, if they have an answer and a citation at all, it will be “4 hours a day” and “Cal Newport’s Deep Work“. The Ericsson paper is in turn Newport’s source. So to the extent people’s beliefs are based on anything, they’re based on this paper.

In fact I’m not even reviewing the whole paper, just this one relevant paragraph: 

When individuals, especially children, start practicing in a given domain, the amount of practice is an hour or less per day (Bloom, 1985b). Similarly, laboratory studies of extended practice limit practice to about 1 hr for 3-5 days a week (e.g., Chase & Ericsson, 1982; Schneider & Shiffrin, 1977; Seibel, 1963). A number of training studies in real life have compared the efficiency of practice durations ranging from 1 -8 hr per day. These studies show essentially no benefit from durations exceeding 4 hr per day and reduced benefits from practice exceeding 2 hr (Welford, 1968; Woodworth & Schlosberg, 1954). Many studies of the acquisition of typing skill (Baddeley & Longman, 1978; Dvorak et al.. 1936) and other perceptual motor skills (Henshaw & Holman, 1930) indicate that the effective duration of deliberate practice may be closer to 1 hr per day. Pirolli and J. R. Anderson (1985) found no increased learning from doubling the number of training trials per session in their extended training study. The findings of these studies can be generalized to situations in which training is extended over long periods of time such as weeks, months, and years

Let’s go through each sentence in order. I’ve used each quote as a section header, with the citations underneath it in bold.

“When individuals, especially children, start practicing in a given domain, the amount of practice is an hour or less per day”

 Generalizations about talent development, Bloom (1985)

“Typically the initial lessons were given in swimming and piano for about an hour each week, while the mathematics was taught about four hours each week…In addition some learning tasks (or homework) were assigned to be practiced and perfected before the next lesson.” (p513)

“…[D]uring the week the [piano] teacher expected the child to practice about an hour a day.” with descriptions of practice but no quantification given for swimming and math (p515).

The quote seems to me to be a simplification. “Expected an hour a day” is not the same as “did practice an hour or less per day.”

“…laboratory studies of extended practice limit practice to about 1 hr for 3-5 days a week”

Skill and working memory, Chase & Ericsson (1982)

This study focused strictly on memorizing digits, which I don’t consider to be that close to thought work.

Controlled and automatic human information processing: I. Detection, search, and attention. Schneider, W., & Shiffrin, R. M. (1977)

This study had 8 people in it and was essentially an identification and reaction time trial.

Discrimination reaction time for a 1,023-alternative task, Seibel, R. (1963)

3 subjects. This was a reaction time test, not thought work. No mention of duration studying.

 

“These studies show essentially no benefit from durations exceeding 4 hr per day and reduced benefits from practice exceeding 2 hr”

Fundamentals of Skill, Welford (1968)

In a book with no page number given, I skipped this one.

Experimental Psychology, Woodworth & Schlosberg (1954)

This too is a book with no page number, but it was available online (thanks, archive.org) and I made an educated guess that the relevant chapter was “Economy in Learning and Performance”. Most of this chapter focused on recitation, which I don’t consider sufficiently relevant.

p800: “Almost any book on applied psychology will tell you that the hourly work output is higher in an eight-hour day than a ten-hour day.”(no source)

Offers this graph as demonstration that only monotonous work has diminishing returns.

Screen Shot 2019-05-16 at 9.08.22 PM.png

 

p812: An interesting army study showing that students given telegraphy training for 4 hours/day  (and spending 4 on other topics) learned as much as students studying 7 hours/day. This one seems genuinely relevant, although not enough to tell us where peak performance lies, just that four hours are better than seven. Additionally, the students weren’t loafing around for the excess three hours: they were learning other things. So this is about how long you can study a particular subject, not total learning capacity in a day.

Many studies of the acquisition of typing skill (Baddeley & Longman, 1978; Dvorak et al.. 1936) and other perceptual motor skills (Henshaw & Holman, 1930) indicate that the effective duration of deliberate practice may be closer to 1 hr per day

The Influence of Length and Frequency of Training Session on the Rate of Learning to Type, Baddeley & Longman (1978)

“Four groups of postmen were trained to type alpha-numeric code material using a conventional typewriter keyboard. Training was based on sessions lasting for one or two hours occurring once or twice per day. Learning was most efficient in the group given one session of one hour per day, and least efficient in the group trained for two 2-hour sessions. Retention was tested after one, three or nine months, and indicated a loss in speed of about 30%. Again the group trained for two daily sessions of two hours performed most poorly.It is suggested that where operationally feasible, keyboard training should be distributed over time rather than massed”

 

Typewriting behavior; psychology applied to teaching and learning typewriting, Dvorak et al (1936)

Inaccessible book.

The Role of Practice in Fact Retrieval, Pirolli & Anderson (1985)

“We found that fact retrieval speeds up as a power function of days of practice but that the number of daily repetitions beyond four produced little or no impact on reaction time”

Conclusion

Many of the studies were criminally small, and typically focused on singular, monotonous tasks like responding to patterns of light or memorizing digits.  The precision of these studies is greatly exaggerated. There’s no reason to believe Ericsson, Krampe, and Tesch-Römer’s conclusion that the correct number of hours for deliberate practice is 3.5, much less the commonly repeated factoid that humans can do good work for 4 hours/day.

 

[This post supported by Patreon].

Epistemic Spot Check: The Dorito Effect (Mark Schatzker)

Epistemic Spot Checks is a series in which I fact check claims a book makes, to determine its trustworthiness. It is not a book review or a check on every claim the book makes, merely a spot check of what I find particularly interesting or important (or already know).

Today’s subject is The Dorito Effect, which claims that Americans are getting fat because food is simultaneously getting blander and less nutritious, and then more intensely flavored through artificial means. This is leaving people fat and yet malnourished.

Claims

Claim: Humans did not get fatter over the last 100 years due to changes in genetics.
True. People are fatter than their ancestors, indicating it’s not a change in genetics (although genetics still plays a role in an individual’s weight).

Claim: Casimir Funk discovered that an extract of brown rice could cure beriberi in chickens.
True.

Claim: In 1932, the average farm produced 63 sacks of potatoes/acre. By the mid 1960s, it was 200 sacks/acre.
True.

KCalGraph.png
(source).

Claim: Everything is getting blander and more seasoned.
More seasoned.
Blander food.
Note that both sources were provided by the book itself.

Claim: “We eat for one reason: because we love the way food tastes. Flavor is the original craving”.
This doesn’t jive with my personal experience. I definitely crave nutrients and am satisfied by them even without tasting them.

Claim: “In 1946 and 1947, regional Chicken Of Tomorrow contests were held.”
True.

Claim: Over time the Chicken Of Tomorrow winners consistently weighed more, with less feed and less time to maturity.
True.

Claim: Produce is getting less nutritious over time.
True (source provided by author).

Conclusions

Extremely trustworthy, and therefore worrisome, given the implication that food is becoming inexorably worse. Dorito Effect is unfortunately light on solutions, so you might just freak yourself out to no purpose. On the other hand, if you’re looking for a kick to start eating better, this could easily be it.

Self Help Epistemic Spot Check Results

In a word: ¯\_(ツ)_/¯

I reviewed several self-help books with a wide range of scientific backing. For posterity:

Polyvagal Theory
The Tapping Solution/EFT
Full Catastrophe Living/Mindfulness
Exercise for Mood and Anxiety
A Guide to Better Movement
There were a few others I never published because I didn’t get very far into.

As a reminder, epistemic spot checks are checking a book’s early claims for truth/scientific validity/coherent modeling, to determine whether it’s worth continuing. After a few books I concluded that scientific backing didn’t seem that predictive of a book’s helpfulness, and started focusing on modeling. But that wasn’t predictive either.

I never officially decided to quit this project, but I can no longer get excited about checking out a new book, because nothing short of trying it seems to have any predictive ability of whether or not it is helpful. This leads me to believe that most of the effects are placebo effect, not in the sense of  “imagined” as people usually use the word, but in the sense that it’s your own brain doing most of the work, and people just have to try things until something clicks for them, starting with the cheapest. I find this answer deeply unsatisfying, but what are you gonna do?

 

Epistemic Spot Check: Polyvagal Theory/Safe and Sound Protocol/Stephen Porges

I read part of the book The Polyvagal Theory and went to a two day seminar by the author, Stephen Porges. I went because I thought there was a strong possibility EFT worked by affected the vagal nerve, and thought maybe polyvagal theory could explain how. I ended up pretty disappointed.

Once I was at the seminar I was very interested in a protocol Porges developed called Safe and Sound, which purports to cure a number of things including many symptoms of autism, plus misophonia (which I have), by playing songs with certain frequencies filtered. Porges showed very impressive videos of autistic children going from non-functional to neurotypical-passing. He bragged about a 50% improvement rate. He played a sound sample and even on hotel sound system speakers, it had a very definite affect on me, relaxing many muscles. So of course I ordered it.

In a failure of order of operations I didn’t look up the results until after I’d ordered it (I really wanted my misophonia fixed, plus the demo had been so impressive). The paper tries very hard to hide this, but what actually happened was not an average 50% improvement in some patient metric, but that 50% of patients showed any improvement. Given that autism is a high variance disease and children are often receiving multiple interventions, this basically means “didn’t make anything worse, probably”.

But I’d already ordered the thing, so I decided to try it. This was kind of an ordeal, btw. Safe and Sound is available only through “trained professionals”, even though the protocol consists in its entirety of listening to some songs on an MP3 player. And I checked, there’s nothing magic about the MP3 player or headphones they send you, you could do it with any reasonably good pair you had lying around. Based on this, I have to assume the 3-digit price tag and gatekeeping are entirely about prestige, because they’re certainly not about helping people or making money (I’m sure he could make more selling the CDs without the gatekeeping).

The protocol did have an effect, in that it consistently made me very sad. It didn’t have any effect on my misophonia, even though I tried it twice. The occupational therapist tried to insist it had worked because I was blunter and more confident in my last conversation with her, but no, sweety, that was because I was more sure your system was bullshit. Then she recommended I give them more money to do other protocols, which I inexplicably declined.

I am fighting the urge to get into the science of polyvagal theory, because it is really really interesting and has a lot of explanatory power. I put off writing this for five months because I wanted to do a more scientific review. But the empirical results are not just bad, they’re bad while proponents are claiming they are good. I can’t trust someone who does that.

For bonus points, when I asked some pointed questions during the seminar, Porges blew me off. So I’m not going to give polyvagal theory any more brain space, even though it would be so cool if it was true.

 

Epistemic Spot Check: The Tapping Solution (Nick Ortner)

This is part of a series called epistemic spot checks, in which I investigate claims a book makes to see if it’s worth paying attention to, without attempting to be comprehensive about it.

Introduction

This is a weird review to write.  I went into reading The Tapping Solution with two beliefs:

  1. The scientific claims would be far less supported than the author implies.  The best case scenario was “as terrible as your average therapy research.”
  2. The book’s prescriptions work for me anyway, in the sense that they make me calmer and happier and enable me to take better actions.

This book is about EFT, which stands for emotional freedom technique. I write that in a very small font in the hopes you won’t notice how stupid it sounds.  EFT is also known as tapping, because the primary action is tapping your fingers against your face.

I originally learned about EFT in a book that went full blown magic about it: you tap your fingers on your face, it changes energy currents in your body, and the universe magically gives you what you want.  There’s no point evaluating the science in books like that; they are what they are.   The Tapping Solution markets itself as the more studious cousin of that book.  It keeps the energy channels but backs off the magic gifts claim, offering the much more defensible explanation that tapping changes something in you that lets you create better outcomes.

The basic idea of EFT is you tap out a pattern on your body, mostly your face, while repeating a statement about something with a lot of negative emotional affect for you, especially ones that activate the sympathetic nervous system (fight/flight/freeze).  Repeat until you feel better.

[There’s a lot of different techniques claiming to be The Best EFT Script and, while I suspect there are individual variations in what works best for each person, I can’t possibly care about the intra-EFT wars.  Any script you use should just be a starting point for making your own anyway.]

Why would tapping improve your mood?  I have some guesses:

  • It makes anxiety et al. boring.  There are a lot of activities where people deliberately activate their SNS (sky diving, horror movies, drugs), so there must be something fun rewarding about being activated.    Plus, lots of the things that happen to you in response to anxiety are quite pleasant.  People cuddle you and bring you ice cream.  You put off doing the stressful thing.  I don’t think many people deliberately push themselves into hysterics for the attention, but I do think these benefits bias how people handle their stress.  Tapping does not offer those kinds of rewards; after two or three rounds of tapping, you are bored.  There are times I have gone and done the stressful thing because I would rather deal with it than have to do another round of tapping.  It’s nice to have my intolerance for boredom harnessed for good.
    • I suspect this is some of how cognitive behavioral therapy works as well.  Having taught myself both, EFT is less work and yet harder to develop an immunity too, although hybrid systems do better still.
  • A sense of control lowers stress.  Having A Thing You Can Do While Stressed that you think lowers your stress level is already lowering your stress level.  You can dismiss this as a self-fulfilling prophecy, but that’s only the point if you’re actually evaluating the concept of energy meridians.  If what you want is to calm down so you can respond to comments on your code review, it doesn’t matter if it’s a placebo.
  • Something something vagus nerve.  The vagus nerve is this weird nerve that skips the spinal cord and runs all over your body, including most major organs and a lot of your face.
    • Its tasks include:
      • Parasympathetic (relaxing) stimulation of all major organs except the adrenal glands.
      • Parasympathetic stimulation of muscles around the mouth and larynx.
      • Possibly reduces systemic inflammation
      • Sympathetic (fight/flight/freeze) stimulation of blood vessels.
      • A bunch of sensory stuff around the face.
    • Activity on your face is already known to affect your body via the vagus nerve.
      • Cold water on the face slows down your heart, and this is attributed mainly to the vagus nerve.
      •  Direct electrical stimulation of the nerve is touted as a cure for all kinds of stuff.  My sense is the science on that is… optimistic, but there is a reason it is being done to the vagus nerve and not something else.
    • There’s an alternate EFT script that involves tapping only on the hands.  I have fond this to be a calming distraction at best.  Hands are also pretty innervated, so this points to the effects being due to something specifically in the face, as opposed to sensitivity in general.
    • I yawn *a lot* while tapping.  Heart problems can cause yawning via the vagus nerve. I’m obviously not damaging my heart by tapping, I mention this just to show that the vagus nerve and yawning are related.
    • So I don’t know what’s going on, but I suspect the effect of tapping is mediated via the vagus nerve.
  • It’s a framework for breaking your problem into bite sized chunks, which is the ideal size for problems to be.  EFT practices vary in how much you work off a verbal script you’re given, vs introspect on your own issues and tap on what comes up.  I predict script-style work to be at best competitive with relaxation exercises, and only introspective EFT leads to actual improvements.
  • Who knows, maybe energetic meridians are a real thing, or at least a workable metaphor for a real thing.  Lots of things sound stupid until you know how they work.
    • In particular, if you mixed up the explanations for EFT and the much more legitimate EMDR (deliberate eye movements rewiring your brain), I’m not convinced anyone could tell which one was the Officially Sanctioned Therapy and which was the crackpot treatment.
    • Mark at meditationstuff.com argues that what gets sensed as energy flow is severe awareness of your own nervous system.  He provides no compelling evidence for this, but it is interesting.
    • Many Properly Credentialed Authorities believe things that are no weirder.

 

How I evaluated this book: usually when doing these checks I evaluate any statement I find interesting.  In this case, I’m sticking to the ones for which the author explicitly claims scientific backing.  For stuff that is essentially running on placebos and metaphors, I find a calm, confident, made up explanation is better than a hedged, hesitant, literally true one, so I’m not going to investigate the obviously exaggerated claims.  But if you’re going to claim scientific validity, I am going to check.

Claim: “The amygdala is the source of emotions and long term memories, and it’s where negative experiences are encoded (p4)”.

True.  Simplified, but obviously trying to explain how the amygdala was relevant to a particular concept, not give a comprehensive overview of our friend the amygdala.  The amygdala is in fact so good at emotional memory that it can be invoked by visual cues even in people blinded by brain damage.  This confused me at first, so let me note that the amygdala is not involved in fight/flight/freeze, but the longer, cortisol-driven chronic kind of stress.

Claim: Stimulating acupoints calms down the amygdala, and this is observable in fMRI and PET machines (p5).

Misleading, either bad faith or credulous.  Both studies cited were done with acupuncture, not acupressure or tapping.  I consider that relevant evidence for EFT, but dislike that he tried to make it even stronger evidence by hiding that both studies involved needles.  The effectiveness of acupuncture appears to have large if weak support; I very quickly pulled up many more studies demonstrating the exact same thing, all of which were tiny (the largest was 18), and used fMRIs, which are suspect.

In general, studies of acupuncture have shown that it kind of works, but Official Legitimate Chinese Medicine Points don’t do any better than a random spot, so this adds more legitimacy to randomly stabbing yourself than it does to meridian points.

Claim: Other studies show that pressure works just as well for stabbing, maybe even better for anxiety (p5).

Seems legit.  I didn’t find any citation for this but I’m willing to spot him that touching works better than stabbing for anxiety.

Claim: A study demonstrated that EFT reduces cortisol levels in the saliva (p5).

True, evidence weak but better than I guessed.  The study cited is real, and with some effort I even found a full PDF.  EFT did better than both a support group and no treatment on both a symptoms assessment and cortisol levels (24% decrease vs 14%).  The differences in symptoms between EFT and the other groups are small, and some were not statistically significant.  OTOH, every one of them goes in the same direction.  I find this pretty compelling, assuming they published every trait they recorded.  As usual, small study, vulnerable to p-hacking, etc.

Claim: This John Hopkins approved doctor agrees with us (p7).

Misleading, possibly very.  The named person (David Friedman) does exist, but he’s a doctor of psychology, not psychiatry.  The level that JHU approves of him is unclear.  On his CV (PDF) he lists himself as “research associate”, “instructor”, and “faculty.”  None of these words are “professor”, which makes me think he was an adjunct and certainly didn’t have tenure.

Claim: Competing systems telling you to never think about the negative are idiotic.  True things are true (p8).  In particular The Secret is bullshit.

Seems legit.  “Make bad things approachable”  just seems like a better tactic than wishing really hard. I also enjoy watching different alt modalities fight with each other.

Claim: Meridians have been scientifically validated, they’re called Bonghan channels (p10).

False.  The official name of Bonghan channels is the primo-vascular system, and there’s minimal evidence it exists.  Given that it’s pretty hard to prove that there’s a link between them and meridians in any scientific sense.  But it’s established fact within the meridian community, so it’s at least well sourced bullshit.

 

A few more notes on The Tapping Solution.

As expected, Tapping Solution has failed the RCT test.  What about the model test?

Well, it’s a fairly vague model, and energy meridians can be used to power anything.  On the other hand it avoids my biggest complaint about heal-yourself-with-the-placebo-effect books, and also religion, certain parts of medicine, and psychology, which is that the solution to failure is often do the same thing harder.  Tapping by and large avoids that trap.  For actual physical problems you’re encouraged to see a doctor first, then tap, and if that doesn’t work see a doctor again.  If a particular tap isn’t working you’re given alternate prompts to try.  Additionally, tapping claims that often it will work so well you’ll forget you will ever upset about something, and the solution is not to hand over money to the nice man to keep the good vibes flowing, it’s to keep track of how upset you are at the beginning of the session.  That level of empiricism shouldn’t make a book stand out, but it does.  Tapping Solution, although not every book on EFT, is also pretty clear that you’re not imposing your will on the universe, you’re calming down so you can take better actions.

I don’t want to write out instructions for tapping because I believe the process of reading a book adds a lot of value over a quick run through (the same way doing yoga is better for you than waving a magic wand and becoming more flexible).  But to help you decide if even starting the book is worth your time, here are some genres of problems I think tapping is most appropriate for:

  • Somaticizations, especially back pain.
  • Emotions you find too overwhelming to deal with, especially anxiety.
  • Legit life problems that are just too big to deal with all at once and need to be broken into bite size pieces.

Model

Simplicity: very low.  “Magical energy currents” sounds simple in that you can explain it quickly, but it takes a very long time to explain what things it can’t do and why.

Explanation quality: poor.  Merdians can power anything.

Explicit predictions: okay. You have to make your own explicit predictions, but the book very much encourages you to do so.

Acknowledging limitations: mixed.

Relative to other heal-yourself-with-the-placebo-effect systems, The Tapping Solution is modest in its claims about what your mind can do.  It goes out of its way to establish that the mind-body connection is in fact a connection, it doesn’t mean your body is a hallucination you can will into whatever form you want.

GemasAugust2015.png
Lesbian space rocks whose bodies are solid holograms  are not representative study subjects.

And then on the next page there’s a story of how a woman cured her lung cancer with EFT.  So it’s not amazing on this axis.

Measurability: extremely good.  This is where EFT really shines.  They claim it’s such a good technique you will forget you ever had a problem, and encourage you to keep track so you won’t forget.

 

Empirics

I’m deliberately not giving a lot of details on how to do it yourself, because I think there might be value to going through the book beyond the technique.

I taught this technique to five people, one of whom had a good response to it. Counting myself, that’s 1/3 successes, which is not great. But it’s cheap enough and has high enough potential I still recommend trying it.