Introduction

One of the challenging things about learning is knowing what sources you should learn from- if you already knew what was correct, you wouldn’t be trying to learn it in the first place. Epistemic spot checks started as a process in which I did quick investigations a few of a book’s early claims to see if it was trustworthy before continuing to read it, in order to avoid wasting time on books that would teach me wrong things. Friends indicated they found these useful, so I started sharing them, and even got a small Patreon running.

Epistemic spot checks worked well enough for catching obvious flaws (*cou*Carol Dweck*ugh*), but have a number of problems. They emphasize a trust/don’t trust binary over model building, and provability over importance. They don’t handle “severely flawed but deeply insightful” well at all. So I started trying to create something better. This post is part of that attempt, and as such contains both checks of a book’s claims and introspection on the process of checking those claims.

But before I started that improvement process, there was another. Even the quick versions of epistemic spot checks are time consuming, and I am only one person, who will not be unemployed forever. I started working with Foretold and Parallel on a project to amplify my spot checks by having people predict how I would evaluate claims- the idea being that if the masses got good at it, prediction markets could be a partial substitute for my investigations. This blog post is also a part of that project, which entails some extra steps. If you’re interested in how this has worked in the past, check out  The Fate of Rome and The Fall of Rome.

Today’s book is The Unbound Prometheus (affiliate link), which aims to explain why the Industrial Revolution happened when and where it did. Spoiler alert: I didn’t like it.

Process

I had three phases of actions: in the first, I read the book, created claims in Foretold, and entered my priors for the claims as predictions. Phase two was much like the only phase of previous checks, in which I had a set period of time (six hours, twice what I had for The Fall of Rome) to investigate randomly selected claims for as long as feels useful, after which I predict what my credence in the claim would be after 10 hours of research. This prediction was submitted as a resolution in Foretold. In practice there were several claims where I stopped an hour in, even though I still had very high uncertainty, because it seemed like it would take a lot of additional time to shrink my confidence bars. 

Answers, here and in the prediction market, are given in Foretold syntax.

In phase three, I had three hours each to answer two questions, randomly selected from those I had answered in phase two. The goal here was to see how good I was at predicting my own answers. The gods of fate were not kind on this one, and I drew the two questions that least benefited from additional time, being fairly strict factual questions with an exhaustible amount of relevant material. These evaluations were not entered in Foretold, but I’ve included them here.

As is my new custom, I took my notes in Roam, a workflowy/wiki hybrid. Until recently I thought Roam was so magic that my raw notes were better formatted there than I could ever hope to make them in a linear document like this, so I could just share my conclusions here, and let people read my notes in Roam if they were especially curious. In between writing most of this post and publishing it I learned that many people find my Roam notes too difficult to read and prefer having them written out linearly. I didn’t have the time or energy to fix this post, but rest assured I’m thinking about how to do this better. In the meantime, Roam notes are formatted as follows:

• The target source gets its own page
• On this page I list some details about the book and claims it makes. If the claim is citing another source, I may include a link to the source.
• If I investigate a claim or have an opinion so strong it doesn’t seem worth verifying (“Parenting is hard”), I’ll mark it with a credence slider. The meaning of each credence will eventually be explained here, although I’m still working out the system.
  • Then I’ll hand-type a number for the credence in a bullet point, because sliders are changeable even by people who otherwise have only read privileges. If the slider and text number disagree, believe the text.
• You may see a number to the side of a claim. That means it’s been cited by another page. It is likely a synthesis page, where I have drawn a conclusion from a variety of sources. The synthesis pages are also what I’ll be linking to in this post.

Another thing that changed this time around is that I learned to use the multimodal function of Foretold, which lets you combine distribution functions. This is great for hedging your bets but not great for comprehensibility, so I’ll include the graphs generated. Unfortunately Foretold still doesn’t have a graphical export, so I’m using hideous screen shots.

Even more unfortunately, WordPress, which has happily accepted these screen shots in the past, will not tolerate them now. So to see my claims and conclusion, please continue to the Google Doc. Please let me know what was useful, useless, or high friction for you. I’m especially interested in how comprehendible/usable my Roam database is.

Many thanks to my Patreon patrons and Parallel Forecast for financial support for this post