Looking For Test Subjects: Breaking Questions Down

As long time readers are well aware, my main focus for the last year has been developing a method for Knowledge Bootstrapping– going from 0 to 1 in an unfamiliar field without undue deference to credentialism. I’m at the stage where I have a system that works well for me, and I’ve gotten feedback from a few other people about what works and doesn’t work for them, but there’s a long way to go. A lot of my knowledge is implicit and not explained on the page, plus I am only one person; what works for me will not translate perfectly for every human. So I’m looking for test subjects.

One particular part of my method is breaking down one large question into many smaller questions. This has several purposes: it forces you to clarify what you actually care about, and makes it more obvious what information is relevant. I describe this process and the reasoning behind it here, but not very well.  I’m looking for test subjects that have a research question, and would like to practice breaking it down into smaller questions, with the goal of refining the technique and my teaching of it.

What This Looks Like

  1. Come up with a question you might like to research.
  2. You book a phone call with me via calendly, or email me at elizabeth -at- acesounderglass.com to set up a time. 
  3. We discuss your question in an attempt to break it down into smaller parts.
  4. I sure hope some people actually go off and research the new questions but there’s no commitment required to do so.

What Are the Expected Outcomes?

  • You will have a better understanding of what you actually want to know and will be better positioned to find answers.
  • You will be better able to break down your next research question, without me.
  • I will make some of my metis on breaking down questions more explicit.
  • I will become better at teaching the technique of breaking down questions.
  • I learn techniques from you I couldn’t have learned on my own.

No Notes Research

I started watching The Vow with a friend, and got inspired to do a bunch of reading on cults in general and Vow’s cult, NXIVM, in particular. I didn’t originally take notes because this was coded in my head as a leisure activity, not real research. Eventually it became clear it was a real research project, but it seemed unfair to introduce real notes halfway through, so I decided to use it as an experiment in research without detailed notes instead (I did end up writing a few, but a far cry from The Algorithm). This turned out to be the right situation for that experiment, because my friend was a check on how much I actually remembered, especially on things we disagreed on, which was a lot.

Observations:

  • Memory is in fact hard. 
    • When I went to share what I learned with my friend, I often had to look back at my (sparse) notes to remember things I wanted to talk to him about. This is true even when I was talking to him the morning of the day after I read the book. 
    • Often he would ask me questions and the answer wasn’t in my notes- sometimes it was firmly in memory and I’d just forgotten to bring it up, sometimes I knew the book had the answer but I had lost it.
  • I mixed up sources a lot.
    • I would frequently assume my friend knew something, only to find out it wasn’t from our shared source. 
  • I’d compress specifics into patterns. This hindered me arguing with my friend.
  • In combination with the source mix-up, this often meant I couldn’t tell apart the following situations:
    • Books A and B repeating the same story with the same source (almost equivalent to one source)
    • Books A and B tell the same story from their own perspectives (stronger evidence the thing actually happened, but not evidence of a pattern)
    • Books A and B tell stories about similar things happening to different people (evidence of a pattern).
  • Losing the specifics that demonstrated a pattern also made it much harder to change my mind in response to new evidence. Is this more credible or a stronger signal than the data my current view is based on? Who’s to say, if I can’t remember the original evidence?
  • This feels way easier with something as emotionally salient as cults than it did with my more distant historical research. And when I did shift to a more history-style book (Mystics and Messiahs), I suddenly had to take real notes.
  • I did go into this with a question, but I didn’t know what it was until I’d read a few books and seen what felt live and what didn’t. 
    • The question was: how do we cultivate instincts/responses that seamlessly antagonize the unhealthy parts of cults while allowing for communities and new ideas.
    • Also: Interpersonal power: How does it work?

I feel like there’s more to learn from this experiment, but I really needed to write *something* today and a draft post I had on a beautiful theory of mine took a hit from an ugly gang of facts, so this is it.

Coping with Cats

I love my two cats dearly and they bring a lot of joy to my life, but they also bring other things, like poop, and fur.  When I tried to research mitigations for any of these it was 100% SEO listicles, so I’m sharing my raw data in the hopes that the next person can find something real-if-incomplete instead of a list of every automated litter box that has ever been manufactured.

I wrote the bulk of this in February, but then one of the cats developed dermatitis and I put it on pause until I could make sure it wasn’t caused by anything on this list. Alas, it wasn’t, she’s just allergic to my new apartment. But then I wanted to test something else, and suddenly six months had passed. So if the tenses seem weird, that’s why.

(Links are affiliate links when possible)

Smell

I screwed up choosing my new apartment (this was written before the allergy thing). I remembered to check for space to put the litter box in the bathroom, but in my joy at finding a place that allowed that at all after so many that did not,  I forgot to check if that space was also where you stood to wash your hands. So I ended up having to put the box in the living room. I’ve done that before and didn’t want to go back to having to keep the window open year round and still smelling poop. I also didn’t love the aesthetic.  So I set up something kind of elaborate.

KittyVent Litter Box Ventilator

This is a long piece of hosing with a gentle fan in it, and a plastic thingy to block off a window that has a hole for the hose to connect to (similar to what you’d use for a portable air conditioner)

The window-blocking thing works vertically but won’t stay up on its own, I had to tape it. The fan is extremely quiet, you can’t tell it’s on from more than six inches away. I’m very sure it works because the first time I noticed a smell, I found the fan had been turned off.

There are instructions for how to DIY this available online (e.g. here), but I just bought it and it worked fine.

Litterbox Holding End Table

To make it less hideous, I got an end table designed to hold a litter box. Amazon is full of these, but most have only a single entrance and no ventilation. I assume these are aimed at people who haven’t thought things through. I wanted something I could connect a vent to and actually get rid of the smell. You might be able to cut a hole in one of the ventless ones to attach the hose to but I didn’t feel like bothering, so I got one with ventilation.

I chose ecoFlex (affiliate link)  because matched my it existing furniture and was the only option with both ventilation holes and a second shelf. For this, I accepted it being really much larger than it needed to be. Also the copytext said something about liquid-proof, non-smell-absorbant material, although who knows how true that is. Then I packing taped the ventilation hose to a hole in the back that wasn’t shown in the pictures, apparently intended for electric cords to automated litter boxes.

My cats adapted stupidly well to this- when I set up the litter box with the door open, they both used it within 10 minutes, despite their other litter box still being available. They did just as well when I closed the door.

Some litter does escape the table, so have a plan for that.

Dog Poop Bags

My weird conservation instincts push me not to empty the litter box until I can fill a bag. Otherwise I’m wasting valuable bag. So I bought a bunch of very small, very cheap bags and stored them in that shelf in the litter box end table, which is why I wanted it so much. I chose Amazon Basics Dog Poop Bags (affiliate link) based on WireCutter’s recommendation and so far they are resoundingly fine.

Dr Esley Cat Litter (affiliate link)

I got this because Sweethome recommended it. I didn’t appreciate how much better it was working until a delivery screw up made me use Arm and Hammer and there started to be a smell where there hadn’t been before.

Smell- Things That Didn’t Work

Litter Robot 3

This is so close to being a truly great product, but isn’t quite reliable enough to be the only litter box. It had a tendency to get stuck in an unusable position, which eventually forces the cats to go elsewhere. The drawer the poop stays in doesn’t completely contain the smell. Also the range of litter between “excess will get dumped” and “too low” is uncomfortably thin. I’d still recommend it as a second litter box to cut down on scooping, or in a populous household where there will always be someone to turn it off and on again.

Two Layer Litter Mat (affiliate link)

This actually worked great at its stated purpose of catching litter the cats track out of the box, but when the cats pooped on it (say, because the litter box had been upside down for six hours), it was almost impossible to clean, especially if I caught it less than instantly. Plus it is ugly.

Open Window with a Fan

This was definitely better than not having an open window with a fan, but there’s no comparison between it and the vent-with-hose.

Cat Hair

Silicone Grooming Gloves (affiliate link)

These collect a lot of hair, plus their squishiness means you can brush bony areas that would be a problem for a metal comb. The cats start purring the instant I pet them with these.

Downside: kind of a pain to get the hair off the glove, does not pick up 100% of the fur it loosens, so be sure to use it somewhere easy to vacuum or at least not your own lap.

The Dryer

Turns out dryers are reasonably good at removing cat hair if you wait long enough. Remember to clean the lint trap frequently.

Reusable Lint Roller

I tried two made of the same material- a paddle shaped one (affiliate link)

and a more traditional roller shaped one (affiliate link)

Of these, I found the wand shaped one easier to use, and the roller easier to clean. I prioritized use, but your mileage may vary. Also, you need to actually read the instructions on how to use it, running it back and forth like disposable lint roller just redistributes the hair.

Here is a picture of a pillow after I ran the paddle shaped one over part of it several times. I think this gives a good idea of what it’s capable of, i.e. it’s a significant improvement but doesn’t get everything.

Note: inbetween writing this and publishing, the paddle brush people offered me actual money to publish a review, which I plan on taking them up on, assuming I can find the form.

Shark Pet-Perfect II Handheld Vacuum (affiliate link)

Sweethome’s top handheld vacuum pick was sold out everywhere and their second is specifically called out as being bad at fur, so I selected based on Amazon reviews. As a vacuum cleaner it is Not Fucking Around- it is super powerful and sucks anything off my floors. It advertises a special whirling brush aimed specifically at pet hair, but I found it choked on soft surface I tried it on, and the bristles were better. The biggest complain on Amazon was the battery life, but I got it for spot cleaning not marathons, so it’s been fine and I appreciate not paying for a feature I don’t use.

All of the anti-cat-hair devices work better if I stay very on top of them. Given time, the fur works itself deeper into the fabric and nothing will work except picking it out with actual human fingers.

Cat Hair- Mixed Results and Stuff that Didn’t Work

The Furminator (affiliate link)

This is another one that was great at its official objective but failed a secret objective. It was absolutely amazing at removing fur from the cats (on areas broad and unbony enough to support the comb), and they liked it, but it left them less soft. I did not get these cats to have them at less than their maximum softness.

Rubber Thingies You Throw in the Washer and Dryer 

I ran an experiment and as far as I can tell these are no better than just running an article through the dryer. Before/after photos (which I unfortunately didn’t save) didn’t show more hair being removed and the thingies are completely free of hair after many uses, which suggests to me they’re not helping much.

I tried wetting them and running them over a pillow case and they indeed picked up something, so I’m leaving them in my washer just in case.

I used LEADTEAM (affiliate link) and didn’t test any others, they looked identical.

Wool Dryer Balls (affiliate link)

Smart Sheep Reusable Wool Dryer Balls, 3-Pack

Testing the dryer inserts is difficult because it’s really hard to get a control group. I can’t put in half a blanket and see what the other half does. I can’t even use my two identical blankets, because they’re not guaranteed to have identical amounts of hair on them. I tried with socks but was kind of out of patience by that point. I was pretty confident the rubber ones were doing nothing, but despite a really silly amount of rigor involving photos and test swabs with the lint roller, I just can’t tell with these.

Disposable Lint Rollers

These were better than nothing but not as good as the reusable roller.

Vomiting

My cats were never great at eating but had gotten to the point where between the two of them someone was vomiting almost every day without cause (cause = “I went too long without eating”, “I ate too fast because I went too long without eating” and hairballs), and one had lost a noticeable amount of weight (her current weight would be fine if she’d started there, but losing almost 20% of her body weight for no known reason is concerning). There was also behavior that made me think they didn’t like their food- things like persistently begging and then when I put new food down, sniffing it, looking upset and begging more. So I talked to the vet.

Prescription Food and/or B12 vitamins (affiliate link)

I started both of these at the same time and now the cats are down to vomiting once a week, tops, some of which is for cause. They don’t love the prescription food by they will eat it. They’re fine with the vitamins.

In terms of specific brands:

  • They hate Royal Canin Duck and Pea protein. When I included this in their diet they ate only two cans of food per day (one Royal Canin, one of their old food, Friskies), which is little enough that the already-underweight one was losing more weight. Also it is even more expensive than regular prescription food. I tried it because it’s supposed to be hypoallergenic, but it turns out the cat’s problem was not food allergies.
  • They will tolerate Hills Digestive Care, although they eat it slower than the Friskies (but I think they eat the Friskies faster now that they’re eating Hills at all).
  • Purina Pro Plan Veterinary Diets EN Gastroenteric Formula is perhaps liked better than the Hills.

In terms of where to buy:

  • There’s a rebel pet store in Oakland that sells prescription pet food without a prescription (thanks federal government, for taking time out of your busy schedule to prevent me from paying more money for food my cats like less without doctor’s approval), but not any cheaper than stores that do require a prescription.
  • For delivery I use chewy.com, which is set up to track prescriptions so you don’t abuse that sweet, sweet easy-digestion cat food. Be warned they do not warn you when they’re about to send an autoship order, so you need to stay on top of that yourself.

Small Radius Carpet Cleaner (affiliate link)

This carpet cleaner deep cleans the 9 inch (?) circle you put it down on. This is the perfect size for cat vomit. I only used this for 9 months until I moved somewhere with hardwood, but I did get the security deposit back on the carpeted place.

Downsides: noisy; downstairs neighbor will complain if you use it too late.

Couch Cover

Between the vomiting and the hair, it was really essential my couch have a removable, washable, dryer-able cover. I got a couch that came with removable, replaceable covers, but

  1. the covers are almost 1/3 the cost of the couch
  2. they were extraordinarily difficult to get off and on

At which point the covers might as well be attached to the cushions after all. I selected a more universal couch cover from amazon with the qualifications that

  1. It handle the fact that my back cushions are removable and sit on top of the seat cushions
  2. It be able to go in the dryer, which is critical to the “don’t live life covered in cat hair” plan.

That left me a single choice, the Chun-Yi one piece (affiliate link).  Elsewhere in this article I’ve stolen Amazon’s pictures to showcase an object: in this case I’m deliberately not doing so, because this cover does not look much like the picture. It fits fine, but not nearly as crispy as implied, and the color is off.

Having a loose cover like this is also a pain in the ass for spot-hair-removals with the vacuum cleaner, because as mentioned above the vacuum cleaner’s fancy twirling brush just sucks the cover off the couch. The with bristles works well enough but still requires two hands to do well with.

Predictions As A Substitute For Reviews

Introduction

There is a lot of praise among my friends for weekly reviews. There is noticeably less doing of weekly reviews, even among the people doing the praising. It’s a very hard habit to keep up. We could spend a lot of time diving into why that is and how to fix it in a systematic way… or I could tell you how I use PredictionBook to get many of the promised benefits of weekly reviews without any willpower.

In a nutshell, when I find myself making a choice about how to spend time or money that’s dependent on some expectation, I write out the expectation as a prediction (with % likelihood) in PredictionBook, which then automatically prompts me to evaluate the prediction at a date I set. This has so many benefits I’m struggling to figure out where to start. If I had to sum them up in a phrase, it would be “more contact with reality”. But to expand on that…

What contact with reality may look like

Benefits

Making the prediction forces me to assess my anticipated outcomes and do an expected value calculation. This in turn forces me to be explicit about what I value, and how I will know if I got it. It also makes it explicit when I’m doing something because I expect the median or modal outcome to be good, vs. when I’m doing it for the long tail of unlikely but super good outcomes. It also gives me a chance to say “huh, that EV is not competitive” and do something else.

Evaluating the prompt gives me more information on how my plans are working out. Andy Matuschak talks about the dilemma of knowledge work, where you can’t make a plan for success but leaving things open ended often leads nowhere. His solution to this is to give himself unstructured time, but then look back and see if it worked out the way he wanted, and if it didn’t, do something different next time. Predictions provide a really natural, lightweight prompt for this reflection.

Sometimes I withdraw a question for being poorly phrased- often when my answer makes the decision sound “bad” but I feel like it was actually “good”, or vice versa. This is an easy way to notice I was wrong about what I valued and look for what is actually driving my decisions.

Then there are the framing benefits. It’s easier to view something as an experiment if you’re explicitly writing down “5% chance of success”, which makes failure feel less bad. I didn’t fail to make something work, I executed the correct algorithm and it failed to produce results this particular time.

Making an explicit prediction and writing it down closes open loops in my head, freeing RAM for other things.

It’s helpful to notice when the habit doesn’t trigger. For example, I have a meeting planned today (when I’m writing this) that I just “didn’t feel inspired” to make a prediction for. When I looked closer I realized that’s because I was kind of eh on this meeting but felt obliged to take it, and was avoiding that fact. I don’t know how this is going to work out because I’m choosing to work on this blog post before tackling that, but I still credit the prediction habit with noticing, which is the first step towards better choices.

Also it seems like this might make my predictions more accurate, which has all kinds of applications. But to be honest that’s not what’s reinforcing this.

Examples

So we’re on the same page, here are some sample predictions, and the decisions that rested on them:

  • Reading Pavlov and his School will lead to behavior change in the area of learning or sleep (20%)

    • Should I buy a physical copy of Pavlov and his School and spend the time to read it?

  • Talking with my friend Jane on 8/1 will be energizing (95%)

    • Should I take a call with Jane?

  • Bob the recruiter will describe a job that I am at all capable of and interested in doing (3%)

    • Should I take a call with Bob?

  • I will judge the seminar on 7/28 will be worth the interruption of flow (40%)

    • Should I spend money and time on this seminar?

  • Sam will finish task Y by 7/31 (20%)

    • No immediate decision riding on this one, but it sure seems useful to calibrate on how well I can predict a project partner’s productivity.

  • I will play with an Oculus Quest at least an hour a week (98%)

    • Should I buy a Quest?

  • California will catch fire to the point I need to keep my windows closed (95%)

    • Should I buy an air conditioner?

  • Company Z will offer me at least an hour of work (15%)

    • No immediate decision

  • Supplement S will improve my sleep to the point I don’t need a nap (3%)

    • Should I buy and use supplement S?

Instructions

If I were starting from scratch, here are the instructions I would want to receive (many of which I did receive, from Raemon)

  1. Create an account on PredictionBook.com

  2. Go to Settings to correct your time zone and set your prediction default to “visible to creator” rather than “visible to public” (unless you’d like them to be public by default).

  3. Create a handful of predictions that will resolve over the next week (PredictionBook will e-mail you when they should be resolved).

    1. The goal is twofold:

      1. Quickly get feedback about how this feels for you.

      2. Get yourself in the habit of making and resolving predictions.

    2. When making predictions, try to hone in on things that are decision-relevant to you; that you would do something different based on if the prediction was true or not.

  4. Create a link to https://predictionbook.com/predictions/new in your browser so you can access it easily.

  5. As you take more actions based on predictions (implicit or explicit), notice that you are doing so, and register the predictions in PredictionBook

  6. Resolve predictions as you are prompted to do so.

  7. After a week or two, check in with yourself and how you feel about the project. For calibration: I found creating the initial predictions kind of a pain but by a week in the project was naturally rewarding and required no will power on my part.

This should take < 30 minutes, and it should take < 45 minutes total to get a good sense of whether this is working for you.

Why Does This Work?

I’ve been thinking a lot about flow and distraction recently. One thing I’ve noticed is that it takes an awful long time, measured in hours, to get into the mindset for certain tasks. Those tasks then feel amazing, unless I’m pulled out of them prematurely, which hurts (and makes it take even longer to get into the next time). There’s a lot of implications of this that I’ll hopefully get to in some other post; the relevance here is that I think weekly reviews might be one of those tasks that requires a lot of time to get into the required headspace and hurts to leave prematurely. This makes them costly- much more costly then I would have estimated before I learned to bill tasks for their prep time. It’s also something of an all or nothing task- doing 50% of your weekly review does not get you nearly 50% of the benefits of a full weekly review.

But predictions, both making and evaluating, integrate into my life pretty naturally most of the time, and scale gracefully. Writing them up can take as little as 15 seconds, and when it takes longer, it’s because I’ve discovered something important I need to work out. Doing this fits in naturally with the process of making plans, so I don’t need to spend time getting into the right head space. Evaluating them is usually trivial- and if it’s not, it’s highlighting a problem with my models.

Tips and Tricks

“I will enjoy X” is very rarely the right prediction for me to make, because enjoyment is a tricky thing for me. I don’t like admitting I didn’t enjoy something, and sometimes I will burn a lot of energy trying to make something enjoyable, which can be a bad decision even if it works. My equivalent is “X will be energizing”

When trying to create predictions, think about what you’ll be spending money and time on in the next week. How do you think those things are going to go? What outcomes would make you change your decision?

Caveats

My original inspiration for this experiment, Raemon, hasn’t gotten the benefits I describe. He’s more focused on improving his prediction calibration and accuracy, and so far hasn’t done as much decision-relevant predictions. I’ve encouraged him to try it my way and hopefully he’ll comment after he’s spent a few weeks on that.

PredictionBook is pleasantly lightweight but it’s already a little cluttered with predictions. If I really do solidify this habit I may need to find or create a better system.

I’ve been doing this for about a month. People have kept up weekly reviews for longer before letting them fall by the wayside. I’m writing this up now because the act of solidifying a habit makes me worse at writing about it. I’ve set a reminder for 12/1 to write an update.

Thanks to Raemon for pushing me to a better version this post and for initiating the experiment in the first place.

This is pretty inside baseball and I suspect boring to most readers. You have my blessing to skip it.

Two weeks ago I published Knowledge Boostrapping v0.1, an algorithm for turning questions into answers. Since then I’ve gotten a moderate amount of feedback, a major theme of which is lack of clarity in certain sections. That’s entirely fair, but not something I immediately know how to fix. In an attempt to make some headway on the problem, I recorded in detail the steps I took in a recent attempt at using a method.

This isn’t really a central case of use- the question was of business norms, not The Truth, and the research I did more primed me to think of the solution myself, more than it provided the answer. But non-central examples are sometimes more useful and the cost to sharing is low given, so here we go.

 

 

  1. 12:30 Working on a project with someone, get to the point where we think “we should have a white paper”

  2. Problem: I’ve never written a white paper, and he has but isn’t really clear on what the goals or audience are. I decide this is worth an hour to get right.

  3. 12:45 Follow my own advice to come up with questions to investigate.

  4. Step 1 is make a mind map of questions. Discover Whimsical’s free plan only allows four creations, ever, even if you delete old work.

    1. Go on FB to complain to friend who recommended it to me

    2. 12:57. Try to create in Roam instead
      Start page currently looks like:

  5. 12:58 Google “white papers 101”

  6. 12:58: first hit https://thatwhitepaperguy.com/tips-best-practices-white-papers/ . Seems…uncomprehensive? Not high level enough? Extremely vague to the point of uselessness?

  7. 12:59 fight with NYTimes paywall https://www.nytimes.com/2009/05/12/business/smallbusiness/12toolkit.html . Article itself is useless but links to some promising pages.

    1. 1:01 https://smallbiztrends.com/2009/04/its-national-write-a-white-paper-month-april-2009.html . It’s basically a link to http://www.rgmcomms.com/whitepapers.html

      1. 1:02 http://www.rgmcomms.com/whitepapers.html no longer has the PDF referenced above

    2. 1:02 http://www.howtodothings.com/hobbies/a4622-how-to-write-a-white-paper.html

      1. Not useless. Begin notes on the top level Roam page, because I expect to only extract a few things from it.

      2. White papers can be aimed at a variety of audiences. Know which one you want and tailor it to them.

        1. 1:06 He doesn’t say it explicitly, but presumably a white paper is aimed at people you want things from. I think about who [partner] and I want things from

          1. This is kind of a dumb revelation. If white papers weren’t aimed at who we wanted things from, we should write a different kind of thing aimed at people we wanted things from.

          2. Rest of page is kind of vapid but I’m feeling pretty inspired.

 

 

 

 

 

 

  • 1:10 Remember to tag parent Roam page as a question. I would show you what Roam page looks like now, but apparently I forgot to take a picture.

  • Take that prompt and think a bit, till I come up with an algorithm I’m happy with

 

Knowledge Bootstrapping Steps v0.1

Introduction

I wanted to have this published weeks ago, but the process of writing it highlighted some gaps, and the experiments to fill in those gaps highlighted more gaps, and… it’s become clear there’s always going to be one more experiment to run, and if I’m going to publish at all, it will be with gaps.  Furthermore, publishing may elicit feedback that alleviates the need to run some of the experiments, or suggests better ones. So with shaking fingers I am going to publish an incomplete instruction set, and hope it does more good than harm.

The goal of this system is to turn questions into answers. It does this by collecting questions and assembling data that relate to them until you can come to a confident answer, or at least an answer with a confidence interval.  This system has worked amazingly well for me, but I am one person. I am very sure that 

  1. It will need to be tweaked for other people 
  2. There are important steps I’m doing without noticing and thus didn’t know to write down. 
  3. This isn’t its final form for me either.

One of my goals in publishing is to get data from other people on how it doesn’t work for them, so I can universalize it. To that end:

  1. You can comment here
  2. You can email me at elizabeth@acesounderglass.com
  3. You can talk to me for an hour
  4. I am offering a limited amount of coaching on using the method. If you’re interested in this, please reach out at that same e-mail address.

The following system was designed to be run on Roam, which when I started was free and freely available. There is now a subscription fee, and off and on a waitlist. For me it’s worth it (well, will be, when they get around to charging legacy accounts), but I don’t know where the line of “worth it” falls for other people. I’ve got a list of untested alternatives at the bottom of this post, but the instructions will assume you’re working in Roam.

 

The Basics

The basic gameplay loop of this system is:

  1. Create a question
  2. Read sources to gather evidence bearing on question, and save it in notes
  3. Assemble evidence new and old into an answer (synthesis).

That looks pretty simple, but raises a number of questions:

  1. How do you create a good question?
  2. How do you choose what sources to read?
  3. How do you know what to write down?
  4. How do you synthesize the information into an answer?

I’ve linked to the questions where I have stabs at an answer; the remainder is still an intuition-based art to me.

 

The Details

Those steps are also pretty vague. Here’s the detailed version:

  1. What do you want to know? Try to be as specific as possible, but no simpler (i.e. leave room to be surprised). Create a Question page for this question (example, template, justification). 
  2. Break that question into smaller sub-questions, and continue until it feels indivisible at your current level of knowledge. Create links on the main question to all of your sub questions (example).
  3. Answer any questions you are confident you already know the answer to
  4. Pick the question that feels most live/interesting to you
  5. Write down everything you already know about the question, including things that don’t seem relevant but keep popping into your brain.
  6. Find sources for it (“How do I do that, Elizabeth?” “Still figuring out how to make that knowledge explicit, reader, but take a look here”).
  7. Pick the most promising looking source
    1. Create a Source page for it (template, example). Fill in metadata, including (in order of declining importance):
      1. URL: If you need to look something up about the work, you want to make sure you’re referring to the same version every time.
      2. Pagination: the type of pagination system you’re using (Kindle location and PDF page number are my most popular) and what it’s total count is in the version of the work you’re using. This is very useful when there are multiple versions or editions of a work and you want to make
      3. Preparatory brain dump: write down everything you already know about this work, or the topic in general (if you didn’t get it out of your head already). 
        1. Things this might include:
          1. Feelings on the author
          2. Why I chose to read this particular work
          3. Questions I have about the book already.
        2. This is important to me for reducing drag while reading: Everything I write in this spot would otherwise be taking up mental RAM.
      4. Subject: Tags for the subjects of the work (e.g., history, biology, economics, the Great Depression, covid). You can have as many tags as you want, and more is generally better, limited by your patience.
      5. Author: This seems really important for collating work from specific authors, but to be honest it’s never come up for me.
      6. Table of contents (long form only): Writing this down (or more likely, copy/pasting it and then correcting the formating) is helpful to me in orienting to topics and structure of the book. Theoretically I could achieve the same thing by reading the table of contents and paying appropriate attention, but in practice I don’t.
      7. Year published
    2. The last section on the form is Notes. 
      1. It will look something like this:
        1. Notes
        2. Introduction Pre-read
        3. Introduction
        4. Introduction summary
        5. Chapter 1 Pre-read
        6. Chapter 1
        7. Chapter 1 summary
      2. A “pre-read” is reading a few paragraphs from the beginning or ending of a chapter, and summarizing what they tell you about the chapter (it’s topic, approach, criticisms the author felt the need to preemptively defend against…). This will make it easier to determine which of the chapter’s claims are important.
      3. The actual chapter content looks like the following:
        1. Chapter 1
          1. Claim: Columbus set sail from Europe in 1492
            1. Pg 1
          2. Claim: Columbus’s expedition was financed by Queen Isabel of Spain
            1. Pg 2
        2. Not sure what to write down? You probably need to refine your question.
        3. Write down something relevant to one of your questions? Create a block reference to it under the Question’s evidence bullet, or copy/paste it if you’re not using Roam
      4. The chapter summary is what you think it is. I typically have to refer back to my notes to create the summary, but sometimes realize in the process of summarizing that I’ve left out something key and need to add it back in.
      5. Rinse and repeat.
      6. Did you record a claim that seems particularly interesting, important to the author’s thesis, or suspicious? Time to recurse. Create a Question for which that claim could be evidence (e.g. “Who financed Columbus’s expedition?”). Repeat this process with that question, from the beginning.
  8. Assemble claims from multiple sources under questions until you feel confident in an answer. 
  9. Change title of page to Synthesis: My Conclusion (example).
  10. Work your way up until you’re at your top question.
  11. You should now have answers to all your sub questions, allowing you to answer it, or failing that be confident in your uncertainty.
  12. CONCESSIONS TO REALITY
    1. If a work is only ever going to be referenced by one question, you don’t need to create a separate Source page for it, just give it a bullet under evidence (example).

 

Addendums

The following are oft-mentioned competitors to Roam, although to the best of my knowledge none of them have block embedding that I consider crucial:

  1. Google Docs
  2. Obsidian (locally stored)
  3. Workflowy (lacks bidirectional links and cross-list embedding)
  4. Notion (I’ve used this for large projects and hate it)

 

EDIT 2020-07-20: Added some small changes based on feedback, and remembered to give credit to the Long Term Future Fund for funding this research.

Breaking Questions Down

Previously I talked about discovering that my basic unit of inquiry should be questions, not books. But what I didn’t talk about was how to generate those questions, and how to separate good questions from bad. That’s because I don’t know yet; my own process is mysterious and implicit to me. But I can give a few examples.

For any given question, your goal is to disambiguate it into smaller questions that, if an oracle gave you the answers to all of them, would allow you to answer the original question. Best case scenario, you repeat this process and hit bedrock, an empirical question for which you can find accurate data. You feed that answer into the parent question, and eventually it bubbles up to answering your original question.

That does not always happen. Sometimes the question is one of values, not facts. Sometimes sufficient accurate information is not available, and you’re forced to use a range- an uncertainty that will bubble up through parent answers. But just having the questions will clarify your thoughts and allow you to move more of your attention to the most important things.

Here are a few examples.  First, a reconstructed mind map of my process that led to several covid+economics posts. In the interests of being as informative as possible, this one is kind of stylized and uses developments I didn’t have at the time I actually did the research.

Vague covid panic@2x.png

If you’re curious about the results of this, the regular recession post is here and the oil crisis post is here.

Second, a map I created but have not yet researched, on the cost/benefit profile of a dental cleaning while covid is present.

Risk model of dental cleanings in particular@2x.png

Aside: Do people prefer the horizontal or vertical displays? Vertical would be my preference, but Whimsical does weird things with spacing so the tree ends up with a huge width either way.

Honestly this post isn’t really done; I have a lot more to figure out when it comes to how to create good questions. But I wanted to have something out before I published v0.1 of my Grand List of Steps, so here we are.

Many thanks to Rosie Campbell for inspiration and discussion on this idea.

How to Find Sources in an Unreliable World

I spent a long time stalling on this post because I was framing the problem as “how to choose a book (or paper. Whatever)?”. The point of my project is to be able to get to correct models even from bad starting places, and part of the reason for that goal is that assessing a work often requires the same skills/knowledge you were hoping to get from said work. You can’t identify a good book in a field until you’ve read several. But improving your starting place does save time, so I should talk about how to choose a starting place.

One difficulty is that this process is heavily adversarial. A lot of people want you to believe a particular thing, and a larger set don’t care what you believe as long as you find your truth via their amazon affiliate link (full disclosure: I use amazon affiliate links on this blog). The latter group fills me with anger and sadness; at least the people trying to convert you believe in something (maybe even the thing they’re trying to convince you of). The link farmers are just polluting the commons.

With those difficulties in mind, here are some heuristics for finding good starting places.

  • Search “best book TOPIC” on google
    • Most of what you find will be useless listicles. If you want to save time, ignore everything on a dedicated recommendation site that isn’t five books.
    • If you want to evaluate a list, look for a list author with deep models on both the problem they are trying to address, and why each book in particular helps educate on that problem.  Examples:
    • A bad list will typically have a topic rather than a question they are trying to answer, and will talk about why books they recommend are generically good, rather than how they address a particular issue. Quoting consumer reviews is an extremely bad sign and I’ve never seen it done without being content farming.
  • Search for your topic on Google Scholar
    • Look at highly cited papers. Even if they’re wrong, they’re probably important for understanding what else you read.
    • Look at what they cite or are cited by
    • Especially keep an eye out for review articles
  • Search for web forums on your topic (easy mode: just check reddit). Sometimes these will have intro guides with recommendations, sometimes they will have where-to-start posts, and sometimes you can ask them directly for recommendations. Examples:
  • Search Amazon for books on your topic. Check related books as well.
  • Ask your followers on social media. Better, announce what you are going to read and wait for people to tell you why you are wrong (appreciate it, Ian). Admittedly there’s a lot of prep work that goes into having friends/a following that makes this work, but it has a lot of other benefits so if it sounds fun to you I do recommend it. Example:
  • Ask an expert. If you already know an expert, great. If you don’t, this won’t necessarily save you any time, because you have to search for and assess the quality of the expert.
  • Follow interesting people on social media and squirrel away their recommendations as they make them, whether they’re relevant to your current projects or not.

Emotional Blocks as Obstacles to Learning

My goal was to come up with a system for reading a book. I eventually identified that as the wrong goal, but came up with a pretty great system for doing the much better goal of “how do I answer a question?” But developing that was not the hardest or most time consuming part of my research over the last 3 months (plus additional time working on covid). I feel weird talking about it, but the truth is, a lot of that time was spent overcoming emotional issues around learning. 

For example, I think I’ve discussed before (but could not find a link on) how I kind of have two modes when reading: too credulous, looking for reasons a work could be true, and too antagonistic, looking for reasons to not only disagree, but dismiss entirely. 

I introspected on this, and eventually figured out that at a deep level I felt I needed to believe books, that I was being bad if I disagreed with them. So of course I developed tools to prove my disagreements, which led to the bifurcation- either I was giving in to the original impulse or its counter, without the option of responsiveness.

This same block on challenging authority was behind my urge to start from a book rather than a question. I not only believed I needed to trust in an authority (as deemed by the publishing-industrial-complex) to give me answers, I needed to let them set the questions. 

A natural question here is “why are you so sure this emotional work helped this specific task?”  My evidence is how the needs-to-be-retitled-epistemic-spot-check project has evolved- I started out having books thrown at me and reading them with the goal of forming a yes-no verdict. I’ve now progressed to starting with a question (such as “What can the 1973  Oil Crisis tell us about supply shocks?”) that serves a specific purpose and finding work that advances it. In that post I derived a model from disparate pieces of information I sought out to answer specific questions. No books, no teachers, just me. I also have pretty extensive notes of the work I was doing and how it tracked to specific improvements, although they’re intensely personal and I will not be sharing them.

It’s not totally solved yet. I really wanted to read a book on the oil crisis, for exactly the reasons they’re a bad solution to the problem. I wanted someone to give me the answer. But I can at least see the desire for what it is, recognize that it’s not a desire to learn, and react appropriately.

Another natural question is “why does this happen?” There’s two answers to that- why did I specifically form that belief, and why did those circumstances have the power to make me form that belief. I have some guesses for the former; my failed state middle school where I was dependent on the goodwill of teachers for my physical safety is a top contender, although the “best” schools arguably use a more subtle stick to inculcate this attitude even more strongly. 

For the latter, I have a very rough theory, dependent on the types of knowledge I described here. Very crudely, I believe trauma instills scientific-type knowledge that is factually false but locally adaptive. False beliefs need more protection to be maintained than true beliefs, so the belief both calcifies, making it unresponsive to new information, and lays a bunch of emotional landmines around itself to punish you for getting too close to it. This cascades into punishing you for learning at all, because you might learn something that corrects your false-but-useful model.

How did I escape these traps? I have some guesses for that, but I can only confidently identify the things I did immediately before breakthroughs. I’ve been building these skills for 10 years, so there’s a lot of background knowledge and skill feeding into that success I don’t have conscious access to. I think instructions for only the immediate predecessors of break throughs could be useless to outright harmful. So my next project is figuring out more about this process, and hopefully finding generalizable techniques for improvement. 

Part of finding techniques that work on people other than me is talking to other people. If you’re interested in ways of contributing to or just using the knowledge, here are some options:

  1. Already done this work? Tell me more. You can comment here, e-mail elizabeth@ this-domain.com, or use this anonymous form.
  2. Want to hear ideas and try them out yourself, ideally reporting back to me? Sign up for this google group.
  3. Want to devote several months of your life to working with me on this intensely? A number of things would have to go right for that work out, but if they all did, I think the potential is enormous. Email me at elizabeth @ this-domain

 

EDIT 7/15: Greetings again, Hacker News readers. This piece is the penultimate post in a long saga. If it strikes a chord with you, I’d encourage you check it out from the beginning, and check in in a few days for climax and epilogue. You can also follow me via RSS, via email by clicking the Follow button at the bottom right of the page, or on Twitter.  Also while this post wasn’t on Patreon, many of mine are, and support is greatly appreciated. You can also Talk To Me For An Hour, although we’ll see how that stands up to the influx of new readers.

 

Types of Knowledge

This is a system for sorting types of knowledge. There are many like it, but this one is mine.

First, there is knowledge you could regurgitate on a test. In any sane world this wouldn’t be called knowledge, but the school system sure looks enthusiastic about it, so I had to mention it. Examples:

  • Reciting the symptoms of childbed fever on command 
  • Reciting Newton’s first law of motion
  • Reciting a list of medications’ scientific and brand names
  • Reciting historical growth rate of the stock market
  • Reciting that acceleration due to gravity on Earth is 9.807 m/s²

 

Second, there is engineering knowledge- something you can repeat and get reasonably consistent results. It also lets you hill climb to local improvements. Examples:

  • Knowing how to wash your hands to prevent childbed fever and doing so
  • Driving without crashing
  • Making bread from a memorized recipe.
  • What are the average benefits and side effects from this antidepressant?
  • Knowing how much a mask will limit covid’s spread
  • Investing in index funds
  • Knowing that if you shoot a cannon ball of a certain weight at a certain speed, it will go X far.
  • Knowing people are nicer to me when I say “please” and “thank you”

 

Third, there is scientific knowledge. This is knowledge that lets you generate predictions for how a new thing will work, or how an old thing will work in a new environment, without any empirical knowledge.

Examples: 

  • Understanding germ theory of disease so you can take procedures that prevent gangrene and apply them to childbed fever.
  • Knowing the science of baking so you can create novel edible creations on your first try.
  • Knowing enough about engines and batteries to invent hybrid cars.
  • Actually understanding why any of those antidepressants works, in a mechanistic way, such that you can predict who they will and won’t work for.
  • A model of how covid is spread through aerosols, and how that is affected by properties of covid and the environment.
  • Having a model of economic change that allows you to make money off the stock market in excess of its growth rate, or know when to pull out of stocks and into crypto.
  • A model of gravity that lets you shoot a rocket into orbit on the first try.
  • A deep understanding of why certain people’s “please”s and “thank you”s get better results than others.

 

Engineering knowledge is a lot cheaper to get and maintain than scientific knowledge, and most of the time it works out. Maybe I pay more than I needed to for a car repair; I’ll live (although for some people the difference is very significant). You need scientific knowledge to do new things, which either means you’re trying something genuinely new, or you’re trying to maintain an existing system in a new environment.

I don’t know if you’ve noticed, but our environment was changing pretty rapidly before a highly contagious, somewhat deadly virus was released on the entire world, and while that had made things simpler in certain ways (such as my daily wardrobe), it has ultimately made it harder to maintain existing systems. This requires scientific knowledge to fix; engineering won’t cut it.

And it requires a lot of scientific knowledge at that- far more than I have time to generate. I could trust other people’s answers, but credentials and authority have never looked more useless, and identifying people I trust on any given subject is almost as time consuming as generating the answers myself.  And I don’t know what to do about that.