Alternate Views On Long Covid

Scott Alexander has published a post on long covid, which he rates as much more frequent and dangerous than I do. Scott and I spent a while hashing this out in private, and our cruxes seem to come down to:

  1. I think his studies are too small and sample-biased to be meaningful.
  2. He thinks my studies (especially Taquet) didn’t look at the right sequelae.
  3. I was only looking at cognition (including mood disorders), whereas he looked at everything.

Scott also didn’t do age-specific estimates, although that’s not a crux because I expect other post-infection syndromes to worsen with age as well.

I intended to include fatigue in my analysis of cognitive symptoms but in practice the studies I weighted most highly didn’t include them. Scott’s studies, which he admits are less rigorous although we differ on how much, did include them. Why the hell aren’t the large, EHR-based studies with control groups looking at fatigue?

Also, this isn’t relevant to the covid disagreement, but I baffled by the medical systems’ decision to declare chronic Lyme in particular as the definitely psychosomatic syndrome, given that Lyme is closely related to syphilis, which we know damn well has a long dormant period and a stunning array of possible long term consequences.

Although I didn’t update much on this particular disagreement, I have a lot of respect for Scott and encourage anyone making decisions based on bloggers’ estimates of the risk of long covid to check out his post as well.

Exercise Trade Offs: Followup

Last week I did some math on the risk/reward profile of exercising indoors (risking covid exposure) vs. outdoors (risking exposure to smoke from the CA fires), and found the numbers for the day I did the math (low-for-fire-season pollution outside, sparsely populated indoor gym) overwhelmingly favored exercise of any kind over not exercising, and any other factor was overwhelmed by how likely it was to create friction to exercising. 

Over on Facebook, a friend pointed out that I’d left out the biggest cost of exercise: the time in which it took place. I then realized a full accounting would also include the time to get to the gym and the risk of getting hit by a car en route. And was I sure the micromort estimate for exercise incorporated the risk of injury? (no, because the data is hidden in an appendix BJM paywalled and sci-hub doesn’t have. If you have BJM access and would like to help me out by emailing me (elizabeth@acesounderglass.com) the appendix for this article it would be much appreciated. EDIT: received and responded to. Thanks Steve!). But exercise has benefits beyond dying later, and I wasn’t fully accounting for any of those either. And time spent at or traveling to work out isn’t exactly lost: zipping around on my scooter is fun, and leaving my house regularly on some sort of schedule has been good for me. This gets unwieldy really quickly.

Nonetheless, time spent traveling and the accompanying risk of car accidents seemed really significant, so I updated the spreadsheet to incorporate it. Ignoring any positive effects beyond the microlives, this was enough to make going to my gym for cardio net costly (note: because the spreadsheet measures in micromorts a positive number is bad), although going to the gym for weights and my nearby friend’s backyard for cardio still come out ahead.

I still think gym cardio is net beneficial for me because I think my exercise is much more impactful than average. But I don’t think it’s so much more beneficial than my friend’s backyard treadmill, so I’m going to emphasize the latter except on very bad smoke days.

Long Covid Is Not Necessarily Your Biggest Problem

Introduction

At this point, people I know are not that worried about dying from covid. We’re all vaccinated, we’re mostly young and healthy(ish), and it turns out the odds were always low for us. We’re also not that worried about hospitalization: it’s much more likely than death, but maintaining covid precautions indefinitely is very costly so by and large we’re willing to risk it.

The big unknown here has been long covid. Losing a few weeks to being extremely sick might be worth the risk, but a lifetime of fatigue and reduced cognition is a very big deal. With that in mind, I set out to do some math on what risks we were running. Unfortunately baseline covid has barely been around long enough to have data on long covid, most of it is still terrible, and the vaccine and Delta variant have not been widespread long enough to have much data at all. 

In the end, the conclusion I came to was that for vaccinated people under 40 with <=1 comorbidiy, the cognitive risks of long covid are lost in the noise of other risks they commonly take. Coming to this conclusion involved reading a number of papers, but also a lot of emotional processing around risk and health. I’ve included that processing under a “personal stuff” section, which you can skip if you just want the info but I encourage you to read if you feel yourself starting to yell that I’m not taking small risks of great suffering seriously. I do encourage you to read the caveats section before deciding how much weight to put on my conclusions.

Personal Stuff

This post took a long time to write, much longer than I wanted, because this is not an abstract topic to me. I have chronic pain from nerve damage in my jaw caused by medical incompetence, and my attempts to seek treatment for this continually run into the brick wall of a medical system that doesn’t consider my pain important (tangent: if you have a pain specialist you trust, anywhere in the US, please e-mail me (elizabeth@acesounderglass.com)). I empathize very much with the long covid sufferers who are being told their suffering doesn’t exist because it’s too hard to measure and we can’t prove what caused it.

Additionally, I’m still suffering from side effects from my covid vaccine in April. It’s very minor, chest congestion that doesn’t seem to affect my lung capacity (but I don’t have a clear before picture, so hard to say for sure). But it’s getting worse and while my medical practitioners are taking it seriously, this + the experience with dental pain make me very sensitive to the possibility they might stop if it becomes too much work for them. As I type this, I am taking a supplement stack from a high end internet crackpot because first line treatment failed and there aren’t a lot of other options. And that’s just from the vaccine; I imagine if I actually had covid I would not be one of the people who shakes it off the way I describe later in this post. 

All this is to say that when I describe the long term cognitive impact of covid as being too small to measure with our current tools against our current noise levels, that is very much not the same as saying it’s zero. It’s much worse than that. What I’m saying is that you are taking risks of similar levels of suffering and impairment constantly, which our health system is very bad at measuring, and against that background long covid does not make much of a difference for people within certain age and health parameters. 

A common complaint when people say “X isn’t dangerous to the young and healthy” is that it implies the death and suffering of those who aren’t young and healthy don’t matter. I’m not saying that. It matters a lot, and it’s impossible for me to forget that because I’m very unlikely to be one of the people who gets to totally walk covid off if I catch it. But from looking at the data, there don’t seem to be very many of us in my age group.

Caveats

Medical research in general is really bad, research of a live issue in a pandemic is worse, you should assume these are low quality studies unless I indicate otherwise.

This research was compiled for LessWrong and Redwood Research, with the goal of assessing safety for their office spaces populated by mostly-but-not-entirely-healthy people 25-40, who were much more interested in the cognitive and fatigue sequelae than the physical. Much of this research is applicable outside that group or the sources can be used in that way, but you should know that’s what I focused on.

There isn’t any data on long covid in vaccinated people with breakthrough delta-variant infections. Neither vaccines nor delta have been around long enough for that to exist. Baseline covid has barely been around long enough to have long-term data. What I have here is:

  • Data showing that strength of acute infection correlates with long term impact, although not perfectly
  • Data on the long term impact of baseline covid, given the strength of an initial infection
  • Data on how the vaccine impacts the strength of acute infections
  • Data on how delta impacts the strength of acute infections

Data

Long term outcomes correlate with short term outcomes

By far the best study (best does not mean good) comes out of the UK, where the BBC coincidentally started an online intelligence test in January 2020 (giving them a pre-covid baseline) and in May began asking participants if they’d had covid and if so how bad a case. When I said “assume the studies are terrible unless I note otherwise”, this is the study I wanted to highlight as reasonably good. Because they can compare test-takers in a given time period with and without covid they can control for some of the effects of changing a study population over time, which would be the biggest concern. Additionally, my statistical consultant described the paper as “not having any errors that affect the conclusion”, which is extremely good for a medical paper. This study was not ideal for determining sequelae persistence, but they did check if size of effect was correlated with time since symptom onset, and it wasn’t (but their average was only 2 months).

This study showed a very direct correlation between the severity of the acute infection and cognitive decline. I don’t trust its absolute numbers, but the pattern that more severe disease -> more severe persistent effects is very clear

A second study in Wuhan, China (hat tip Connor Flexman) examined long term outcomes of hospitalized patients, based on the intensity of their care (hospitalization, supplemental oxygen, ventilation) found an increase in acute severity was correlated with an increase in sequelae, although it didn’t hold for every symptom (there are a lot of symptoms and the highest-intervention group is small), and they barely looked at cognitive symptoms.

Taquet et al used electronic health records to get a relatively unbiased six figure sample size, that also showed a strong correlation between acute and long term outcomes, which we’ll talk about more below.

From this I conclude that your overall risk of long covid is strongly correlated with the strength of the initial infection.

Odds of acute outcomes

Sah et al estimate that 35% of covid cases (implied to be baseline and pre-vaccination) are asymptomatic, with large variation by age. Children (<18) are 46% likely to be asymptomatic, adults 18-59 are 32% likely, adults >=60 are 20% likely. I’m going to round the non-elderly adult number to ⅓ to make the math easier.

The Economist has a great calculator showing your pre-vaccine, pre-Delta risk of hospitalization and death, given your age, sex, and comorbidities. Note that this calculator only includes diagnosed cases, so it excludes both asymptomatic cases and those that did have symptoms but didn’t drive people to seek medical care. Here’s a few sample people:

  • A healthy 30 year old man has a 2.7% chance of hospitalization, and <0.1% risk of death
  • A healthy 30 year old woman has a 1.7% chance of hospitalization, and <0.1% risk of death
  • A 25 year old man with asthma has a 4.2% risk of hospitalization, and <0.1% risk of death
  • A 40 year old woman with obesity has a 6.5% risk of hospitalization, and 0.1% risk of death.
  • Risk of hospitalization rises steadily with age but the risk of death doesn’t really take off until 50, at which point our healthy man has a death risk of 0.4% and our health woman has a risk of 0.2%

If you’d like, you can use your own numbers in this guesstimate sheet.

And again, that’s only for officially diagnosed and registered cases. If you assume ⅓ of infections in that age group are asymptomatic, the risk drops by ⅓.

If you are hospitalized, your risk of being ventilated is currently very, very low even if you’re in a high risk category. The overall average percent of hospitalized patients who were ventilated was 2.0% in the last week for which data was available (2021-03-24), after dropping steadily for most of the plague. We can assume that’s disproportionately among the elderly and people with severe comorbidities, so if that’s not you your odds are better still. I’m going to count the risk of intubation for our cohort as 0.5%, although that’s likely still an overestimate.

How do vaccines change these odds? According to CDC data from a time period ending 2021-05-01 (so before delta took off), 27% of breakthrough infections that reached the attention of the CDC were asymptomatic, and only 7% were hospitalized due to covid (another 3% were hospitalized for non-covid reasons). It’s very likely that the CDC is undercounting asymptomatic cases, so we’ll continue using our ⅓ number for now. The minimum age of reported breakthrough infection deaths was 71, so we’ll continue to treat the risk of death as 0% for our sample subjects. Additionally, given the timing most vaccinated participants would be elderly or front line workers, raising their risk considerably. A CDC press release goes much farther, saying vaccinated people > 65 had 7% of the hospitalizations of age-matched controls. 

How does delta change these odds? A Scottish study estimated delta had 2x the risk of hospitalization as alpha, which a Danish study estimated as having 1.42x the risk of hospitalization as baseline covid. So very roughly, we’re looking at 3x the risk of hospitalization from delta, relative to baseline.

So for our sample cases above, we have the following odds (note I updated these on the night it was posted, due to a math error. Thanks to Rob Bensinger for catching it):

Risk given vaccine, deltaHospitalizedIntubated
Healthy 30yo man0.38% = 2.7*.07*3*2/3.002% = 0.38*.005
Healthy 30yo woman0.24% = 1.7*.07*3*2/3.002% = 0.24*.005
Asthmatic 25yo man0.58% = 4.2*.07*3*2/3.003% = 0.58*.005
Obese 40yo woman0.92% = 6.5*.07*3*2/3.005% = 0.92*.005

That’s not so far from the rate of hospitalization in that age range for the flu (0.6%), with some caveats (the CDC sample includes unvaccinated people and the bucket is 18-49 years old, with the higher end presumably carrying more of the disease burden).

There is concern that vaccine effectiveness wanes over time, which I haven’t incorporated here.

Odds of long term outcomes

In general I ignored studies that merely tracked number of persistent sequelae but not their severity or type, which made it impossible to distinguish between “sense of smell still iffy” from “permanent intellectual crippling”, and studies that didn’t track how long the sequelae persisted. This was, unfortunately, most of them.

We talked about the Great British Intelligence Test above. I initially found this study quite scary. The study used its own tests rather than IQ, but if you assume a standard deviation in their tests is equivalent to a standard deviation in an IQ test, the worst category (ventilation) is equivalent to a 7 point IQ loss. That’s twice as bad as a stroke in this study (although I suspect sampling bias). I suspect the truth is worse still, because the worse your recently acquired cognitive and health issues are, the less likely you are to take a fun internet test advertised as measuring your intellectual strengths. However as I noted above, you are extremely unlikely to be put on a ventilator. 

For people with “symptoms, but not respiratory symptoms”, the cognitive damage is ~equivalent to 0.6 IQ points. For “medical assistance at home”, it’s 1.8 points. These are both likely to be overestimates given that the study only included known (although not necessarily formally diagnosed) cases. Additionally, while the paper claims to control for education, income, etc, bad things are more likely to happen to people in worse environments, and it’s impossible to entirely back that out.

Taquet et al used electronic health records to get a relatively unbiased six figure sample size, and found unhospitalized diagnosed covid patients (pre-Delta, pre-vaccine) had a 11% likelihood of a new neuro or psych diagnosis after their covid diagnosis, hospitalized patients had a 15% likelihood, and ICU patients had 26% likelihood. The majority of these were mood disorders (3.86%/4.49%/5.82% for home/hospitalized/ICU) and anxiety (6.81%/6.91%/9.79%). This seems quite bad, until you compare it to the overall numbers for depression in the time period, a naive reading of which suggests that covid had a protective effect

These numbers aren’t directly comparable. The second study is much lower quality and includes rediagnoses (although the total depression diagnosis numbers for the covid patients are 13.10%/14.69%/15.43%- still under the total increase in depression in the general population study). 

Overall this seems well within what you’d expect from getting a scary disease at a scary time, and not evidence of widespread neuro or psych impact of covid. Even if you take the numbers at face value, they exclude most people who were asymptomatic or treated at home without a formal diagnosis.

A UK metareview found the prevalence at 12 weeks of symptoms affecting daily life ranged from 1.2% (average age: 20, minimum 18) to 4.8% (average age: 63). The cohort with average age 31 had a mean prevalence of 2.8%., which is is well within the Lizardman Constant. This is based on self-reports on survey data, which will again exclude asymptomatic cases, so even if you treat it as real, you need to discount it down to 2.8%.

On the other hand, medicine is notoriously bad at measuring persistent, low-level, amorphous-yet-real effects. The Lizardman Constant doesn’t mean prevalences below 4% don’t exist, it means they’re impossible to measure using naive tools.

Comparison to other diseases

The Taquet study did compare covid patients to those with other respiratory diseases (including the flu, not controlling for disease severity or patient age), and found covid to be modestly worse except for myoneural junction and other muscular diseases, where covid 5xed the risk (although it’s still quite low in absolute terms). Dementia risk is also doubled, presumably mostly among the elderly.

Additionally, cognitive impairment following critical illness, and especially following intubation, is a well known phenomenon. This puts the Great British Intelligence Test numbers in perspective- being/needed to be ventilated is quite bad, but it’s always been that bad, there doesn’t appear to be any unique-to-covid badness.

Conclusion

My tentative conclusion is that the risks to me of cognitive, mood, or fatigue side effects lasting >12 weeks from long covid are small relative to risks I was already taking, including the risk of similar long term issues from other common infectious diseases. Being hospitalized would create a risk of noticeable side effects, but is very unlikely post-vaccine (although immunity persistence is a major unresolved concern).

I want to emphasize again that “small relative to risks you were already taking” doesn’t necessarily mean “too small to worry about”. For comparison, Josh Jacobson did a quick survey of the risks of driving and came to roughly the same conclusion: the risks are very small compared to the overall riskiness of life for people in their 30s. Josh isn’t stupid, so he obviously doesn’t mean “car accidents don’t happen” or “car accidents aren’t dangerous when they happen”. What he means is that if you’re 35 with 15 years driving experience and not currently impaired, the marginal returns to improvements are minor. 

And yet. I have a close friend who somehow got in three or four moderate car accidents in < 7 years, giving her maybe-permanent soft tissue damage (to answer the obvious question: no, the accidents weren’t her fault. Sometimes she wasn’t even driving). Statistically, that friend doesn’t exist. No one gets in that many car accidents that quickly without it being their fault. And yet the law of large numbers has to catch up with someone. Too small to measure can be very large.


What this means is not that covid is safe, but that you should think about covid in the context of your overall risk portfolio. Depending on who you are that could include other contagious diseases, driving, drugs-n-alcohol, skydiving, camping, poor diet, insufficient exercise, too much exercise, and breathing outside. If you decide your current risk level is too high, or are suddenly realizing you were too risk-tolerant in the past, reducing covid risk in particular might not be the best bang for your buck. Paying for a personal trainer, higher quality food, or a HEPA filter should be on your radar as much as reducing social contact, although for all I know that will end up being the best choice for you personally. 

Change my mind

My own behavior and plans have changed a lot based on this research, so I’m extremely interested in counterarguments. To make that easy, here’s a non-exhaustive list of things that would change my mind:

  1. Evidence that long covid gets worse over time, rather than slowly improving (note that I did look at data from SARS 1 and failed to find this).
  2. New variants increase the risk to what it was or was feared to be in April 2020
  3. Evidence of more severe vaccine attenuation than we’re currently seeing.
  4. Credible paths through which the risk could drop sharply in the next six months.

Thanks to LessWrong and Redwood Research for funding this research, Connor Flexman and Ray Arnold for comments on drafts, and Rob Bensinger and Lanrian for catching errors post-publication that did not affect my overall conclusion.

Exercise Trade Offs

Woman Exercise Bike - Free vector graphic on Pixabay

Update 9/2: A friend pointed out that I was ignoring the time costs of exercise, which ended up being pretty significant. See new numbers here. I then double checked the math on the microlife numbers and the news is not good.

Tl;dr: under my current conditions, outdoor exercise is slightly safer than indoor for me, but the risks of both are dwarfed by the benefits of exercise.

Recently I’ve been weighing trade offs around exercise. At the gym I’m risking covid exposure. I can reduce that by wearing a mask, at the cost of making the exercise less effective or enjoyable. I could use my friend’s outdoor gym, but it’s fire season here in California so there are prolonged periods where I don’t want to be sucking in all that unfiltered air. This is also addressable with a mask, but at the same cost. I could exercise indoors in my own home, but I do not have that much space and it gets miserable really fast. I could not exercise until conditions improve, but that has its own health costs. So I did some math.

Wikipedia says 10 minutes of exercise = 1 micromort lost (as in, you live longer). That’s obviously going to depend a lot on the type of exercise but we’ll use it.

This calculator translates time * AQI into cigarette equivalents. At 50 AQI, it takes 12 minutes to generate .01 cigarettes. I’m going to treat that as 10 minutes because exercising is slightly worse than merely existing out doors and it makes the math much easier.

Wikipedia lists an equivalent of 1.4 cigarettes = 1 micromort.

N95 masks block 95% of PM2.5 particles (which is what the AQI is based on). I couldn’t immediately find a translation of that to micromorts so let’s assume it’s linear discounting. EDIT: On Twitter Divia Eden points out that 95% assumes a perfect seal, which you probably don’t have. This isn’t material at my current air quality; I did this whole thing without including masks at all and then added them in afterwords, but when you do your own math you should include that.

That means that 10 minutes unmasked outdoor cardio, at 50 AQI = .01/1.4 = .007 micromorts, which is clearly dwarfed by the 1 micromort lost from exercise (even if you assume it’s 10x worse for me due to the existing chest congestion, and don’t give the exercise a corresponding impact bump). If I wear a mask the risk is probably below the significant figures I’m allowed. It’s so negligible compared to the benefits that if allowing myself to go outside increases total exercise by any amount at all, it’s obviously worth it.

How about covid risk?

My gym is personal training focused with a single cardio machine, which you must schedule in advance. If I’m doing cardio there will be at most two clients doing weight training and two trainers in the room, plus me, all > 10 feet away, in a large room with filtration they claim is good. If I’m doing weight training there’s me, my trainer (fairly nearby), and potentially a farther away client and trainer pair. In theory there could be an additional person on the cardio machine but I’ve yet to see it happen.

Under an excessively conservative set of assumptions (City-average vaccination, no mask, constant talking), my cardio scenario is 7 microcovids. If I give everyone masks it’s 0.5. My weight training scenario is <=10 microcovids (7 for the other pair, which may or may not exist, and 3 for my trainer. Note that weight training is 2.5x as long as cardio). But microcovids are not micromorts. The Economist calculator (pre-delta, pre-vaccine) has the risk of dying of acute covid at my age and sex as immeasurably low, despite it being prone to overestimate because its denominator is only diagnosed cases. Long covid is a concern (although I’ve tentatively concluded its overblown: more on that soon hopefully), but lack of exercise is bad for long covid in particular. If we generously use my age/sex hospitalization rate as the discount factor (2.6%), the micromorts from my indoor cardio are <=0.16, and my weight training is <=0.23. These are not quite as negligible as the pollution, but still very safely under the benefits of exercising.

Some caveats: I didn’t examine any of these numbers that closely because the verdict was so overwhelmingly clear; the values would need to be off by orders of magnitude to change my conclusion. But that is always an option, and when I tried to follow up on the 0.1 micromort/minute of exercise number, I hit a dead end.

I’ve made a very crude spreadsheet with sources linked in comments so you can make a copy and play around with your own numbers, based on your local air quality, covid prevalence, etc.

Book Recommendation: The Cruel Prince

This is not normally the venue I use for discussing fiction, but I’ve recommended The Cruel Prince (affiliate link) to ~10 people in the last year and a half, and every single person loved it. Every last one. And not just a little bit; multiple people put off important things they needed to be doing because TCP was so good and did not regret their decision. This level of agreement and enthusiasm among my friends is completely unprecedented. So if you can possibly see yourself being interested in “YA fairyland with unusually agentic protagonist”, I suggest you give it a shot.

The first five or so people I gave this recommendation to, I warned that I hated the third book in the series and thought that it undid some of what was most important to me in the first two (cannot reveal without spoilers, but happy to discuss out of band). No one was able to resist after enjoying the first two so much, and thus far no one has agreed with me. So you should probably read it too, and if you happen to agree with me I would be extremely happy to have someone to complain about this with.

Antidepressants and Medical Uncertainty

I’ve occasionally talked about how great my experience with antidepressants was. First one (2015) worked great and reduced my trigeminal neuralgia to boot. But it wasn’t enough so I started a second one (2017), which was also great and also helped my trigeminal neuralgia with no other side effects. I knew this experience wasn’t universal, but I would occasionally share it so people would have data on the best case scenario.

Welp.

I tried lowering both meds last summer, but each time my neuralgia got worse so I resumed my regular dose. Then in January USPS delivered my pills two weeks late, and I was forced to go off them (luckily I saw this coming as was able to taper with my remaining pills, so it wasn’t cold turkey). Like over the summer my neuralgia got worse, as did my anxiety, but the depression definitely did not come back.

A bunch of stuff happened here. First I was fine, then I was sleeping an awful lot, then I was sleeping quite a bit less (and felt fine about it). Every time I felt bad I would wonder “is this a transient reaction to an external stimulus, a sign of returning depression, or both?” When I had my checkup with my psychiatrist and mentioned that I no longer needed a daily nap, she immediately said “Oh that’s [medication 2]”.

I’ve talked to multiple doctors about my previously quite extreme sleep needs, including her. They all tried a bunch of complicated stuff, and they all had access to my list of medications, but no one ever said “hey, it might be this medication you’re on, let’s try adjusting that”. It’s possible the medication was the right choice for a period of time even when it was eating two hours of of my day, it’s possible it wasn’t eating two hours of my day and it’s a coincidence things have improved. But when my doctor is so sure it was the problem now and nobody even mentioned it before, something has gone deeply wrong somewhere.

There was a time this would have really freaked me out and possibly triggered a panic attack, but TBH it’s about what I expect from medicine at this point. So the point of this post is mainly to be an accurate data point for people assessing antidepressants for themselves, because God knows the medical community isn’t going to give them to you.

I still think medications were mostly a success for me. I really needed them when I started them, I might really need them again in the future and would consider taking them even with the risk of nap. But I would pay a lot for a doctor who even raised this possibility.

PS. Given what I do for a living, “why didn’t you research this yourself?” is a reasonable question. The answer is:
1. I was not in a good place to do this when I started them
2. The fatigue did not kick in immediately- I was actually hypomanic for a bit, and the mandatory naps kicked in at least a year later
3. I did in fact know that fatigue was a possible side effect of the medication, but I was taking it off label at a drastically lower dosage than is typically prescribed (<1/20th), so it didn’t seem very applicable. I don’t think better scientific studies are the solution here, humans are too variable. What’s needed is a system that is responsive to individual feedback, including doing experiments to get that feedback.

What can a covid test tell you?

The following is extracted from research I did for a client (which I’m sharing with their permission). An important thing to keep in mind while reading this is that the information was gathered to answer particular questions (Primarily “quarantine procedures for very risk sensitive, cost insensitive people”), and that biases what I looked at in weird ways. But I think it’s still much more useful for some set of you to have this information than not, so here it is.

The Tests

Symptoms

Of young, healthy people caught by prophylactic screening (typically contact tracing), 30-50% who test PCR+ will never report symptoms (in some cases these subjects are proactively screened for temperature, in others it’s not clear. Additionally, other studies have found that some percentage of people who report no symptoms will go on to develop CT lung anomalies). Additionally, most people who do develop symptoms spend several days being contagious before they develop symptoms.  So symptoms are not a very good metric at all for determining if someone is infectious.

 [Full notes and sources]

PCR

PCR tests look for specific viral RNA sequences and amplify them to make them easier to detect. They can vary in a number of ways:

  • Specific sequence searched for
  • Number of amplifications done
  • Collection site and mechanism of sample

PCR false positives are quite rare, and typically have to do with sample contamination. False negatives are more common, and can come from a variety of sources:

  • Covid was present but did not have the particular sequence searched for
  • Did not do enough amplifications to notice effect
  • Patient has covid but the particular site sampled does not.

How common are false negatives? That’s hard to define, because currently a nasopharyngeal PCR is what gets you diagnosed as having covid- if you fail that, you’re assumed to have one of the many things that produces similar symptoms. However, we can make some guesses.

The following graphs show the calculated chance of testing PCR+ on a nasopharyngeal test on a given day after exposure, given frequent testing and an eventual positive result.

But that’s just for nasopharyngeal (NP) tests. Data on sicker patients (the only ones studied) show it’s possible to have a negative NP test while another area of the body (lower respiratory or gut) tests positive.

Figure thumbnail gr1

But these are all people who tested PCR positive eventually. What about people who get sick without ever testing positive? A very small study found that of 24 health care workers who developed positive antibodies over a 3 month period, 10 never had a positive PCR test despite being tested twice weekly. The paper offers multiple explanations for this, and I’m very reluctant to draw conclusions from such a small study, but it is concerning.

[Notes on NP tests]

[Notes on stool tests]

[Notes on spit tests]

[Notes on lung tests]

Antibody tests

Antibody tests look for an immune reaction to viral proteins. They can be negative when a person is contagious (because they haven’t formed a large enough reaction yet) and positive when they are not (because they successfully fought off an infection). 

A reasonable question I have not investigated is “does an antibody+ test mean I’m immune?” 

[Full notes and sources]

LAMP tests

Like PCR, LAMP tests look for specific viral RNA sequences and amplify them to make them easier to detect. LAMP tests go through many fewer amplification cycles, making them less sensitive but much faster: 30 minutes vs 2-3 days for PCR, plus the transit time for the PCR samples. 

A LAMP test every three days is much more valuable than one PCR test, because catching the peak is much more important than the sensitivity. I’m not sure if a LAMP test taken today is more or less useful than a PCR test taken to give results today. I stopped digging into this because there were not yet any LAMP tests on the home market.

[Full notes and sources]

Antigen tests

Antigen tests look for specific viral proteins in a sample. There’s no amplification, which makes them less sensitive, but manufacturers report catching 88% of cases caught by PCR, so maybe this is fine, especially since you get results days faster? 

The 88% sensitivity number is much, much higher than you see in literature (which is something like 20%-50%), I assume because they used more abundant samples. There’s some controversy over whether that’s because many PCR positives are driven by dead virus, in which case the antigen test returning a negative result is a feature, not a bug. This may well be true, however at least one study was able to culture live virus from a PCR+/antigen- sample, so it’s not foolproof.

Of the studies looking at antigen tests, all that I found either started post-symptoms, or took a random sample of people showing up at a sampling site. None were in a position to determine how good antigen tests are as an alarm system for catching an early infection (as opposed to diagnosing symptoms or determining when someone has ceased being contagious)

However antigen tests return results in minutes, while PCR tests take 2-3 days to complete not including shipping time. Given how quickly covid multiplies, it’s possible that an antigen test now is more sensitive than a PCR test from four days ago, especially if the exposure occurred in the intervening time.

[Full notes and sources]

What do any of these mean for contagiousness?

That’s a good question, which with the exception of asymptomatic spread can’t really be answered without human challenge trials, which the entire world declined to do. I would be surprised if any of these tests’ thresholds lined up perfectly with the threshold of infectiousness, because there’s no reason to expect they would. I expect culture tests to be much closer, but still not necessarily exactly on line (plus AFAIK they’re not available for diagnostics, even through a doctor). Given that I asked…

How long do I need to stay in quarantine after potential exposure?

The best data on this is from New Zealand, which has a very strict isolation policy for entrants: 14 days isolation, with PCR tests on days 3 and 12 of isolation (a pre-departure test is required only if you’re coming from the US or the UK). By my earlier estimations, day 3 of isolation is already likely to be past the peak of detectability, unless travellers were exposed right before they left. Nonetheless, New Zealand reports only a handful of import-related covid cases after this policy kicked in, and a model attributes those to people who caught covid in isolation (e.g., a couple who shares a hotel room and one partner gives it to other on day 4), rather than people who entered New Zealand infected.

Transmission type of COVID-19 cases

I did look for similar data from other countries, but NZ had both the best quarantine and the best data.

So despite the tests’ low sensitivity, 14 days + a test at the end really does seem to be a long-enough isolation period.

[Full notes and sources]

Thanks to my original clients for funding this research, and my Patreon patrons for contributing to the cost of preparing this for publication

Follow Up: Predictions as a Substitute For Reviews

In August I wrote about using PredictionBook as a substitute for weekly reviews. Several months later, I am rather predictably not using it, for the same reason weekly reviews stop working: it reminded me of something I endorsed doing but didn’t finish, and that put an ugh field around the whole thing.

Weirdly, I re-picked-up that thing a week ago. This was completely independent: I haven’t looked at PredictionBook since September. I don’t think this has any bearing on the viability of predictions substituting for reviews, it’s just a cute coincidence.

Interview with former cult member Duncan H.

As my twitter followers already know, I started a deep dive on cults back in September. To the extent I had a goal for this, it was “figure out behaviors that differentially push you away from actual cults, without throwing the baby out with the bathwater”.

I mentioned this to a friend, who suggested I talk to a friend of hers, Duncan. Apparently Duncan gets this request a lot, because his condition for talking to me was that we record the conversation so he can share it with the next person who asks. We had mixed success at this- the audio quality was horrible for the first section of our conversation, and the transcriptionists had limited ability to save it.

But the second half got quite interesting. Some highlights:

  • Duncan got a lot from the cult and would make the same decision again (and then leave again).
  • If power is creepy, only creeps will have power.
  • How society interacts with strong belief.

Part 1 video (low quality) and transcript
Part 2 video (much better quality) and transcript

These calls lasted 2 hours and I still had more I wanted to talk about, so I suspect you do too. Duncan has indicated a strong willingness to answer more questions, as long as they are interesting. So if there’s anything you’re dying to know after listening/reading these, comment here, @ me on Twitter, email me at elizabeth@acesounderglass.com, etc.

Thanks to Patreon patrons and Duncan for helping to defray the cost of the transcripts.

Looking For Work

Three and a half years ago I announced I was leaving software to try to work on longtermist projects instead. A year ago I announced I was focusing specifically on research on the meta level. Today, I’m sad to announce that neither of these have worked out like I hoped; I’ve failed to find a way I’m confident has a large positive impact on the future. Three years seems like long enough to give this project, especially given that I don’t know what I’d try next if I wanted to. In light of that, I’m stepping back from big Save The World projects to focus on finances for a while.

What this means for the blog: I still like reading stuff and talking about it, so that will probably happen sometimes just like it did before I took this on ~full time. Maybe even more, now that the pressure to do the Most Impactful Research is gone. I don’t plan on putting more effort than is fun into refining knowledge bootstrapping or learning-and-trauma studies for the foreseeable future.

What this means for me: I’m now looking for employment that isn’t “the most impactful thing I can do”, and would appreciate any leads you can give me. My primary skills are as follows:

  • Research
  • Was once a programmer. I didn’t like it then but ironically the learning-and-trauma stuff is making it much easier to do and I may hit enjoying it again on a reasonable timeline.
  • Was once an SDET. I’m going to be even pickier about SDET work than dev work, but it is a skill I have.

The research thing is tricky. There’s not a lot of call for uncredentialed laymen who can read books about the history of childhood really thoroughly. But those skills are applicable to a wider range of problems, if people know to conceptualize the problems the right way and know that they have the option to turn them over to an expert. I spent one of the last three years at a company that *did* know how to use those skills, and was fantastically useful as I twitter-stalked investors, found the unwritten rules for getting on exchanges, and forecast the economy of Argentina. But of course I would claim that.  Instead of taking my word for it, here’s a description from my former boss at Reserve, a for-profit company where I worked as Head of Research for a year:

From managing Elizabeth at Reserve, I would strongly recommend her as a researcher to other companies, and even more so to startups, that need help making difficult highly complex decisions.

Early on at Reserve we faced many very complex challenges – e.g. navigating a labyrinth of legal issues, identifying product-market fit, and making major strategic decisions about how to fundraise. A major recurring bottleneck to make decisions was that we lacked information on various key unknowns, and so to find this critical information we tried out hiring Elizabeth as a researcher. Initially her scope of research at Reserve was pretty limited, but we quickly realized that her skill and autonomy meant she could do alot more for us and so we kept giving her larger and larger research tasks and she quickly became a pivotal member of the team.

Elizabeth has high agency at learning about an extremely wide range of topics and enjoys research work, and so she happily took on any research challenge that we gave her. She’s very competent – I remember we needed background info on someone, and she found a lawsuit against them hidden away on the internet. She will also research to any depth that you want – she can do 1 hours worth of research on something and then report back, or 5, or 10 hours. Best of all she’s highly autonomous and needs very little oversight, so once she knows what you need she can independently make decisions about what’s worth researching, and at what point is it best to stop and report back, which makes her especially well suited to a startup.

[Said manager didn’t want his contact info plastered all over the internet but it is available on request]

[If you’re wondering why I left a job I enjoyed and was good at and appreciated me that much, the answer is a combination of “changing business needs” and “crypto winter”.]

If there’s somebody you think this could be useful to, I’ve also included this in my Hire Me page and LinkedIn, so you can share it with them without the rest of this post.

Some other traits that make things a good fit for me:

  • Not make the world worse on net
  • Not be a bullshit job, in the Graeber sense of the word. One of the hardest things about the last three years has been the lack of feedback loops, and one of the things I’m most looking forward to is having concrete contact with reality.
  • < 25 hours/week on average (large week-to-week fluctuations are fine)
  • Working with other people who can give me feedback. One of the other hardest things about the last three years has been doing it more or less alone, and worrying about what I missed is just exhausting.

This would have been a really devastating announcement if I was making it any earlier than I am, but after my last grant rejection, I came to terms with it pretty rapidly. I tried something, I failed, and I’m pretty fortunate to have had both the option to try something unlucrative and a lucrative thing to return to. I don’t think my current orientation towards money and mental space is permanent- I just don’t currently have anything compelling enough to justify further burning of runway.

If you’d like to contact me directly, my e-mail address is elizabeth@acesounderglass.com.

Thanks to everyone for all their support and encouragement over the years.

-Elizabeth