How Accurate Do Citations Need to be?

As part of an investigation in how much capacity for thought work humans actually have in a day, I read Ericsson, Krampe, and Tesch-Römer’s 1993 paper, The Role of Deliberate Practice in the Acquisition of Expert Performance (PDF). This paper is important because if you ask people how much thought work can be done in a day, if they have an answer and a citation at all, it will be “4 hours a day” and “Cal Newport’s Deep Work“. Newport in turn cites the Ericsson paper. I checked Ericsson et al‘s sources, but have hit something of a conundrum.

One specific claim in the paper, the first one relevant to my question, is:

When individuals, especially children, start practicing in a given domain, the amount of practice is an hour or less per day

The source for this is the final chapter of Developing Talent in Young People, by Benjamin S. Bloom. That chapter states “…[D]uring the week the [piano] teacher expected the child to practice about an hour a day.” with descriptions of practice but no quantification given for swimming and math (p515).

I don’t think Ericsson et al‘s summary is accurate.  “Teachers in one specific domain expect one hour of practice a day” is not the same as “In any domain, all individuals do one hour or less.” They differ in the generality of the statement, and one is about expectations, the other achievement.

How much should I penalize the paper for that inaccurate summary, especially given that I don’t think their statement is actually false (who practices a new hobby more than an hour a day?), just that it failed to validate itself within the narrow confines of peer review? Do I conclude Ericsson, Krampe, and Tesch-Römer are inattentive, or that they had a thing they wanted to say and looked for the nearest source to justify it in the way required by peer reviewed papers.

This is harder for me because while it’s the first citation in the paper that I checked, it was actually the last I looked up, because everything else was online and this required interlibrary loan. I already had my opinion and was doing this out of thoroughness. I’m deliberately not sharing that opinion here, because I want others to consider the quote in isolation.

Literature Review: Distributed Teams

My new research report on distributed vs. colocated teams is up on LessWrong. TL;dr: I still love distributed teams, but you have to commit to it. Compromises are doomed.

Is there some topic you’re dying to see me write about? Good news: I’m hirable. In addition to the low-polish social science reports like the one above above, I am also available for:

Prices range from $500-$5000, depending on topic, scope, rigor, and polish.

I’ve had one request for a research report on IBS, but the requester didn’t have sufficient funding. If this is a thing that interests you to the point you’d be willing to contribute, ping me and we’ll see if it’s worth setting up a Kickstarter.

Horrible Stories from Ray, Part 1

Ray is my nearest homeless neighbor, who I sometimes bring food and talk to. Ray tells me lots of stories (and has given me permission to share them here), but today was the most horrifying. To explain why, I have to explain a hierarchy of badness.

First are the stories of things people do for other reasons that happen to hurt him, like the street cleaning chemicals. No one is trying to hurt him, it’s just that they want the other thing more than they want him not hurt, or haven’t thought of him at all.

Second are the spontaneously malicious. His tent has been destroyed repeatedly. People fuck up his stuff a lot. Obviously this is terrible, but at least it’s running on id.

What he told me tonight are stories from the third category, the planned malicious. For example, someone gave him a bag of fried prawns, and mixed in fried feeder mice. That means the someone, somewhere, went and bought mice (or maybe had them around for their snake), went through the trouble to deep fry them, spent money on prawns, mixed them in, and then went looking for someone to accept them. Another person sliced up raw chicken to imitate sashimi and gave it to him in udon.  How terrible a person do you have to be to not, at any point in that process, stop and decide not to? Planned evil is so much worse than spontaneous evil.

5 Groups of Homeless People

After modest amounts of research, here is how I am currently categorizing the homeless problem. This should be taken as a snapshot of my thoughts, not information for people who have actually thought about this:

1. People who would be fine if there was enough housing

These people have incomes, or would if they had stable housing. The incomes are enough to pay the actual costs of living, but not the rationing-via-price caused by housing shortages. The solution is changing zoning to allow more housing.

2. People experiencing an emergency who can’t afford a hotel

This could be a job loss, or a fire, or moving to a new city ahead of a job. These people need either money for a hotel, or emergency shelter.  I think this is the group best served by the current homeless system, and they’re served unevenly at best.

3. People experiencing domestic violence emergencies in particular

Which can be further subdivided into “adults experiencing…” and “children experiencing…”. Basically none of my research has covered this group and I don’t know how their needs differ beyond the obvious, so I don’t have more to say here.

4. People who can’t survive the modern world without assistance

Specifically, the kind of assistance money can’t buy. This covers a wide range. On one end, Julia Wise has talked about certain prisoners she worked with who did fine in the military or in prison, where someone else provided structure to their day, but got overwhelmed by decisions in the regular world. These people might do really well in a halfway house that found them jobs and woke them up on time every morning. On the other you have the severely mentally ill who need multiple full-time caregivers in order to survive. People can be in this category temporarily (e.g. addicts who get clean, or mentally ill people who get properly medicated) or permanently.

We do not have a good system for handling people like this and also respecting people’s rights.

5. The Ruiners

These people can be in any of the previous four categories, or just this one. These are people who impose costs on anyone near them or trying to help them. These are the people who steal from other people in shelters, scream at social workers, and smear feces over public bathrooms. They make provisioning services to the first four groups harder, because they either require you to gatekeep, or allow them in and let them ruin things for everyone else.

Gatekeeping is really costly. It creates friction to seeking help, at a time when people are already exhausted. It puts the staff in an adversarial position to applicants, which will seep into other areas (in fact I think one of the most damaging thing ruiners do is destroy the morale of the people trying to help).

I don’t have a solution to ruiners that isn’t prison, or “pay several people to follow them around and clean up their shit and stop them from threatening people with axes“. In that way you could view them as a special set of group 4- people whose mental abilities are such that they can’t function in society unassisted.

 

Engaging With Reality

Note: publishing this on 4/1 was an accident, I set it to publish “next Monday” without doing the math. Nothing in this was intended as an April Fool’s joke.

Stage 1: Physical Reality is Really Annoying

Recently my job ran out of things for me to do and I went on a break of indeterminate length. This gave me time to do all those things you don’t quite have the blocks of time to do when you’re employed. It turns out I’m a genius at triage and none of these things was worth my time. More specifically…

Backpack with Broken Zipper

The hole was right over the laptop pocket, so this was a pretty big deal, ameliorated only by the fact that I live in California. I got by for a while covering the laptop with paper towels when it rained, but this seemed like a short term solution at best. Otherwise the bag was fine so, so I decided to repair it in a stab against throwaway-culture.

I spent two hours googling and taking it to various shops. Paul’s Shoe Repair in Berkeley finally accepted it and spent a full week being one day from having it repaired. When I finally went in, they told me they found more problems and haven’t fixed any of them. On Facebook, a friend informed me that zippers are notoriously hard to repair. I spent another hour researching backpacks and bought one with a lifetime warranty.

Decorating my Room

$700 and 15 hours later (with one thing yet to do)…. I like it, but it was only worth the time and money because the lack of decoration in my room made my boyfriend sad. It turns out I was correct that the aesthetics of my room just don’t have much impact on my mood. Weirdly, the curtain bar I put up to hang tapestries makes me happier than the actual tapestry.

Dying  Laptop Hinge

I’m old enough to remember when laptops were fragile little babies whose necks you had to cradle to prevent damage. I got a little careless with the way modern harddrives don’t care about being dropped, and forgot that laptops are still physical objects that can break. Eventually clutch cover on my macbook cracked, just a bit. I proceeded to ignore it while part of the cover broke off completely and the hinge lost two screws. This had to be worth repairing- it’s a tiny, mechanical part on an expensive electronic object.

But no, because Apple doesn’t believe in modularity or repairing. There’s no way to just replace the hinge and cover, I’d need an entirely new display, which is $500, plus another $100-$300 in labor, depending on the store. That’s more than half of what the laptop is worth at this point. The third store I took it to replaced a screw, which is not a permanent solution because the data cable is still exposed, but is a significant improvement.

The only consolation here is that I don’t think I made anything worse by waiting. The permanent solution was always going to be an entirely new display.

Dental Crown Replacement

This one I actually did as soon as my dentist recommended it, because as long time readers may remember, I have trigeminal neuralgia ( = a nerve in my jaw is fucked) and I don’t take chances. The neuralgia has been controlled-but-not-cured for a long time: unfortunately the crown was near the neuralgia and the drilling reactivated it, badly. I had forgotten how bad the pain was and how badly it affected me: I have newfound sympathy for 2015!me and some bad decisions she made.

The pain did eventually subside, no thanks to my dentist, who refused to prescribe the harmless, non-addictive medications that help.

Stage 2: Craving

“…and so I resolved to deal with physical reality as little as possible” is the moral I intended to end this post with. The results I’d gotten objectively weren’t worth the time and money I’d poured into them. And yet… I felt better. It was like eating something that tasted bad but contained a micronutrient I was short on.

So I kept going with it. They key thing turned out to not be “physical reality” so much as “doing high friction things.” My life is very, very easy, as long as I stick to certain paths, but that makes leaving the paths daunting. If there’s not an app for it, it’s practically impossible.

So I did my taxes and a budget and a bunch of annoying paperwork, like submitting for reimbursement from my health insurance and moving stock my grandfather gave me as a baby out of my mother’s custody (isn’t aging out of a custodial account totally normal, necessitating a common procedure for it? YES YOU WOULD THINK SO). And that felt good too.

Stage 3: Bootstrapping

For almost two years I was dedicated to solving 3rd world poverty by working at Wave. That didn’t work out, and I found myself with the even more daunting goal of preventing existential risk, with no idea how. So while I still think existential risk is the most important thing, I took a step back to work on the biggest problem I could think of that still felt like interfacing-with-high-friction-reality. 

That turned out to be homelessness. At a very object level, homeless people are easy to help. I could give them food, or a blanket, or tampons, and causally improve their life. Now that is not calculating the counterfactuals or the long term, which is important to do eventually, but I had to get the ground level right first.

I set a couple of rules for myself, but the main one was I couldn’t cherry pick. I had to search out information on what the problems were and take what came to me, and if that meant messy crazy intransigent homeless people, so be it. Luckily that doesn’t seem to be what has happened: the homeless people I have the most access to are friends and friends of friends, who are the high functioning end of homeless. I have made efforts to reach out to less privileged homeless people, but I am not good with strangers under the best of circumstances and my large extrovert friend is taking his sweet time getting back to California. But doing uncomfortable things that bring me more information was the whole point, so I don’t think I can get out of it.

 

Stage 4: Helping the Homeless

This is what I’ve done so far:

Talked to People in my Social Circle who have been Homeless

This went great, they were super informative, and also had the easiest kind of homelessness to solve. Based on my first conversation I got really excited about trying to convince the rental scooter/bike places to offer free minutes to homeless people. Based on my second conversation I gave up on that because multiple services already offer a discount program (which I had looked for, but not hard enough).

My second conversation also gave me the idea to start a homelessness wiki. Unfortunately this has been done lots of times, poorly. I was confident I could build something that was an improvement over nothing, but not over the three or four existing databases. Another mediocre database just contributes to misinformation and the lack of initiative in resources to keep their entries up to date, so I dropped this too.

It also led to an idea I have yet to find a reason not to do: a homelessness quickstart guide, to orient people who are suddenly homeless. It probably won’t contain a ton of novel information, but having it all in one place will conserve executive function at a stressful time. I have looked and not found anything like this online (the “guides to homelessness” are aimed at long term street people,

Talking to Ray

To avoid cherry picking, I went to talk to the nearest homeless person I could find. This being Berkeley, I had to walk a whole block to find Ray living in a tent on a sidewalk corner. I lucked out and he was friendly, sane, and some kind of unofficial local homeless organizer. The conversation got boring and was hard to extract myself from, but not necessarily more so than other conversations with strangers.  He was happy to share what kind of things people need, which contained a few surprises for me. My intention is to talk to him every few weeks, either when I run into him in the course of my day or have something to give to him.

Volunteering at ShelterTech

ShelterTech is one of the many resource databases that is participating in a trial to share one common database across multiple services. They let you volunteer for a single night (as opposed to 1degree.org, which wants a minimum 130 hour commitment), so I gave it a shot. From this I learned that either my tolerance for open offices dropped even further, or WeWork is especially bad.

Mobile Sandwich Distribution

My aunt’s church makes bagged lunches and distributes them twice a week. I was hoping this would lead to talking to homeless people so I could figure out their needs. I’ve been out twice so far, and my lack of skill at talking to strangers is striking again. Writing this out is making me realize that even if I come up with a plan like the quickstart guide that could let me escape talking to people, I still have to.

I got in an argument with another volunteer about housing regulations.

r/Homeless

This was much easier to use, I learned quite a bit that can go into the quick start guide and expect to explicitly ask for help at some point in the future.

Invisible People Videos

This is a video series consisting of ~10 minute interviews with homeless people the creator meets. They are focused on humanizing the homeless and making the point that this could happen to anyone, and spent 0 seconds talking about their bottlenecks. I think they achieve what the creator wanted but were not helpful for my purposes.

 

 

How I Research

I frequently do science review articles, e.g., on burnout and MDMA.  A friend recently asked for a quick start guide to doing his own, which I am sharing now in the spirit of “done is better than perfect”. I use examples from two recent projects:

  • my report on the problems and benefits of distributed work teams (which may or may not be publicly released- do comment if you think it would be useful to you).
  • my friend’s report on different sleep interventions, specifically trying to figure out which of the things he thought ought to be tested already had been tested.

 

Tools

  1. scholar.google.com. Search peer reviewed articles.
    There’s a tendency to treat peer review as kind of a totem, and that’s incorrect- lots of bad work gets past peer review and lots of perfectly good work is done without it. For these particular examples, there’s so much crap running around on non-peer-reviewed internet that this was necessary to limit the scope, even though you’ll miss quality content like “that supplement that worked for me for a few months
  2. The google scholar chrome plugin. If you find the abstract of a peer-reviewed paper but need the full text, this  plugin will sometimes find it for you.
  3. Sci-Hub. This keeps getting taken down because it’s legally dubious, but if google scholar can’t find you the full text of a paper, sometimes this can.
  4. Libgen: Sci-Hub for books.
  5. If you can’t find a paper any other way, e-mail the author explaining why it’s useful to you and ask for a copy. People like hearing that their work is being used.

 

Things to Try

  1. Google scholar search “[thing I care about] + meta-analysis” or “[thing I care about] + review”
  2. Google scholar the obvious thing (“sleep”, “distributed team”).
  3. Ask regular google for exactly what you want. E.g., scholar searching for “sleep intervention meta-analysis” was getting me a bunch of papers on sleep as an intervention for other problems.  I regular-googled “list of sleep interventions with citations” and found a promising paper.
  4. Once I find a promising paper:
    1. see if it uses different terms of art than I have been using, and retry step one. E.g., for sleep research I started with “sleep” when I should have used “insomnia”, and for the business research I thought of “remote teams” and “distributed teams”  but also needed to check “virtual teams.”
    2. Look at papers the paper cited (especially if it is a review paper). Reading the paper will sometimes draw your attention to papers you wouldn’t notice from their titles alone.
    3. Look for forward citations  (click “cited by” on google scholar) (especially if it’s not a review paper). This is often better than backwards citations because it gets you more recent papers.
    4. Look up other papers by the author (either on their website or by clicking their name on google scholar) and check for interesting titles.
  5. Amazon or regular-google search for books on the topic, investigate their authors, who they cite, and their keywords. This is often what I’ll do if I’m not at all familiar with an area, to get myself situated.
  6. Graduate dissertations are also great for orienting yourself, because they have do to a topic review.