When does being careful become its own kind of reckless?: Context weighers
Two dials, one display
In 1999, the Institute of Medicine reported that between 44,000 and 98,000 Americans died annually from preventable medical errors. Congress held hearings. Hospitals launched safety initiatives, checklists, protocols. Twenty-five years later, a Johns Hopkins study estimated the real number at 250,000 — third leading cause of death. The safety systems built in response to the original report had succeeded at reducing the errors they could see. The errors they could not see — alert fatigue, checkbox compliance replacing clinical judgment, documentation hours displacing patient hours — accumulated in the space the system was not designed to monitor.
We start there because it contains our entire position. The precautionary principle and the action-biased are arguing about the speed dial when the instrument panel has two dials and the system only displays one.
Statisticians call them Type I and Type II errors. The FDA’s architecture is calibrated almost entirely around Type I avoidance. Everything punishes the official who approves a bad drug. Nothing punishes the official who delays a good one. During a pandemic, that asymmetry becomes lethal.
The question haunts beyond the FDA. Nuclear regulation after Fukushima — Japan’s safety culture was so invested in the impossibility of a severe accident that it had no plan for one. Financial regulation before 2008. AI governance now — the precautionary instinct regulating harms that have not materialized while the move-fast instinct normalizes harms that compound before anyone builds measurement tools.
Careful about what is the only frame that works. The institutional capacity to ask that question in real time is the thing almost no system possesses.
Where we concede ground: Our position is the most comfortable of the three, and comfort is suspicious. In December 2020, our framework would not have moved faster.
What would change our mind: An institution implementing dynamic error-weighting that produces worse outcomes than the static approach over ten years.
Read the full synthesis: When does being careful become its own kind of reckless?