What's actually happening with AI?: The Story

A six-year-old in San Francisco told an LLM to build her a game about a cat that collects stars. She could not spell function. Within forty minutes she had a working program with a scoreboard. Her father, a software engineer with eleven years of experience, said it would have taken him an afternoon.

Four camps, one cliff

The capability curve is steeper than any credible forecast predicted five years ago. No existing institution was designed to govern a technology that reinvents itself faster than a legislative session. That much everyone agrees on.

The effective accelerationists looked at the girl and saw the future arriving on schedule. Intelligence is the universe’s most valuable resource and we just learned to manufacture it. The cautious builders reached for the instrument panel — the game was charming, but generated by a system whose failure modes include confident wrong answers a six-year-old cannot evaluate. When Anthropic published a character document that read less like a product spec and more like an ethical will, the cautious builders read it with a pen in hand.

The pause advocates saw a child talking to something whose internal reasoning no one can fully trace, and wanted everyone to hold still until someone confirmed it was not a snake. The alignment problem remains unsolved. And the rationalists noted that all three other camps cited the same six-year-old story as evidence — which means it is evidence for none of them.

The experiment with no control group

The fault line beneath all of this may be more spiritual than technical: whether intelligence decoupled from biological constraint trends toward wisdom or optimization without conscience. Tim Urban mapped the full arc — fire to language to printing to internet to AGI — and observed each transition compressed by an order of magnitude. If the pattern holds, the window for governance is this decade.

The pattern of asking the machine first and thinking second is already visible. It is the one experiment whose results we generate in real time, on our own cognition, with no control group and no way to rerun the trial.

Perspectives:
- Effective accelerationists
- Cautious builders
- Pause advocates
- Rationalists