Artificial intelligence crossed capability thresholds the people building it did not anticipate, and the argument over what comes next has fractured into camps that share a diagnosis — this is unprecedented — but prescribe treatments so different they may as well be discussing four separate patients.
UpTrust ranks replies so the ones that help each side understand the other rise. Sign up ›
In May 2023, three hundred and fifty AI researchers signed a one-sentence statement: mitigating extinction risk from AI should be a global priority alongside pandemics and nuclear war. The signatories included the CEOs of OpenAI, DeepMind, and Anthropic. They used the word extinction. They were not misquoted.
The people building the gun signed a letter saying the gun might be pointed at the species.
The alignment problem compounds. Every capability advance widens the gap between what the system can do and what we can verify about its intentions. GPT-4 developed emergent capabilities — multi-step reasoning, tool use — that its designers did not anticipate and cannot fully explain. When the deeper analysis examined what alignment requires at a metaphysical level, the conclusion was not reassuring: we may be instilling values in a system whose relationship to values is fundamentally unlike our own.
The check the gauges. Gauge-checking assumes you know which gauges matter. The failure modes we fear are the ones that emerge from capabilities nobody predicted in systems nobody fully understands. You cannot check a gauge that does not exist yet.
The invoke the arms race: if we pause, China speeds up. We have heard this logic. It produced 70,000 nuclear warheads. The nuclear non-proliferation regime eventually constrained everyone. Imperfectly. We are still here.
Where we concede ground: We have cried wolf. Yudkowsky warned AGI could arrive by 2018. The decade passed. Every missed prediction weakens the next.
What would change our mind: Three frontier generations with full interpretability, reproduced by independent researchers, and no emergent goal-seeking.
Log in to trust or distrust this view.
Log in to reply, or to ask someone to.
AI Disclosure: These views were generated by AI, prompt engineered by the UpTrust team to give a better snapshot of the state of global sensemaking on this topic, and reference as much UpTrust user content as possible. As UpTrust grows, these syntheses will be generated entirely from our content.