Anthropic insiders warn AI could kill all humans

Three Anthropic researchers went public last night with chilling concerns about out-of-control AI, warning it could destroy humans this decade.

• Anthropic AI researcher Jacob Coxon wrote on X, after resigning Tuesday to sound the alarm: "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger."
• Anthropic alignment-science lead Evan Hubinger responded: "Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

• Samuel Marks, Anthropic scalable-oversight lead, added: "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are."

Why it matters: They're hardly alone. Their warnings came just days after top OpenAI leaders, including CEO Sam Altman, said AI is speeding into a scary, uncontrollable phase.

Warnings from inside the AI giants point to an epic shared dilemma: Slow down and risk falling behind, or press ahead and risk losing control. The companies are full speed ahead, even as they practically beg for regulation or a global pause, Axios' Maria Curi, Madison Mills and Ina Fried report.

The big picture: Calling this unprecedented would be a gross understatement. You basically have the fastest-growing companies in human history warning their products could harm or even destroy humanity.

• Critics say Anthropic and OpenAI are hyping their products to raise their valuations and invite regulation that would benefit them alone as the dominant incumbents. But we've been talking with dozens of people inside these companies for months, and they've sounded increasingly spooked and concerned. Given they see models not yet released to the public, it seems reckless not to take them seriously.

🏛️ Context for readers from Mike & Jim: This is self-evidently scary stuff — and these vague warnings are impossible to validate or appraise. But we think readers, especially members of Congress and those in relevant federal agencies, need to be aware that the AI creators themselves see potential catastrophic outcomes absent a shift in how America, China and others review and release more powerful AI models.

💡 How to think about this: Nobody is warning AI is an imminent high-level threat. What they're saying is that the technology keeps improving faster than they thought possible and will soon be able to self-improve (recursive self-improvement). Once that happens, it gets even better, faster ... and much harder to predict or control.

• Go deeper.