No, AI isn’t going to kill us | Andrew Orlowski | The Critic Magazine

Could I persuade you to believe that AI will kill us all if I gave you $1.5 billion? What if you just say you believe? I think we may have a deal.

That’s how much has been spent on promoting the idea of existential risk from artificial intelligence in recent years. David Sacks, the venture capitalist and former White House AI czar says it has created a “doomer cult”. Almost every penny of that $1.5 billion comes from relatively new philanthropic organisations associated with Effective Altruism, a radical utilitarian movement with deep pockets. The companies OpenAI and Anthropic both emerged from EA rationalist circles. The personal and professional worlds are deeply entwined.

The researcher Nirit Weiss-Blatt, whose AI Panic site documents the sprawling complex of EA entities, companies and philanthropic institutions, thinks the true figure may be even higher. It’s a “well organized, well funded campaigns”, Sacks says.

Looking at the list of grantmakers, the mind boggles at just how many there are, and by how remarkably similar they look and sound. Welcome to the world of EA philanthropy, which handles around $1 billion a year, sending it to causes ranging from shrimp welfare to political marketing exercises like “Progress Studies” and Yimbyism. But in recent years, it’s AI doom that has absorbed the most capital.

This week, a researcher called Jacob Coxon played his part in the narrative, resigning from the AI company Anthropic with a theatrical flourish. Coxon’s parting shot asserts that progress in AI is accelerating and “we’re gambling with our lives”.

“Jacob is correct here – we really do earnestly believe AI could kill all humans!” echoed Anthropic’s “safety” chief Evan Hubinger.

Coxon isn’t the first such dramatic resignation from an AI company. The neural network pioneer Geoffrey Hinton resigned from Google in 2023, so he could “speak freely” for the EAs on AI existential risk, and he hasn’t shut up since. Daniel Kokotajlo and William Saunders resigned from OpenAI the following year, citing identical concerns.  There are several others, too: the flounceouts have become as regular as the spectacle of paint being thrown on old masters by the apocalyptic climate catastrophists of Just Stop Oil.

Coxon’s resignation can be viewed as an instalment in the story that AI is going rogue, which has hogged headlines all summer. Open AI and its spin-out Anthropic were founded to address fears of AI doom. Elon Musk instigated OpenAI in a huff at Google buying DeepMind, according to the author Sebastian Mallaby in his biography of Sir Demis Hassabis I reviewed for The Critic. Musk believed that Google wouldn’t take existential risk seriously enough. When senior executives left OpenAI, to form Anthropic, they cited the same reason. Both have conducted exercises which purport to show AI “going rogue”, displaying agency and collaboration to break into external systems. The minders then expressed “surprise” at the results.  Even Whitehall has joined in, with the EA-dominated AI Security Institute conducting a similar exercise.

Cybersecurity experts, however, deride the claims. They point out these AI models were trained to break into systems, and then instructed to hack while their human minders looked the other way. I asked two global security experts to review AISI’s material: they concluded it was deliberately deceptive. The “surprise” expressed by all three was fake.

The stunts have been criticised by the founding chief of the National Cybersecurity Centre (NCSC), part of GCHQ, Professor Ciaran Martin in The Economist. He has come under attack from the EAs and even their most reliable attack dog, Dominic Cummings.

“They were not going rogue. They were doing what humans had told them to do, with the precocity and indiscipline of talented but unsupervised children. They did no real harm,” Martin wrote. These were irresponsible exercises, explains Dr Heidy Khalif, a systems security expert who was recruited by OpenAI in its early days, and who works at AI Now, one of very few organisations looking at AI safety critically — and from outside the EA multiverse.

Stopping a runaway computer software programme is easy as pulling the plug from the wall. A killer AI would need to be able build and harvest its own power sources and resist a military attack. Such a machine is not feasible at present, or for a very long time.

So why would so much money be devoted to what is an absurd, but admittedly compelling science fiction story?

Khalif believes that existential risk, which the EAs abbreviate to x-risk, is “a regulatory capture mechanism”. The EAs co-opt the language to elbow aside incumbent experts and well established safety practices, and advance their own power base, something she has observed in both nuclear and AI regulation. The gambit “places AI technologists as arbiters of life or death”.

Eschatological narratives like catastrophic climate change and AI going rogue and killing us all both inspire extreme reactions, inducing a kind of mental episode. Sam Altman’s home has been attacked by EA doomers, some EAs went on hunger strike outside Google’s London HQ, and it has even spawned a vegan murder cult. Mostly nonbinary, and describing themselves as “vegan anarchotranshumanists”, the Zizians emerged from rationalist and EA circles, and murdered six people before being arrested.

You alone are blessed with an insight: you alone can see the end of the world

Both preoccupations fundamentally appeal to narcissism. You alone are blessed with an insight: you alone can see the end of the world. Others can’t, and so the urge to share the prophecy is overwhelming. AI doom has been the more persuasive with our political and media elites, for whom faith in AI is a deeply high status opinion. This is not just because they lack the technical grounding to assess such claims independently, but because they desperately want to believe in the prospect of an overwhelmingly powerful AI. They paint the ghost into the machine.