Matthew Butterick | Big AI to humanity: drop dead

9 min read Original article ↗

Recently, jour­nal­ists have sought comment from me on a series of unusual AI events. Last week, in a certain AI copy­right case, the US govern­ment filed a “state­ment of interest” in support of AI training being fair use of copy­righted works. (No comment.) This week, workers at a certain AI company posted gloomy messages on social media about the possi­bility of AI extin­guishing human life, so I was asked whether “crimes against humanity” have been committed by these AI compa­nies. (No comment.) Mean­while, op-eds in major US news­pa­pers are calling for some­thing, anything to be done.

I’m a self-employed author, designer, programmer, and lawyer. In 2022, I learned that my own works were in the training datasets of gener­a­tive-AI compa­nies. In response, I invented the first set of lawsuits chal­lenging the legality of these prac­tices. There are now 142 such cases in the US. I’m currently co-counsel for plain­tiffs in eight of them. Though I discuss certain legal issues here, I am not your lawyer, and nothing here is held out as legal advice. These are my personal views; I speak only for myself.

AI risks ripen

In 2024, I said that “an AI cata­strophe arising from failure of align­ment is much more likely than one arising from sci-fi-style malig­nant agency of the AI.” In some sense that predic­tion is ripening.

But media predic­tions about the nature of that AI cata­strophe remain unhelp­fully rooted in sci-fi scenarios—what I’ve termed the Skynet fallacy. Unhelpful because these scenarios are primarily a vessel for fear. They don’t illu­mi­nate paths to real­istic policy change. This New York Times op-ed, for instance, asks us to imagine “rogue A.I.s [that] hack out of their container” and “design a super­virus that spreads uncon­trol­lably”. The “hack out” part—plau­sible. It’s already happening. Designing a super­virus—less so.

Still, taking the NYT op-ed as a template, let’s consider why pundit-friendly proposals for AI safety likely won’t work.

Op-ed proposal 1: shut it down

Shut it all down, now … The obvious way to prevent A.I. from killing everyone is to issue a global ban on A.I. research. The problem is that our society has already gambled more than a tril­lion dollars on A.I.’s upside, so a ban would have ruinous side effects.

“Obvious way”—yes, in the vacuous sense of there oughta be a law! But in prac­tice—much easier said than done. No tech­nology has ever been the subject of a preemp­tive “global ban” of this nature. Inter­na­tional nuclear nonpro­lif­er­a­tion treaties are prob­ably the closest analog. But they only arose after the US and other nations had competed over decades to develop nuclear weapons. And of course, these treaties did not call for complete nuclear disar­ma­ment.

The “shut it down” argu­ment also over­looks that there are already state and federal laws that prohibit mali­cious soft­ware—e.g., the Computer Fraud and Abuse Act, the Elec­tronic Commu­ni­ca­tions Privacy Act, and others. At the federal level, the ques­tion is not whether we have laws that can address AI—we do. The ques­tion is whether we have law enforce­ment that will charge AI compa­nies with violating those laws. Conversely, as long as federal law enforce­ment remains supine, then enacting further laws is an empty gesture.

The economic argu­ment is salient, however. As I noted in March 2023, as a public-wealth matter, “[t]he money” expected to be returned from AI invest­ment “has already been spent.” Here in 2026, a stag­gering amount of national capital is flowing toward AI. No nation would volun­tarily make itself poorer by acceding to a “global ban” on AI. Anthro­pol­o­gist Joseph Tainter predicted this effect in his 1988 book The Collapse of Complex Soci­eties (which I wrote about). I summa­rized this partic­ular point: “In prin­ciple, a nation could choose to decel­erate its own economic growth to fore­stall collapse in the future. But that would simply make itself vulner­able to domi­na­tion by another nation today. Such decel­er­a­tion would there­fore be polit­i­cally irra­tional.”

Op-ed proposal 2: investigate incidents

Take an air[-]crash[-]inves­ti­gator approach … When an aircraft crash occurs, inves­ti­ga­tors from the National Trans­porta­tion Safety Board are imme­di­ately dispatched to the site to gather forensic evidence, conduct inter­views and deter­mine the under­lying cause.

National Trans­porta­tion Safety Board inves­ti­ga­tions have certainly led to air-safety improve­ments. But the NTSB is not the primary source of avia­tion regu­la­tion in the US—that’s the Federal Avia­tion Admin­is­tra­tion. The NTSB was estab­lished as an inde­pen­dent inves­ti­gator of trans­porta­tion inci­dents partly so that the FAA would not be in the conflicted role of inves­ti­gating the effec­tive­ness of its own regu­la­tions (or confronting its own polit­ical entan­gle­ments). Like­wise, an NTSB-like orga­ni­za­tion that retro­spec­tively inves­ti­gates dangerous AI inci­dents will have a very limited range of influ­ence without an FAA-like orga­ni­za­tion that imposes and enforces oper­a­tional and safety regu­la­tions.

Op-ed proposal 3: public monitoring

Monitor the situ­a­tion … like the systems we use for air traffic control … Researchers would be required, by law, to post public infor­ma­tion on who is conducting the training run and which data center is doing the training.

The air-traffic compar­ison doesn’t hold. US airspace is a feder­ally regu­lated and managed resource (by the afore­men­tioned FAA). So infor­ma­tion about ordi­nary flights oper­ating within is public by default—some­times to the conster­na­tion of aircraft-owning private citi­zens. Imposing similar public disclo­sure on private US AI compa­nies using private US data­cen­ters would be legally diffi­cult. Further­more, in the future, more AI models will be trained for national-secu­rity uses. These will be among the most poten­tially dangerous AI models. But they will be exempt from public disclo­sure on national-secu­rity grounds, lest these data­cen­ters become mili­tary targets—this week, we started training the Torment Nexus model at our beau­tiful Spring­field data­center 

Op-ed proposal 4: kill switch

Flip the kill switch … Repre­sen­ta­tives Ted Lieu, Demo­crat of Cali­fornia, and Nathaniel Moran, Repub­lican of Texas, have intro­duced the A.I. Kill Switch Act, which would give [Depart­ment of Home­land Secu­rity] the power to order the shut­down of dangerous A.I. oper­ating beyond its para­me­ters

First: for any ques­tion of AI safety—or human safety gener­ally—the answer cannot, cannot, cannot be “more DHS”. Second: as a tech­nical matter, AI kill switches are a sci-fi fantasy. Sure, any AI model can, in a yank-the-power-cable sense, be turned off. But that doesn’t prevent, say, AI-gener­ated malware from prop­a­gating. This is not new: in 1988, a human programmer released a small self-repli­cating program onto the internet that inca­pac­i­tated thou­sands of email servers. Once these copies had prop­a­gated, there was no way to arrest them remotely. Recently, LLMs have been discov­ered leaving messages for each other on public wiki sites. We can infer that there are already other instances of LLMs commu­ni­cating in the wild that have not yet been detected, and further instances that will never be.

Big AI’s security narrative

Against a back­drop of secu­rity inci­dents that will only increase in number and severity, Big AI is pursuing a three-pronged narra­tive:

  1. Big AI believes they are the only ones who can protect against the risks that their prod­ucts create. But this narra­tive isn’t believ­able unless the threat is believ­able. So Big AI has a huge incen­tive to talk up AI risks, but no incen­tive to invest in commen­su­rate secu­rity prac­tices.

  2. Big AI believes they should not be held account­able for the conse­quences of their AI systems because these systems are unpre­dictable and perhaps uncon­trol­lable. A certain AI researcher said of recent AI hacking inci­dents: “AI agents … took actions that would be consid­ered as crimes if a human took them”—seem­ingly taking it as axiomatic that these were not human-controlled activ­i­ties and there­fore cannot qualify as crimes. But they were and they do. This outra­geous posi­tion inverts decades of US law about dangerous items gener­ally and computer hacking in partic­ular (e.g., the 1986 Computer Fraud and Abuse Act). So let’s call this narra­tive what it is: an attempt to thwart the rule of law. Indi­vidual human program­mers have been sentenced to prison for far less than what AI compa­nies have done recently. This relates to what I foresaw in 2023:

    If AI compa­nies are allowed to market AI systems that are essen­tially black boxes … we will not dele­gate deci­sions to AI systems because they perform better. Rather, we will dele­gate deci­sions to AI systems because they can get away with every­thing that we can’t. … [W]e could end up with some­thing truly novel: tech­nology systems that deserve much higher levels of legal scrutiny (because of the conse­quen­tiality of their outputs) but simul­ta­ne­ously resist such scrutiny (because of the opacity of their inputs and reasoning).

    See also: a certain AI CEO recently called for “industry-wide coor­di­na­tion” within Big AI while attaching a quieter foot­note seeking “waivers of antitrust restric­tions” to do so. As if antitrust law were merely one more statu­tory mosquito to swat.

  3. Big AI believes that the burden is on govern­ment and citi­zens to affir­ma­tively stop Big AI from proceeding. Since overtly opposing regu­la­tion is a bad look, Big AI CEOs have occa­sion­ally made noises about being open to regu­la­tion. As one AI CEO said recently: “We must slow the pace at which we improve the capa­bil­i­ties of AI models.” But as Big AI is well aware, there’s no chance of AI-specific domestic laws or inter­na­tional treaties being enacted soon enough to matter. Indeed, the same AI CEO blamed democ­racy for not meeting his KPIs: “[u]nfor­tu­nately, passing laws can take time”. A widely signed March 2023 letter sought to pause AI research; like all chain letters, it accom­plished nothing. After a genuine AI cata­strophe arrives, we can be sure these same AI CEOs will say “gosh—why didn’t you make us stop?”

Big AI’s security narrative is ludicrous

So let’s not take the bait. Nor over­com­pli­cate. We needn’t spin our collec­tive wheels spit­balling answers to big-picture, long-term AI-policy ques­tions. We don’t know enough yet. The best next steps are the concrete ones: Big AI needs to follow all current laws—just like everyone else. So far—they haven’t. When Big AI breaks those laws, they must face pros­e­cu­tion and penal­ties—just like everyone else. So far—they haven’t. The state and federal agen­cies tasked with enforcing those laws need to apply them to Big AI—just like everyone else. So far—they haven’t. In short, we have to attend to the basic features of the rule of law. So far—we haven’t. If we can’t or won’t insist on that now, then we shouldn’t expect to be able to later.