The internet is abuzz with debate about “pacing the frontier” of AI development. This conversation was sparked in part by an Anthropic employee resigning and accusing AI companies of “gambling with our lives,” followed by another employee putting the chance of AI causing human extinction above 10%. We’ve heard Chicken Little before.1 But the capabilities of these systems, the inability of these companies to contain them, and the decisions being made about who gets to use them, make the debate increasingly difficult to ignore.
I find there are a few crucial things missing here.
The first is this: we shouldn’t let distrust of AI companies reassure us that AI is harmless. There are plenty of reasons to distrust AI CEOs when they make bold claims about the end of humanity. These claims can be great marketing for their products. They can justify regulations that slow competitors or make their own business models more favorable. And the people making them have not always been trustworthy messengers.
But we should be as skeptical of our own cynicism as we are of AI CEOs. It’s comforting to believe that people claiming to have the capacity to end the human race are lying to us for self-serving reasons, and finding those reasons can feel like resolving the question of whether they’re right. It doesn’t.
We could ignore them and try to answer this question in other ways, but AI CEOs and employees are privy to nonpublic information, and what they say about AI development is worth considering. So I’m trying to pay attention to where these beliefs cost something. When Daniel Kokotajlo resigned from OpenAI, he refused to sign a departure agreement containing a nondisparagement clause while believing he would lose substantial equity. I believe he sincerely holds his concerns (though sincerity doesn’t mean he’s right).
It’s harder to see where it’s costing the CEOs right now. But I think it’s a more useful way to evaluate sincerity than continually finding new ways fatalism benefits the people hawking it. It’s easy to do, it’s a cop out, and I think it’s taking up too much of the conversation right now.
Taking the danger seriously, though, doesn’t require accepting extinction as the organizing frame.
We should demand more from AI companies and AI safety researchers than the promise not to kill us. I want to know how they expect the social contract will change in the presence of superintelligence. Personal agency and self-determination are at risk as much as our lives are, and I think placing them at the center of the alignment conversation has several benefits for everyone involved, including AI companies.
First, resolving the tension between personal agency and AI capability solves the existential risk issues and solves a harder and more important problem than mere survival.
In some ways, personal agency is amplified with AI. I can use it to build things I couldn’t build before, understand things I couldn’t understand, and finish projects that would otherwise require a team of people.
But agency is easy to argue for when it’s free of consequences. The difficult question is what room people should have to act when they might do something idiotic, defy authority, or cause some trouble for someone else. (This room for personal judgment is especially critical when it’s least likely to be exercised well – during adolescence).
We already limit that freedom, through laws and other institutions. In principle, fine. We live in a society, after all. But the social contract may begin to strain as AI raises the stakes. One person’s bad judgment could have global consequences. As downside risks increase, we might find ourselves restricting the room for personal judgment with new rules and new laws, enforced at the scale of a surveillance state.
And the stakes don’t have to increase for that room to shrink. Our existing rules operate with limits on human attention, and room for discretion. Making every small violation detectable and punishable changes what it means to live under those rules, even if the rules themselves don’t change.
Determining who can take certain risks, and when, and not have society collapse, is the AI alignment problem. The “aligned” vision of the future currently proposed, in which we are all infinitely healthier and wealthier, is simply one instantiation of this. But if this vision asks us to sacrifice personal agency for survival, it falls short of true alignment with human values. People have fought and died for the right to self-determination.
This brings me to my second point: self-determination is a more accessible way to bring people into the conversation about AI risk.
I think AI doomers ask too much of the public when they make the end of humanity the entry point to this conversation. If someone finds the scenario implausible or doesn’t trust the messenger,2 it becomes easy to dismiss the concern along with it. I think this same mistake was made by climate change activists.
On its face, the threat to personal agency is more abstract than any of the doomsday scenarios. It doesn’t grab a headline as easily as Skynet does. But it’s here, and we can talk about it in concrete ways.
In my industry, software engineering, the stratification of power and trust is already becoming acute. Gratifying parts of our jobs are being automated away, all in the name of velocity.3 It adds insult to injury that the frontier labs tell us “you’re not to be trusted” when they reserve the most capable models — the models that make fewer of the mistakes we have to clean up — for themselves.
As a member of the general public, I already feel my behavior is steered through automated bureaucracy, customer service, hiring funnels, and recommendation algorithms on Spotify, YouTube, Netflix; surveilled at scale by adtech, social media, and the proliferation of cameras, whether CCTV, smartphones, or placed surreptitiously in sunglasses.
I think people feel these things far more acutely than they feel that some future AI might engineer the next plague.
Centering personal agency in this conversation asks more of AI companies than the promise not to kill us. It asks how we’ll retain the freedom to exercise judgment, challenge authority, and make mistakes when these systems make all three more consequential (and easier to restrict).
We shouldn’t let cynicism reassure us that AI is harmless. But we shouldn’t accept a despotism of surveillance, automation, and bureaucracy as the price of making it safe, either. If increasingly capable AI requires increasingly tight restrictions on personal agency, those restrictions are themselves a cost of development. That belongs in the conversation about “pacing the frontier.”