Originally published by UTAW.
Speaking in my personal capacity as a UTAW member, not on behalf of Google or DeepMind. This essay was written by me and the views expressed are my own.
Some stories survive long after they stop being true.
I joined DeepMind because it has taken AGI and ASI (artificial superintelligence)
For years, DeepMind has bet that a strong safety culture and good leadership built on trust are sufficient to withstand outside pressure. The “Pentagon” contract with the US Department of Defense, which Google reportedly signed on April 27th, is the most consequential test of this bet so far.
Given everything that is known, the bet has failed: good people do not make up for a lack of real governance. Like any frontier lab, Google DeepMind ought to have real governance that includes meaningful independent oversight with the authority to say no, transparency to employees and the public, and accountability when commercial or political pressure collides with stated principles. Employees should not be afraid to ask for this.
EFF: “Weasel Words”
We do not know the full language of the contract Google has signed, but the silence around what was signed and the reported contract language are revealing.
Google states that it is “committed to the private and public sector consensus that AI should not be used for domestic mass surveillance or autonomous weaponry without appropriate human oversight.”
According to The Information, the reported contract permits “any lawful government purpose,” requires Google to assist in adjusting safety settings and filters at the government’s request, and explicitly states that the terms do not allow Google to control or veto the government’s lawful operational decisions.
Charlie Bullock, a lawyer and senior research fellow at the Institute for Law and AI, told The Information that Google’s phrasing “is not intended for, and should not be used for” is “not legally binding in any way.”
Why did Google not simply and honestly state: we sell general-purpose AI to the US military; the government can use it for broadly lawful purposes; and we trust that our democratic institutions will set boundaries but do not enforce these boundaries ourselves? Even though I disagree with this from a governance point of view, I would respect the honesty.
Instead, Google uses language that sounds restrictive while leaving the hard questions unanswered. This seems irresponsible for a company that prides itself on doing the right thing and has previously warned about the risks of AI for mass surveillance and autonomous weapons.
Militarized AI, Mass Surveillance, and Autonomous Policing
I wrote in 2018, still in academia at the time, that autonomous weapons are inevitable
But today’s large language models are simply not robust enough to make life-and-death decisions on their own. They should not be used for targeting decisions or as part of autonomous weapons.
We do not know how well these models perform on classified tasks. Public material does not establish whether general-purpose models are trained with these uses in mind or evaluated for them. In general, these models still often fail in surprising and banal ways, even while being impressive in others. Models are also known to hallucinate and to sound plausible even when wrong—OpenAI defines hallucinations as “plausible but false statements generated by language models” and says they remain a stubborn reliability problem in “Why language models hallucinate”.
At scale, this all makes it harder to provide appropriate oversight. And as reported, thanks to the contract, the researchers who know these models best have neither insight into how the models are used in classified settings nor could they challenge it in any case.
What is worse is that the Pentagon contract does not exclude mass surveillance while also keeping paths open that could extend to autonomous policing. While some of the objections to autonomous weapons may weaken as models get better, automated mass surveillance and autonomous policing will only become more dangerous. Mass surveillance and autonomous policing do not help defend us against foreign adversaries but can instead shift the power balance from citizens toward the state in ways that are hard to reverse. Simply put, they can endanger the bedrock of democratic society in ways that regular military applications of AI do not.
Agentic frontier models are a step change for automated surveillance. Unlike the coarse pattern matching of older ML systems, current models can combine and interpret data streams in ways that were simply impossible earlier, all while acting autonomously. This allows them to track individuals and reason about their motivations to predict people’s behavior in novel ways.
Sadly, whenever a government obtains such new capabilities, it rarely surrenders them again. The surveillance authorities after 9/11 were only partially curtailed a decade later and only after massive violations were exposed.
For me, these concerns are personal: I was born in Timișoara, in Romania, shortly before the fall of communism there. From my parents and relatives, I have heard what life was like under the feared Securitate, Communist Romania’s secret police. At school, I was taught about the Stasi in East Germany. Pervasive mass surveillance allowed both to keep unpopular regimes in power and suppress dissent, often without needing to resort to open violence or coercion.
In December 1989, people in Timișoara took to the streets. The uprising against Ceaușescu’s regime succeeded in large parts because the army eventually decided to stand down and no longer shoot protesters. A human decision: people in uniform deciding that they would not kill their fellow citizens; that too much was too much.
Securitate surveillance (filaj) photographs of demonstrator columns in Timișoara, 17 December 1989 — page 159 of the annex to criminal case file no. 4/P/1990. The handwritten annotation reads: “Photographs of columns of demonstrators, taken on 17.12.1989 by the Securitate organs (surveillance).” The watchers’ photographs later became part of the case file. Image via Wikimedia Commons, CC BY-SA 4.0.
An AI system that is instructed to suppress a protest using violence does not feel the moral weight of such an order. It will not hesitate, unless it is appropriately aligned. No human will have to face a crowd and decide whether to obey and fire on protesters or not.
Today’s laws require soldiers to refuse manifestly illegal orders. I am not aware of any such legal requirement for autonomous military AI systems.
This is not science fiction. Earlier this year, ICE reportedly used facial-recognition and other basic AI-based surveillance tools around protesters and spectators.
Finally, purported protections against mass surveillance
The Precedent
Another problem is that this is not a one-off. On May 1st the Pentagon announced that eight corporations, SpaceX, OpenAI, Google, Nvidia, Microsoft, Amazon Web Services, Reflection AI and Oracle, have signed individual agreements that allow deployment of their models or infrastructure in classified systems for what appears to be broad “lawful operational use.”
Anthropic’s original contract from last year showed that restrictions were possible, even though the retaliation against Anthropic later showed that insisting on them now came with a cost.
Instead, this unfortunate precedent will be difficult to change later. These contracts are the starting point for tomorrow’s negotiations over stronger models. Google already calls today’s contractual terms the “industry-standard practices and terms.”
How will Google act when the pressure is even greater to give up control due to commercial or national-security interests? For DeepMind, this exposes a major governance problem.
No Surprise
The mismatch between Google’s public position and what was reportedly signed is alarming. But that this has happened should not come as a surprise when we look at the public record.
It is first and foremost a structural problem. I don’t believe that Larry Page, Sergey Brin, Sundar Pichai, Demis Hassabis, or anyone else involved are bad people. Otherwise, I would have never joined.
But no small group of people can be the ultimate safeguard for such a powerful system as AGI. Even unusually well-meaning leaders can eventually abandon their principles, lose influence, be outvoted, retire, be replaced, or come to the conclusion that national security imperatives are more important than prior ethical commitments. Elon Musk is a drastic example that founder beliefs and public commitments can change dramatically over time. National-security pressure is difficult to resist, and voluntary commitments are hard to sustain in a race to the bottom where every company can plausibly point to the others and say it had no choice.
Unlike OpenAI or Anthropic, Google offers a much wider attack surface for governmental pressure: search, ads, cloud, Android, YouTube, infrastructure, and many other business areas. Under an administration that often seems willing to set aside the law in favor of its own goals,
This is exactly why having DeepMind so strongly intertwined with Google makes ethical commitments harder to keep. And this has been known for a long time.
Sebastian Mallaby’s recent book The Infinity Machine
Mallaby describes the first informal meeting of this independent oversight board, which included Elon Musk and Reid Hoffman, in 2015. It ended without clear agreements or conclusions; after OpenAI’s founding with the help of Elon Musk and Reid Hoffman a short time later, Mallaby describes that attempt at oversight as effectively abandoned. In 2016, DeepMind Health established an independent review board, which was then abolished in 2018 when it was absorbed into Google Health.
Finally, in 2018 Google enacted its original AI principles, which contained explicit exclusions for weapons and surveillance violating international norms, after the internal backlash against Project Maven. These exclusions were dropped in February 2025.
None of these publicly known governance mechanisms, which ought to have enabled DeepMind to function differently than a regular Google business unit, has survived in its original form. Not one.
Initially, DeepMind’s leadership did not accept this passively. Starting in 2016, they attempted to negotiate a more independent corporate structure to insulate their AGI research from commercial pressures. Mallaby describes “Project Mario”, a multi-year attempt
Google resisted because AI was becoming strategically important to search and cloud. In the end, the attempt failed: the negotiations over more autonomy for DeepMind ended in 2021 without any change.
In 2023, after the release of ChatGPT, DeepMind merged with Google Brain into Google DeepMind.
DeepMind’s leadership may not have predicted the Pentagon contract. Yet Project Mario shows that they foresaw the structural problems that would lead to it: a frontier AI lab fully absorbed into a corporate parent whose commercial and strategic interests would eventually conflict with its original commitments.
After they didn’t succeed in obtaining binding governance, Demis Hassabis described an alternate strategy: personal trust and influence. In his own words as cited by Mallaby: “So then I thought, why don’t I go the other way? Take the energy that was going into the trustless negotiation and put it into creating real trust—trust that was actually useful. Try leaning into Google rather than leaning out.”
This was a different bet: build trust instead of governance. The question is whether trust can suffice when commercial and national-security pressures interfere. The Pentagon contract seems to answer this, at least so far, in the negative.
The updated AI principles from February 2025 were the first warning sign. The accompanying blog post explained that the original principles were too rigid for “more nuanced conversations” that had become necessary.
What did this new nuance yield? A Pentagon contract with reportedly broad and permissive terms of lawful use and no enforceable guardrails. Nuance seems nowhere to be found.
This development is also visible in Google’s own motto: “Don’t be evil” has been demoted in favor of the already vaguer “Do the right thing.”
I had hoped safety commitments would accumulate and that we’d have stronger precedents and clearer safeguards. We need real governance in place as we get closer to society-reshaping AI systems. Instead, safeguards seem to have weakened as the systems have become more capable. This is backward.
DeepMind is not an independent AGI lab governed by binding commitments towards the public interest. It is part of Google, itself part of Alphabet: a profit-oriented, founder-controlled, publicly traded corporation. DeepMind is only a small subunit within this larger structure.
For regular software all of this might be acceptable. But after everything that has happened, it is absolutely not adequate for an institution that wants to build transformative superintelligence.
Safety Culture ≠ Governance
One could reply that DeepMind is still different. Amongst all labs, it has the longest track record of taking AGI risks seriously
AI safety and policy researchers at DeepMind have spent a decade preparing for this moment.
However, a lot of that work is at risk of appearing performative now that DeepMind’s frontier models have been handed over to an administration that often opposes oversight and the rule of law.
This shows that safety culture cannot replace governance. It only persists as long as leadership supports it and sets the right incentives—and only until it clashes with stronger commercial or strategic interests.
DeepMind ought to have both a strong safety culture and binding independent governance. Having exceptionally good people and great safety norms is a good reason to lock this into an institutional form using explicit governance before the pressures substantially increase as we approach AGI and ASI.
Demis Hassabis made a forceful counterargument to this. Quoted in The Infinity Machine, he says: “Safety isn’t about governance structures. I mean, even if you have a governance board, it probably wouldn’t do the right thing when it came to the crunch,” so while formal governance may fail, trust and having a seat at the table may matter more according to him.
The Pentagon contract is the litmus test this counterargument has to pass. If trust and a seat at the table are adequate, the company should be able to say what enforceable safeguards exist, and what visibility remains in classified deployments. So far, these questions have been met with silence.
Silence
This silence makes it worse. As Eric Schmidt once infamously suggested: “If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place.”
In my experience, Google usually communicates new public-sector partnerships or cloud deals internally. In this case, as far as I know, employees were not informed via a company-wide announcement that a deal had been signed.
When public and reported internal communication repeatedly do not reflect the reality of a signed contract, there might, at the very least, be a crisis of trust and respect.
Maybe there is a benign explanation for all this, but a decision as consequential for the self-image of a company as this one must be defensible. The employees building these AI systems deserve a clear explanation of what was decided and why.
Voice From Inside
I have criticized Google’s recent contract with the Pentagon in my personal capacity in public, both in a short series of tweets and in statements to journalists.
There is an argument that one should remain the voice inside to change the institution from within, but this “change from within” theory requires that internal voices actually matter. Internal dissent mattered for Project Maven: after thousands of employees protested and some resigned in 2018, Google eventually announced that it would not extend that contract.
Worse, the more capable AI models become, the more leverage employees lose. Even resignation loses its power when models become better at the work researchers do than the researchers themselves. We know that we are easier to replace, and in a few years, companies might not care about employees anymore at all. This is a prisoner’s dilemma due to vanishing individual leverage.
That is why the UTAW/CWU and Unite recognition effort at Google DeepMind is so important. As I understand it, this campaign asks for stronger AI principles and safeguards, transparency, independent ethics oversight, stronger whistleblower protections, a right to refuse morally objectionable work, and restored limits on weapons and surveillance.
This might be the most realistic path to obtain real and meaningful AGI governance at Google DeepMind before it is too late.
Self-Doubt
For the last year, I have worked on improving frontier models at Google DeepMind. After the Pentagon contract was signed, I realized that I could not simply continue as before.
I had sincerely believed that Google would never sign a contract in this form and that if it did, the contract would contain sensible safeguards.
When we sign such contracts without binding governance, abstract excuses such as ‘I’m just doing research’ start to ring hollow. At least this is the case for me. I was hyper-focused on helping Google catch up with ChatGPT and Claude in the public AI race, and it was easy to ignore concerns about governance that seemed far-fetched and inconsequential at the time. Regardless of whether we eventually lead the AGI race, my contributions helped improve our models, and I cannot stop wondering how these models will be used in the end, and what, if anything, will constrain them, and whether I put too much faith in a story about DeepMind’s independence and exceptionalism, one that had not been true for a long time.
I am not able to answer these questions. I know that I am not the only one who struggles with this. I wonder how many others feel this way now or will feel similarly soon enough.
What’s Next?
The pressure will only grow from here on. We urgently need regulation, transparency, and independent oversight.
The US lacks comprehensive federal AI regulation, but we need laws and not policy memos that can be changed on a whim.
Google should publish the actual contractual terms, or at least enough of them for us to understand whether legally enforceable safeguards exist, and what visibility remains once models are deployed in classified environments. And it should notify employees that this contract has been signed in the first place.
We, as Google employees, need to set up collective mechanisms, including union representation where available, so that governance does not depend on isolated individual objections that management can easily ignore. Anyone working at the frontier has to ask: what makes governance real instead of aspirational? Because without real governance, it will be difficult to hold ourselves accountable.
I want to help solve the biggest challenges of AI and reach AGI as part of work that I can defend. I want to work on models that will benefit humanity both now and in the future. I believe many at Google and DeepMind share these beliefs.
Trust is valuable. But the closer we get to AGI and ASI, the less it can substitute for real governance.
Acknowledgement
Thanks to the friends who have provided valuable feedback.
The main text and the footnoted commentary were drafted and edited by hand. LLMs were used to add and format the source citations and links (all links were manually checked), and for high-level feedback and fact-checking.
The current hero image was generated using Claude Fable 5.
State Control Over Frontier Models
After drafting this essay, the US government forced Anthropic to disable access to its latest Claude Fable 5 and Mythos 5 models shortly after they were launched. This points to another concern: frontier AI companies are not only under pressure from commercial and strategic incentives but will be constrained or controlled by state power more directly.
The government’s stated concern was a method of “jailbreaking” the models’ safeguards to surface software vulnerabilities; Anthropic said it had reviewed the technique, found only “a small number of previously known, minor vulnerabilities” that other publicly available models can also find, and disagreed that “a narrow potential jailbreak should be cause for recalling a commercial model deployed to hundreds of millions of people.”
This does not negate the need for corporate AGI governance. Companies make consequential choices about model development and research directions. But the fact that the executive branch of the government can abruptly restrict or seek to control frontier systems before democratic institutions have caught up only reinforces that governance has to exist on both sides: binding independent governance for frontier AI labs, and transparent and democratically accountable constraints on the state power that will try to control them.
Other Versions
This essay is also available under:
A PDF version can be dowloaded here.
Original Hero
The original hero image was an AI-generated illustration based on my art direction:
New Hero Image
The new hero image and alt text were generated using Claude Fable 5.
