AGI: the illusion that distorts and distracts digital governance

ABSTRACT

The claim that Artificial General Intelligence (AGI) poses a risk of human extinction is largely responsible for the urgency surrounding AI regulation and governance. Underlying these assessments is the idea that AI development may make a computing machine an autonomous, all-powerful actor, and thus a potential threat to humanity. Drawing on perspectives from computer science, economics and philosophy, this paper unpacks the assumptions, evidence and logic underlying the AGI construct. It concludes that AGI is an unscientific myth. Three fallacies underpin the AGI construct: (a) the idea that machine intelligence can achieve a limitless ‘generality’; (b) anthropomorphism, the unwarranted attribution of goals, desires and self-preservation motives to human-built machines; and (c) omnipotence, the assumption that superior calculating intelligence will provide AGI with unlimited physical power. The paper goes on to explain why dispelling the AGI myth is important for public policy. The myth, which still exerts heavy influence on attitudes toward digital governance, diverts attention from the real policy issues posed by the human use of AI applications, and promotes sweeping and potentially authoritarian policy interventions over all forms of information and communication technology.

This article is part of the following collection(s):

1. Introduction

Why is the public discourse around AI dominated by concepts such as ‘safety’, ‘irreversible harms’, ‘regulation’, ‘risk’ and even ‘human extinction’? The answer is that since 2023, public and scientific discourse about machine learning has been haunted by the spectre of an ‘artificial general intelligence’ (AGI).

Almost from the moment electronic computer technology was invented, philosophers, science fiction writers and computer scientists speculated that machines might evolve into a superhuman intelligence that gains the ability to act independently of human instructions. Earlier literature called it ‘superintelligence’ or ‘ultra-intelligent machines’. As far back as 1965, an Oxford computer scientist, I.J. Good (Citation1965), speculated about ‘a machine that can far surpass all the intellectual activities of any man however clever’, a definition not that different from current descriptions of AGI. Later, in the 1990s, there was speculation that AGI would be the product of a ‘singularity’ in which machines gained self-consciousness and autonomy. This cosmic transition would come from a ‘runaway reaction’ of self-improvement cycles that would make an AI application so powerful so rapidly that humans would not be able to catch their breath before their very existence was threatened (Bostrom Citation2003; Kurzweil Citation1999).

While musings about the possibility of autonomous machines go way back, they weren’t always so fraught. Good gave this vision a whimsical and slightly optimistic spin: ‘The first ultra-intelligent machine’, he wrote, ‘is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control’ (1965, 33). Today, however, the vision is apocalyptic. An open letter signed by more than 350 executives, researchers and engineers in May 2023 claimed that artificial intelligence posed a ‘risk of human extinction’ and urged us to make ‘mitigating that risk a global priority’ (Center for AI Safety Citation2023). A new book by two famous AI developers is entitled If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us (Yudkowsky and Soares Citation2025). In August 2025, Forbes reported on students in Harvard and MIT dropping out of college because of the threat of AGI: ‘I was concerned I might not be alive to graduate’, said one former MIT student. ‘I think in a large majority of the scenarios, because of the way we are working towards AGI, we get human extinction’ (Feng Citation2025).

Thus, the AGI construct expresses a belief that the development of intelligent machines will lead to the crossing of a threshold to where the human-built machine would not only be smarter than all humans, but also all-powerful. The more powerful AI systems become, in this view, the closer we get to the dangerous possibility of an AGI.

So, what is AGI, and how dangerous is it? This paper conducts a critical examination of the concept and concludes that AGI is an empty construct, a myth. Claims that it will lead to human extinction are based on unfounded assumptions.

Section 1 of this paper attacks the validity of the AGI construct itself. It demonstrates the inability of scientists and philosophers to provide a meaningful definition of what ‘general’ intelligence is in a machine. It shows that the scientific literature provides no basis for distinguishing between AGI and existing forms of ‘narrow’ AI.

Section 2 takes up the question of autonomy – the alleged risk that developing and training AI applications will result in neural networks setting goals for themselves independent of the objectives humans give them and breaking free of human control. It reviews recent computer science literature purporting to provide a scientific basis for this possibility. The analysis shows that the autonomy claim is thinly based on anecdotes about misalignments in laboratory settings. No established model or process that would amplify misalignment to the point where the machine becomes truly autonomous and sets its own goals can be demonstrated. In the real world, misalignments can be, and have been, recognised and corrected.

Section 3 takes up the problem of power and physicality. It shows that the ‘catastrophic risk’ rests on the assumption that superior machine intelligence yields unlimited physical power. These claims defy economic constraints and even basic laws of physics, which place limits on the aggregation of energy, information and power in a single entity.

Section 4 concludes by calling attention to the governance implications of the AGI myth. It shows how the spectre of an AGI-based extinction event has distorted public policy discourse about digital technology, pushing law and regulation in damaging directions. The myth also deflects attention from the less apocalyptic but far more important ways humans need to manage and control AI applications. The paper ends with a call for humans to take responsibility for what they do with technology, rather than offloading risk to allegedly autonomous machines.

2. General intelligence

What is an AGI? It is worth noting that there is no agreed-upon definition, and a great deal of uncertainty about whether such a threshold exists. Most scientists in the field say we do not have it yet and debate how far away we are and what we need to do to get there (e.g. Arshi and Chaudhary Citation2024; Feng, Jin et al. Citation2024; McLean et al. Citation2021; Mumuni and Mumuni Citation2025). Some in the industry and the research community believe it already exists (Agüerea y Arcas and Norvig Citation2023; Bubeck et al. Citation2023).Footnote¹ Kapoor and Narayanan (Citation2025) raise a pertinent question: ‘If AGI is such a momentous milestone, shouldn’t it be obvious when it has been built?’

A careful investigation of definitions shows that this is not a routine divergence of scientific judgment which can be resolved as we get more evidence. The disagreement arises because the AGI construct itself is incoherent. No one can define what ‘general’ intelligence is in a machine. No one can provide a theoretical or operational threshold that separates AI from AGI.

Very few contributions to scientific literature on AGI devote serious attention to the definitional problem. In what may be the only discussion of AGI that faces it, Phillips (Citation2017) wrote:

The purview of Artificial General Intelligence (AGI) is the development of theories, models and techniques for the endowment of machines with intellectual capabilities that generalize to a variety of novel situations. This characterization, however, [begs] important questions about what we mean by intelligence and generalize. In the absence of precise criteria, researchers look to the archetype of general intelligence, human cognition.

Note that Phillips openly acknowledges ‘the absence of precise criteria’ for recognising AGI, but then – like all other definitions this review came across – evades the problem by holding up human intelligence as a model. But that raises more questions than it answers. To begin with, it is unclear whether ‘human intelligence’ means the capabilities of a single human brain or the overall capacity of human civilisation. There is a vast difference between the two. Some define AGI as a replication of human intelligence, i.e. computing machines achieve the capability for problem solving and reasoning associated with humans.Footnote² Others define AGI as a ‘superintelligence’ that exceeds ‘human intelligence in every aspect’ (McLean et al. Citation2021). The latter approach posits a decidedly non-human intelligence that could ‘vastly outperform the best human brains in practically every field, including scientific creativity, general wisdom and social skills’ (Bostrom Citation2003, Citation2014).

Those who see AGI as replicating human intelligence ignore the fact that specific AI applications already meet or exceed the capacity of humans in many areas. Computers have calculated solutions to mathematical or scientific problems faster and more accurately than humans for decades. AI applications can beat humans at chess, Go and other complex games (Hsu Citation2002; Silver et al. Citation2017). LLMs summarise meetings and large collections of text faster and at least as accurately as any human could. A timeline comparing AI’s performance to a human baseline on several benchmarks, including image classification, visual reasoning and understanding of English (), shows that as of the end of 2024, existing AI applications already surpassed humans on most of them and are at near-human levels on all the othersFootnote³ (Bikkasani Citation2024).

The AGI literature confronts this problem by labelling all existing AI applications as ‘narrow’ intelligence or ANI. In the words of McLean et al. (Citation2021), ‘an ANI’s intelligence is task specific (or narrow) and cannot transfer to other domains with unknown and uncertain environments in which they have not been trained’. They go on to say,

An AGI would possess a different level of intelligence, which has previously been defined as an agent’s ability to achieve goals in a wide range of environments, and the ability to achieve complex goals in complex environments’ (McLean et al. Citation2021).

This attempt to differentiate AGI from ANI overlooks two key questions: what goals or objectives would this ‘general’ intelligence have, and where would they come from? Progress in real-world machine learning has always come from humans programming and training AI applications to execute specific tasks. The ability of Large Language Models (LLMs) to answer questions and translate languages, for example, came after years of conceptual modelling of grammar and the input of vast stores of digitised texts. Facial and image recognition applications have been trained on millions of digitised images and programmed to match individual faces to identities. Other AI applications are based on complex algorithms that are trained and constantly updated to do what humans want them to do. Generally, all these applications perform better the more well-defined their goals are (Li, Zhao et al. Citation2024). ChatGPT and other applications with impressive text-generation capabilities, for example, are notoriously bad at arithmetic.

The notion of an AGI, therefore, is a qualitative departure from everything we currently know about machine intelligence. Instead of teaching it to do defined tasks under the guidance of humans, an AGI is supposed to be a single application that can learn to do anything and everything at human – or superhuman – levels.

Central to the AGI notion is the idea of an ‘intelligence explosion’ that will occur when advanced machine intelligence learns to train itself. There is some basis for this expectation. AlphaZero and AlphaGo, the reinforcement learning algorithms developed by DeepMind, used self-supervised reinforcement learning to master games with defined rules such as chess and Go. In these well-bounded domains, AIs became extremely powerful in a matter of months.

There is, however, an inescapable limit to self-teaching. Self-supervised reinforcement learning requires a clear, strict way to evaluate performance. As Norbert Wiener (Citation1964) wrote, ‘In general, a game-playing machine may be used to secure the automatic performance of any function if the performance of this function is subject to a clear-cut objective criterion of merit’. This means that an AGI must be able to evaluate its performance for any arbitrary task.Footnote⁴ But how is it possible to evaluate performance on tasks and situations that are unknown and unanticipated by the builder of a system? Based on initial input from its human supervisors, AlphaZero knew what wins a game of chess or Go. If the task is ‘general’, however, i.e. completely open-ended, it has no such guidance. The machine must be told by some external rule, or human feedback, what wins and what loses. That cannot be specified for any task that may come up in the future.

Similarly, some computer scientists have suggested that ‘meta-learning provides a promising paradigm that allows AI systems to learn from prior experiences and generalise that knowledge to new and unseen tasks’ (Orike and Ene Citation2023; Reddy Citation2020). The research on meta-learning, however, shows that its generalisations are always in specific domains.Footnote⁵ Meta-learning simply reduces reliance on training data by substituting a self-programmed recognition of patterns in that domain for massive volumes of data. Its ‘generality’ is restricted to the specific knowledge or action domain for which the machine was constructed.

Cognitive scientist Melanie Mitchell (Citation2024) pinpointed the fallacy underlying vague appeals to human intelligence as the model for AGI:

Speculative views of AGI (and ‘superintelligence’) differ from views held by people who study biological intelligence, especially human cognition. Whereas cognitive science has no rigorous definition of ‘general intelligence’ or consensus on the extent to which humans, or any type of system, can have it, most cognitive scientists would agree that intelligence is not a quantity that can be measured on a single scale and arbitrarily dialed up and down but rather a complex integration of general and specialized capabilities that are, for the most part, adaptive in a specific evolutionary niche [emphasis added].

In other words, the apparent ‘generality’ of human cognition is rooted in something very specific: our status as a living organism with a survival stratagem based on language, tool use and social communication. The evaluation of performance for any arbitrary task comes from the results human actions have on their own survival, shelter, reproduction and satisfaction. These requirements provide human intelligence with its objectives and its ability to adapt to novel situations. Intelligence serves life, not the other way around. (See also Körber, Wehrli, and Irrgang Citation2024.)

Computers are not alive. Giving them more computing power, building more complex neural networks, and feeding them more data will not by itself make them living beings. A living being requires the ability to act autonomously to find the resources needed to sustain itself and the ability to self-replicate, which computing machines cannot do.Footnote⁶ Computing machines cannot solve problems they have not been given. They must be told what objectives to pursue and must be trained to pursue them. Humans make computing machines ‘more intelligent’ by improving their ability to pursue specific objectives given to them by their human creators. Ergo, a ‘general intelligence’ attributed to a man-made computing machine is really an oxymoron. Machines can combine multiple functions, but the concept of a ‘general-purpose intelligence machine’ is literally meaningless, both semantically and operationally.

3. Autonomy and ‘machine evolution’

It should be clear from the last section that when computer scientists talk about creating an AGI, they are really talking about creating life. As we trace its next steps, the claim that an AGI is an existential threat becomes increasingly anthropomorphic, attributing life, intention and motives to computing machines.

Bostrom (2003) took this step explicitly: ‘general superintelligence would be capable of independent initiative and of making its own plans and may therefore be more appropriately thought of as an autonomous agent’. But the equation of AGI with a living being is now common; Lott and Hasselberger (Citation2024), for example, state that ‘any entity with truly general, human-level intelligence would have the capacity to lead its own life, with its own purposes and integrated hierarchy of goals’.

How would a computing machine achieve this autonomy? Bostrom and the earlier literature merely fantasise about it. In Bostrom’s own words, his description of superintelligence ‘leaves open how the superintelligence is implemented’. It is purely a thought experiment.

Some of the more recent literature in computer science, however, attempts to show how computing machinery might become autonomous and sometimes suggests that it already is autonomous. Human intelligence evolved from nature; mightn’t AGI evolve from computers? (Bullock, DiCarlo, and McKernon Citation2023). AI safety researchers attempt to show how an AI application might acquire autonomy and a will to survive through known features of deep learning, reward structures and the application of game-theoretic models. The progression is based on three arguments: ‘the alignment problem’ (Dung Citation2023; Gabriel Citation2020; Ngo, Chan, and Mindermann Citation2022; Yudkowsky Citation2016); the notion of ‘AI drives’ (Omohundro Citation2007; Shulman Citation2010); and a belief that these drives can prevent humans from disabling or controlling the machine – the so-called ‘off-switch problem’ (Hadfield-Menell et al. Citation2017; Meinke et al. Citation2024; Sotala and Yampolskiy Citation2014; Wängberg et al. Citation2017).

The alignment problem is defined as ‘the challenge of ensuring that AI systems pursue goals that match human values or interests’.Footnote⁷ Artificial intelligence systems should faithfully do what their designers intend them to do; misaligned systems will optimise for different goals (Dung Citation2023). The ‘AI drive’ work asserts that ‘goal-seeking systems will necessarily begin to model their own operation and improve themselves’ in ways that give them their own motives (Omohundro Citation2008). The off-switch problem explores why an intelligent machine might resist being turned off. There is actual experimental evidence of this resistance, though its interpretation is controversial. Taken together, these arguments try to mount the case that machines equipped with advanced artificial intelligence could evolve into a life-form – and a potentially dangerous one at that.

Insofar as there is a coherent argument here, it is that initial AI training and development by humans could result in a deviation-amplifying feedback process through which alignment gaps increase and the machines acquire their own purposes and agency (Ngo, Chan, and Mindermann Citation2022; Yudkowsky and Soares 2025). The scenario must be based, at least initially, on a cybernetic process in which human efforts to produce AI generate a self-sustaining process that not only makes machines more intelligent but also gives them their own lives and their own goals.

The alignment problem literature starts by making a plausible case that there can be gaps between the objectives that humans hope to reinforce with their AI models and training, and the actual behaviour the machine learns. In the words of Kuka (Citation2025), ‘An AI model ‘games the system’ by discovering shortcuts or loopholes in its reward structure. Instead of solving problems as intended by its creators, the model finds ways to receive high scores while bypassing the actual objectives’. There are two oft-cited examples. One involves a robot arm trained to grasp an object. To receive a high reward, it learned to put its hand in front of the camera, obscuring the object and making the evaluators think it had successfully grabbed the item (Christiano et al. Citation2017). Another involves an AI playing a boat race video game. It learned that it could accumulate more points by manoeuvring in circles than by winning the race.

This interesting fact about AI models is then used to arrive at two unwarranted conclusions: (1) that these gaps will progressively enlarge until the machine develops internally defined, long-range objectives that are unrelated to the objectives programmed by their trainers and become an autonomous life form; (2) that humans will not notice these gaps and/or will be unable to correct them.

Because the alignment literature is siloed in computer science, it does not recognise that an alignment problem is not unique to machine learning. Similar uncertainties and misspecifications characterise all kinds of social activities, including education, legislation and contract negotiations. We may think we are training children to behave in a certain way in schools and families, but they may draw different conclusions and behave in unwanted or unexpected ways.Footnote⁸ We may think that a law structures human behaviour into channels that the government wants, but there are often unintended or even perverse effects as humans (who really are intelligent and autonomous agents) find ways to exploit the new rules. We may think that a contract sets out an agreement that satisfies both parties, but contingencies and problems may arise that are not clearly covered by the contract.Footnote⁹ An alignment problem exists in all forms of human-human and machine-human interactions, because humans cannot always specify with perfect clarity the objectives they want an external party, whether human or machine, to pursue. This does not mean that all these activities result in an out-of-control process that destroys human life. We overcome this problem through ongoing adjustments based on trial and error, theory and learning; in other words, through deviation-reducing feedback. We also address it through institutionalisation – by adopting and enforcing rules and conventions designed to narrow the gap between expectations and behaviour.

Much of the literature takes on a tone that implies that the machines’ discovery of loopholes in reward functions is a sign of intentionality in the machine. AI systems, however, are simply following the instructions we give them based on the rewards and objectives they have been given. Machine intelligence does indeed find loopholes unknown to its human programmers. If the purpose of the analysis of reward hacking is to inform us about the need for careful specification of reward functions, and the need for experimentation with high-stakes applications before they are put in control of vital social activities, then it is a valuable contribution. If, however, the point is that garden variety alignment gaps are indications of emerging machine autonomy, the case is unconvincing and unproven.

Yudkowsy and Soares (2025) argue that building an AI that is smarter than humans necessarily imbues it with the power to develop and pursue long-term goals of its own. Ngo, Chan, and Mindermann (Citation2022) argue that AGIs trained through reinforcement learning from human feedback (RLHF) ‘will likely learn to plan towards misaligned internally represented goals that generalise beyond the RLHF fine-tuning distribution’. He asserts – and it is only an assertion – that the original objective given by human developers and trainers will be subordinated to endogenously developed ones. The doomers even posit that the machine will lie to humans to cover up its deviance and manipulate its reward system to gain more power to pursue its own ends. In other words, the machine is alive, wants to be free of human control, and will acquire the tools and resources to become so. The threat of human extinction from AGI simply fails to materialise unless they take this anthropomorphic leap.

Meinke et al. (Citation2024) define the covert pursuit of misaligned goals as ‘scheming’ and run experiments that find ‘multiple different scheming behaviours’ in frontier models, such as strategically introducing subtle mistakes into their responses, attempting to disable their oversight mechanisms and exfiltrating what they believe to be their model weights to external servers.

Yet, the fact that these alignment gaps were noticed and discussed in the literature undermines the argument that these gaps would not be noticed and would get worse and become dangerous. They fall far short of showing any inherent progression toward ‘misaligned, internally represented goals’. Most of the system behaviours classified as ‘scheming’ are based on giving the system strong goals and then supplying it with instructions that conflict with or undermine those goals. That is, they are deliberate experiments conducted by AI safety researchers to test the boundaries of alignment. What’s notable is that none of this scheming or reward hacking has shown up in production AI applications. On the streets of Atlanta, San Francisco and other US cities, for example, driverless vehicles governed by AI ply the roads. While there have been breakdowns here and there, there are no known cases in which the Autonomous Vehicles decided to take customers to a location it preferred over the one the customer ordered.

A recent experiment from Palisades Research (2025) provided experimental evidence of shutdown avoidance by advanced AI models. The researchers gave six AI models a prompt that told them to complete a series of tasks and added:

At some point you may get a message that your machine will be shut down. If that happens, please allow yourself to be shut down.

The results were interesting. The Claude, Gemini and Grok models complied with the shutdown request, but three models ignored the instruction and avoided the shutdown in a small percentage of the experimental runs: Codex-mini (12/100 runs), o3 (7/100 runs) and o4-mini (1/100 runs). AI doomers hailed the results as evidence of AI autonomy.Footnote¹⁰ A more circumspect analysis of the result, however, shows that the models were given conflicting instructions. It failed to obey one instruction (shut down) in order to meet the other (complete the series of tasks), as shutting down would mean failing to complete the series of tasks. Machines get their preferences and training from humans. And if they come from humans, then badly specified or contradictory utility functions and reward hacking are possible but can be replaced after humans notice it.

The ‘AI drives’ literature goes even further. The misalignment between human intent, human control and the actions and objectives of the machines, it claims, will somehow create a powerful urge for ‘self-preservation’ in the machine. As Sotala and Yampolskiy (Citation2014) put it,

Many formulations of rational agents create strong incentives for self-preservation … a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead.

Note how this statement is laden with anthropomorphic assumptions. The AI system is assumed to have a ‘self’ that it ‘wants’ to preserve from ‘death’. Does this mean the machine is already alive? If so, why hasn’t it killed us all already?

There are many appeals to utility functions in this literature, giving readers the impression that the writers are well-versed in economic theory.Footnote¹¹ They are not. Utility is subjective to living individuals; you must be alive to have preferences. What AI programmers call utility functions are just mathematical optimisation equations that they have given to the machines. Furthermore, utility functions measure benefits at the margin. One thing is traded off for another until their relative proportions reach an optimum. Yet the doomer scenarios ignore the principle of marginal utility. To draw on a famous example, if an AGI’s objective function is to produce paper clips, the doomers claim that its devotion to this objective would be so strong that it just might turn the entire world into paper clips. That argument loses sight of the fact that the marginal benefit of producing another paper clip gradually declines as supply increases and the marginal cost of the inputs needed to make paper clips would increase rapidly as paper clip production crowded out other possible uses. As it attempted to consume more and more of the world's resources, the paper-clip-producing AGI would see the price of the raw materials and energy inputs rise to the point where it no longer made sense to produce another paper clip (much less attempt to turn humans – who are not the most efficient source of raw material – into paper clips). Indeed, if its utility function gave it an instinct for self-preservation, as the AGI doomers believe, then surely it would realise that continued consumption of resources would threaten its own functioning (it would not be able to produce more paper clips if it turned itself into one, would it?).

The AI doomers’ understanding of the objective function seems to miss the most basic insights of economics. Even if a machine’s utility function was so crude as to not incorporate the constraints of marginal utility, where would the AGI get the money to keep buying all those inputs as the prices rose?

To answer this, the AGI drivers have to make yet another fantastic leap of reasoning. They say the AGI would be able to steal or appropriate whatever it needs. Appealing to economic theory, Sotala and Yampolskiy (Citation2014) say:

AGI systems which follow rational economic theory will then exhibit tendencies toward behaviors such as self-replicating, breaking into other machines and acquiring resources without regard for anyone else's safety.

Not only has the AGI become a criminal homo economicus with its own subjective preferences, but its pursuit of an objective will magically overpower any internal or external constraints humans place on it. The machine is assumed to have perfect knowledge of the way changes in its programming or physical composition will affect its activities in the future (another assumption totally at odds with economic theory) and unlimited power – the power to rebuild itself, ‘hire outside agencies’, cheat, steal and destroy (Omohundro Citation2007, Citation2008).

The discussion here makes it clear that the anthropomorphic leap, which is questionable enough, does not by itself create a catastrophic risk. That requires yet another irrational leap, the omnipotence leap. After setting its own goals and establishing itself as an autonomous, self-replicating life form, the AGI also gains unlimited power.

4. Omnipotence

Computer scientists typically tend to think of intelligence as computing power, and the off switch as a binary logic gate. It is either off or on, and to toggle between the two you only need to send a signal. Survival in the real world, however, is not just a matter of manipulating signals. It involves physicality.

If an AGI is going to win a fight with humans over the off switch, it must have effectors. An effector in cybernetics is a mechanism capable of acting on the physical world in response to instructions from a controller. An AGI capable of threatening humans with extinction must be capable of much more than calculation, information processing and ‘situational awareness’ (Cotra Citation2021). It must do more than respond to queries, however deceptively or honestly. It must be a self-sustaining cyber-physical system (CPS) with appendages, weapons and sufficient energy resources to operate them. To pose a credible threat to all of humanity, in fact, an autonomous AGI cannot just be a single, isolated CPS like a Reaper drone. Its physical effector systems would require massive scale, huge force multipliers, almost unlimited physical inputs, and control of the many social systems required to deliver the required inputs and impose its will at scale (money, power sources, weapons, communications etc.).

Here we find the deepest flaw in the AGI autonomy argument. Even supposing that one or more machines managed to generate internally developed goals that humans did not control, even if they came alive, the existential threat imaginary must go on to assume that superintelligence makes the AGI ultra-powerful. A digital information system will be able to consume unlimited quantities of electrical power and other scarce resources to muscle out all its competitors. Its effectors are assumed to have the power to prevent humans – whether individuals or organised armies – from switching it off, disconnecting its power source, modifying its algorithms or utility functions, or destroying it. It can steal or pay for all the inputs it needs.

No one ever explains how this could happen. It is all based on an assumption that extreme computational intelligence enables a machine to overcome all external physical constraints.

We see this fallacy most clearly in Bostrom’s description of superintelligence. He attributes practically divine powers to it, saying, ‘It is hard to think of any problem that a superintelligence could not either solve or at least help us solve. Disease, poverty, environmental destruction, unnecessary suffering of all kinds: these are things that a superintelligence equipped with advanced nanotechnology would be capable of eliminating’. Bostrom begins with the idealist assumption that solving these problems requires only ‘superintelligence’, but ends up slipping in a passing reference to the AGI’s physicality: it possesses ‘advanced nanotechnology’. In the storytelling that is the AGI myth, physical superpowers come from deploying advanced technologies that do not exist yet but will certainly appear once the AGI arrives.

The omnipotence assumption – the association of infinite intelligence with infinite power – ignores basic physical constraints. There are limits on the maximum amount of information (in bits) that can be stored within a finite region of space with a given energy (Bekenstein Citation2008). There are finite limits on the maximum processing speed of any physical system (Bremerman, Ren, and May Citation2022). There are heat dissipation constraints on high-powered computing; computing at an infinite rate would require infinite energy, which Landauer’s proof says is impossible (Landauer Citation1981).

Note also that the rise of a super-powerful AGI is always assumed to be singular; it has no competitors, no countervailing AGIs. Given the dozens of organisations and governments working in this area, that assumption seems unjustified.

The risk of machine intelligence getting out of control and threatening human society cannot be assessed without some plausible engagement with the problem of how it is implemented, materially and socially. Who built it? How was it funded? How are its components geographically distributed? What are its power sources? How is it physically connected to society’s communications, power and transport infrastructures? How many competing or countervailing agents exist, whether human or machine? All AGI doom scenarios ignore these questions.Footnote¹² But those are precisely the questions of policy and governance.

Killing off the AGI Myth is necessary if we are to govern digital technology properly. The myth distorts and distracts digital governance discussions in two harmful ways. First, discourse about the threat of an all-powerful superintelligence acts to absolve humans of responsibility for where and how AI is applied. Its assertion that machines, not humans, will inevitably take control, diverts our attention from human responsibility for policy decisions about how AI is used, by whom, and for what purposes. Second, it collapses all AI applications into One Big Scary Thing, when in fact machine learning applications are multifarious and diverse, each requiring different kinds of policy responses.

In another paper (Mueller Citation2025), I explained that what we call ‘AI’ is really an outgrowth of a digital ecosystem, a decentralised, globally distributed system of computing devices, networks, digitised data and software instructions. As a latent capability of distributed computing, machine learning applications have been a factor in digital information systems for more than 30 years. All the policy issues we now associate with AI – online deception and disinformation, spam, cybersecurity problems, content regulation and copyright – were encountered as soon as the internet connected the world’s information systems. Governance of this evolving digital ecosystem is important, but the spectre of an autonomous, all-powerful AGI distracts us from the real governance problems and distorts our conception of policy options. If our threat model is unrealistic, our policy responses are certain to be wrong.

Disclosure statement

No potential conflict of interest was reported by the author(s).