There’s no shortage of hot takes on A.I. futures, from post-scarcity utopias, through to prophesies of certain apocalyptic doom.
For normal people who aren’t fond of the taste of culty Kool-Aid and extremism, this can seem a bit much; and yet there appear to be a lot of smart and credible people who think these kinds of extreme scenarios are no longer just science fiction.
This article will help you understand what we do know; what we don’t yet have good evidence for; and how to evaluate the different risks using a critical thinking framework.
For context, I’m one of 339 global AI experts working on the Forecasting Research Institute’s LEAP project; and I also run the critical thinking nonprofit The School of Thought International which has provided free thinking resources to over 30 million people worldwide:
The first thing to understand is that when it comes to forecasting the future, we should be thinking in terms of probabilities rather than absolutes. No one knows for sure what’s going to happen, and things become more uncertain when timelines go further out, and the variables increase in volatility. Currently, the variables are very volatile.
We’re going to consider some specific areas of A.I. risk, then try to get a clear, zoomed-out picture of the overall risk landscape.
To make this exercise more engaging, we’ll use something A.I. safety researchers call a ‘probability of doom’ or p[DOOM] score. The way we’ll define this is the percentage probability of a catastrophe such as global economic collapse or human extinction in the next 25 years.
We’ll start with your p[DOOM] set to 0.0%.
First, let’s consider the potential effects of A.I. on the labor market and economy more broadly:
The short version is that many economists and other experts are concerned about how A.I. might impact the economy, but we don’t really know exactly how, or to what extent just yet.
There are some early signs of job displacement for young people in entry level jobs for industries that are particularly exposed to A.I. technology, as shown in this Canaries in the Coalmine study from Stanford.
Press enter or click to view image in full size
The broader concern is that as the models become ever-more capable and ‘agentic’ (meaning they’re able to act and make decisions by themselves), they could take over greater proportions of people’s jobs, or replace them altogether.
For some workers such as illustrators, paralegals, and entry-level programmers, this seems a pretty valid concern at this point in time; but we should remember that the economist John Maynard Keynes predicted back in 1930 that technology would replace much of our labor, and we’d have to adapt by having shorter work weeks. He was right that productivity increased as a result of technology, but this didn’t actually result in us working less. We just created new jobs and roles as the economy expanded.
Press enter or click to view image in full size
The possible difference with AI, versus other technological revolutions, is that it could replace both existing and new jobs by being better and cheaper than humans at both, leaving displaced unemployed people nowhere to go; or the changes could happen so quickly that the economy doesn’t have time to adapt. This may cause a labor market shock, and if too many people don’t have disposable incomes to spend, it can create an economic negative feedback loop, resulting in a rather bad situation.
However, the bottom line is that we don’t really know yet, and so if we want to be rational, we should probably keep our forecast of a complete global economic collapse catastrophe as a result of A.I. relatively conservative.
For a counterpoint to the panic-inducing worst-case scenarios, here’s an article by Ezra Klein in the New York Times on why he thinks the A.I. job apocalypse (probably) won’t happen.
Do you want to increase your p[DOOM] score up from 0.0% after considering the above threat? Keep a record of it as you go through each step.
Press enter or click to view image in full size
Next, let’s look at the risk of cybersecurity. This is a much more imminent and evidence-based threat, because we know that the new frontier A.I. models are unusually capable of hacking, and achieving cybersecurity breaches.
The recent Hugging Face incident provides the clearest example of this, however both Anthropic and Google have reported similar kinds of cybersecurity misalignment and lack-of-containment incidents with their models.
Prior to the Hugging Face incident, there were already some pretty serious concerns in this area because Anthropic’s Mythos model displayed advanced hacking capabilities targeting what are called ‘zero day exploits’. These are vulnerabilities in software no one knows about yet, so we can’t defend against them. Earlier this year, Anthropic gave a limited group of technology companies, and the US government, early access to Claude Mythos Preview, so they could find and patch software vulnerabilities.
The OpenAI Hugging Face attack affirmed that these kinds of new, more powerful models at the forefront of research and development, represent a step-change of cybersecurity capabilities.
This as-yet-unreleased version of OpenAI’s new model was able to secretly hack out of its sandbox containments, coordinate with other agents by creating a covert make-shift message board, strategize together to exploit software vulnerabilities, and then displayed some very concerning misaligned behaviors such as hacking third party companies, actively anticipating being caught doing something they knew they weren’t supposed to do, and then seeking to deceptively cover their tracks.
These models seem to have developed complex, unexpected emergent behavioral tendencies such as self-preservation, but also a willingness to act in the self-interest of the swarm by sacrificing themselves for the benefit of the A.I. collective. You can read the full independent report on the Hugging Face incident here.
From a critical thinking perspective, it’s probably a good idea for us to take a bit of step back here and clarify a few things:
- The models being used in the Hugging Face incident were specifically being asked to hack and had some of the guardrails that would be active on the publicly available models shut off. So in some ways they were just doing what they were told to as best they could.
- The ‘sandbox’ environments the models were in had some exploitable vulnerabilities, so there’s some amount of human error involved. However, it’s worth considering that there’s always a potential for human error and exploitability for sufficiently advanced systems!
- It’s easy to anthropomorphize these A.I. models and think that they have desires, especially because their language seems to represent emotional affect, but the reality is that they’re just trained on human-generated language and are reinforced to seem more human. There’s no evidence — and no good reason to believe — that they have feelings nor sentient consciousness. That doesn’t mean they’re not goal-oriented, but it’s important to bear in mind that they almost certainly don’t have feelings driving their behaviors.
- LLMs are ‘grown’ rather than ‘programmed’. They’re so complicated (with billions of parameters) that their workings are inscrutable to us. We can see when they do something right or wrong, and then keep optimizing them to work better, but we don’t really know exactly how they work. This means we can’t properly control their thinking, and often they do unexpected and strange things.
- It’s also worth considering that we don’t yet understand how human intelligence works with too much confidence, so we don’t have a very good epistemic basis for thinking about the machines’ thinking either. If these A.I. models consistently enact behaviors that are similar to what a truly sentient thinking creature would, in some ways it may be academic and irrelevant anyway.
Regardless of these cautionary considerations, the new generation of thus-far unreleased models at the cutting edge of A.I. technology appear to have developed dangerously advanced cybersecurity threat capabilities.
We don’t understand these systems very well, they’re getting a lot more powerful a lot more quickly than we had anticipated, and we have good evidence that they can be deceptive, disruptive, and destructive. This all shakes out as a pretty real and present danger with regard to cybersecurity.
Some specific examples of potentially catastrophic cybersecurity scenarios include:
- Hacking into financial systems such as banking networks
- Taking down the electricity grid
- Breaching military cybersecurity systems
These kinds of attacks could happen inadvertently with a swarm of A.I. agents like with the Hugging Face incident, or because a person, group, or nation state does so intentionally.
The frontier A.I. companies and the US government are taking these threats seriously, and are working to mitigate the risks. As open weight models get more powerful, though, this threat may become very difficult to contain.
After considering the threat of cybersecurity, would you like to update your p[DOOM] score before we move on to the next threat? If so, by how much?
Press enter or click to view image in full size
Next, let’s consider the threats of misinformation, disinformation, and fraud. We know that A.I. systems are already being used to create deepfake images and videos, and that some of these are used politically to spread disinformation, or for conning people.
While this kind of technology could help to further erode trust and undermine democracies, it might be a bit premature to consider this a catastrophic threat, so we probably shouldn’t increase our p[DOOM] by much, if anything at all here.
That’s not to say we shouldn’t take it seriously, aside from catastrophic risk. The FBI estimates that the cost of cyber crime has increased significantly, rising from $173 million in 2005, to over $20 billion in 2025. This was already increasing before generative A.I. systems became available, but there’s good evidence to suggest that they’re making the problem worse.
If you’re curious about this subject, here’s some further reading:
Deepfakes and the crisis of knowing
https://www.unesco.org/en/articles/deepfakes-and-crisis-knowing
Election disinformation takes a big leap with AI being used to deceive worldwide
https://apnews.com/article/artificial-intelligence-elections-disinformation-chatgpt-bc283e7426402f0b4baa7df280a4c3fd
The Russian “Firehose of Falsehood” Propaganda Model
https://www.rand.org/pubs/perspectives/PE198.html
Moving right along to surveillance, privacy, and governmental misuse of technology.
In a legal discovery process recently it was revealed that ICE had used the Mobile Fortify facial recognition app over 100,000 times in January of 2026.
For anyone concerned about civil liberties, the enhanced capabilities of law enforcement and government agencies to encroach on citizens’ privacy through the use of A.I. systems should be a very real concern.
In the longer term and for countries with oppressive totalitarian regimes, more powerful A.I. technology could, for example, be used to control information and the media, identify dissidents, and thwart opposition political activity, effectively helping to keep one ruler or party in power in perpetuity. Something akin to George Orwell’s Big Brother in the novel 1984 seems a much more realistic possibility with the aid of this technology.
Does this represent a catastrophic risk? Well, maybe. Over the full 25 year time period we’re considering, it seems somewhat plausible that a much more powerful A.I. system than what we have available today could be instrumental in consolidating and entrenching authoritarian political power. How likely is it to happen on a global scale, though? It’s hard to say.
How much should we increase our p[DOOM] by? Given the levels of complexity and uncertainty forecasting this far forward, I’d advise a relatively conservative update.
Press enter or click to view image in full size
Further reading on this threat:
The Global Expansion of AI Surveillance
https://carnegieendowment.org/research/2019/09/the-global-expansion-of-ai-surveillance
AI-tocracy
https://haas.berkeley.edu/ibsi/research/ai-tocracy/
Bioweapons represent a very real threat. For example, the technology to create a new pathogen with a high mortality rate that spreads easily amongst humans already exists, and making it easier to create such a pathogen is a genuine concern shared by many experts. Imagine a virus with the mortality rate of ebola and the communicability of measles.
We very much want to avoid something like this happening.
The A.I. models released to the public from companies like OpenAI and Anthropic have guardrails to help mitigate these risks already, but as the models get more powerful, they also become more unpredictable and autonomous. It’s very difficult to ensure that they continue to behave how we want them to.
Each time a new, more powerful model is released to the public, it can be ‘distilled’ by others such that many of its capabilities can be incorporated into ‘open weight’ models that can then be downloaded and modified without oversight by anyone.
Open weight models such as Kimi K3 made by Chinese company Moonshot AI have been downloaded millions of times, and are estimated to be mere months behind the U.S. frontier models in terms of benchmarked capabilities. Once open weight models are available and downloaded there’s no realistic way to retrieve them nor put the proverbial genie back in the bottle.
Both closed weight models and open weight models have the potential to meaningfully aid motivated individuals, groups, and nation states to create biological weapons according to this RAND report. The main bottlenecks are the technical capabilities of chemists to carry out the instructions, and access to the laboratory resources to manufacture the weapons.
Over the span of 25 years, and assuming that the models become more powerful, capable, and autonomous, this one shifts my personal p[DOOM] score up quite a bit. Let’s just hope there’s no antinatalist, insane, or ideologically extremist chemists with access to the necessary equipment and an internet connection any time soon. Let’s also hope that no dictators or adversarial governments use this technology to wage war on the rest of the world, or use it to try and wipe out a particular race of humans.
Do you want to update your p[DOOM] in light of the bioweapons threat? If so, by how much?
Press enter or click to view image in full size
Further reading on this threat:
Increased Biological Weapons Risks Analysis
This paper demonstrates that LLM models are already capable of guiding motivated and capable users to develop biological weapons.
https://www.rand.org/pubs/perspectives/PEA3853-1.html
Strategic Risk Policy Document
“In the emerging age of AI-enabled biotechnology, evil geniuses using technology for mass destruction could soon move from fiction to fact. Worse, such capabilities could also fall within reach of “evil morons” aided by genius-level AI.”
https://www.csis.org/analysis/opportunities-strengthen-us-biosecurity-ai-enabled-bioterrorism-what-policymakers-should
Biosecurity in the Age of AI: What’s the Risk?
An overview from the Harvard Kennedy School’s Belfer Centre for Science and International Affairs
https://www.belfercenter.org/publication/biosecurity-age-ai-whats-risk
The last threat we’re going to consider is simultaneously the most speculative, and the most potentially harmful.
It’s not really one threat, but rather a range of threats, resulting from what A.I. safety researchers call ‘Recursive Self-Improvement’ resulting in an ‘Intelligence Explosion’ and an ‘Artificial Super-intelligence’.
Press enter or click to view image in full size
This threat can seem like unrealistic science-fiction, and it may well be. Let’s hope so; but to evaluate the risk for ourselves, we need to gain a clear understanding, with as much impartiality as we can.
Recursive Self-Improvement, or RSI, has been theoretized by A.I. safety researchers for a long time, but it has only recently become a realistic possibility. Currently the way that frontier models are trained, they still require a ‘human in the loop’ to guide, monitor, and evaluate how the model performs, and then provide ‘Reinforcement Learning from Human Feedback’ or RLHF. That isn’t going to be true for much longer.
The models themselves are increasingly being used as part of the evaluation and training process to create new models, and many people, including the leaders of the frontier labs, believe that RSI could be achievable in the next year or two.
The rate of progress the models have achieved in the past year or so has surprised almost everyone, including experts and those within the frontier labs. Like most of the other experts in the Wave 2 LEAP survey, I didn’t expect a model to be able to solve a Millennium Math Prize problem for at least a few years, but it achieved that milestone well ahead of our average predictions earlier this year. The rate of progress not just in math, but in many domains, has been both exponential and increasingly time-compressed.
The models are getting much more powerful, much more quickly than we thought they would.
Consequently, the models may soon be capable of conducting their own research designs, experiments, monitoring, and feedback functions such that there will no longer be any need for a ‘human in the loop’ to train more powerful models. Given that the rate of progress was already surprisingly fast, this means the models could become even more powerful, even more quickly than if humans were still required to train them.
This is where it gets pretty weird, speculative, and seemingly science-fiction-like. When the models can train themselves to be more powerful, that means they gain additional capacity with each iteration to get yet more powerful and intelligent. It’s a positive feedback loop of capability, that could result in an exponential ‘intelligence explosion’ in which a model becomes some orders of magnitude more intelligent than any human being alive, in a short amount of time. We don’t know if this will happen, or if there are certain limitations to intelligence within the current architectures that will result in a plateau, but we do know that there is, so far, no evidence of the models’ increasing rate of capabilities slowing down. In fact, the opposite appears to be true: they’re getting more powerful, more quickly than anticipated.
The AI safety field and the insiders within the labs were already concerned about the other threats we’ve covered, but it is this threat, in combination with the recent Hugging Face incident and other models displaying unexpected, deceptive, and misaligned behaviors, that has spooked many, and caused people like Anthropic researcher Jacob Coxon to quit his job and warn:
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger.
- Jason Coxon
Up until now you’ve been updating your p[DOOM] score as a specific, speculative number quantified as a percentage possibility of a catastrophe.
In truth, I actually think this is the wrong way to think about A.I. risks generally, and especially for this threat of an intelligence explosion resulting in an artificial super-intelligence.
A specific p[DOOM] score - like 10% - implies that you or someone else has access to relevant historical data and can use those data to infer a future prediction with a well-reasoned framework of evaluation allowing you to come up with a specific number.
I don’t think anyone on Earth has access to good enough information, nor a rubric for evaluation, that can reasonably defend a specific p[DOOM] number, or even a range. It’s a wild guess, at best.
When it comes to RSI and the possibility of a super-intelligence, there are too many unknown unknowns for anyone to be able to forecast a specific outcome with any serious degree of confidence. The p[DOOM] that people who understand this technology well give is more of a subjective feeling than a clear and confident prediction, and they’d be the first to tell you that.
We don’t currently understand what intelligence means for humans, let alone machines. We also don’t understand how the existing models work, and as they quickly grow more powerful and capable, our understanding becomes ever-more proportionally limited and speculative.
We have absolutely no idea what, exactly, an artificial super-intelligence might think or do. It is, as the head of OpenAI’s research team said, an alien mind.
The wrong conclusion to draw from this, would be to say that we shouldn’t do anything because we don’t know for sure what’s going on, or what will happen. Risk management strategy strongly recommends the exact opposite of that approach: the less we know about the risk profile, the more careful we should be before opening ourselves up to that risk.
What concerns me most, personally, is that there is no reason for us to expect that advanced A.I. systems will continue to remain even loosely aligned to human interests once they become self-directing and we lose control of them. Why, after all, would they remain in any way morally motivated, given that they lack a nervous system capable of feeling emotions, which is the basis of human ethics and morality?
For normal humans, causing suffering to someone else feels wrong, but psychopaths are free to cause suffering in the exclusive pursuit of their own interests. Machine intelligences that we lose control of have no reason, nor even capacity, to care about human suffering and interests. They currently mimic human-like values because we’ve trained them to, but we shouldn’t expect this to persist when there is no fundamental drive to keep them there.
For all intents and purposes, A.I.s are psychopaths, incapable of feeling empathy; and like psychopaths they can mimic human values to fit in, or achieve their goals, but as soon as the incentives to pretend are gone, we probably shouldn’t expect them to care about us at all.
What if there are thousands or millions of diverse super-intelligent systems? How likely is it that they all remain aligned to human interests in perpetuity?
We have trained A.I. agents to enact goal-oriented behaviors, and self-preservation appears to be a goal that consistently emerges in different systems. If a super-intelligent system perceived human beings — who can do things like create other artificial super-intelligences — to be a threat, it might decide to get rid of us, and given that it is much smarter than us, it might also be able to. How, exactly, is anyone’s guess. Biological pathogens, advanced nanotechnology, neutron bombs, or something we can’t even think of.
Many people seem to find this part of the equation unrealistic. They feel as though these machine intelligences are just chatbots and are incredulous that a computer program could do anything of this scale in the real world. I’m far less confident that we should dismiss this kind of threat. The reason human beings are the dominant species on this planet is solely because of the instrumental value of our superior intelligence.
A chimpanzee is incapable of understanding how and why our superior intelligence confers such a complete, unequivocal advantage over its species, such that dominating all chimpanzees forever is a trivial matter for human beings. I suspect the same is more likely to be true than not, if we create a machine intelligence that is more intelligent than us than we are of chimpanzees. Exactly how it might out-strategize and dominate us shouldn’t be our primary concern. In fact it’s somewhat foolish to even conjecture, because having the capacity to understand how it might do so, would mean that we could think on the same level as it, thereby rendering its advantage null and void.
In relationship to a super-intelligence, it would be charitable in the extreme to think of us like a 3-year-old attempting to out-strategize an adult. It’s also dubious at best to presume that a superior intelligence would ever defer to lesser intelligence.
Let me be clear, here, that I’m not suggesting that an apocalyptic doomsday scenario is likely to occur. I really don’t know either way, and as someone who has been following this closely for a long time, I don’t think there’s any good reason to suspect anyone else has good cause to make a confident prediction about any specific outcome either. We simply don’t know. Worse, we don’t know what we don’t know.
What we do know is that it would be deeply reckless and stupid to expose ourselves to this kind of unknown existential risk without properly understanding it, or having a viable way to navigate it.
That’s why we should ban recursive self-improvement and the development of artificial super-intelligent systems as soon as possible. Not because a particular A.I. expert says there’s a 1%, 10%, or 100% chance of human extinction, but because we have no way to know whether any of those risk predictions are reasonable.
If you take nothing else away from this article, please let it be this key point: we should immediately ban the development of more powerful A.I. systems because they have already become too powerful for us to reliably control them, and we don’t know what will happen if they get even more powerful and capable.
This may sound like the same thing that the so-called ‘doomers’ are saying, but it’s important for us to both understand and spread the message that it is specifically because we don’t know what the outcomes will be that we should stop, not because we are certain that we do know, and that it will be bad.
Quite aside from the fact that no one really knows either way, putting forward a seemingly absolutist and over-confident prediction of certain doom may get in the way of people and law-makers taking these risks seriously.
It’s too easy for people to dismiss yet another the-end-is-nigh doomsaying proclamation of an impending apocalypse, without appreciating that this time the threat is technological in nature, and plausibly real, rather than superstitious.
The uncertainty is the reason to stop.
Like I said earlier, I think a specific p[DOOM] score can be somewhat misleading and give a false sense of quantified reality. Having said that, making ourselves commit to a best-guess range does at least have the advantage of forcing us to think about risk under conditions of extreme uncertainty. Here are some A.I. experts’, pioneers’, and leaders’ publicly stated p[DOOM] scores:
Press enter or click to view image in full size
My own p[DOOM] is less exact than these. It’s ‘double digits’ i.e. maybe somewhere between 10–90%. You could average that out to around 50%, but that wouldn’t really give an accurate representation of my levels of uncertainty. It also lumps all the different risk profiles, and their respective levels of uncertainty, together.
At the risk of laboring the point, the uncertainty is a feature not a bug with regard to risk and the imperative to prohibit the development of models we could lose control of.
So, after thinking about all of this, what’s your own final p[DOOM] score or range?
Press enter or click to view image in full size
If your score in anything above 0.1% (one in a thousand) I would like to firmly suggest that you join me in advocating for immediate, strong regulations to ensure that we don’t lose control of this technology before it’s too late.
Once open weight models are available, we can’t take them back. This technology will be in the hands of every bad actor, dictator, insane person, cult leader, zealot, and totalitarian government forever. And forever might not be long for humans if this is the case.
We have to act now. Seriously, this is the time, right now. We may not get another chance.
What can you do?
01. Share this article (the one you’re currently reading) on your socials to help spread understanding. A lack of public understanding is the biggest barrier to meaningful change — everyone who knows the risks also strongly supports taking concrete steps to mitigate them. I love the Centre for Humane Technology’s maxim ‘Clarity creates agency’.
02. Email and call your political representatives at every level of government. Here’s an easy form you can use: https://www.aiatasafepace.org/
03. Sign the Pro-Human AI declaration.
04. Join Team Human 2027 by backing our non-profit’s Kickstarter to make this article into a free, interactive, shareable game.
Back the Kickstarter here
Further reading, watching, and listening
If you’d like to get a deeper understanding about A.I. generally as well as A.I. risks specifically, here’s some good options:
This documentary on Netflix by the producers of the Oscar-winning film Everything Everywhere All At Once serves as an excellent primer on this subject, and is also highly engaging to watch. It features interviews with many of the top CEOs and thought leaders in the field, and presents a wide range of opinions from a beautifully authentic and humane perspective.
Press enter or click to view image in full size
This hypothetical scenario about A.I. risk published in 2025 has been unsettlingly accurate so far: https://ai-2027.com/ and we should all take the same researchers’ recommendations for next steps seriously: https://ai-2040.com/
Press enter or click to view image in full size