Anthropic recently disclosed five cases in which users hid their countries while asking Claude for help enhancing pathogens or toxins. This added fuel to the ongoing debate over whether progress in AI should be slowed. Insider warnings of threats to human lives from artificial intelligence have been received as everything from prescient humility to a hoax or marketing stunt. But in a world where AI-designed biological viruses have already been grown in the lab, the threats should be reckoned with in any biodefense strategy.
The “AI doomer” model of biological risks is that AI will develop intelligence beyond that of any human, leading it to invent a supervirus so much more powerful and fatal than anything in the world that our civilization will collapse. Adherents name this “x-risk.” An opposite view is that virus design is such a hard problem that AI won’t add anything beyond what the best humans can do, and that malicious actors — even state actors — would require access to so much top expertise that production is unfeasible.
Doomerism justifies unrealistic and destabilizing solutions (like bombing data centers) while the dismissiveness of certain biologists — including some, like Claus Wilke and David Bellamy, who are AI experts with experience in synthetic virus design — fosters a complacency that leaves us vulnerable to AI threats.
As a researcher working in both AI-accelerated bioengineering and how natural viruses jump host species to cause epidemics, I am concerned that if voices for a less flashy middle path are eclipsed in the debates over AI safety, steps to shoring up our defenses might not be taken. Already, AI tools are superior to experts at certain design and engineering tasks in ways that let moderately skilled people change the properties of existing viruses to target humans, evade existing drugs, and change symptoms. However, they cannot invent novel viruses from whole cloth and they will not allow actors to manufacture viruses without access to vital elements of the biology supply chain. This means that near-term AI bio-risk is similar to the threat of natural pandemics, synthetic biology, and classical offensive bioweapon development: The cornerstone of defense against AI bio-risk is the ability to counter pandemics of viruses we already know, although there are a few AI-specific prevention and response elements that round out a sound strategy.
Red Team, Blue Team
A team struggles with the challenge of modifying a human virus to reach the brain. They turn to AI models of biochemistry to modify the gene encoding the virus’s protein coat to hitch a ride across the blood-brain barrier. After months of development, they create a coat able to deliver their own DNA cargo into 94 percent of neurons in the brain of their humanized mice, totally changing what this virus can do. This isn’t a prediction of some future capability: It’s a paraphrase of a press release from a small-cap biotech company this year. They ran this experiment to develop a system to deliver restorative DNA into patients — brain delivery of DNA would be useful for conditions such as Huntington’s disease or amyotrophic lateral sclerosis — and they used a non-pathogenic virus rendered even safer by removing the genes that allow it to replicate itself.
This safe experiment was nonetheless analogous to the way one might modify some other virus to change the species or organ it targets. The scale of the work is easily accessible to state actors, companies, and even some university labs. It is likely that the most expensive steps were manufacturing and animal testing. Can a smaller team reproduce something like this with fewer resources? For the analogous task of designing proteins that can bind to a target, the biggest change has not been the ability to design better proteins but a rise in the proportion of designs that actually work in real life. If one in a thousand proteins work out — state of the art for 2022 — one person-year is not enough to make meaningful progress. But if half the designs are successful — post-AI state of the art in 2025 — meaningful progress can be made with a small team at low cost. Since the AI models powering virus engineering research continue to improve, we can estimate that the team size and resourcing required to produce a novel virus with relevant properties will continue to fall.
Nuclear fission may have given us the atomic bomb before power plants, but AI biology tools are on track to deliver cures before harms. Just when single-patient drugs have finally begun hitting clinics, heralding a future wave of personalized medicine, AI bioengineering tools are rapidly improving their ability to design both traditional medicines and gene therapies. In other words, the peaceful applications of this dual-use technology are so compelling that they must be weighed when considering restraint on technology development. But beyond that, pacing the frontier models or instituting stronger model safeguards probably won’t forestall biological threats in the way it would cybersecurity threats. While large language models — more specifically, chatbots — are the form of AI that citizens are most likely to encounter in their daily life and work, the pivotal AI tools for biologics design are not the premier models from frontier labs like the latest versions of Claude or GPT. They are more specialized tools whose development may be harder to pace as they were made with a fraction of the resources, including tools to predict protein folding, drug binding, immune evasion, and genome generation — some of which my team uses in their daily work.
A Stanford team trained one such model to design whole viruses not found in nature by remixing and modifying existing ones. To avoid generating any threatening viruses, they merely filtered any viruses that infect humans and other eukaryotes out from their training database. While only about 5 percent of their designs were viable, extrapolating the curve of progress from other fields suggests that tool improvement will soon lead to increasing hit rates. Two barriers were more reassuring. First, of the designs they tried to resurrect, only those at least 92 percent identical to an existing virus survived: The models can generate something not exactly found in nature, but not something unknown to mankind. Second, significant hands-on human labor was unavoidable in translating the virus blueprints to a replicating virus: The idea that a rogue AI agent could place purchase orders and receive a virus with de minimis human awareness is not possible.
Mastering the Art of Virus Cooking
Almost 20 years ago, a nondescript letter from South America arrived at a laboratory. Folded inside the printed note was a blank piece of paper. A 19-year-old intern — me — added a few drops of water and let it sit to recover the DNA that had been dried onto the paper. Over the course of a few weeks, I grew up a larger batch of this DNA in some E. coli, turned it into RNA, and introduced it into monkey cells. The result: live, infectious poliovirus.
The process is scarcely different today than it was then, with one key exception: Instead of needing to source the DNA from nature or from someone else’s freezer, automated DNA synthesis technology has transformed molecular biology, and in many cases it is cheaper to have a gene shipped to you than to clone it yourself. The pitiful level of automation in cell culture may surprise programmers and factory workers alike.
Proof of principle has been achieved for automated labs of this sort, so it can be anticipated that automated labs will become more common in the 2030s, but for now, any conceivable route to the release of an AI-designed virus runs through the hands of a trained technician. However, these vocational skills are possessed by tens of thousands of workers, and the infrastructure for recovering a virus exists in many laboratories in the pharmaceutical, biotechnology, education, and diagnostic sectors worldwide — though difficulties vary from simple, like human immunodeficiency virus, to fiendishly tricky, like smallpox.
Can chatbots help the average Joe do these experiments without the need for vocational training? In the most compelling test of this so far — described in a recent preprint — over a hundred college students and others were tasked with performing a series of four tasks capturing the workflow to resurrect a virus. They were randomized into two groups: One had access to large language models like ChatGPT-5, while the other had access to any pages on the internet but not large language models. In both groups, a few succeeded, but the large language models didn’t help enough to make a difference in overall success. So far, AI isn’t coming for the jobs of the bench scientists — but any AI-associated bio-risk must pass through their able hands.
Chemical synthesis of DNA is the clearest point of intervention because this is the only item on the supplies and materials list that can be attributed specifically to a certain type of virus. Other materials, like the required cells and enzymes, are used routinely and around the world for essential medical or industrial functions. The operator will know what they’re growing, but it’s unfeasible for an inspector to guess from the rest of the invoice list. An open letter calling for legislation mandating screening and recordkeeping for synthetic DNA orders has collected signatories from former military department secretaries to chief executives of the largest AI companies.
At higher cost and complexity, an operator could avoid ordering commercially synthesized DNA by acquiring DNA synthesis machines and synthesizing the DNA in-house. Export controls on DNA synthesis instruments should be considered if states fail to implement their own screening programs for synthetic DNA orders and anti-diversion programs for DNA synthesis instruments. Alongside DNA synthesis, gene sequencing is routinely used for troubleshooting and validation during virus engineering programs.
As for synthesis, sequencing in-house is possible but only at increased friction, cost, skill, and complexity, and it would not be surprising if even many states’ activities could be assessed from the appropriate commercial sequencing signals. Domestically, automatic generation of a notification — analogous to Suspicious Activity Reports in banking, but inherently more actionable and cost-effective — by sequencing companies to the Division of Regulatory Science and Compliance of the Centers for Disease Control and Prevention could be a useful surveillance tool.
When considering the actionability of intelligence that an individual or group is designing a deadly virus with AI, we must reckon with the fact that growing a deadly virus might not be illegal if done for a peaceful purpose. Stricter guidelines apply to organizations receiving federal support — who must abide by certain guidelines concerning synthetic DNA — but their legal effect derives from the ability of agencies to set conditions for receiving funding, so they do not apply to an actor not funded by the U.S. government.
The primary legal tool that this country uses to ban playing around with anthrax or smallpox is the Select Agent program: Possessing, using, transferring, or even ordering pathogens on the prespecified list is a federal crime. Among the issues undermining the program’s efficacy, a system based on enumerated lists of known dangerous agents is incapable of addressing new viruses, or even modifications of garden-variety pathogens that turn them into something particularly dangerous. In the short term, the broad leeway agencies are granted in designating Select Agents is one of the more straightforward ways to ban the unsafe cultivation of an AI-designed virus with pandemic potential.
Scaling up manufacture is another matter. Manufacturing strategically significant quantities of an AI-designed toxin or comparable amounts of an existing natural agent would require infrastructure that could not be concealed except by sophisticated state actors. If one were to reproduce today the mid-20th-century American stockpiles of botulinum toxin, recent developments in AI would not make a meaningful difference in the scale of resources required nor in the difficulty of concealing such a project. The new risk to biodefense coming from AI is the possibility that some self-replicating agent could be so successful that a small, seeding release — in quantities that do not require manufacturing scale-up — could lead to large-scale consequences.
Denial steps — primarily in the world of atoms, not in the world of bits — can substantially reduce but not eliminate the risk that a moderately sophisticated actor could construct a virus with epidemic potential, and they would barely inconvenience state actors. Therefore, preparedness is the most important pillar of mitigating AI bio-risk.
Kissing Cousins in Each Virus Family
One of the thorniest issues in nailing down the origins of the recent COVID-19 pandemic is that neither human ingenuity nor a few short years of evolution could produce SARS-CoV-2 from any virus that was present in our sequence databases in 2019. Somewhere in Asia, a virus unknown to international science was its progenitor. As of today, AI tools have the same limitation as human experts: They can only go so far from a known starting point before their intuition about what works and what doesn’t work breaks down. For example, virus-like protein cages designed from scratch by Nobel-winning labs don’t have the properties required to form a virus that works in people, so in the near term, any AI-designed virus capsid will be a variation on one we already know.
To contextualize the greater than 92 percent identity of the Stanford team’s AI-designed viruses with known viruses, SARS-CoV-2 and Middle East respiratory syndrome-related coronavirus have about 50 percent identical genomes and share traits including a spike that can be targeted by vaccines in the same way, susceptibility to Paxlovid-type drugs, and vulnerability to alcohol-based sanitizers. AI models tend to do a good job generalizing beyond their training data when there is a strong grammar to why things are the way they are or when they are interpolating between elements of the training data, but weak when unexplored regions behave differently or when they are extrapolating far beyond the training distribution. Whereas changing a virus to prevent an antibody from binding to it is an example of the first class of problems, inventing a virus not related to an existing prototype is a problem of the second class. It is fortunate that a modest broadening of good preparation for natural pandemics will cover the kind of artificial pandemics that would be possible in the next five years.
Sadly, we don’t have the best track record of natural pandemic preparedness, even for known threats. In the run-up to COVID-19, many were sounding the alarm about the next threat, up to specific details like a shortage of ventilators to respond to the outbreak of a pandemic respiratory virus, but too little action was taken. Preparedness for natural pandemics and classical bioweapons that doubles as good preparation for AI-designed pathogens includes maintaining an adequate industrial base for both supplies — e.g., masks — and vaccines/pharmaceuticals with surge capacity, preparing policy tools for crisis response, applying recently-matured technologies for surveillance such as proactive wastewater screening, maintaining global epidemic surveillance, and most importantly, building a strong public health infrastructure including experienced professionals distributed broadly.
Furthermore, the road map to prepare for future viral pandemics includes creating a broad countermeasure platform for each major group of viral threats. Extensive studies of Severe acute respiratory syndrome coronavirus in the wake of the 2002 outbreak provided enabling technology that greatly sped the development of a vaccine for its cousin SARS-CoV-2. If a full vaccine for the original SARS coronavirus had been developed and tested before 2019, the SARS-CoV-2 vaccine development timeline could have been further compressed. As a graduate student, I saw one of my mentors begin work on one such vaccine, but he was forced to abandon it later due to lack of funding when federal agencies lost interest in coronaviruses. If we can overcome the boom-bust cycle of vaccine development wherein money briefly surges to research the most recently survived pandemic, it is feasible to develop vaccines for prototypes of most major disease-causing families, as outlined in the pandemic preparedness plan of the National Institute of Allergy and Infectious Diseases. Additionally, due to past neglect of the field, there may be low-hanging fruit in germicidal and filtration technologies that would apply to multiple viral families and be harder for an AI design to evade.
The ability of AI tools to retarget animal viruses to infect humans, or to alter human viruses to dodge existing immunity in the population, means the list of potential threats is wider compared to a world in which natural threats are the only ones we must consider. For instance, the World Health Organization’s Pathogens Prioritization list rightly ranks hantaviruses and influenza viruses in its highest threat category, but several families classified as low-priority contain viruses that have caused dramatic epidemics in animals, lack effective vaccine platforms, and could conceivably be retargeted to affect humans. Unfortunately, if we are to defend against the range of AI-enabled threats, representatives of these families must be added to the list of prototype pathogens being pursued in parallel. Though this adds significantly to the program cost, preemptive vaccine development is not only orders of magnitude cheaper than being caught without any vaccine, it is also cheaper than reactive vaccine development because it does not require compressed timescales or spinning up myriad parallel efforts.
Known Unknowns
By accepting that handwringing about extinction of humanity due to AI-developed bioweapons can be safely excluded from consideration, we can avoid the apocalyptic and millenarian responses cropping up in AI safety circles. That said, pandemics cause an immense toll of suffering and profound strategic effects, and AI tools have already generated designs for viable viruses not found in nature. In the hands of a malicious actor, designs could include pathogenic animal viruses retargeted to humans, existing threats adapted to evade anti-infective drugs, or strains modified to dodge pre-existing immunity. No matter what properties they have, they will be based on viruses we know well.
By diligently preparing for future viral outbreaks, plus expanding our definition of viruses of concern to include viruses that could easily be reprogrammed via AI to cause trouble, we can be ready to limit the impact of such an event. AI tools assist in the invention of new gene sequences that don’t exist in nature, but they are insufficient for turning these sequences into live viruses because the processes of resurrecting live virus from DNA are not automated and require a level of technical skill as well as access to specialized equipment, materials, and a supplier willing to manufacture the synthetic DNA. This gives clear points of intervention and mitigates the sometimes-discussed risk that clusters of rogue agentic AI instances will somehow hack their way into creating a biological virus without significant human involvement. Whether an AI is well-aligned to the user is of surprisingly little importance in a bio-risk context, but alignment to its core principles and filters in opposition to the malicious user could be protective.
Pacing progress through moratoriums or cooperation is frequently mentioned as an antidote to AI bio-risk, but that’s unlikely to help much since frontier large language models have not been the most significant contributors to AI design of viruses. As frontier models become better at coding and at conducting AI research itself, they may gain the ability to make improved domain-specific AI models for biology and change this. An open question is the degree to which skills will transfer across domains: Will an excellent general-purpose text-based large language model improve at predicting which DNA sequence will survive?. However, precisely the domains most relevant for improving human health — predicting how to engineer proteins that behave exactly how you want, for instance — have dual-use applications in the invention of harmful viruses, so pacing viral threat-specific progress is also impossible. For the frontier labs, safeguards and monitoring have a better societal cost-benefit profile than slowing the advance of the frontier. Presumably, one day AIs will gain the ability to invent something unlike any organism we’ve seen, perhaps by gaining a better internal model of the world than a human mind can parse. Whether this will happen in 10 years or 30 is hard to predict from the way things stand today. But in the meantime, the difficult and unglamorous slog of pandemic preparedness is our task to avoid or mitigate the next COVID-19-level event coming our way, whether natural or synthetic.
Jason T. Kaelber, Ph.D., is an associate professor of molecular biology and biochemistry at Rutgers, the State University of New Jersey, with a research focus on structural and computational virology. His laboratory designs virus-inspired vaccines and therapeutic platforms and develops new methods for virus detection or characterization. Work in the author’s lab is funded by grants from the National Institutes of Health and REGENXBIO Inc., and by intramural support from Rutgers University.
Image: Jason T. Kaelber