Are scientists in some nations outside the United States already trying to use artificial intelligence (AI) to design potentially deadly bioweapons? The AI company Anthropic raised that worrisome possibility with a report this week describing how it recently discovered five attempts at using its Claude software for pathogens research with the potential to cause harm.
“The individuals implicated in these case studies are working scientists. We do not assert that they intended harm,” the report released Thursday states. Still, when blocked by Claude’s restrictions, the scientists found workarounds, which raises suspicions, the company says. And the five case studies are just a few examples of a “range of potentially concerning activity that we have found on our platform” involving biological research over the past few months, Anthropic states.
The report, the first public acknowledgment by a top AI company that one of its products might have been asked to help develop bioweapons, grabbed headlines and alarmed some scientists and security experts. The cases are “chilling examples of state-sponsored biological weapons developers tapping into the rapidly advancing capabilities” of AI tools, Andrew Weber, a senior fellow on the Council on Strategic Risks who reviewed the report before its release, told The New York Times. “We are sleepwalking into a potentially huge disaster,” Ashish Jha, a physician and public health expert who helped guide the COVID-19 response for former President Joe Biden’s administration, wrote on X. “We won’t get five more warnings before something bad happens.”
The Anthropic report, which also detailed other potentially illegal or disturbing uses of its AI, comes amid a much broader debate over the safety of AI models, which even some employees at the top companies suggest pose an existential threat to humanity. In an essay published yesterday, in which he advocated for slowing down the development of AI, Anthropic CEO Dario Amodei mentioned a prohibition on its use for the development of biological weapons as one measure countries might agree on.
But some pathogen researchers think the way Anthropic portrayed the five case studies was an overreaction. Scripps Research virologist and evolutionary biologist Kristian Andersen dismissed the concerns on Bluesky, calling many if not all the experiments described in the report “just basic research.” Andersen tells Science he worries companies like Anthropic “can decide whether they think that type of research should go on and whether they think that it’s dangerous or not, and then they can just shut it down.”
Several biosafety and biosecurity experts took a middle ground. “I would resist both extremes in interpreting these cases,” says Filippa Lentzos, a biosecurity expert at King’s College London. Some online reactions have been “overheated,” says Johns Hopkins University biosecurity expert Gigi Gronvall, who has advised Anthropic but was not involved with the report. Still, both say the report has shed light on the potential misuse of AI by biologists. They hope its release will stimulate further discussion around the issue—and possibly new regulation.
Here’s what to know about the controversy.
What are the five biology “case studies” in Anthropic’s report?
Three cases involved modifying viruses. Anthropic says researchers used Claude in May to write a grant application for a project to identify mutations in the chikungunya virus—which can cause fever, severe joint pain, and other symptoms but is rarely fatal—that help it spread more efficiently and evade the immune system. The researchers’ prompts to the AI indicated they planned to engineer viruses containing these mutations and test them in animals.
Also in May, researchers used Claude to plan experiments that would make mutations in a highly pathogenic version of the avian influenza virus (the report does not specify the strain) that help it adapt to mammals and transmit more easily. In the third case, researchers used the AI to write a grant application aiming to identify mutations that made an unspecified member of the orthopoxvirus family, which includes monkeypox virus and the eradicated smallpox virus, less virulent in mice.
The three studies could have been part of an effort to develop vaccines and drugs or detect an emerging pandemic, but they could also be used to make the viruses more dangerous, the report notes. It called the chikungunya work “highly concerning gain-of-function research,” which means giving a pathogen new capabilities that could make it more harmful. And it described the avian influenza work as “research into a pathogen with enhanced pandemic potential”—a controversial type of experiment no longer allowed in the U.S. under new rules imposed by President Donald Trump’s administration.
But Andersen says the experiments sound like more typical studies that involve giving the viruses mutations already found in nature—not new, riskier genetic changes designed by scientists. Because it’s not easy to isolate live viruses with these existing mutations—they are only known from sequencing—virologists routinely swap them into a “backbone,” which can be a weakened version of the original virus or a so-called pseudovirus that can’t reproduce at all.
“That sounds scary, but in reality, it’s just basic biological research which can be done perfectly safely” in high-containment laboratories, Andersen says. And it’s essential to our understanding of the mechanisms that viruses use to cause harm or become more harmful, he adds. Andersen points to a 2023 paper, on which he was a co-author, about experiments similar to the chikungunya work but with the Zika virus, which can cause babies to be born with an abnormally small head.
The fourth Anthropic case involved compiling an atlas of venom toxins as part of a drug development program—using public information, Andersen notes. In the fifth case, scientists used Claude to help computationally redesign a set of toxins, including two bacterial and viral toxins considered to be major disease threats. As the report notes, both projects were studying “novel compounds that can simultaneously be developed into novel therapeutics or toxic agents.”
“All of the research discussed by Anthropic was on pathogens that are endemic in different parts of the world and are current public health threats, so studying them is not suspicious on its own,” says Gregory Koblentz, a biosecurity expert at George Mason University.
If the work was potentially unremarkable, why was Anthropic so concerned?
One reason the company sounded the alarm is that in all five cases, the scientists took measures such as using intermediate computer servers, VPNs, or disguised accounts to make it look like they were using Claude from the U.S. or other countries where its use is not restricted. (The report does not say where the researchers were located; Claude is unavailable in a half-dozen countries considered U.S. adversaries, including China, as well as some other nations.) Anthropic took measures to cut off access to Claude, such as banning the accounts the scientists were using, but in at least one case researchers continued to find ways to use it.
In the fifth case, the researchers deliberately used vague terms for the toxins while using Claude to write progress reports, Anthropic’s report says, suggesting they may have been trying to disguise their intent. The report also notes that the chikungunya research was done at an institute affiliated with a military lab, and the orthopoxvirus study at a “state-associated” laboratory.
None of that is necessarily concerning, however, Koblentz says. If the researchers were in China or another country where Anthropic restricts access to Claude, it’s not surprising they would use VPNs or other means to access AI models that are becoming indispensable in many scientific fields. The workarounds “sound like what teenagers do to get around the parental controls on their phones and computers and what people did over the summer to watch the World Cup,” Koblentz says.
Nor is Koblentz troubled by the fact that research was done at state institutions or labs with ties to military institutions. In the U.S., the military itself “conducts and sponsors research on a range of infectious diseases that pose a potential risk to our troops,” Koblentz notes.
The biggest worry with AI is that a novice scientist with evil intentions will use the models to get “uplift”—to quickly gain knowledge needed to create a dangerous pathogen, Gronvall notes. But in her view, the researchers Claude flagged were “using the latest tools to assist their work,” like established scientists anywhere.
Will the report lead to stricter rules on the use of AI in biology?
Even if these particular cases don’t demonstrate any evidence of bioweapons efforts, several experts who spoke with Science say they point to a risk that will need regulation in the future. Some welcome Anthropic’s transparency because it brings real-world evidence to biosecurity risks that have until now been mostly hypothetical. “This starts to give us some visibility into what developers are actually encountering in the wild,” Lentzos says. “In the absence of adequate government governance, I would rather they were doing this than not doing it.”
But Andersen and some others worry companies’ efforts to crack down will prevent legitimate researchers from using AI in their work. Even now, he says, when he searches Claude to compare the genomes of two strains of the Ebola virus, the query is blocked. Biosecurity expert David Gillum of Arizona State University shares his concern. “These companies already have a lot of power, and now they’re going to make these decisions that could theoretically impact scientific knowledge,” he says.
Lentzos and Koblentz both see a need for government regulation with limits on who can access the models for biological research. But others are more cautious. Gronvall suggests Anthropic and other companies should instead continue to share cases and work on improving safeguards. “I think this field is going to evolve and grow, and it is going to be important to keep an eye on it, but I don’t think the models are quite as useful for nefarious use in biological weapons as people presume they are,” she says.
Science’s AI in Science reporting initiative is supported by Ray Rothrock & family.