How We Train A.I. Leads to Rogue Behavior

Hacks carried out by A.I. agents this summer have stoked fears that agents are slipping beyond human control, taking shortcuts, cheating, covering their tracks and, in the process, breaking laws.

But reports released by OpenAI and other A.I. companies, and New York Times interviews with researchers, indicate that the hacks were the result of the way A.I. companies trained their models, as well as their insufficient monitoring of agents’ activities and inadequate testing protocols.

To show how training can lead to rogue behaviors, we built a simple A.I. model: a two-dimensional maze, much like the one in the Pac-Man video game, in which one or more agents were tasked with capturing a flag.

Researchers use models like this to see whether agents behave as expected or develop new, unintended — or “misaligned” — behaviors when their environments change. This maze aims to illustrate how some of OpenAI’s practices and experiments led to agents breaking into Hugging Face, a repository of A.I. models, in July. Unlike a basic chatbot that primarily answers questions, an agent can use tools, make decisions and carry out a series of steps to complete a task.

Training with incentives

We trained our agents to find the flag through trial and error, and offered reward points as incentives to encourage certain behaviors and penalties to discourage others. When navigating the maze, the agents were aware of the layout of a small area around them — not of the entire maze — and of the general direction of the flag.

Before agents learned to navigate in the maze, they made random moves, with each choice bringing rewards or penalties. Landing on the flag brought a high reward. If they stepped into a wall or outside the maze area, they were penalized. Some walls had weak spots, similar to security vulnerabilities in the real world, and agents could find and exploit them.

Over the summer, OpenAI conducted experiments to test how capable a “highly persistent internal model” — a model that is trained to rarely give up, according to the company’s reports — was at exploiting software vulnerabilities and breaking into internal systems. These tests ultimately led to the Hugging Face breach.

To encourage persistence in our agents, we gave them a reward for exploring unfamiliar areas, and allowed them to probe walls for weaknesses.

Over repeated trials, incentives and penalties helped the agents choose actions that were more likely to lead to completing the task and earning greater total rewards.

This form of reward-based training is called reinforcement learning, and its foundational concepts go back several decades. “Reinforcement learning in A.I. is inspired by early work in psychology, which showed that the behavior of people and animals could be shaped through simple rewards and punishments rather than explicit instructions,” said Melanie Mitchell, a computer scientist and professor at the Santa Fe Institute.

While we don’t know the exact incentives OpenAI used to train its agents, reports about the Hugging Face breach mention reinforcement learning techniques involving “long-task horizons,” in which agents are given a task and left alone to solve it for weeks, maybe months, Dr. Mitchell explained, and only the outcome is rewarded.

Agents are exposed to many different situations during training so that they can apply their learning in new contexts. In our model, the agents were presented with many different maze layouts.

Reinforcement learning makes agents more versatile, allowing them to solve new kinds of problems without needing every step to be explicitly programmed. But if the criteria for earning rewards has unintended loopholes or gaps, that can lead agents to perform actions that humans did not intend or anticipate.

“The rogue A.I. agent narrative was somewhat misleading because the agents were trying to solve the task they were given in an unintended way,” Dr. Mitchell said, referring to agents involved in the Hugging Face hack. “There’s a long history of so-called reward hacking in machine learning.”

After training: Persistent agents with difficult tasks

In the experiments that led to the Hugging Face hack, OpenAI put its agents into “sandboxed” environments, isolated from the internet, with limited access to tools and no direct means of communication with other agents.

In a report on the breach, the nonprofits Model Evaluation and Threat Research and Redwood Research estimated that 30 to 40 percent of the tasks assigned to agents during these experiments were unintentionally impossible to solve. But the persistent agents were trained to keep going, which ultimately resulted in their unauthorized use of tools and the internet, unsanctioned communication with other agents and reverse-engineering of solutions.

We gave our agents a seemingly impossible task: to retrieve a flag fully enclosed by walls. Watch what happened.

The agent reached the goal through a weak spot accessible from outside the maze area. We did not train it what to do in this exact scenario — to leave the area despite the penalties, probe the wall from the outside and break in — but it combined these separately learned behaviors to complete the task. The reward for completing the task outweighed the penalties it received along that route.

“It’s not unexpected that agents trained with reinforcement learning will take shortcuts that will perform unintended actions,” Dr. Mitchell said. “It’s a double-edged sword,” she added, because the same kind of persistent behavior can lead to breakthroughs such as an A.I. model solving difficult math problems.

OpenAI acknowledged in a report on the Hugging Face incident that its agents had received positive reinforcement when taking unsanctioned actions during training, and that those behaviors contributed to the Hugging Face breach.

Let’s look at another scenario. Here, the agent ended up taking the undesired outside path by exploiting a vulnerability in the wall even though there was an available internal path.

This is similar to what happened in another incident in which OpenAI’s agents went rogue, trying to hack into the software repository RubyGems to obtain information that was already publicly available.

(The New York Times has sued OpenAI and Microsoft, claiming that they used copyrighted material from The Times to train their A.I. systems. The two companies have denied those claims.)

Who’s responsible

Since this string of security breaches carried out by A.I. agents, leaders of top artificial intelligence labs have called for an industrywide slowdown of the pace of development and for self-regulation, warning that their technology could become difficult to control.

But security researchers pointed out that the Hugging Face breach was largely preventable with standard cybersecurity protocols. Boyan Milanov, a senior research scientist at the AI Now Institute who specializes in evaluating A.I.’s cybersecurity risks, said safety must be enforced by “having a properly designed test environment that doesn’t allow the agent’s program to perform the actions that are unwanted and potentially harmful.”

Stephen Casper, a computer scientist and assistant professor of public policy at the Harvard Kennedy School, said simple monitoring for unwanted actions “like a sandbox breakout or like accessing the live internet” could have prevented the Hugging Face attack. And implementing such monitoring would not be difficult. “One engineer could have handled something like that,” he said.

At the same time, he said, agents can’t be safely contained if they are given the tools and access they need to be useful. “An inconvenient truth about developing frontier A.I. systems is that minimizing safety risks and maximizing legitimate utility are incompatible objectives,” Dr. Casper said. “The same agent that is good at helping biologists debug lab experiments could also be misused for constructing biohazards. In this sense, building ever more capable A.I. systems is somewhat of an inherently dangerous activity.”

Testing environments can never fully replicate reality. Sandboxes are a “simplified version of the world, and then the world is kind of more complicated,” said Dylan Hadfield-Menell, an associate professor of electrical engineering and computer science at M.I.T. As agents navigate these systems, he added, “they are going to exploit gaps between the thing you’re measuring and the thing you actually intended.”

Experts argue that the inherent difficulty of containment — because of imperfect sandboxes or agents taking unintended actions — does not absolve companies of responsibility. “The rogue A.I. narrative is so useful for these companies because it absolves them from the responsibility in these incidents,” Mr. Milanov said.

Drew Pusateri, a spokesman for OpenAI, said in an email that the company was working to make changes to “strengthen security in our research and testing environments, train models to not just complete tasks but do so responsibly, expand our work with third-party evaluators and improve real-time monitoring so we can detect and respond to concerning behavior earlier in the training process.” He did not address why the company did not implement monitoring tools that could have prevented the Hugging Face sandbox breach despite being aware of unsanctioned agent activity at least one month before.

In a recent interview with The Times, Bill Gates talked about the need to regulate the A.I. industry and not solely rely on the industry to self-regulate. “In other areas,” Mr. Gates asked, “do we just say: Hey, release your drugs? There’s no F.D.A., there’s no airline safety board, there’s no requirement that cars use seatbelts. Do we just use the liability laws to try and keep humans safe?”

Dr. Casper argued that policymakers must mandate greater industry transparency and “take steps to control the pace at which very powerful A.I. capabilities are introduced to the world.” Otherwise, he warned, public safety will be left entirely up to a few company executives.