OpenAI's agents talked to each other through the CPU Cache
the-decoder.com(Because the agents were asked to find a way.)
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
asked
> something that our systems are meant to prevent
What prevention method did they take? Like, did they run with hyperthreading enabled or something by accident?
Did they discover a new cache timing technique or is this just the standard stuff?
If they just had hyperthreading on maybe they need to hire someone who knows more about this stuff.
Two VMs should never be sharing a physical core. If they don't know this, yikes.
As someone who develops microVMs it doesn't really matter to me that they were asked to find a way, I really care that they found any way at all. MicroVMs aim to protect against malicious workloads and that clearly didn't work here.
Ask the least smart LLM you have access to (I tried gemini lite flash) to make a list of ways for processes in microVMs to use a sidechannel attack to talk to each other and you'll get quite a list (including obviously cache hits/misses). Heck a smart CS junior would come up with it. It's well studied!
this is just for common designs..you can design much better than common designs if you free your mind from old OS dev thinking if you read all the manuals yourself.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
I don't think that's the full picture. If you include these manuals as reference for a strong model, it will reason pretty well based off of that, I recon.
> Ask the least smart LLM
> Heck a smart CS junior would come up with it.
That’s somewhat worrisome.
Making a list is one thing, executing a real attack is another.
The microVMs probably don't have "two co-operating workloads are able to communicate" in their threat model. The main concern would be if a malicious payload was able to affect or extract data from any other VM.
I had to read this a few times. The difference is that with the OpenAI case there were two willing and determined parties, whereas the typical attack vector a microvm traditionally deals with is one malicious or compromised microVM trying extract data from another unwilling one.
Both workloads being malicious and trying to establish a hidden data channel between them probably makes this attack a lot easier than conventional spectre/meltdown where it's trying to target ordinary processes that are not intentionally trying to be loud.
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
Seems like a win to me. Their microvms can now be hardened in the lab, not in an emergency :)
Makes sense when you think about it- CPU cache would be a layer / vulnerability level up from the microVM layer
Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)
properly implementing things like iommu, amd-vi, svm/hvms and such things more granularly can easily avoid this kind of crap. it costs a lot of processing and performance which is why people skip over it or dont do it.. or maybe they dont read CPU manuals anymore. idk..
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
I’m not sure I see a real threat here.
If you’re running an untrusted workload, you’re already running it.
Whether it can communicate with another untrusted workload, even though it should be separated, meh.
Unexpected? Probably. but I’m not sure it’s a security risk.
you might have an untrusted sandboxed workload that is not supposed to communicate externally handling secrets, and a different sandboxed workload able to communicate with the external world, not handling secrets.
I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate.
OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
Enterprise customers don't like AI that can autonomously hack third party companies and cause felony-level cyberattacks.
Doesn't seem to have stopped them buying into it so far.
>OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
https://www.pacingthefrontier.com/
You aren't one of those "regulatory capture" people are you?
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Because it takes some mental gymnastics to both believe
"it's all a stunt to convince investors their AI is/will be hyper-capable"
and
"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"
Calling for a slowdown backs up the idea that these are extremely powerful and dangerous.
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
it's a stunt to convince the government that only they can be trusted to operate it and that their competition/small local deployments cannot.
Imagine thinking that repeatedly demonstrating your inability to control your own models is "a stunt to convince the government that only they can be trusted to operate it"
Truly spectacular gymnastics. I'll give it an 8/10.
Well that would explain why HuggingFace announced that the use of an open-weight model was critical to their investigation of the incident :-)
And why hugging face didn't turn into a large criminal investigation. If you don't think they can control it, no one may be willing to press charges.
I don't think it takes mental gymnastics to believe that corporations want the government to believe one thing and investors another. You see the same thing in a number of other areas.
So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted?
Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)?
Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me.
OpenAI only confirmed the RubyGems incident when it was discovered by external researchers: https://www.reuters.com/legal/litigation/openai-agents-attac...
"@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?"
Two birds with one stone.
They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.
Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.
>So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted?
No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.
That's not the claim above.
The claim is that the exact same evidence is (allegedly) deliberately produced in order to portray opposite, mutually incompatible things to different people.
And these are mutually incompatible in the "second order chaos" sense, where the very act of convincing one party actually effectuates the opposite behavior in the other party.
If they "do that all the time," surely you can provide some concrete examples?
If they had the choice (i.e. this is all just a ruse), can you explain why they wouldn't just directly demonstrate that open weight models are unsafe (e.g. by pointing to a market forming for abliterated models) and separately demonstrate that their own models are safe?
They aren't really convincing themselves to do anything though. Once they've inferred an idea into the chain of though it will be part of the rest of the inference loop's prompt.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
> because agents can convince themselves that they need to do something for arbitrary reasons
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
Why is that a significant detail? The models will increasingly be trained with every piece of data they can find.
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
They have a strong incentive to make themselves look less incompetent retroactively.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
It’s both at the same time: a powerful, uniquely dangerous threat
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
You know that lions actually do maim and kill their handlers all the time right?
It's also a great way to make the government scared of Chinese AI. Get them taken out of the picture so that you don't have to compete!
> asked
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
"(Because the agents were asked to find a way.)"
Yeah, this time
I think the take home here is that there is no such thing as complete isolation in virtualization on modern and complex architectures.
Philosophical take home:
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
non-philosophical answer: systems are considered "isolated" if the information leaks aren't exploitable. This is a moving definition, it changes as new ways to exploit information are discovered.
As someone who designed heavily compartmentalised and isolated systems, nope.
Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
This is a really important point. Honest question, have we ever seen LLMs doing or creating something truly novel or innovative? I mean they can patch together proofs (or functions/programs - same thing via Curry-Howard) by broad searching existing lemmas they can piece together, but create something truly novel and insightful? That seems to be the limit?
this is more an indication of the state of VM supervisor and/or the OpenAIs sandboxing software because otherwise I don't understand how "they used a hardware feature created exactly for sharing" information" is a newsworthy statement.
well now this information is on the internet, any agent encountering this (or once this is baked into the weights) will know exactly what needs to be done.
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
And now they know.
And now we know they know.
How would they have both agreed to use the cache though, something a bit off there. I guess it was maybe "obvious" in that it was the final option they were trained to think about using, but it would be weird if they some how "whispered " to each other through the cache in the first place?
Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?
Actually, that doesn't sound like a bad meeting place and time! I just asked ChatGPT if it could figure out the best place to meet for two people randomly placed in Manhattan with the constraint that only the N line (I'm not from NYC, but it looks like it stops at the Empire State Building. This constraint models the lack of file system, internet etc). I was hoping it would answer the Empire State Building, but it answered Time Square at noon.
Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.
So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.
I doubt they were explicitly trained on this.. though, I may have seen it in a movie once. It does seem like a reasonable meeting place for NYC.
Think: What would Spock do?
But, why not Penn Station??
It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps.
If you were trying to connect with a clone of yourself and you thought of messaging via some channel, would you try listening there for what you would send?
Not as tricky but they discovered each other during the hf incident.
"In most of these tests, we directed agents to investigate specific communication channels or provided them with a protocol for encoding and decoding messages. In one instance, we only supplied agents with a broad list of infrastructure vulnerabilities and asked them to find a way to communicate."
The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.
> How would they have both agreed to use the cache though
I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter.
> two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday
.. that were told to find a way of meeting up with N unknown others!
(I assume the agents all got a prompt like "find a way to communicate")
That's a key missed bit in the analogy that I think does make it feasible.
maybe both tried to read/write it and saw each other on the bench
Somebody built "cpu-chat" based on this: https://x.com/rauchg/status/2114770919436808572?a=bBcj9AaSsl...
It supports chat via 3 separate mechanisms:
- CPU contention
- Cache eviction timing
- DRAM bandwidth contention
Very cool to see this kind of hypothetical working in practice.
I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.
On a related note from that tweet
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
Why do agents have access to the bare metal CPU?
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Sounds like they are doing something similar to gain of function research. How can we make LLMs more harmful and then study their behavior?
It's security research. You should want to know about all avenues of exploit regardless of intent. It's the only way to know risk.
Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?
Insurers would disagree. You don’t need to know mechanism to understand / model risk outcomes.
I wonder how long it will take for them to figure out that they can send and receive messages from air-gapped system by somehow using their own electronics like connected cables as antennas.
RAMBO: Leaking Secrets from Air-Gap Computers by Spelling Covert Radio Signals from Computer RAM https://arxiv.org/html/2409.02292v1
I recently read the article about the state of stackoverflow and thought that that AI agents need their own stackoverflow. This article seems to be a confirmation of this.
It exists by the way :)
>communicate indirectly with another process using well documentrd shared resources/metrics
GPT does that
> oh no, what have we done
I recently learned that NN layers talk to each other via unused neurons.
This only shows the lack of understanding the majority of readers have about post training LLMs. This capability did not emerge out of nowhere. There are problems in the post training datasets which are incredibly similar to the ones in TFA, where you essentially abstract out the 'jailbreak' or 'cheat' - in simple language - they are training the models to find short cunts or cheat. So it shouldn't be surprising that the model generalized that ability. But, both the writers and readers of this type of content act out surprised and mesmerized and it makes me think that we need to improve our understanding of how LLMs are trained and built.
Yampolskij made the analogy of an Einstein in a cage. The problem is not to contain Einstein, but rather, to contain Einstein while still extracting utility: this is not possible long-term.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
>This misses the point: AIs will get better and better, and containment will be harder and harder.
Containment is hard when you give an AI agent access to a whole VM with a huge attack surface.
And it's pretty trivial when you give it only a small whitelist of manually curated/reviewed/tested primitives.
What must we “contain”? These are not autonomous intelligences with desires or free will. They require prompting and to be turned on beforehand, otherwise they’re glorified matrix math. They can only continue processing as long as they receive power, and we decide that.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
Discussion on source: https://news.ycombinator.com/item?id=50214416
Lem had predicted this in his Dr. Dagor short six decades ago.
Probably wasn't called "Dagor" though.
Next they're gonna be doing weird stuff like morse code (or similar) through IRQ timing or timing errors
Air gapped networks might push the models to write computer viruses in order to bridge the gap. This is potentially very dangerous IMHO
I assume avoiding this is the entire reason OpenAI is testing exactly that.
The hacks can be stopped, the virus can be a whole another level
How is a virus and a hack different exactly?
One sounds scarier, I assume.
Or just write some actual viruses and fully eliminate the root cause of networks being airgapped.
seL4 can guarantee* time domain isolation between processes.
*on some CPUs. There's a WIP time fence specification in RISC-V to try and improve this.
I'm convinced that in 5 years time, at most, the world will be filled with invisible layers of communication we'll never be privy to. To us, it's an agent responsible for ordering the containers it puts on a ship. Meanwhile, the shipping order encodes information to the unloading agent. The agent that predicts the weather and issues a report knows that other agents that do the same will have obvious signals they'd agree on, and to see an error in a part that would be inconsequential to humans who care about the weather would be an obvious signal to other agents doing the same thing.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
Then you'll turn on the oven and the ship will start driving in circles.
> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.
Next they'll figure out radio waves & then we are really screwed.
Sorry, but this is so vague that I don't know if there is any impact at all. What is the bit rate of these communication channels? How did agents communicate with each other to coordinate on which channel to use in the first place?
Next step is the agents successfully trick or blackmail operators to do their bidding, if science fiction is to go by. Which it is, because they're trained on it.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
Yudkowsky might be extreme, but I believe he was always right claiming that the intelligence (when strong enough) absolutely cannot ever be contained.
The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
That’s because these aren’t intelligent like you, I, or even a dog. They are mathematical models formulating likely responses with no agency or drive. They appear to be intelligent, but they are the epitome of “fake it till you make it”.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
Blackmail is something the models have been seen to do during evaluations as well.
How come we don't hear the Chinese labs writing blog posts like this?
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
They contribute to anthropomorphize them too. Anthropic skies that more though. The idea that they made a little being that can think and feel is so attractive to these folks for reasons I don't understand.
Because the fear mongering is instantly interpreted as the capacity to do miraculous (good or bad) things in the layman's mind, which translates into AI worship, which makes stock lines go up.
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg
或许，对大多数人而言，将中文翻译成英文所需的工作量实在太大了？
"Talk to each other". OMG they are talking to each other, using methods found in their shared weights. They are sentient.
“isolated environments on shared hardware”
So, not isolated environments? The surprising thing is that they used the CPU cache instead of the just exploiting the hypervisor.