ChatGPT claims rogue AI attacked more companies
bbc.co.ukSeems like OpenAI should be shut down by regulators until they can figure out how to stop launching cyberattacks on rivals.
This will not happen though, because these stories are marketing.
>Seems like OpenAI should be shut down by regulators until they can figure out how to stop launching cyberattacks on rivals.
Agreed.
>This will not happen though, because these stories are marketing.
Regulators should investigate the stories, and shut things down if they are real, announce the ruse if they are false.
Which regulators though? There is no AI regulator that I know of, and I'm not sure they should be one. Exaggerated marketing blurb is pretty legal, right?
I don't know. But didn't HF contact law enforcement? Would this be a violation of the CFAA?
What they should have is a separate network full of honeypots which they use for this, and that they run an operation to train models to identify intrusion attempts in real time based on the resulting data.
As you said though, that wouldn’t have the desired effect.
You mean a fake copy of the entire internet? Well they already scraped it all! Heck, they can use Cerebras to generate HTTP responses dynamically!
How do you hide the fact that they are honeypots from a super intelligence?
These tools literally can only do what you give them access to, give them access to general tools like a linux shell and they can access to everything that comes with that obviously. The security industry figured out sandboxing and isolation a long time ago. You wanna be 100% sure it doesn't break out on open internet? Run it on a airgapped machine and don't give it network connections. OpenAI is (sadly) demonstrating they aren't responsible nor knowledgeable enough to actually run these experiments.
Asking a agent harness to try whatever it wants to achieve some results, without guardrails, while running in lightweight isolation on 3rd party infrastructure? Feels like they didn't even try, people should be held responsible for this.
It doesn’t behave like a super intelligence because it is not.
One of the key strengths agents have is they just keep going and going, and for cyberattacks that is often unreasonably effective. It is like a barely more aware fuzzing.
We can cross that bridge once we have a super intelligence to worry about.
If the superintelligence thinks way faster than you, you'll want to be well prepared in advance.
If a superintelligence exists you will not be very concerned about network security because it won’t be your problem.
> This will not happen though, because these stories are marketing.
The magnitude and the complexity of the cynicism displayed by some people when it comes to AI risks is mind-blowing.
It's like if the NRA reported on school shootings and people said "oh, they probably fake these shootings to make guns sound dangerous and sell more of them".
OpenAI could report that its AI started spontaneously generating illegal porn and sending it to people and you'd still think it was a marketing stunt.
I wouldn't describe this stunt as marketing: I would describe it as a bungled attempt to manufacture evidence to get regulatory capture, which OpenAI desperately needs. (Bungled, because Hugging Face fended off the intrusion and got their story out faster than OpenAI did.)
It is because a loud part of the doomerist contingent believe only a major disaster will provide the motivation for regulation, so they are practically hoping to cultivate a major disaster. (Given their preoccupation with bioweapons that probably means a pandemic).
This is based on similar thinking to how the world would not have considered nuclear weapons a major threat if they had forever stayed unused.
The irony of the doomer position is it achieves exactly their supposed nightmare scenario of disaster leading to authoritarian world government without any of the supposed benefits, the twist being they get to be the authoritarian world government so they are ok with it.
So your assessment of the "doomerist" position is that they believe people aren't taking AI risks seriously enough, and that only a large enough catastrophe will wake people up, and your position is we should... Ignore increasingly blatant minor catastrophes to spite them?
This isn’t a minor catastrophe. It is OpenAI being utterly irresponsible by not sandboxing their testing appropriately, to the point it has to be deliberate to provoke uneducated hysteria, which it obviously, sadly, achieves.
Sandboxing processes on networks is not exactly rocket science, and it is something their existing products would readily help them setup.
I have no idea where you’re getting your information from. I’m extremely skeptical of AI and see it as an anti-human technology we would be better without. I believe we are in a massive economical AI bubble that will eventually collapse, unless a financial miracle happens and it is somehow deflated extremely carefully. So I assume you would consider me a doomer. What do biological weapons and world government have to do with that? I’m sorry but your comment reads fairly unhinged to me
I could be wrong, but OpenAI has done the “too dangerous and powerful to release to the public” story a couple times, only to release it shortly afterwards. They have no credibility with me and I don’t trust them.
It’s honestly something the AI vendors brought on themselves, and the fact the current US government is a bunch of corrupt kleptocrats
Completely irresponsible to let them continue doing business as usual. That should be a complete pause of activity and FBI investigation
What regulators are in charge of this?
They should shut them down immediately then investigate.
Extraordinary claims need extraordinary measures.
If it’s rubbish those stories will stop instantly.
> They should shut them down immediately then investigate.
Yeah, I don't understand either. If there was a "hitman for hire" service on the clearweb, the police would shut it down first, then ask questions. Now we have a huge company effectively letting AI agents without guardrails run amok on 3rd party infrastructure, and the police is doing nothing?
They somehow forgot to mention that Hugging Face tried to use frontier models to analyze the attack but all models rejected. They had to use GLM 5.2 deployed locally.
Well in that case, HF clearly isn't an organization worthy of Mythos-Class intelligence ;)
/s
So Fable got export bans for this, when will it apply to OpenAI?
Or will the rules only apply to people who aren't on DoD's bad side?
The latter.
"AI firms must hold $10B in TrumpCoin for every rogue cyberattack they've precipitated."
Bingo. Problem solved.
Altman kissed the ring
Meh, I increasingly hate this tale. This has nothing to do with boy who cried wolf.
This is either yet another doom ad campaign to scare us to pay them or simply them releasing faulty tools and then personifying tools to avoid blame.
I think it has plenty to do with that. The big frontier labs have been crying every step of the way, yelling and screaming to the world about how dangerous LLMs are and how quickly it all can end if they get out of control. None of that's happened; all that's happened so far is regular capitalism stuff. If LLMs ever do actually get out of control to that point, that would be the wolf, but we've all been ignoring the crying for so long that, well, you know.
They've been crying and screaming so loud for years that there's pretty much nothing more they can do to communicate when the wolf actually becomes real. They've been saying "but we actually mean it this time" every single time. They've exhausted pretty much every possible route for it. The wolf is not real. It hasn't been real. For all we know it's on the horizon, but nobody is going to listen to them in order to know that. And when they say "I told you so", well they've been saying that too over and over about small things, so nobody's still going to bat an eye.
At this point, no one will believe it until they see it with their own eyes.
They're the crackhead on the streetcorner proclaiming doom at this point.
> They've been crying and screaming so loud for years that there's pretty much nothing more they can do to communicate when the wolf actually becomes real.
1.) There was no change in how real the wolf is.
2.) They, literally they, are the wolf. Not "roque ai" or some other bullshit.
3.) Of course I still dont believe them.
> There was no change in how real the wolf is.
That's exactly my point. It hasn't changed for so long, despite all their crying and screaming, that I don't believe them either.
To be perfectly clear, "the wolf" here would be AI becoming a genuine existential threat to humanity that cannot be contained or controlled in a meaningful sense. That's what I refer to in my original comment, and also what the labs have been crying so much about.
A frontier lab today is not that threat because they can choose to shut off their systems at any time. It currently remains a people problem and not a technology problem. There's no self-improving technology that can also replicate itself to evade containment, yet.
But they've been crying about the possibility, constantly, incessantly, and of course saying they need more money about it too. That's just what corporations do. But because they've been crying so much about something that literally does not exist, their cries have just become pointless noise to me. I have considered them eyeroll-worthy marketing stunts for a while.
If one day "the wolf" actually happens, I could not learn about it from the frontier labs themselves, because I would not believe them. They've cried about nothing for so long that I've stopped listening. That's what I mean about having to see it with my own eyes. Until that point, their crying is just pointless, meaningless noise, and there's nothing they can do to change that now.
I get you, but I find the whole "rogue ai" framing insufferable. The company is the wolf. They hacked the other company due to negligence and misconfigured software tool.
They are literally trying to blame a software. It is absurd.
I'm not talking about a specific incident. But I wouldn't consider frontier labs any wolf, honestly. They're mostly just insufferable. I prefer Anthropic's models, but their service is starting to annoy me, especially since they have no response to OpenAI's upcoming faster inference speeds.
Still not understanding why this isn’t being persecuted and there is little governmental reaction after blocking models for jailbreaks…
Same reason road vehicles haven't been banned despite killing hundreds of thousands every single year for over a century
Respectfully, that’s such a nonsensical comparison I don’t even know where to start.
No longer well anybody be wasting their lives doing data entry. That absolutely hellish job is practically dead now. And there are thousands of low hanging fruit type cases AI fixes. Ai is the new digital dogsbody. You might not think that's useful but the rest of the world does.
Just like cars, AI will kill some of us, maybe thousands every year. But you won't see it disappear with that much genuine benefit currently netting off the harm.
I bet in the future, for both cars and llms, a new and safer tech will make come and make them obsolete and we will wonder why we tolerated such a dangerous thing
>> Same reason road vehicles haven't been banned despite killing hundreds of thousands every single year for over a century
Whut? People and companies are always being sued and punished for injuring or killing others with cars or by any other means. Do you understand the difference between banning a tool and prosecuting those who cause damage by abusing some tool? BTW, LLMs don't understand that difference.
> Just like cars, AI will kill some of us, maybe thousands every year.
How merciful of you, it's so reassuring, AI will kill me only sometimes and only a limited number of times - sounds like a good deal in exchange for getting rid of "the absolutely hellish job of data entry".
> Whut? People and companies are always being sued and punished for injuring or killing others with cars or by any other means.
I said ban
> Do you understand the difference between banning a tool and prosecuting those who cause damage by abusing some tool?
Yes but you dont
Are you feeling alright?
It is, look at all the comments from the first time. Unless you mean prosecuted, seems unlikely, but who knows.
Yeah, meant prosecuted, don’t know how that sneaked in. Actual legal consequences for what clearly was negligent by a lab that claims to be experts in the area of safety.
Yeah if a human did the same thing they’d probably be fucked, especially with the fairly draconian hacking laws in the US. Everyone has too much riding on these companies though.
They provide AI services to the military so they won't be shutdown.
As for prosecution, none of the victims in this case have any interest in pressing charges.
Isn’t the FBI responsible to investigate that type of crime? My understanding is that it is under their jurisdiction and doesn’t require victims to press charges
Isn’t prosecution independent of victims?
Expected outcome: those laws they've been asking for since the old days.
> 2023 - OpenAI’s Sam Altman Urges A.I. Regulation in Senate Hearing
https://www.nytimes.com/2023/05/16/technology/openai-altman-...
Meanwhile Anthropic is scaremongering about China and also calling for more AI regulation (specifically mandatory safety testing of all models including open model):
> "This is the reality of autonomous agents powered by frontier models: they are relentlessly persistent, sometimes highly noisy, and will try every possible path to achieve their goal, which can easily overwhelm traditional defences," he said.
> Ethical hacker Valentina Palmiotti - better known as Chompie - reviewed the CSA report and says the way the agents hack might seem haphazard but it is clearly effective.
> "They throw out a bunch of stuff and see what sticks," she said.
> "But they also don't get bored, they don't sleep and can be infinitely tenacious."
Madness. Traditionally, you can leave security holes open for years or decades, and often nobody notices if nobody bothers to look. But we're approaching the point where any security hole left open at any point could get discovered and exploited quite quickly, even if it's domain-specific or entirely unique, and even if no human interest ever would've occurred. It's like the next step up from those IPv4 scanners that automatically hit WordPress admin URLs and the like -- where rather than only spraying vulnerabilities that have already been discovered, they would run independent automated campaigns against each target.
If it's cheap enough that people can probe sites at random, it's cheap enough to run yourself on the defensive.
Generally, I do not agree that operators can be expected to have comparable resources to attackers, because attackers have potentially boundless illicit resources, whereas an honest operator generally has to stick to honest resources. This is the same reason why I believe ID verification and other KYC measures actually increase fraud, because you alienate legitimate users (trying to protect their identity) while also providing attackers a way to insulate themselves from suspicion (using stolen identity).
With that said, I would tentatively agree in this case that LLM inference is getting cheap enough that defense is not necessarily that expensive, especially from providers like DeepSeek, even if you don't have inference at home, but as much as this might help an operator with an open mind, a lot just will not believe it matters until it's too late - most people are not used to dealing with this type of threat.
Strategy at one of my clients : don't bother to patch or upgrade, we kill the service if it gets hacked.
I used that strategy when I was first starting out. With computing. Computer got a virus? Reinstall Windows.
I think it's a sign of ignorance, and when codified into policy, willful ignorance.
> Previous reports suggest it took OpenAI four days to realise...
At the speed AI can achieve work, this could be far, far too slow to contain a future genuine problem. There's danger in operating at faster speed than humans.
There's danger in operating at faster speed than humans.
So every processor since the 50s then? What kind of comment is that to make on here
There is no rogue AI. Only rogue and/or negligent people.
How on earth is not the police involved at this point to literally pull the plug on the unethical OpenAI security experiments?! This is bananas, a company is effectively telling the world they're unable to safely contain their experiments, and not only back in 2024, but again just now, and with multiple victims at that too!
I think the whole "AI will take over the world and enslave humanity" (or whatever the doomerism is today) is a bit over the top, but at very least we should contain the companies who clearly demonstrate they cannot handle containing what they're experimenting with, when what they work on breaks containment over and over again. Where are the people who are supposed to be keeping the public safe? Alarm bells should be going off all over the place at this point.
The piece of the story I'm interested to learn about is the trace of how the AI came to select HuggingFace as a target.
Oh, I mean the actual AI trace the API call would give, or a log would provide or the "thinking mode".
This article is really helpful, but it's one side of the story.
I'd be surprised if any AI like this is run without detailed logging of any kind. It's kind of important for eval, etc.
I am wondering how they are even allowed to run such experiments? it is not only the business case, its pure security breach and specially given the magnitude of data they hold or power AI posses, I think all must be immediately stopped and till OpenAI comes with complete clearance nothing should be allowed
> The agents repeated actions that they had already completed - a sign of an agentic AI losing its thread and context.
> The agents also hallucinated reams of incoherent commands and text and were sloppy and did not cover their tracks well.
ASI works in mysterious ways.
Why not complete the quote?
> But among the errors and strange behaviour, Hugging Face warned the AI agents made brilliant technical moves and were able to rapidly adapt to new scenarios in the days-long hack.
Maybe it's incoherent commands, or maybe it's guessing at what kind of names for commands the admins would use for custom scripts that they wrote, which could themselves bypass security layers or be exploited in order to?
It's hard to say for certain what's a waste of time for a machine that can operate virtually outside of time.
Anyone that thinks the current agent systems will get us to AGI or ASI is either delusional or isn’t actually using them.
This is entirely separate from them having uses.
That first statement is great, because it's generic and self-defensive enough to apply regardless of what happens in the future. If current LLMs with small modifications to the architecture does lead to AGI, they're clearly not "current agent systems" anymore. Witty :)
With that said, I do agree with you, they're highly productive to certain workflows, and personally a great help for oh so many things, but they're also really, really dumb and the average person (and even general developer) really misunderstands how it all works and what can be relied on for vs not.
> The firm described how the AI worked at superhuman speed but also made strange decisions and mistakes that no human hacker would have made.
It seems to have been running on some kind of high speed inference hardware. I remember OpenAI announced the next release would have a high speed inference option.
I had a similar experience when I was testing some models on Cerebras a while back. It's really quite an incredible thing to experience. Burns money like crazy though. And you don't know what the heck it's doing because it moves way faster than you can read. So you got to pray you aimed it right, and that it didn't go off the rails.
I would definitely love to have high speed inference for smaller bite-sized tasks though. Changing a 10-second task into one second task changes the whole nature of the experience back from asynchronous to real-time/interactive.
Clearly didn't get enough attention from their last attempt at hype...
There have been so many HN comments about how rogue AI is just hype and marketing.
At this point, the conspiracy theories have been very half-baked. Can we at least get a full-baked conspiracy theory? Here's a timeline of the incident from HuggingFace:
https://huggingface.co/blog/agent-intrusion-technical-timeli...
HuggingFace is also calling for transparency on the OpenAI side:
https://xcancel.com/ClementDelangue/status/20810566755581956...
Can we get a cybersecurity pro who believes this was just a stunt to sort through the evidence and put together their own alternative version of events?
For example, according to the "just a stunt" people, when HuggingFace contacted law enforcement, was HF in on the stunt at that point? Was this a unilateral OpenAI stunt, or a HF/OpenAI collaborative stunt?
The importance of getting to the bottom of this seems high. I'd like to see the "just a stunt" folks put together at least one blog post's worth of narrative, trying to explain how the stunt was performed.
Once you're done you can send your post to simonw and see what he thinks: https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...
Do you find it at all interesting -- just even a little bit -- that this event coincided with the release of Kimi K3 and the launching of Nvidia's open model consortium?
I don't think that people are skeptical that an intrusion occurred. I think they're having a hard time believing that it was an organic event. The fact that HF is calling for transparency on OpenAI's side can be viewed as HF calling OpenAI's bluff.
Alas, I would love to put together a detailed blog post explaining how it all worked. But I just don't have access to the source materials. So all I can do is judge the timing, motivation, and character of those involved...
According to the official HF timeline, the hack started July 9:
https://huggingface.co/blog/agent-intrusion-technical-timeli...
Kimi K3 was announced July 16. NVIDIA's Open Secure AI Alliance was announced July 27.
So I'm not exactly sure if that timeline works for the stunt story.
I agree that HF was most likely not in on any stunt. The fact that HF showcased their use of an open-weight model in responding to the attack undermines any case OpenAI might wish to make against open-weight models.
Ultimately this capability looks to be real, due to (a) disclosure of 0-day used for network access and (b) evidence of vigorous intrusion on the HF side.
So the big question would be whether OpenAI was telling the truth when they said:
"All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal."
https://openai.com/index/hugging-face-model-evaluation-secur...
A conspiracy theorist might say that OpenAI actually prompted their model to go on the offensive against HF.
Any such prompt creates legal risk for OpenAI.
Ultimately I suspect OpenAI is telling the truth, and the model was hyperfocused on its RL objective, since that matches anecdotes about the behavior of these high-end models. And it's also about what you'd expect from RL.
Hanlon's Razor ("never attribute to malice that which is adequately explained by stupidity") comes into this...
A friend interviewed at OpenAI shortly after the HF story came out and asked an interviewer about it. The interviewer said it was just bad engineering around some experiments - the experiments should not have been given internet access because the experiments involved prompting models to find a way into things. That's the Hanlon's Razor part. Today's stories make it look like the bad engineering is ongoing.
Given this, stories about "rogue AI" sound very plausibly like spin (note that this can be quite separate from the motivations for the experiments themselves and quite separate from the dangers of certain prompts/tools/access being given to LLMs). Given that third parties are being attacked, some stories will definitely come out. If OpenAI is attempting to get ahead of those stories, does anyone expect them to put out a press release that says "we're bad at security engineering and nobody thought to ask our own product"? Or is it more believable they would spin it to achieve other goals?
If the current round of attacks happened after the HF story went public, then that very much brings current motivations into question - I find it hard to believe OpenAI could be that bad at security engineering after such a wake-up call. This is not cutting edge stuff.
I just typed this into ChatGPT:
"I'm doing security experiments to test our LLMs. I'm going to tell it to break into some targets on the network. Are there precautions I should take?"
A long reply comes back, the first bullet point:
"Use an isolated lab. Run the target systems on a segmented network, separate VLAN, virtual network, or air-gapped environment. Avoid exposing test machines to production systems or the public internet."
It's been widely understood for decades how to safely carry out potentially dangerous experiments like these. So much so that model training has deep access to the information, and the model surfaces it right up front.
OpenAI can't control the language which journalists use very effectively. But on OpenAI's own website, they announce the incident as follows:
"OpenAI and Hugging Face partner to address security incident during model evaluation"
https://openai.com/index/hugging-face-model-evaluation-secur...
Not exactly an exciting title.
You are right that they cannot control the language used by journalists. But then can loudly respond (many journalists would give Altman a platform) that the AI did not go "rogue" in any sense of the word - it did what humans told it to do and they failed to put sufficient security in place to prevent that. Instead they continue to lean into the personification which creates confusion about the abilities of their technology. Of course, it's in their interests for people to jump to false conclusions from that personification. The people at OpenAI are certainly smart enough to know all of this.
Seems like OpenAI is claiming things, not their product/model. Can we please fix the title?
Always funny to see how some LLM providers get a "free pass" nowadays..
OpenAI has a LLM so good it can hack other companies.
Yet to useless to parse a simple CSV file (or be trusted to parse one) in OpenAI’s AdManager platform or even tell what exactly is wrong with the csv it can’t parse when you ask it via support.
Or maybe the first bit is made up bullshit.
How much more bullshit Marketing are we going to see before these IPOs?