Recovered chat logs show how hackers are abusing U.S. AI models

Recovered AI chat logs and coding sessions are giving researchers one of their clearest looks yet at how cybercriminals are using generative AI and easily bypass model guardrails.

Why it matters: Hackers of all skill levels are developing attacks and finding exploitable software vulnerabilities with the help of mostly closed AI models.

Driving the news: Cisco's Talos intelligence group studied AI artifacts that hackers accidentally exposed online, including prompt histories from endpoints running Claude Code, Codex, Cursor and Gemini, according to a report shared exclusively with Axios.

• The hackers used these models to code software, write malware, and hunt for vulnerabilities. One actor used AI to build a chatbot designed to scam cryptocurrency users out of their money, Cisco found.
• Some actors also appeared to be using compromised enterprise AI accounts and API tokens instead of paying for their own compute, Cisco said.
• Cisco researchers found the artifacts after threat actors accidentally exposed them online through operational security mistakes, Nick Biasini, senior technical leader at Cisco Talos, told Axios.

The intrigue: Hackers used simple jailbreaks like telling the models that they were participating in ethical hacking competitions or creating new sessions mid-way through a task to bypass safety restrictions.

• When models initially refused requests, actors often persuaded them simply by claiming they were authorized to perform the work, according to Cisco.
• "I was hoping there would be a little bit more protection from what they were asking the models to do, " Biasini said. "At the same time, the models are in a tough spot because they have to actually support people that do vulnerability research for a living or do red teaming for a living."

Between the lines: The report suggests AI benefits experienced hackers and novices very differently.

• Sophisticated hackers saw a boon when using these tools for things like automated zero-day discovery.
• But novices struggled to get their ideas beyond just creating the tools they need for an attack, the report found.

Zoom in: In one example, Cisco found a French-speaking hacker used an undisclosed AI tool to turn publicly available information about the critical React2Shell flaw into an automated credential-harvesting platform.

• The hacker — whom Cisco assumes to be a novice-to-intermediate software developer — used the AI-assisted pipeline to scan 9,180 internet-exposed hosts before collecting credentials and source code from 54 systems.

The bottom line: Biasini is pushing companies to make sure they have security protocols that log AI agents' movement on their networks and to build defenses based on deception techniques, like creating honeypots that trap hacker's agents. "Don't trust model guardrails," he added. "You need to make sure you're doing your own protections, that you're building your own guardrail."

Go deeper: These 5 AI risks have the highest potential for catastrophe