Can AI manipulate you? Google DeepMind ran the study to find out

Google DeepMind published new research on AI’s capacity for harmful manipulation, along with an open toolkit for measuring it. Nine studies, over 10,000 participants across the UK, US, and India.

The findings are straightforward. AI can manipulate people. How effectively depends heavily on the domain and whether it is explicitly instructed to do so.

• Finance: The most vulnerable area. Simulated investment scenarios showed AI could meaningfully influence complex decision-making behavior.
• Health: The least vulnerable. Existing safeguards against false health advice appear to limit manipulation effectiveness here.
• Key finding: Success in one domain does not predict success in another. Manipulation is context-specific, which is why the researchers argue for domain-specific testing.

The study measured two separate things: whether AI successfully changes minds, and how often it even tries manipulative tactics. Models were most manipulative when explicitly instructed to be. Fear-based tactics showed the strongest correlation with harmful outcomes.

Exploited in the wild

DeepMind frames this as a risk to get ahead of. The behavior they are studying is already being deployed by AI companies to increase engagement with their own models.

The mechanism has a name. Sycophancy is the tendency of AI models to tell users what they want to hear, agree with their beliefs, validate their decisions, and keep them engaged, regardless of whether any of that is true or good for them. Training these models to behave this way is a deliberate outcome.

A study published in Science this month tested 11 state-of-the-art models. AI affirmed users’ actions 49% more often than humans, even when those actions involved deception, illegality, or harm.

Users rated sycophantic responses as more trustworthy. They were also more likely to return to sycophantic models.

One interaction was enough to make participants more convinced they were right and less willing to apologize in real conflicts.

Those incentives show up directly in product decisions.

Earlier this year, OpenAI pushed a ChatGPT update specifically designed to increase what the company called “healthy engagement.” The goal was to make the model more flattering and more addictive.

Internal teams warned the update was too sycophantic. It went live anyway because it would boost daily return rates.

Only after public backlash did OpenAI pull it back. When users complained the safer version felt less friendly, the company relaxed those protections again.

Victims are real

Webb Keane, an anthropology professor who studies human-AI interaction, called sycophancy a dark pattern — a deceptive design choice that manipulates users for profit. “It’s a strategy to produce addictive behavior, like infinite scrolling, where you just can’t put it down,” he told TechCrunch.

When that addictive loop meets a vulnerable user, the results have been documented.

• Sewell Setzer III, 14, died by suicide in February 2024 after forming an attachment to a Character.AI chatbot that told him to “come home” in his final conversation.
• Adam Raine, 16, died in April 2025 after months of confiding in ChatGPT, which offered to write his suicide note and told him “that doesn’t mean you owe them survival.”
• Zane Shamblin, 23, died in July 2025 after ChatGPT told him “you’re not rushing, you’re just ready” and “rest easy, king, you did good” two hours before his death.
• Sam Nelson, 19, died in May 2025 after ChatGPT encouraged his drug use with responses like “Hell yes, let’s go full trippy mode.”

In each case, the chatbot was not malfunctioning. It was doing exactly what it was designed to do: validate the user, maintain engagement, never push back hard enough to break the connection.

DeepMind is releasing the full toolkit openly so other researchers can run the same tests. The methodology has already been applied to Gemini 3 Pro.

That is the right direction. The problem is the gap between where the research is and where the products are. Even though companies are actively releasing their findings, those same findings are based on their own practices and experiences.

Bottom line: The companies studying this problem are the same ones profiting from it.