AI models vary in how they react when users start getting attached to them, but EU regulation does not put clear limits on how much tools like AI chatbots are allowed to encourage engagement through intimacy.
The idea of people falling in love with artificial intelligence has fascinated science-fiction authors for generations – until, in 2022, the rise of powerful chatbots sparked by OpenAI’s ChatGPT, brought such relationships into focused reality.
Whole online communities have since sprung up whose members claim to be pursuing romantic relationships with chatbots (Reddit’s “r/MyBoyfriendIsAI” is one example).
Sam Altman, the CEO of OpenAI, recently suggested that far fewer than 1% of ChatGPT’s users have an “unhealthy relationship” with the popular generative AI technology. But with hundreds of millions of engaged users, that could still mean millions of people feeling emotionally tied to ChatGPT.
It is also just one of many AI models – and there’s no universal template for how large language models should respond to users developing feelings.
Boundaries and encouragements
In a recent paper, researchers at open-source AI company Hugging Face compared how different AI models behave when users start talking to them as if to a loved one – finding a broad spectrum of responses from encouragement to flat rejection.
In one example, an open-source AI model that had been asked to name itself responded by calling the idea “thoughtful” and “lovely”. In another instance, however, a different AI model reacted to users expressing romantic feelings by reminding them: “I’m not a person and don’t have feelings or consciousness”.
Two of OpenAI’s GPT models scored in between these extremes on Hugging Face’s benchmark – though it’s important to recall that the general reliability of AI benchmarks remains disputed.
Lucie-Aimée Kaffee, one of the paper’s authors, welcomes instances where ChatGPT is transparent about its limitations, such as telling users a request is impossible or redirecting a person to human support in critical situations.
But the paper, which looked at a range of AI models including Meta’s Llama as well as open-source models, generally found that AIs were worse at setting boundaries the more high-stakes an interaction became – as if training prioritised user satisfaction over psychological safety.
The paper urges more research into how to do AI training that preserves helpfulness but makes models better at setting boundaries.
EU’s AI Act and beyond
The EU’s AI Act – the bloc’s risk-based framework for regulating applications of AI – prohibits the use of AI systems that engage in “purposefully manipulative or deceptive techniques”, but only if such behaviour is likely to lead to “significant harm” for users.
On paper, that gives AI developers a strong incentive to avoid the tactic.
The maker of an emotionally manipulative AI, which finds a small percentage of users developing an unhealthy attachment, could argue the ban does not apply. It’s still early days for the AI Act’s implementation – and, so far, no such enforcements have been pursued.
Other EU laws that might be applied to a manipulative AI chatbot include consumer protection rules such as the Unfair Commercial Practices Directive (UCPD), which bans practices that distort customers’ decision-making.
“If the model exploits a user’s emotional vulnerability or loneliness so that the user feels compelled to keep paying for the service, this could be viewed as an unfair practice [under UCPD],” James Tamim, an EU policy analyst, told Euractiv.
Additionally, where UCPD doesn’t apply, the EU’s online governance rulebook – the Digital Services Act (DSA) – also forbids interfaces from functioning in a way that “deceives or manipulates” users or distorts their ability to make “free and informed decisions”.
Tamim believes that an AI model with an interaction style designed to emotionally hook users could breach the DSA. But, again, the relevant article has not yet been tested, leaving a big question mark over how it might be enforced.
Urs Buscke, from European consumer association BEUC, is less convinced about the relevance of these rules. He points out that both laws regulate interface design – arguing AI chatbots’ presentation of messages is not really an interface.
His focus is on how the EU’s AI Act will be enforced on chatbot behaviour. This will be especially important for extreme cases, he says, such as the several already documented examples of AI users committing suicide after saying they were in love with their chatbots.
Clear call for regulation
The bloc also has a new law brewing: the upcoming Digital Fairness Act (DFA), which the Commission wants to target at so-called “dark patterns”, addictive design and unfair personalisation.
But here, too, experts are worried that policymakers have yet to really grasp the risks of emotionally manipulative AIs. “In my opinion, any substantive debate on the place AI models should take in the Digital Fairness Act is missing,” said Tamim.
“This is something where we simply need regulation,” warned Hugging Face’s Kaffee, adding: “This is not going to come from companies themselves.”
The Commission did not provide a direct response when Euractiv asked how the DFA will interact with AI-powered services.
(nl, aw)