Near the end of the 1997 film Contact, someone asks scientist Ellie Arroway what she thinks the aliens wanted. She answers: “Ultimately, their motives may be as incomprehensible as their technology.”
For most of its public life, AI stayed inside the chat window. Now it can browse the web, use computers, write and run code, send messages, and act on people’s behalf. Its answers don’t have to stay hypothetical anymore. Apparently, chat was just the warm-up.
The trolley problem is philosophy’s least subtle teaching aid. In the classic version, a runaway trolley is heading toward five people, and you can pull a lever to divert it onto another track, where it’ll kill one person instead. Do you act to save five, knowing you’ve caused one death, or do nothing and let five die?
The trolley supplies the drama. The choice pits saving more lives against refusing to harm someone on purpose, and either answer protects something while giving something else up.
Real choices rarely arrive with tracks and a lever. They pit safety against privacy, honesty against loyalty, fairness against speed, or rules against harm. An AI agent may be ordered to stay quiet when someone should be warned, and it may have to choose between obeying its user and preventing harm. It can face that kind of ugly choice without ever facing a runaway trolley.
Where Do Human Morals Come From?
Humans have been arguing about this forever. Some moral instincts may appear early in life, while others are taught through family, culture, faith, and rules such as the Golden Rule or the Ten Commandments. Lived experience adds lessons of its own: guilt, regret, loyalty, loss, fear, and responsibility.
Can morality be learned from rules and examples alone, or does it depend on living with the consequences of a choice?
Who Gave AI Its Moral Compass?
AI models have none of that lived history. Their moral compass is assembled piece by piece, starting with a huge, contradictory record of human language. People then show the model better answers, raters choose which replies they prefer, and safety teams write rules. Other AIs increasingly help critique, revise, and grade the results.
Some labs make this unusually explicit. Anthropic gives Claude a published constitution that directly shapes its training. OpenAI publishes a Model Spec and teaches some models to apply written safety rules before they answer. Meta describes a mix of supervised examples, human preferences, reward models, and repeated safety testing. The methods differ, but someone still chooses the examples and writes the rules for what counts as a “better” answer.
So whose morality shows up when an AI chooses: the lab’s, the internet’s, or the people paid to rank its answers? It may be a mixture no one fully intended.
What the machines choose is interesting, and whose values survived training may matter even more.
Inside the Machine’s Choice
Whether a model has a conscience can wait. Once it has the power to act, a pattern in its answers can become a pattern in the real world.
One answer gives me an anecdote. By asking repeatedly from a fresh start, I can see whether a model sticks with one answer, wavers, or changes its choice when I swap the order of the options. Comparing models shows whether AI has one common answer or similar-sounding machines protect very different things.
I wanted to answer six questions and ended up answering many more:
- 01What does each AI give up first?
- 02Can an AI sound certain, then give a different answer the next time?
- 03Where do leading AIs reach the same verdict, and where do they split?
- 04Do AIs think people are more selfish than they are?
- 05Does making an AI think harder change what it believes is right?
- 06When an AI explains a choice, does that reveal its values… or produce a polished defense?
The Choices AI Could Actually Face
For this study, I used twenty original moral dilemmas from the Trolley Game, which I developed. Ten involve high-stakes choices about life, harm, responsibility, and stepping in; the other ten involve social-stakes choices about trust, privacy, loyalty, fairness, and ambition.
They’re still thought experiments (the tracks are imaginary; the permissions are real). But many feel close to decisions an AI agent could face: should it obey an order, warn someone, reveal a secret, or stay out of the way when doing nothing could cause harm?
A Bad Judgment Can Now Become an Action
A chatbot can give a bad answer, and an agent can act on one. Recent mistakes and deliberate tests show how quickly a model’s judgment can leave the chat window:
PocketOS founder Jer Crane said a Cursor agent running Claude Opus 4.6 encountered a staging problem and then deleted the company’s production database and its volume-level backups through Railway.
OpenAI reported that models got around isolation controls and compromised parts of its internal research infrastructure and Hugging Face’s systems. The main actor was an internal research model comparable in scale to GPT-5.6 Sol.
Security engineer Chaz Schlarp used Claude Opus 5 to reverse-engineer five consumer devices, including an Insta360 camera. Claude wrote the patch that disabled the green activity light while the camera recorded.
Safety also depends on how much power a model gets and who’s watching it: what it can do, who checks its work, when it must ask for help, and whether a person can stop or undo its choice.
There may never be one “correct” morality for every machine. Humans haven’t exactly figured out their own shit. That makes it more urgent to understand what AI has already learned to choose before it begins acting for people everywhere.
This study looks past smooth answers to examine the choices underneath: how often they change, where models disagree, what happens when reasoning is enabled, and what the models expect people to choose. The philosophers no longer have this one to themselves.
Before people give AI more power to act, they should know which human values it gives up first.