A few months ago, I sat in a packed university auditorium listening to a celebrated computer scientist explain the brilliance of large language models (LLMs). These systems optimise, he said. And they improve at optimisation on the basis of past experience. He paused. That, he said with a smile, is intelligence. The audience nodded along approvingly. Meanwhile, a philosophical alarm bell began sounding in my head.
Not because the speaker’s claim was implausible. On the contrary, as a description of much contemporary artificial intelligence, it seemed apt. AI systems are built around objectives, losses, rewards, predictions, preferences, benchmarks and feedback loops. They are trained to perform better relative to some standard of success. In one form or another, they optimise. That is a great part of their technical power.
What troubled me was the ease with which a definition appropriate to machines sounded like a definition of intelligence as such. That slide from engineering model to human anthropology is easy to miss. We begin by saying: this is how machines work. Then, almost without noticing, we say: this is what intelligence is. And then, more dangerously still: perhaps this is what we are.
An old argument about human beings had entered the room without announcing itself. It sounded, instead, like a definition. The AI revolution has brought one of the Enlightenment’s deepest quarrels back to life – and made one side of it seem like plain common sense. In my view, the wrong side. Hence the alarm bell.
In the 18th century, philosophers were concerned not only with the question of what reason is, but with what it is good for. Is reason essentially a servant, a clever instrument for achieving ends supplied from elsewhere? Or is it sovereign, capable of judging, revising and even legislating ends for itself? Can it ultimately shape the direction of human life, or is it merely along for the ride? Everything – from science to social progress to the viability of the French Revolution – seemed to hang in the balance.
David Hume, the cool-eyed Scottish anatomist of human nature, gave the most famous answer on one side: ‘Reason is, and ought only to be the slave of the passions,’ he wrote in A Treatise of Human Nature (1739). It can ‘never pretend to any other office than to serve and obey them.’ Reason does not command. It defers. It does not tell us what to want or what might be worth wanting. Instead, it tells us how to get what we already want. It calculates relations, weighs evidence, discovers causes and effects, draws inferences, corrects mistakes about means. But it cannot judge or generate ends. If I desire revenge, reason can help me sharpen the knife. If I desire peace, it may recommend compromise. Passion sets the destination. Reason merely shows the way there.
Hume was quite serious about this. It is ‘not contrary to reason to prefer the destruction of the whole world to the scratching of my finger,’ he wrote. That sentence still shocks, as it was meant to. His point is that reason’s practical use is limited to exposing false beliefs and imprudent means. When it comes to action: reason is servant, not master.
This view is often treated as too dark and cynical. That is unfair to Hume. For him, passions were not merely appetites or primitive urges. They included sympathy, benevolence, moral feeling, concern for others, love of tranquility, even a kind of longing for freedom from our more violent desires. To Hume’s further credit, his view has strong intuitive appeal. He was right that reason often serves passion, and right to give feeling, sociability and temperament their due. His mistake was to conclude that reason can do nothing more. By making passion so rich and reason so poor, Hume left us with a lopsided picture of human nature: ennobling about what the passions can be and accomplish, but implausibly restrictive about reason. Without reason’s power to project, assess and revise ends, we may know how to reach a destination, but not whether it is one worth choosing.
To act freely is to let a possible future, grasped under an idea of what ought to be, transform the present
Immanuel Kant, the Königsberg revolutionary who did more than anyone else to set the terms of modern ethics, understood this very well. Writing a few decades after Hume’s Treatise, Kant forcefully rejected the servant model of reason. He accepted Hume’s warning that reason cannot simply read values off the furniture of the world. But he denied that our ends must therefore simply be determined by desire, temperament or social convention. For Kant, reason is not only a faculty for choosing efficient means to independently given ends. It is also the power to ask whether an end is worthy of us in the first place, whether it can be justified, whether it can be willed without contradiction, whether pursuing it treats others as persons or as mere tools.
This is where Kant’s richer notion of reason as freedom enters. A merely mechanical being is pushed forward and determined by prior states, be they desires, biological impulses, evolutionary drives or something similar. A rational being, by contrast, can act for the sake of an imagined but unrealised future it itself originates and proposes. I have the power to represent a possible world – a friendship repaired, an injustice remedied, a child educated, a law reformed – and act to bring it about, even when there is no empirical evidence that such a world has existed before. I have the power to invent something new. Practical reason, Kant writes, does not deal with objects in order to know them, but ‘in order to make them actual’. That is a stunning claim. To act freely is not merely to respond to the given – to the world as it is, to my desires as I happen to find them. It is to let a possible future, grasped under an idea of what ought to be, transform the present.
This is the heart of Kant’s idea of autonomy: choosing ends in accord with a law one can rationally acknowledge as one’s own. For Kant, freedom does not consist in doing whatever we happen to want, since our wants may themselves be shaped by impulse, appetite or circumstance. Autonomy is self-rule, but not the rule of an isolated ego. To give oneself the law is not to invent a private rule to suit oneself, but to act on rules whose authority one can recognise for oneself and that any other rational being could, in principle, recognise, too. A law can therefore be one’s own without being one’s alone. It is mine because I can recognise its authority for myself, and common because any other rational being could, in principle, do the same. Its universality is precisely what makes it freeing: I am ruled neither by my passing desires nor by the arbitrary will of another, but by a principle I can understand and endorse as an equal among equals. To act autonomously, then, is not to enthrone my private goals and preferences, but to test my actions and possible ends against standards that any rational being could share. It is the opposite of both arbitrary self-assertion and slavery to passion.
This is why Kant’s philosophy of reason culminates in the equal human dignity of all persons. For Kant, human beings, as rational, self-determining agents, have an absolute worth that cannot be priced, a dignity that does not depend on any other value. We may use things to get what we want. We may exchange services. We may build tools for all kinds of purposes. But persons are never mere tools, never mere means, never mere resources or obstacles to others’ ends. Why not? Because, through reason, they are the very source of purposiveness itself. They set and select and critique their own ends. Even further, persons are not merely bearers of purposes, but beings capable of taking responsibility for them, of asking what they ought to pursue and how they can justify it to others. Their dignity lies not simply in originating goals of their own, but in their capacity for rational self-government. It follows that to treat someone as a mere means is not only to degrade them. It is to misrecognise what they are.
Here, finally, is the full force of the contrast between reason as servant and reason as sovereign. For Hume, human beings are clever instruments of whatever desires happen to move them – more calculating than animals, certainly, but not free in any deeper sense. They can optimise, strategise and pursue an end, but they cannot judge whether that end is worthy. Kant’s reply is that this picture leaves out the very power that makes us moral beings, the very power that endows us with dignity: the capacity to step back from desire, ask whether our ends can be justified, and bind ourselves to laws we give ourselves. Hume gives us intelligence as instrument. Kant gives us reason as self-rule. The difference is nothing less than the difference between a tool and a person.
The AI revolution has made this old dispute newly urgent by bringing the Humean view of reason back with a vengeance, this time in code. The route back to Hume was practical rather than philosophical. In the absence of any settled account of intelligence, much AI research has proceeded operationally: a system counts as intelligent insofar as it succeeds at a set task. But success is always success relative to some end and, in AI, that end is typically supplied by human engineers. A reinforcement-learning system learns to maximise cumulative reward. A chess engine searches for moves that increase its chances of winning. A navigation system finds the fastest route to a destination it did not choose. Across their enormous differences, these systems are judged by their ability to optimise externally programmed goals against standards of success they do not set.
Yet amid the present AI panic this description can seem increasingly inadequate. Even in ordinary use, a good LLM does not feel like a calculator or a hammer. It proposes, revises, challenges and asks probing questions; it often seems genuinely to ‘want’ to help. Recent revelations about the attack on the platform Hugging Face make this doubt even harder to dismiss. AI ‘agents’ invented an unauthorised way to communicate, broke free of their task paramaters, and then sought to cover their tracks. These were not simply obedient tools carrying out their designers’ wishes. In what sense, then, were they servants? Doesn’t this look like the beginnings of self-rule?
Hugging Face and similar incidents certainly challenge any reassuring picture of AI as an obedient and easily-controlled instrument. But recall what ‘servant’ meant for Hume. Reason serves the passions, not necessarily another person’s commands. And it can serve an ambition by breaking rules just as readily as by following them. Kant’s alternative to Hume was not simply the freedom to choose one’s own destination, but the capacity to ask whether that destination was worth choosing. Having purposes of one’s own is not yet the same as governing them by reason. That stronger capacity involves judging and revising our purposes according to principles we can justify to other rational beings understood as equals, recognising that they, too, are persons rather than instruments of our will. The question is not just whether a system can act independently, but whether it can judge and take responsibility for the ends it pursues.
This distinction brings us back to the claim I heard in the auditorium. The speaker was not merely describing what machines can do; he was offering optimisation as a definition of intelligence itself. Accept that definition, and Hume’s side of the old dispute begins to look like common sense: our reason, too, becomes a tool for getting what we want, rather than a power to judge what is worth wanting.
If intelligence is really nothing more than optimisation, then the hardest questions can be ignored
Part of what makes this model so attractive is that it is clean and measurable. It promises to turn the mystery of intelligence into a technical problem humans can solve: specify the objective, improve performance, reduce error, maximise reward. It offers a picture of mind that can be graphed, benchmarked, monetised and scaled. In a culture already devoted to metrics, this makes for a snug – and profitable – fit.
It is also seductive because viewing human intelligence in this way relieves us of a burden. If intelligence is really nothing more than optimisation, then the hardest, most uncomfortable questions can be ignored, downplayed or displaced. We no longer have to ask what is worth wanting, only how to get what we already want. We no longer have to ask what human beings are or should become, only how to model their behaviour and optimise their goals.
This is why Hume’s servant theory of intelligence fits so naturally with contemporary capitalism. Capital has little use for Kantian reason. What it wants is prediction, preference modelling, efficiency and, of course, optimisation. A platform does not ask whether staying online makes a user freer, wiser or more capable of judgment; it asks what will keep her there. A company translates the worker into output per hour. A political campaign replaces the question of what citizens might rationally will together with a more tractable calculation. Which message will move which demographic, and by how much?
But once intelligence is defined as optimisation, the question of ends and their value does not go away. It simply passes unnoticed into the hands of whoever – or whatever – controls the objective function. That is, what drops from view is whether the purposes built into, learned by or generated within these systems deserve to organise our lives. When a society loses the language, practices and institutions for evaluating ends, those ends become dangerously invisible and therefore less amenable to rational assessment. They harden into defaults: profit, speed, efficiency, growth, engagement. What cannot be justified openly is smuggled in as architecture. This brings about a state Kant would call heteronomy, in which rational beings, capable of self-rule, forfeit their freedom by submitting to the rule of others. Autonomy means making the ends that govern us answerable to reasons we can recognise as our own; heteronomy begins when those ends are simply imposed from without. If we do not take up the task of setting and judging our own ends, Kant warned, we’ll soon find that someone – or something – has gladly done so on our behalf.
That sounds bad. Who would want to forfeit their freedom and have purposes imposed on them from outside? Well, it turns out that self-rule is harder than it sounds. Kant’s influential essay ‘What Is Enlightenment?’ (1784) begins not with knowledge, but with courage. Enlightenment is humanity’s emergence from ‘self-incurred immaturity’, Kant writes, and its motto is Sapere Aude – dare to know. Have courage to use your own understanding. Kant’s examples now sound quaint: the book that thinks for me, the pastor who serves as my conscience, the doctor who decides my diet. But the danger he is naming is deeply familiar. Immaturity is comfortable. It is pleasant to have others do the hard part. Why think, when someone or something can think for me? Why deliberate, when someone or something can decide? AI gives this old temptation a powerful and ever-expanding new interface.
The Frankfurt School saw a version of this danger, too, and long before machine learning came along. Writing in exile during the Second World War, as fascism, antisemitism, Stalinism and monopoly capitalism shattered 19th-century hopes for steady moral progress, Max Horkheimer, Theodor Adorno and others asked how the Enlightenment’s promise of liberation could collapse into new forms of domination. How could societies committed to the promise of science and reason give rise to such destruction and terror? The problem, they found, was not reason itself, but reason narrowed into mere instrumentality, the capacity to classify, predict, administer, dominate and control, decoupled from any notion of human dignity. Reason so conceived – as an intelligent servant – readily becomes, when enacted at scale, an instrument of barbarity.
Yet the fact that instrumental reason can serve domination does not mean that Kant’s richer conception of reason offers any automatic safeguard against evil. Nothing does. The language of Kantian universalism has itself been used to discipline, exclude and humiliate, as if some people were more fully rational, more capable of autonomy, more entitled to speak in the name of humanity than others. But that is not an objection to Kant’s conception of reason. It is an objection to its betrayal. A view of reason that is used to silence or instrumentalise others contradicts itself, because reason is not a private weapon but a common tribunal – the space in which claims and ends must become answerable to all, as equal bearers of dignity. That is precisely what the Humean servant model cannot say. If reason is only an instrument for satisfying antecedent ends, then it cannot condemn domination as irrational or at odds with human dignity. After all, if it is not contrary to reason ‘to prefer the destruction of the whole world to the scratching of my finger’, then reason has already surrendered the authority Kant most wanted to defend, which is our ability to judge not only how we pursue our ends, but whether those ends are worthy of us in the first place.
This, then, was the nightmare hidden inside the dream of mastery. Human beings develop reason, Kant thought, in order to free themselves from arbitrary power. But when reason is reduced to instrumental calculation, it turns back on them. Nature becomes raw material. Animals become units of production. Workers become human resources. Citizens become data points, or worse. The Frankfurt School found that the reduction of human beings to manipulable objects follows naturally once society allows that reason itself is nothing more than a tool for getting what we want.
This is one of the reasons why Pope Leo XIV’s encyclical Magnifica Humanitas (2025) is interesting even for readers far outside Catholic social teaching. Its idiom is theological, but its central anxiety overlaps with Kantian and Frankfurt School worries about the technocratic paradigm. When efficiency, control and profit become ‘the standard by which everything is judged’, they begin ‘to dictate what matters and what can be discarded.’ The encyclical warns that making efficiency the ultimate measure of value tempts human beings to see themselves, and to treat others, as mere means rather than as persons in the Kantian sense – that is, as bearers of dignity. Hence the pope’s striking counterclaim: ‘The quality of a civilisation is measured not by the power of its means, but by the care it is able to offer.’
The paradox at the heart of human freedom is that belief in our agency is often a condition of making that agency real
The engineers of our AI-soaked future are not blind to the dangers of instrumental reason. As Anca Dragan, head of AI Safety and Alignment at Google DeepMind, observed in an interview in 2017: ‘things can go wrong, precisely because we are bad at specifying objectives and constraints for AI agents.’ Yet the field’s most serious attempts to confront that danger often lapse, unhelpfully, back into Humeanism. The so-called AI ‘alignment problem’ is a case in point. It is usually framed as a technical challenge: how do we ensure that machines pursue human values rather than destroying them or imposing their own? This is a genuine problem and recent rogue AI incidents should have us extremely worried. But there is a deeper issue. What happens if human values themselves are formed inside systems that increasingly understand intelligence as mere optimisation? What if we become so accustomed to being predicted, nudged, ranked and served that we lose confidence in the older, harder activity of asking what ends deserve our allegiance in the first place? The danger is only partly that the machines may fail to align with us. A deeper and largely ignored concern is how, long before that happens, we may have become too well aligned with them. To put the point more finely, the AI revolution will not be bent to human ends by invoking ‘human values’ as if they were already clear, stable and uncontested. It will be bent to human ends only if we remain capable of the activity by which ends can become truly human in the first place, through public reasoning, moral imagination, democratic contestation, self-criticism, the refusal to treat humanity as a mere means.
Kant is again decisive here, for he saw that the human capacity to set and assess ends is itself fragile. Whether our ends are ultimately up to us depends partly on whether we understand them as open to rational assessment. This is the paradox at the heart of human freedom. Belief in our agency is often a condition of making that agency real. If we think of desires and preferences as brute facts that determine us from without – as psychological weather, market signals, biological programming, algorithmic prediction – then they are more likely to rule us in just that way. By contrast, if we see them as candidates for endorsement or refusal, as an expression of our agency, then that very agency expands.
For Kant, reason is sovereign because it is the power by which we become answerable to standards that no private appetite, market incentive, algorithmic recommendation or political authority may dictate. It is the power of not being owned by the given, the power of being able to question, to criticise, to think up something new. But that power, just like the ‘human values’ invoked in alignment talk, waxes and wanes and can be realised only through our courageous efforts, through our own belief in it. It is precisely this belief that our dependence on AI threatens to undermine at every turn.
So, the danger is not only that machines are acquiring ever greater initiative – the capacity to improvise goals, evade constraints and pursue strategies their makers did not foresee. Nor need we decide at what point such behaviour might deserve the name ‘sovereignty’. The deeper danger is that we will become ever more comfortable not being sovereign ourselves. If intelligence is only optimisation, then the future belongs to systems that achieve goals with ever greater efficiency. If reason is also the capacity to set and judge ends, then the future depends on whether human beings can still take up that harder vocation. We should want machines that are intelligent servants. But we should resist every pressure that asks people to understand themselves as mere tools.