Neoclassical economics assumes rationality. The corollary of "If you're so smart, why aren't you rich?" is "you're rich, so you must be very smart!" Thus it is that many people assume that if powerful, well-compensated CEOs insist that "AI is changing everything," well then, AI must be changing everything.
But the evidence for this "changing everything" thesis is thin on the ground. Despite a global mania that has reduced the real, pressing need for digital sovereignty to the imaginary need for nations to create "sovereign AI", no one can really articulate the case for "sovereign AI". If Donald Trump ordered Big Tech to turn off all of your country's chatbots tomorrow, nothing would change. Every one of your country's ministries and corporations would chug on with nary a hitch. Households, too, though perhaps a few of the younger members of those families would have to do their own homework again.
Contrast this with what would transpire if Trump directed his tech giants to switch off your country's Office 365 access, or to brick your Android and iOS phones, or to killswitch your John Deere tractors. Your country would effectively cease to exist. If "digital sovereignty" means anything, it means doing something about this urgent fact.)
The world is full of people who insist that "AI is changing everything" but who – when pressed – have to admit that what they mean is that they're pretty sure that AI will change everything. Eventually. After we allow it to consume all the planet's energy, carbon, water and financial resources.
Maybe.
(They're pretty sure.)
One person who's had a lot of opportunity to observe the shear between the stated business/AI situation and the real business AI situation is Nikhil Suresh from Hermit Tech, a consulting firm of "radically ethical data wizards" (that is, tech consultants). For a year and a half he has been talking to hundreds of executives — and, more importantly, their subordinates — about what, if anything, AI is doing for their businesses. The "head of AI" at a billion-dollar firm told him that his job was "totally fraudulent" – but that it was "the only promotion pathway remaining" at the organisation.
“We’re facing a coordination problem around executives being honest around the AI gains they’ve witnessed – if they co-operate, they keep their jobs. If they defect, they will possibly be fired by their embarrassed peers (who have now been implicitly called liars, cowards, or incompetents) and then replaced with someone that will toe the line anyway. If they could all admit the truth at once there might be some hope, but there is no way to coordinate that event.”
In other words, corporate leadership is starting from the premise that AI has (or will) radically change the business, and they're working backwards from that premise to find the evidence to support this article of faith.
❝
This has created a situation in which everyone has a strong incentive to lie about how much AI is delivering for their companies
Suresh describes this as a literal religious mania. In the 500-plus-employee businesses he studied, the only people who were promoted – or even spared from being fired – were people who professed "religious declarations of faith" about "the transformative power of AI". Employees who voiced honest, informed objections to AI in the workplace were passed over for promotions or targeted for layoffs.
This has created a situation in which everyone – "boards, executives, employees, vendors, consultants" – has a strong incentive to lie about how much AI is delivering for their companies. Suresh says he's seen announcements from publicly traded companies about their AI triumphs that he knows for a fact never took place.
Suresh says he's never seen a successful enterprise AI project: "Every single one – we have seen 0% success in a year and a half." Not one of their clients would face a business challenge if OpenAI went out of business tomorrow. The problem most companies struggle with is that they're "terminally bad at running software projects effectively". Adding AI to the mix doesn't solve this problem – it just adds a whole new range of ways that software deployment can fail.
Chatbots don't help. The internally facing chatbot that's supposed to help employees figure out how to navigate the business sucks because it is only as good as its training data – the business's documentation of its own processes. Businesses suck at documenting their processes. Customer-facing chatbots also suck. They either can't solve your problem, or, when they seem to solve your problem, the "solution" goes nowhere.
Suresh recounts his sole positive customer-service chatbot experience: a Mitsubishi chatbot with a natural-sounding, responsive voice politely took all the details of an automotive failure and promised him a callback. That callback never came, but Suresh is certain that Mitsubishi has logged this as a chatbot success story, even though the experience convinced him not to buy a Mitsubishi car.
He says he frequently encounters people who reflexively utter the AI catechism: "AI is changing everything." But when he presses these people for details, they admit that their organisation "does not currently use LLMs for anything, and indeed, that they cannot name a single thing that has changed other than they get some use out of ChatGPT".
This shear ("AI is changing everything"/"Well, OK, we're not using AI for anything") is so extreme that Suresh once met an exec who confessed to crafting an AI-centred strategy for a $2bn-a-year business, even though that exec "had never even used ChatGPT or any AI tool in their life”.
Some people have privately admitted to Suresh that they've embraced AI in order to earn a career-boosting corporate reputation for "thought leadership". But many other people (especially non-technical people) sincerely believe that AI is about to "change everything". As Suresh says, if you're in business with a liar, you might be able to reason with them in private – but you can't reason with a true believer.
The true believers are in charge. Suresh points out that it would be very weird for the CEO of an engineering firm or a hospital to mandate "specific procedures or building techniques without explicit agreement from the professionals on staff". But when it comes to AI, business leaders will confidently demand that the skilled professionals who perform the business's core functions use AI, even if those professionals don't think it will help.
As an aside: I remember the dotcom era, when the business press was full of articles about the conflict between CEOs and a new workforce that demanded the right to use the web on the job. Today, the business press is full of articles about the conflict between the workforce and CEOs who demand that they use AI.
Suresh describes workers who feel they have to "AI wash" their work: "They just do the work, the same way they have for decades, and say Claude did it." To add verisimilitude to this sham, they write circular processes in which one chatbot prompts another, and then the process repeats itself in reverse, for the sole purpose of consuming AI tokens to score a high rank on corporate "token leaderboards".
I want to reflect a little on two questions that Suresh's essay raises but doesn't answer. The first is why? Why are people in power such easy converts to this religious mania?
I have my own theory. The most important discomfort that powerful people experience is having ego-shattering conflicts with subordinates who know how to do things they do not know how to do. The fact that you're "in charge" is hard to reconcile with the fact that the people you're nominally in charge of tell you that all your ideas are impossible, illegal, immoral or lethal.
The other question Suresh implicitly raises is: "How can you reconcile the failure of AI in the enterprise with the individual claims of skilled technologists who insist that AI is helping them do great work?" The answer is that these AI users are "centaurs" – experienced workers who are assisted by automation on terms that they set for themselves.
Thanks to their skill and experience, these workers possess discernment, the ability to tell good code from bad, and (more importantly) good uses of code-generation tools from bad. They demonstrate the adage that worker-driven automation improves quality, while capital-driven automation improves throughput.
An automation technique that requires close supervision by skilled and experienced workers isn't going to be a raw productivity powerhouse. You don't "100x" your code this way, at least, not in the sense of firing 99 of your coders and having the remaining programmer pick up all their work. Rather, an automation tool that requires the continuous and conscientious exercise of discernment will let individual practitioners improve their work in extremely satisfying and useful ways. It's a way to spend more on operations in order to produce better outputs. It's not a way to cut your workforce, realise a gigantic saving, and still produce comparable goods and services at a far lower cost.
That is why some individual coders report such delight with their AI tools. They engage with those tools on their own terms, to improve their work in the ways that they, in their expert judgment, consider beneficial. No one ranks them on a "token-maximisation" scoreboard. No one tells them they can't do a project if it isn't "sufficiently AI". When they set out to do a project, no one makes them prove that it couldn't be "done by AI".
As ever, the most important fact about a given technology isn't "what it does" but "who it does it for" and "who it does it to."
Quantity has a quality all its own. These businesses aren't just wasting billions – they're replacing skilled workers with defective chatbots. As I've written before, AI is the asbestos we're shovelling into the walls of our technological society. Our descendants will spend generations digging it out again, and the longer the bubble goes on without popping, the longer it will take to repair the damage.
Cory Doctorow, who was born in Toronto and now lives in Los Angeles and London, is the Nerve’s tech columnist. His new book is The Reverse Centaur's Guide to Life After AI. He will be appearing in conversation with the Nerve’s Carole Cadwalladr at Brighton Dome on 8 September; tickets are available here