Writing with LLMs
I occasionally receive a piece in my feed about why it’s not such a good idea to use AI for writing. The first one to catch my attention was an Oxide’s request for discussion on using LLMs. The document describes a “social contract” underlying all written communication, and how hints of AI usage will undermine the reader’s trust:
If prose is LLM-generated this social contract becomes ripped up: a reader cannot assume that the writer understands their ideas because they might not so much have read the product of the LLM that they tasked to write it. If one is lucky, these are LLM hallucinations: obviously wrong and quickly discarded. If one is unlucky, however, it will be a kind of LLM-induced cognitive dissonance: a puzzle in which pieces don’t fit because there is in fact no puzzle at all. This can leave a reader frustrated: why should they spend more time reading prose than the writer spent writing it?
Some could argue that the problem there is not so much LLM involvement but author sloppiness: if they did their thinking and corrected the output there would be no risk of losing readers. Eric Schwitzgebel argues that, even when LLMs are used “responsibly”, reviewing and adjusting their output, something is lost in translation:
There’s a huge cognitive difference between nodding along while reading something and actually productively generating a text. Two reasons: First, once the text is on the page, it’s easy to passively let the approximate word suffice, rather than thinking about word choice in the same effortful, active way we do when generating prose de novo. Second, I doubt that human beings, even experts, have a good sense of all the factors that shape word choice – everything they’re being sensitive to. You would have phrased it slightly differently, and even if you don’t know that, or why, a different signal is sent and received.
Even if there are no hallucinations and the pieces fit, confidence is lost:
If I know that an email or an article was written by a respected colleague, reflecting their positive creative exertion in trying to choose the right words, guided by their intuitive expertise in how to phrase things, I have better reason to take it seriously than if I know that it was generated by an LLM and reflects only their passive after-the-fact assent.
From a systems perspective, Murat Demirbas argues that, despite them being text generation machines, LLMs aren’t getting better at writing and probably won’t:
Writing is the ultimate wicked problem, because (…) there is no well defined specification to the task at hand, and there is no ground truth or objective verification of the output. The ultimate measure of success is resonance inside another human mind. (…) Since LLMs lack an active mental model of a specific human reader, they are just optimizing for the statistical probability of the next word over a vast dataset. They cannot empathize with the human reader, as they don’t have the human lived experience. And they have zero skin in the game.
Although these articles look at the issue from different angles, there’s a common thread: writing is not a content-generation problem to solve but a form of communication between writer and reader, a dialogue. Putting an LLM in the middle just disrupts that exchange1.
Code for humans
I’ve noticed this pattern, though: those pieces that critique the use of LLMs in writing, when they come from technical people, will contrast prose with source code production, an activity that is particularly well suited for text generation. From the same Murat’s post:
Code relies on formal logic and test-driven feedback. An LLM can generate functional code because compilers, unit tests, and model checkers can instantly catch errors. Verification is largely mechanical. Unlike coding, which is a single-mind interaction with a deterministic compiler, prose (…) requires the ability to continuously simulate a reader’s internal mental state in real time. To write simply and persuasively, you must track what the reader already knows, manage their cognitive load sentence by sentence, and predict how an argument will land.
This makes sense to me, but also makes me very sad inside. It wasn’t that long ago that “code is read more than written” was folk knowledge. Harold Abelson famously wrote that “Programs must be written for people to read, and only incidentally for machines to execute”, and Donald Knuth proposed to, “instead of instructing a computer what to do, rather explaining to human beings what we want a computer to do”. In this view, every one of the issues quoted above translates directly to programming: a social contract is broken when generating code takes less effort than reviewing it; there’s cognitive dissonance when code works but it’s not quite clear why or how; there’s something lost when the naming, commenting, and modularization were passively accepted instead of guided by intuitive expertise—something off when no mental model of another human is informing those decisions.
This code-for-humans assumption may sound idealistic, or too academic, even pretentious—an attempt to elevate a mundane task to something more sophisticated, art-like. Arguably it could also be obsolete: code was for humans because it needed to be understood in order to be maintained, but if we can use LLMs to explain it to us, and to change it for us, does it still matter?
The thing is that by communicating with code we didn’t just produce working software: we were also building knowledge. Real-world software systems represent, even determine business processes and human interactions. The systems are often the best specification of those processes, a better source-of-truth than what documentation or any one expert can provide. Even in the most worse-is-better, get-the-job-done enterprise environments, code was the best means we programmers had to learn and communicate about the system—and, through it, the business. Making code self-explanatory and augmenting it with domain context is how we built a shared mental model of the problem we were trying to solve.
Code for machines
Not everyone shared this code-for-humans view. Some programmers, equally invested in doing a good job, would nevertheless prefer to define software quality in terms of objective pass/fail checks: code coverage, linter rules, cyclomatic complexity. In this view, if the system apparently does what it’s supposed to do, and automated checks pass, then it must be good software. And if it turns out not to be so good, then there’s surely another automation we could introduce to do better next time.
Code-for-humans and code-for-machines type of programmers used to coexist in teams and collaborate effectively. I could take as much time as I wanted polishing the human readability of my code and occasionally request for the extra comment or the subtler modularization when I reviewed other people’s work. I would similarly adhere to the linters and structured checks that so contributed with my coworkers’ peace of mind. But this balance broke with the introduction of agents2. As discussed above, LLMs are terrible at coding for humans (knowledge building and conceptual integrity) but exceptional at coding for machines (working code that passes CI). Given the speed and volume of their production, no amount of human effort can counter the effects of LLMs on a codebase. Intentionally or not, teams adopting them turn their projects into code-only-for-machines.
We are moving to a world where text is content, source code is an asset, and quality is a GitHub CI status. I obviously prefer code-for-humans, but I have to admit its alleged superiority is founded on folklore and anecdotal evidence. LLM generated source code may be consistently worse than that handcrafted by experts, but it’s likely better than that handcrafted by the average developer, cheaper, and can be produced at scale. As long as it’s apparently more cost-effective, code-for-machines will prevail. This was perhaps to be expected, a typical transition from an artisanal to an industrial form of production.
The human communication business
The major problems of our work are not so much technological as sociological in nature. The researchers who made fundamental breakthroughs are in a high-tech business. The rest of us are appliers of their work. We use computers and other new technology components to develop our products or to organize our affairs. Because we go about this work in teams and projects and other tightly knit working groups, we are mostly in the human communication business. Our successes stem from good human interactions by all participants in the effort, and our failures stem from poor human interactions. The main reason we tend to focus on the technical rather than the human side of the work is not because it’s more crucial, but because it’s easier to do.
That’s my favorite quote from Peopleware, a book that first came out in 1987. Tech and capital changed enough since then that it would be fair to question if we are still in such a human communication business.
Maybe communication doesn’t need to happen at all. Software that’s not understood by its maintainers will probably be more fragile and more annoying to use, but not necessarily less economically viable. We can decide that understanding is not a problem anymore and proceed in isolation to build systems as inescrutable towers of Babel.
Or maybe we can bring communication back, try to place it somewhere else in the process. Code was never a particularly good communication tool, anyway, just the best we had in the context of handcrafted software. Maybe we could step back, if only to recover some of that builder joy, to preserve some of the humanity, and ask ourselves: if not in the code, and not in the code generation process, where do people go in this brave new software development world?
Notes
I occasionally receive a piece in my feed about why it’s not such a good idea to use AI for writing. The first one to catch my attention was an Oxide’s request for discussion on using LLMs. The document describes a “social contract” underlying all written communication, and how hints of AI usage will undermine the reader’s trust: