Special note for subscribers who get this in the mail: this is an extremely long post because it contains two entire python scripts I use (and that some may want to download) and a large document that is part of my ‘contextual harness’. There is nothing I can do about that. This one is easier to read on my site where these long segments are within a small scroll box.
To find out more about the reality of all the (generally overblown) promises about LLMs, I have been doing a very serious experiment on the poster child for Generative AI: coding. I wrote one critical post, and I am close to writing up my conclusions, but I am hitting some work that I have to do first before I can publish that (e.g. create a demo video which is a lot of work, and before that I need to finish some final functionalities and improvements.
This is Lab Note #6. Some basic explanations about my setup are in Lab Note #1. Some basic technical notes (on tokens, context) are in Lab Note #5.
This is an extremely long and detailed note. Sorry for that. So, let’s do a
TL;DR
A review based on the IfSQ method brought to light that LLMs are better than IfSQ in unearthing certain problems, but also that the code that LLMs generate have serious problems that were unearthed by IfSQ. Attacking those problems is possible, but it slowly degrades the effectiveness of coding with LLMs.
For a software engineer, the experience of coding with LLMs remains a somewhat awe-inspiring experience, but that is in part typical shallow human judgement. Underneath, reality bites. By now, more than 50% of my energy goes to quality and defects, not features. I can do more than I ever did, but for how long?
Next time I will report on how much better the code got after Claude reworked it based on the inspection results.
And if you get this by mail as a — very appreciated — subscriber, this one you should read on the web as the large amounts of code aren’t in scroll-boxes in that email.
A month ago, Michael Larabel wrote AI Made A Lot Of “Hideous” Code But Found Major Bottlenecks For Faster Linux Compilation. In it, he relates the observations by a Linux kernel engineer, Lorenzo Stoakes, about the combination of the effects of LLM-coding that has struck many: the code may work but it can be hideous. Another example is this item from a Linux kernel graphics development GitLab repository: iMac 5K shows only 1 tile (2560×2880) instead of a 2 tile display (5120×2880), which started out with a 1500 lines of code LLM-assisted patch, which was brought back to 50 lines and might even become only 25 lines of code. This kind of echoes the paper OpenAI released on a quasi-Riemann proof which is an important result, but the 200-page paper produced by the LLM would be rejected by a journal because, while the math checks out (mechanically checked by LEAN), it was extremely poorly written. A 49-page version from OpenAI with a slightly different proof with a smaller but still important result was edited by a human. According to reports it might be possible to bring that size substantially down further.
We all know AI-slop by now. AI-generated results that are so poor that — often without humans noticing — they do more harm than good. But in mathematics we have those results mentioned above. Results that check out (are correct) but the form is poor. I am reminded of the use of the word ‘sludge’ as defined by Cass R. Sunstein of Harvard University here. Sludge as a form of friction.
If LLMs generate, we might classify them into the following three categories, broadly:
My project is an experiment to get vakwerk (solid results) from coding with LLMs, which in themselves are stochastic, so there is a fundamental contradiction we are confronting.
As I wrote in Lab Note #4, I recently had the code from Claude inspected at IfSQ Level 2. IfSQ stands for “The Institute for Software Quality” and it is a method based on visual inspection by an experienced engineer that provides a measure of maintainability: analysability, modifiability, testability, modularity, reusability.
IfSQ is directed at inspecting code created by humans and to be maintained by humans. The latter is important as I expect the economics of LLMs is going to be a fundamental problem that limits their usefulness in many (if not most, we will see) cases. LLMs are hideously inefficient in more ways than one, but they are currently extremely heavily subsidised, mostly by venture capital and private credit. So, my project has a stated goal: the code must be maintainable by humans (I try to be prepared for when the serious models become unaffordable for me when the party is over, and we also might take future software engineers’ conundrums into account — more on that a next time).
An overall rating goes from AAA (best), via AA,A,B,C,D,E,F,FF, to FFF (worst), based on the estimated number of defects per thousand lines of code. I’ve worked on such an audit myself once, about twenty years ago and I’ve been using one during my time as Lead Architect for the Judiciary of The Netherlands two decades ago. I then found IfSQ to be one of the better ways to audit code quality, so I decided to ask originator of this method — Graham Bolton — if he wanted to play along today. He did.
Graham has created some tools to support this form of an audit, but in the end the judgement remains a human judgement. Supported by the tools, however, he could quickly inspect about half of the code.
The idea of this kind of audit is not just simply a measurement. It is a result that is meant to be discussed with the engineers. When we used such an audit twenty years ago, you could see the junior engineers fight its findings (“come on, how bad is that”) while experienced, senior engineers shifted nervously in their chairs as they recognised how bad the findings were for the future of a system. Graham confronted them with what they already understood but had ignored. So, I worded the result of the audit and gave that back to Claude to work with.
Caveat: Graham did a quick and rough version of the inspection (still a lot of work — thanks, Graham!) so the tools he uses bring up candidates to look at, but as a human he has to check these if the candidates are really issues. So, we will have some false positives as Graham did a quick scan and not a one-by-one deep analysis of the candidates. As an example, an if statement not followed by an else statement is a potential issue (an issue with defensive programming, an unexpected result is not handled and the program simply runs on in a broken state after which everything can go wrong). A switch statement without a default: label is too. But there are many situations where this is not true. Using a hard coded number (e.g. distance from border: 6 points) in code is a bad sign (for maintenance, because if you change it in one place, will you not forget it to change it somewhere else), while code that finds the center of a box by dividing height and width by 2.0 is not an issue.
The result of the IFSQ inspection on my LLM-generated code was interesting and helpful. To start with: the overall score was FF.
Which looks bad, obviously.
So, why was it so bad? What did Graham’s IfSQ audit flag? And given that the audit is based on human creators, how bad is this result if you just keep using an LLM which maybe isn’t bothered by these scores at all? Let’s answer all of these.
IfSQ’s WIP-detection is very limited. It looks for things like dead code, TODO statements in comments, that sort of thing. That is behaviour that is typical for (poor or too much under stress) coders, not for LLMs generating code. An LLM is ‘often wrong, never in doubt’, so it will not leave something like that in the code like an uncertain human would.
My code quality document that is part of Claude’s context (which I will share), already is built to direct Claude at producing robust and self-documenting code. As the development has proceeded I have often had to correct Claude to improve the quality, and such findings then found their way into my ‘stable documentation’. In the case of if and else, my instructions before the audit were already that any if should either have an else, or a comment documenting why an ‘else’ is not required here. Example:
There is no ‘else’ branch here because the routine returns, nothing else needs to be done in that case. When I looked into a few of these obstacles found, I saw several false positives which a thorough human review would have spotted. Say that about 1/3 of the DP-3 findings were real based on my quick scan.
The results of SPM surprised me in part. Not the hard coded numbers and the ‘string literals’. I knew they were there and most of these were things like keys in JSON, which are strings, and I use JSON schema to check correctness in the data. But that isn’t really a good way to be ready for human maintenance. While an LLM will definitely not mistype ‘default’ as ‘defualt’, humans might. And default will be caught by the compiler, string literals will not, so using them is a liability. I already knew I had to fix that.
The LLM-generated code is — despite all the attention paid by me to quality, probably more than 50% — extremely complicated. Take this snippet from inside a large routine:
I defines a variable, the defines a routine inside the outer routine and uses that. That gets you a lot of depth in your logic. Even harder to read: sometimes these extra routines were defined as anonymous ‘lambdas’. A routine is called with parameters, and one of these parameters is the address of an on-the-fly defined function, i.e. the [&]() { part of the first line is the start of a complete function definition inside the parameter list of the function call:
You can guess how complex that can become.
Quite well, in fact. I first gave it this:
Claude generated this first version of a document (Its finding ‘F1’ is funny, it found out that the audit it had built itself was pretty broken):
Graham reacted to this when I showed him: “My impression is that your agent has reacted better to the findings than any development team I ever spoke with, and that it shows a remarkable amount of common sense in its analysis. Glad to see that the review led to the agent finding actual bugs and fixing them!”. This document became the ‘temporary documentation’ for the tasks regarding code quality as is my usual modus operandi.
I had Claude address the quality problems with the flows (like unreachable code etc.). I had it address the ‘magic strings’ and ‘magic numbers’ it had used widely in the code.
I had experienced DRY already in code generated by Claude already: e.g. some user actions in the application were reachable in three ways: via the main menu, a context menu, and a properties panel. At some point, Claude had changed two out of three (so when I used the third it didn’t work properly). You can simply fix the problem, but a better way is to fix the situation in a way that the problem cannot happen as easily in the future: remove as much DRY as you can.
At the time, I had Claude create a script that could detect DRY (you get all these temporary side tasks):
Then it was time to address the biggest issue: complexity. To attack this I first had Claude create my own tool that measures the complexity:
[By the way, this part of developing with an LLM is great. A frontier model generates a ‘small’ Python script like this easily, and that improves my power to work with a large code base]
As it turned out, Claude had generated code with extremely long and complicated logic. Here is that previous example from smack in the middle of a routine of several hundred lines — already some levels deep — again, where Claude has defined routines inside routines.
After the complete rewrite this became a separate small named routine at the top level:
As a result we get more code (about ~10%), but it is easier to read and understand. We lose local complexity, we gain readability. Every function has a clear job and its job is not written as part of another job.
CfC is not part of a Level-2 IfSQ inspection. It requires much more in depth work for an auditor. It is a pure manual human thing and it is hard. Take hundreds of lines of code in a product and try to find missing functionality.
But here we find that a review by an LLM actually improves on IfSQ here. Because, the pattern matching of an LLM is capable of catching WIP based on code alone. I’ve had the LLM create reviews that came up with “the implementation of X is incomplete, it is missing from Y and Z”. To be honest, this is a mix of WIP and a — ‘missing functionality’ — ‘Cause for Concern’. IfSQ CfC’s are hard to spot for humans, the auditor’s trained eye is better at detecting it than ordinary software engineers, but it is still a matter of chance. I’ll get to that below.
Now, the LLM created all that code itself and this leads to an important observation:
I’ve seen this before. LLMs do not understand code, because if they would, they would not produce code they themselves flag as very bad, and that is what I (and others) have seen.
As promised: This is my current code quality document that is part of my ‘stable project documentation’ (the ‘context harness’), which is the material Claude can draw from for its context:
Next time I will report on how much better the code got after Claude reworked it based on the inspection results.
A few observations related to (future) cost and models (these are lab notes after all):
Subscribe? I don’t know if I am going to publish the appearance of new notes every time everywhere (it is a drag), so best to keep up to date with my observations and analysis on the Information Revolution is to subscribe (top right). You’ll get an email whenever I publish. These Lab Note are on a pretty high frequency schedule so far, but normally I haven’t been doing more than 1-2 a month. (I also like to be read. Share?)
[You do not have my permission to use any content on this site for training a Generative AI (or any comparable use), unless you can guarantee your system never misrepresents my content and provides a proper reference (URL) to the original in its output. If you want to use it in any other way, you need my explicit permission]