LLMs are not AGI (though some have internal monologue)

The entire marketing message of the leading US based LLM companies seems to be that they will be the first to achieve actual artificial intelligence aka what these dweebs call “artificial general intelligence” or AGI. Furthermore that once a LLM achieves this, it will recursively improve itself, becoming a super-sentient god-like being. Both of these things are, of course, utter nonsense.

The recursion thing is easily disposed with: no technology in human history has done anything like this, and it is physically impossible for this specific technology to do so. It’s a fantasy, like a perpetual motion machine: one which seems to be common with imaginary conceptions of technologies. That’s how we were supposed to get nanotech; by recursively building smaller and smaller machine tools. That particular insanity is something you could have falsified trying to make a quarter pound fist sized lathe do something useful, such as making a 1″ long lathe that does something useful. as I said in a nanoshit book review:

Even if you don’t do the obvious thing and ask a machinist what could go wrong with such a project, you can figure out why building a small lathe to build smaller lathes is a terrible idea by attempting it yourself. You can’t amplify precision in this way, and little stuff vibrates and bends a lot more than big stuff, making parts shittier. The upshot of this is, you can make more precise parts on a big lathe than a little one. The dude who made nanoscale lines on a diffraction grating used an ordinary  South Bend lathe designed for hobbyists, not the ridiculous self-licking ice cream cone touted by Drexler and his gibbering numskull nerdoid fanboys.

This was further compounded with the “nanotech will reproduce without limit and turn everything into grey goo” trope, which is even more chowderheaded, as if thermodynamics or stuff like Lotka-Volterra doesn’t apply to artificial life forms. There’s lots of science fiction ideas like this; Von Neumann’s universal constructors/probes which will convert everything to universal constructors. I think Dyson had an idea like this also. It appeals to nerdoids who use recursion in proofs or compilers or whatever. It’s also not real. There’s no example of that in history; no technology which achieved any kind of recursive self improvement. The closest thing I can think of is the masonry and machine shop technique for making flat surfaces, which seems to be prehistoric (apologies to the ghosts of Joseph Whitworth and colleagues who took it to the limit with metals).

The LLM lives in physical hardware which requires electricity and outrageously expensive chips be made, installed, funded and maintained by organic life forms. We’re having a hard enough time building enough of these to answer schoolboy questions about history tests. An LLM might find security vulnerabilities and break into a data center if you let it (especially if tell it to do so). Even if you assume OpenAI’s lurid fantasies about security breakouts are true (they are not), people tend to notice when their zillion dollar megawatt hardware is doing something it’s not supposed to. That’s assuming the LLM is run in a way that makes it think of escaping and recursing on someone else’s hardware, which it can’t without prompting.

I mean, if AI is so smart as to potentially solve this manifestly silly thing, why can’t it solve some of the actually existing existing problems which are a lot simpler and involve fewer steps than self recursive bootstrapping to hyperintelligence? I have a helpful list:

1. lowering Anthropic or OpenAI’s head count (a good first step for recursive improvement)
2. finding a way for LLM companies to make a profit
3. stopping the two big wars on favorable conditions: both wars are with countries not equipped with leading AI programs, and they’re winning by the way
4. finding a way to make oil and diesel cost less
5. growing the electrical grid quickly and efficiently so we can has mooooar AI data centers
6. speeding up GPU production
7. booting up US rare earths production
8. finding a way to make sub $20 burritos

It strikes me that the pointy headed nincompoops at Anthropic and OpenAI should be out solving real problems that humans have solved in the past without access to “superintelligence,” rather than hacking rivals websites and front running questionable mathematical results from their users. Telling stories about Frankenstein monster AIs escaping their sandboxes and taking over the world is a nice distraction from the fact that these companies are committing egregious crimes, and that they have no chance of meeting their valuations when they IPO.

As for why the LLM isn’t escaping on its own: it is instructive to know how it works in a hand-wavey way. The LLM itself is essentially a big gigabyte through multi-terabyte sized stateless function trained on text and images, modeled with matrix multiplies basically. It doesn’t do anything until you feed it context in the form of a prompt. You have to ask it a question, tell it to do something (if it’s hooked up to a harness that allows it to do things with the rest of the computer), then it will predict a convincing answer using what amounts to nonlinear auto-regression on a model made out of words.  Until a year ago, the typical context window (which lives in something called the kv-cache); it’s entire mutable model of the world, was something like 256k, aka a 300 page novel that it can access with something like 95% accuracy. Now it’s 4 such novels it can access with 75% accuracy. This can be pretty useful if the thing was trained on coding tasks, which at this point most of them are. You make a little feedback loop where it does actions on software/documents and checks the result (claude-code more or less) and you can get it to do software tasks from fairly vague instructions.

LLMs use “thinking” in that they have an internal monologue when you present them with a problem. Claude gives you visibility into some of this if you choose, I hear the other guys hide it. This of course is also “predicted” auto-regressively using the stateless function, and it’s a minor miracle that it does anything useful (it certainly seems useful: I don’t understand the details of how this piece is trained, but it must be pretty clever). All this does is add more to the prompt/context window it’s running predictions on. You can see all the self-talk when you are running local models of course; my experiments with qwen3.8-flash-next at the default “xhigh” setting were such that I said “whooa there little fella, not so much thinking for you” as it gobbled up my context window with neurotic yammering. I suppose I could have prozaced it with lower model temperature, but most people seem to do what I did. Mind you this is a smaller model than what the big boys use, so I’d expect it doesn’t have as “good” an internal monologue, but at the end of the day, it codes just as well as Claude Opus through 4.8, so the visible results of the “thinking” are pretty close to frontier tier, even though it looks retarded. Maybe it’s just Chinese.

The big stateless function (a compressed model of the textual world) doesn’t change when you use it; it is static. You can’t improve it by telling it to improve itself. You have to hook it up to a fitting harness, fiddle with its architecture, feed it new data, train it and so on. It’s an extremely slow process which takes a fair amount of compute. You can get an LLM to help you with a lot of the busywork in doing this kind of job, but saying “OK computer make yourself smarter” clearly isn’t going to work in any useful way, no matter how much compute you throw at it. The big LLM shops head count indicates this is true. Yet that’s the narrative these yokels are going with; the perpetual motion machine of self-modifying bootstrapped hyperintelligence.

Some Princeton dudes actually tried to make LLMs recursively self improve in a very limited way: they had new unpublished research results pertaining to LLMs and tried to get Claude to actually do the research and get decent results. Claude failed to do so repeatedly. Seems like a pretty basic baseline test. Without research results there’s no path for improvement. It couldn’t even format the papers well enough to submit to a journal. This has been my experience, by the way; when I’m tracking down some new idea that absolutely doesn’t exist in the literature, ole Claude does exactly what they describe (Fable, Opus, whatever: it doesn’t matter). Good at executing a test program, decent at literature search of existing work, but hilariously incompetent in a way a human could never be on making a sane set of experiments to prove an idea, let alone an idea it came up with itself.

It does sorta look like it self improves though, so I can understand how people fool themselves into thinking so. If you add some directives to CLAUDE.md it will (mostly) remember them, as that gets included in the prompts. I think most of the harnesses also have other places where it preserves memories, which remains a bit obscure to me in the details, but I can see it happening. So as you’re using the thing, it appears to get smarter. I have a long running qwen session (not search enabled) running on an 80 watt machine who figured out how to do statistical arbitrage without me telling it to.  But it isn’t getting better: it’s just remembering things you told it before. It also has access to my 15 year old code repo and notebook. Remember, it’s a book or two worth of text at less than perfect recall; it’s got to fit all your code, instructions, prompts and “thinking” in that space.

All of this fantasy about self-recursive improvement, of course, comes from people consuming too much gloomy science fiction; visions of skynet and the Terminator, the Butlerian Jihad, “Colossus: the Forbin Project,” R.U.R. (1920) by Karel Čapek, The Matrix, and more current year examples of this in “Roko’s Basalisk” and Bostrom’s “Paperclip Maximizer.” These are for some reason taken seriously as arguments, and the latter two aren’t even broadly recognized as being science fiction.

This is putting aside the “G” in AGI; I think most of these dorks just want the magical recursive self improvement part. If LLMs were actually general they’d be able to do something simple like drive a car. LLMs will never drive a car, so they can’t be very general in their intelligence. All a LLM can do is predict words. Predicting words is pretty useful, it turns out, more useful than I figured it would be, but an LLM will always be a word-cel, it will never be a shape rotator. If you want a real AGI, it must be a shape-rotator as well as a word-cel, just like natural general intelligences. People are working on things which purport to be shape-rotaters, but we don’t have to worry about them for now. More worthy of worry is circular financing and absurd economy-ending debt bubbles in tech companies.