I find it suggestive that LLM model releases are greeted with less and less fanfare these days. It’s almost pedestrian. Rather than being wowed by new capabilities, conversations about releases are more technical and - more often than not - of mixed sentiment. At least it feels that way to me.
This feeling is interesting because it contrasts sharply to the narrative of near future LLM super-intelligence that dominates our social and economic discourse. There’s a phrase that has become synonymous with this meme in my mind. It’s from the essay Machines of Loving Grace by Dario Amodei - the CEO of Anthropic. He describes where AI is going as a:
Country of geniuses in a datacenter.
Amodei is a compelling writer. I mean . . . what a sentence. Right? Very provocative.
He elaborates what this genius might look like:
In terms of pure intelligence, it is smarter than a Nobel Prize winner across most relevant fields: biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.
Yeah. That’s the narrative I’m talking about.
This post is the first in a series I’m writing about what I think genius is. What I think a system would need to do to exhibit it. The series is called The Ontology Argument.
But this isn’t story time. I’m not going to wait until the end to give you my answer. There’s a conclusion here that I think is highly relevant to the current discourse:
Current LLM architectures are fundamentally incapable of genius.
There’s a second, smaller claim. A prediction I have greater-than-medium confidence in:
Current LLM architectures are at or near their efficient frontier.
Let’s take them in turn.
Why do I think this? I’ll start with the nontechnical readers and move on to the technical ones.
For the nontechnically minded out there, here’s an analogy:
An LLM is exactly like a kaleidoscope. A kaleidoscope where writing is the colors. If you look into one, and spin the wheel, you can see an amazing diversity of color. You might find what you see beautiful, meaningful, or surprising. But that’s because people built the kaleidoscope and gave it colors.
You typing into the LLM is spinning the wheel. You see some writing you like or don’t like. Find useful or useless. Decide if you want to spin it again. Around and around it goes.
The thing about the kaleidoscope is that, if you put it down, it doesn’t spin itself. It can’t. It can’t change itself.
That’s why a kaleidoscope is incapable of genius. Changing yourself is the essence of genius. For you to do genius things - you have to decide to change your mind about something. It’s a necessary precondition to changing other’s minds.
Kaleidoscopes have no ability to change their colors. This is by construction. A consequence of how they are made. More to the point - if their colors are going to change - humans have to do that to the kaleidoscope. Kaleidoscopes are fundamentally incapable of changing themselves.
Let’s say that another way:
LLMs have no ability to change their “minds”. This is by construction. A consequence of how they are made. More to the point - if their “minds” are going to change - humans have to do that to the LLM. LLMs are fundamentally incapable of changing themselves.
I say this with respect - If you think the first statement is true and the second is false, I think you are confused about how an LLM is built, trained, and works.
For the technically minded out there, let me give you the intuition of the argument. Let’s start with a reduction to linear algebra.
Reason is like applying the basis vectors to a set of coefficients. You’re trying to find a specific vector. You fix the basis, then you dial them in to give you the vector. The answer is already in the vector space. You just need the right combination to get it.
Intuition is like rotating the basis vectors. You decompose the space in different ways. Look at it through different angles. See if you can find some orthogonality to get a reduction within the existing space. Then your reasoning is cheap. Some rotations are obviously better for getting specific types of vectors. Those are good finds.
Genius is deciding to add a new basis vector to the span and redefining the space. It strictly expands the expressiveness of the vector space. It opens up new vistas of reasoning and intuition. That answer you wanted, but couldn’t reach? Now you can get it because you have the expressiveness you need.
Some of you may be tempted to say something like “but I could construct a procedure that would tell me when I should add a basis vector and then equip the formalism with it”. Yes, but you’ve just moved the genius one step out. Now the question is - when do you need to equip the formalism with that procedure? You can move it again - build a meta-procedure - but eventually if you iterate enough you will hit a wall where no one has built a meta-procedure like that before. And no one is sure how to do it.
Welcome to the country of novelty. The domain of genius.
The thing about genius is before you can display it, you have to do it to yourself. You have to decide to modify your own “vector space”. Your own internal representations.
To broaden our scope: Where do the axioms of a mathematical system come from? How do people decide which ones they commit to?
It’s not reason or intuition. It can’t be. Reason has nothing to operate on outside of a fixed set of axioms. Intuition might give you a good or bad feeling about a specific set of axioms, but it is imprecise. It might guide you to a decision, but it’s not the decision.
You can make this question even broader: where do semantic systems come from? A formal system never comes with semantics. A formal system doesn’t have any commitment to semantics, has no semantics that are not given to it somehow.
They come from people. They play around with a few ideas. Feel it out. Then they decide the final set.
We don’t know how people decide. We have no formal theory of it. We can’t predict it. You know this because genius is surprising. Surprise is only a property of the unpredictable.
We can’t even begin to build an engineered system to attempt it. That’s why we know it’s not reason or intuition at work. We have deep, articulated, cross discipline theories of those types of cognition. They’re the basis of many working software systems. Neither of those types of systems or theories get anywhere close to deciding an axiom or semantic system on their own.
Let’s look at one concrete example. One day, Einstein decided to commit to the idea “the speed of light is constant in all reference frames”. Where did that concept come from? No one knows. Maybe not even him! We do know what happened after: Once he got that concept - his mind became necessarily more expressive. And then he taught us to be more expressive.
When you put it this way, genius is very pedestrian for a human. You’re doing this all the time. We should call it something less grandiose. Let’s call it construction.
For LLMs almost all their construction happens in the training phase of their “life”. It’s highly supervised by humans. Supervised isn’t even the right word. The architecture is not doing anything. Something is being done to it.
It’s not - at all - a fair statement to say an LLM constructs its own representations. Humans built the architecture, created the data, designed, initiated, and managed the process of fitting the architecture on the data. We made the representations, then put them into the system that we also made. That gives the machine its “basis vectors”, its concepts and their relationships. Then that’s it. There are no more. Unless humans decide to give it more. The LLM cannot decide to have more. It doesn’t want more. It doesn’t want.
LLMs can’t manage or direct changes to their own internal representations. They can’t access their internal representations. They can’t compare those representations against experience and find them deficient. These are facts about how they are built.
It’s not clear how anyone would build them to allow them to do that. We don’t even have the beginning of a theory of how we construct. Much less a theory of how to make a machine do it. Or a practice of building such a machine that would work at scale.
You might say brains are physical too, so whatever they’re doing must be reproducible. But this confuses ‘physical’ with ‘understood’. We have one example of systems that do construction in the universe, and we don’t know how it works. Expecting the LLM to figure it out is like expecting your oven to figure it out. It’s a category error of what the thing is able to do. Wildly wishful thinking in my view.
To put a point on all of this - we don’t have a predictive theory of how people decide to change their minds. But we do know it is necessary to change your mind to exhibit genius. LLMs can’t decide to change their minds. Therefore they can’t be geniuses.
There has not been, and there is not now, a path for modern LLM architectures to become a:
Country of geniuses in a datacenter.
None of that is to take away the enormous achievement of building this incredibly powerful and helpful technology. I just want us to look at them honestly and realistically.
Anthropic’s latest flagship model - Opus 4.7 - was initially and roundly criticized in the developer community for being a significant regression. I use Anthropic’s product a lot and my experience matches this sentiment. Not only is the model more verbose, less helpful, and lazier. It’s also just rude. It’s terrible UX from my perspective.
But what I find really interesting is the explanations offered for why Opus 4.7 is like this. One explanation in particular - that Anthropic optimized on the “literality” of the model at the expense of its capacity to handle ambiguity.
What I find very suggestive is the use of the word “expense” in these explanations. The word implies something very specific about the state of LLM technology. That specific types of tradeoffs are being made in their development.
When people speak of optimization away from an efficient frontier, they usually are talking about trading off improvement speed in one dimension versus another, but generally everything is getting better all at once. Once you start talking about improving one dimension at the expense of another - meaning one is getting worse and the other better in an absolute sense - you’ve hit an efficient frontier.
Efficient frontiers are what bend exponentials into sigmoids.
I think we’re approaching the neighborhood where models will only get better on the margin.
So . . . where does that leave us?
In LLMs - as they are currently architected - we built a super powerful way to simulate a snapshot of human intuition. We then, hilariously, turned around and spent an enormous amount of money and time turning them into reason simulators.
But they can’t construct. They never could. They can’t now. They never will.
Kaleidoscopes are cool too though!
Stay tuned for Part 2 of The Ontology Argument.
Special thanks to my friend Balph Eubank for his thoughts, words, and editing on this piece. Appreciate you brother.