The Asymptote of Reality: The Hard Limit of Multimodal Models

There is a classic thinking exercise in computer science classes. It usually happens in the first week. The professor asks the students to write down simple, step-by-step instructions for making a peanut butter sandwich.

Then, the professor stands at the front of the room and follows those instructions. Literally.

If a student writes, “Put the peanut butter on the bread,” the professor places the unopened jar on top of the slice. If the student writes, “Spread the butter,” the professor uses their fingers instead of a knife.

The result is always a mess. The class laughs, the point is made. The lesson is supposed to be about the importance of precise specifications.

But I think about this exercise differently now. It isn’t just about coding. It is about the fundamental failure of text as a medium.

When you write “spread the butter,” you are relying on the receiver to have a massive amount of “common sense” - a world model that understands jars, lids, knives, and friction. You are compressing a physical reality into three words and hoping the person on the other end can decompress it without data loss.

Right now, in the AI industry, we are effectively trying to teach a computer to make that sandwich. We are feeding it terabytes of text and distinct visual data, hoping that a “world model” or real-world understanding will naturally emerge.

But we are discovering what that CS professor proved decades ago: Text is a terrible way to describe physical reality. It is lossy compression.

The Asymptote of Reality

To understand why this approach fails, we need to talk about fidelity.

In the audio world, fidelity is a measure of faithfulness to the source. “High fidelity” means the playback sounds exactly like the original concert recording while standing in the best spot in the theater.

In AI, I’m using the term “fidelity” to measure how accurately our data represents the messiness of the physical world. In this context, fidelity is an asymptote.

We can probably get closer and closer to reality, approximate it better and better, but the ability to actually touch it, to get there, is questionable.

When we try to teach an AI to make a sandwich by pushing it along this asymptote, the progression looks something like this:

Level 1: Text (The Recipe) This is pure abstraction. As we saw with the professor, it fails because it relies entirely on the user’s internal world model to fill in the blanks.

Level 2: Images (The Snapshot)We show the model a photo of a sandwich or maybe a series of photos for each step. Suddenly, the ambiguity collapses. Now the model solves spatial ambiguity - it knows the butter goes between the slices..

Level 3: Video (The Tutorial) Now we have time and motion. The model sees the angle of the knife and the speed of the hand.

Level 4: Simulation (The Matrix) We create a digital twin. We might even add navigability, sound, or simulated physics. We can run the scenario a million times.

We look at this progression and assume that if we just keep adding data - if we hit a Level 5 - we will reach 100% reality.

We won’t.

The “Gentle Press” Problem

The problem isn’t the resolution of the video. The problem is Tacit Knowledge.

Michael Polanyi, who coined the term back in 1958, argued that “we know more than we can tell.” There is a gap between Explicit Knowledge (what we can write down) and Tacit Knowledge (what we know).

You can watch 1,000 hours of video on how to spread peanut butter, but you still don’t know the haptic feedback of a cold spread tearing soft white bread.

This is the gap:

• Video captures motion, but not friction or temperature.
• Images capture shape, but not tensile strength or smell.
• Text captures intent, but not pressure of the knife.

This is the “Uncanny Valley” of AI agents. They can plan the task, they can identify the objects, but they cannot “feel” the consequences. They don’t know that if you press too hard, the bread rips.

The Agent Delusion

Why does this matter to us as engineers?

Because we are currently selling “AI Agents” that promise to interact with the world, but we are building them on a foundation of text and pixel prediction.

Think about embeddings. A text embedding is a vector in space. A multimodal embedding puts the text “dog” and the image of a “dog” near each other. That is useful.

But there is no dimension in that vector space for “how hungry it is?”, “does it look mean?” or “will it bite me?”

When we design architectures for decision intelligence, whether it’s for a startup or an enterprise environment, we need to stop assuming the model understands the weight of a decision just because it ingested the PDF manual.

We are building systems that can do a lot, but we need to acknowledge their limitations.

Conclusion: The Ceiling of Compression

We need to be honest about the ceiling.

Multimodal models possess incredible, detailed, and rich knowledge. They are powerful tools. But they are not magic, and they do not understand the world the way we do.

There is a fundamental limit to how much of reality you can capture through lossy formats. We can expect a lot from these systems : better planning, better retrieval, better synthesis. But, alas, we must stop expecting a ghost to spontaneously appear in the machine.

True understanding requires being in the world, not just processing data about it. AGI will not arise simply from a projection of thought onto a compressed medium.