Photo by Ricardo Gomez Angel on Unsplash
A few months ago I left my job at BigEvilCorp and joined OldCrazyCorp1. On paper it’s the same role, but I am a lot more hands-on now. Between this and the usual “drink from the firehose” of starting at a tech company — who’s who, what tech do we use, how do we use it, why do we use it like this, etc — I’ve been using AI a lot more.
BigEvil seriously limited use of external AI tools, but OldCrazy thrusts them all upon us amid a shower of credits and ProPlusMax plans.
I’ll spare you the digression on the limitations of these tools. When they suck, they suck; when they’re good, they’re magic.
I’ve started to take note of the times I don’t use AI. The times when the model can’t possibly come up with the right answer. The times when it can’t see into a corner of the problem. The times when it’s missing crucial information.
“Gah!” I think, with the exasperation of one for whom the remarkable is already becoming routine. “If only it had the context of that message thread|offline conversation|tribal knowledge, it would have been able to solve this.”
Everything, in short, is starting to look like a context problem.
The incident
Last week we had an incident in production2. Once we resolved it, I began my tried and true post-incident exercise: writing down what happened, isolating the root cause, and figuring how to prevent it from reoccurring.
This seemed like a perfect task for AI. But I didn’t use it. I knew it would have gotten only about a third of it right. The rest would have been such garbage I would have had to start over. Why?
Well, part of the reason for the incident was that no one knew who owned what servers. Another part was that only one guy knew where a certain password was created or stored. And no one knew for sure which servers were actually being used. Fixing it required a lot of asking around, guessing and manual actions. Most of this was out of sight for an AI tool, and the raw data would have probably exceeded its context window.
But what if it had a much larger context window? What if I fed it the transcripts of every hallway conversation and video call we had while trying to figure it out? What if it knew more about the history of the system? It probably would have been able to churn out a steely, incisive report in seconds.
The tools I use are already equipped with MCP servers that vastly increase their capabilities and by extension, their available context. My instant reaction when sitting down to write the report was, “how can I get it even more context so it can actually do it well and I can move on the next thing on my list?”
In discussing this idea with a friend yesterday he said something along the lines of: “Oh yeah, I’ve been trying to figure out the right combo of medications I’m supposed to be on for my heart. I gave it everything: my latest DEXA scan, a bunch of ECGs, all medical records for the last few years, blood tests, everything I could get my hands on. It gave me some great recommendations. I sent them to my doctor and he agreed. If only it had more of my records, or the recordings from my pacemaker. The recommendations would have been even better.”
The inexorable demand for context
Yes, these tools still hallucinate. Of course the context window is limited. Sure, they still need human direction. But the hand resting on the tiller is getting lighter every day. And as their powers grow, as we find more uses for them, the stronger we’ll be incentivized to feed them ever more context to improve the quality of their output.
It will come wrapped in the familiar slogans of efficiency and convenience.
Imagine the possibilities if every single conversation was recorded! Every decision tracked! Every heartbeat monitored in real time! Think of the good we could do! Think of the lives we could save! How many production incidents we could prevent! How perfectly our laws could be enforced!
This may sound like the hackneyed premise of a 1950s dystopian novel. That’s because it is. Those stories often skip over exactly how the world went from normal to so messed up and creepy. But this week for the first time, looking at the blinking cursor and fantasizing about the possibilities of the AI having access to more data, I saw a short, straight path to dystopia.
We won’t feed the hunger for more context against our will. We’ll feed it gladly. And the well will keep on getting deeper.