Observability is supposed to help us understand our systems. But somewhere along the way, we confused more data with better understanding. The result? Most engineering teams today are drowning in telemetry they never look at, paying enormous bills for storage they don’t need, and still struggling to debug production issues.
The biggest problem in observability today isn’t a lack of data. It’s that the value-to-noise ratio is approaching zero.
And the primary culprit is auto-instrumentation.
OpenTelemetry has been a massive win for standardization. Having a vendor-neutral, open standard for telemetry is genuinely important. But the way most teams adopt it — flip on auto-instrumentation, watch the traces pour in — has created a perverse outcome.
Auto-instrumentation generates enormous volumes of traces that capture the skeleton of a request but almost none of the meaning. You get spans for every HTTP call, every database query, every gRPC method. What you don’t get is the business context: which user was affected, what feature flag was active, what the cart total was, why the recommendation engine chose those results.
The traces are structurally complete but semantically empty.
So teams end up with millions of spans per hour, storage costs that make finance teams nervous, and dashboards that still can’t answer the question: “why did this specific user have a bad experience?” The signal that matters is buried under an avalanche of mechanical noise.
Honeycomb recognized this problem early. Their approach — wide, richly-attributed events — flips the model. Instead of generating a span for every mechanical step in a request’s lifecycle, you emit fewer events that are packed with meaningful context. A single wide event might carry dozens of dimensions: user ID, tenant, plan tier, feature flags, request parameters, result counts, cache hit/miss, business logic outcomes.
This is a fundamentally more useful signal. Wide events let you slice and dice production behavior along the dimensions that actually matter for debugging and understanding. You can group by any attribute, spot outliers, and ask novel questions you didn’t anticipate when you built your dashboards.
The problem? Wide events require engineers to think about what context matters and then write code to emit it. There’s no “just turn it on” equivalent. You have to go into your codebase, identify the meaningful decision points, and manually enrich your events with the right attributes.
That friction is real. And it’s the main reason this approach hasn’t displaced auto-instrumentation as the default, despite being objectively more useful. OpenTelemetry’s auto-instrumentation wins on adoption because the activation energy is near zero. Honeycomb-style wide events win on value but lose on effort.
I believe the next major leap in observability will come from solving this exact tension — getting the richness of wide events without requiring engineers to manually instrument every meaningful code path.
The question is: how?
The most obvious idea is to use AI to analyze codebases and automatically suggest or insert instrumentation. An LLM could read your service code, identify the business-critical functions, determine which arguments and return values carry meaningful context, and generate the instrumentation code for you.
This is plausible and some teams are already experimenting with it. But it still fundamentally requires code changes. Someone has to review the AI’s suggestions, merge them, deploy them. It reduces the cognitive effort but not the operational effort. It’s a better workflow, not a different paradigm.
A more radical approach is to skip code changes entirely and tap into code execution itself. Imagine a runtime layer that observes function calls as they happen — capturing the function name, its arguments, and its return value — and intelligently emits events from that raw execution data.
The key word is intelligently. Naively capturing every function call would recreate the same noise problem auto-instrumentation has. The breakthrough would be in selective, context-aware emission: understanding which functions represent meaningful business logic vs. internal plumbing, which arguments carry user-facing context vs. internal state, and which return values indicate outcomes worth tracking.
This could work through runtime hooks, eBPF, bytecode manipulation, or language-specific instrumentation APIs — the mechanism matters less than the principle. You’d essentially be generating wide events automatically by observing what your code does rather than requiring developers to describe what it does.
The hardest part is the intelligence layer: deciding what’s signal and what’s noise at the execution level. But this is exactly the kind of problem where a combination of static analysis (understanding code structure), runtime heuristics (tracking which functions are on hot paths), and yes, AI (classifying business relevance) could converge into something genuinely useful.
The observability industry has spent years optimizing for volume and collection. The next era needs to optimize for meaning. The tools that win will be the ones that generate high-context, low-noise telemetry by default — without asking every engineer to become an instrumentation expert.
Auto-instrumentation was the right idea at the wrong level of abstraction. Wide events are the right signal at the wrong level of effort. The breakthrough lives in the intersection: automatic emission of semantically rich events, derived from understanding what your code actually does.
Whoever cracks that problem doesn’t just build a better observability tool. They fundamentally change how we understand production systems.