High-performance AI agents are distributed systems

original talk

This post is based on my SingleStore x PyDelhi talk on building high-performance AI agents in Python.

Code and artifacts from the talk are available in the pydelhi-talk repository.

The slide deck is archived here.

lets get to it

"Codex took 6 hours to implement this seemingly simple refactor".

"I think Research mode on Perplexity is stuck."

We all know LLM APIs are slow, and are content with staring at a spinner while the model slowly emits tokens.

But what happens when you're building AI agents that need to be low latency?

We hit this while building FixBugs, an AI debugging agent that reads bug reports, logs, code, screenshots, traces, and issue comments, then reproduces the bug and finally generates a validated fix.

Bug reports and their associated logs/metrics/traces can contain too much context for one model call. A repository can have hundreds of files. The final answer may need thousands of output tokens. And if the agent takes ten minutes to say anything useful, the user assumes it is broken.

Summarization, also referred to as compaction, is the usual way to work with huge context. However, summarization is slow and often loses essential context.

Modern coding agents like Claude Code and Cursor rely heavily on blindly grepping through log files and reading from specific offsets. The effective context window the coding agent is allowed to process at once is smaller than the total context window. GPT 5.5 for example has a context window of 400K tokens but it's 'input context' is closer to 258K tokens.

Once you step beyond conversational agent loops, other interesting patterns become usable.

start with token math

Most agent performance discussions start in the wrong place.

They ask:

Which model is fastest in terms of tokens/sec?

That is a useful question later. The first question is:

How many input tokens and output tokens does this task need?

LLM latency has two different pieces that matter to the user.

Time to first token (TTFT) is how long the user waits before the model starts responding.

Token throughput is a measure of how much time it takes to get the full answer.

They are not the same problem.

In the prefill phase, the model processes the input context and prepares the key/value cache used to generate the first new token. In the decode phase, the model generates output tokens one at a time autoregressively.

For a practical agent, a crude mental model is enough:

Input tokens are not free. They hit prefill and therefore time to first token.

In our case, debugging agents are usually output-heavy. They do not just answer "yes" or "no." They produce hypotheses, evidence, reproduction test cases, code diffs, and validation notes.

the 10-minute file search

The biggest bottleneck in early FixBugs was not repository parsing.

It was asking the LLM which files were relevant to a bug.

The naive version looked reasonable:

1. Gather the bug context.
2. Gather the repository files.
3. Put all relevant context into one prompt.
4. Ask the model to rank files and explain why.

For a small repo, this works.

For 50 files, this algorithm is horribly slow.

If the model emits 30,000 output tokens and the endpoint gives you 50 output tokens per second, you are waiting about 600 seconds. Ten minutes. That is before retries, rate limits, or any downstream fix generation.

To get faster performance, we realized we had to use as much parallelism as possible.

Each file was decomposed into chunks. Each chunk got its own relevance call. Those calls ran concurrently.

If one call gives you 50 output tokens per second, 16 independent calls on a 16-vCPU VM can expose roughly 16 times the useful throughput to the workflow.

This drops the file-relevance stage from roughly 10 minutes to roughly 40 seconds.

This particular problem was a "needle in a haystack" type. One that lends itself well to the MapReduce algorithm.

"Do the same analysis over many independent records, then combine the results."

This is what Hadoop does and why it is so useful for data analysis.

the "reduce" part can be complicated

Suppose you're given:

• A: A bug report.
• B: Log files and Traces.

Now you've got to extract log snippets relevant to the bug report.

Mapping a log file is simple. You can chunk it greedily.

Reducing is a bit more complex.

Without putting the snippets in sequence by time, grouping them by span, separating them by service name, the snippets would not be useful at all.

In my experience, the "Reduce" phase is often messier than people would like it to be.

The rule I use now:

Chunk for independent data units.
Merge for judgment calls.

time to first token

For interactive debugging, time to first token matters more than total completion time.

The engineer does not always need the whole final report immediately.

They need the first useful hypothesis, the first file name, progress updates.

Streaming helps.

In the demo, streaming reduced time to first token from 13 seconds to 2.4 seconds.

That is a huge UX difference.

But throughput got worse: 486 tokens/sec without streaming versus 244 tokens/sec with streaming.

Streaming can affect throughput, but it is a great user-experience optimization.

For chat-like workflows, it is usually worth it.

For batch stages inside an agent pipeline, it may not be.

FixBugs uses both modes. User-facing stages stream progress. Internal worker stages optimize for total job completion, retry behavior, and queue throughput.

concurrency has a ceiling

The first time you parallelize LLM calls and see a 5x or 10x improvement, it is tempting to conclude the next fix is "more workers."

That works until it does not.

Source

At low concurrency, the system has idle capacity. Adding workers improves utilization. Throughput rises quickly.

At higher concurrency, the slope changes. You are now fighting shared bottlenecks: provider rate limits, GPU scheduling, KV cache memory, network overhead, queueing, retries, and your own post-processing.

While throughput plateaus, TTFT gets significantly worse with concurrency beyond a point.

queues beat request chains

Once the workflow has more than one stage, a single request chain becomes fragile.

Analyze the bug. Then reproduce it. Then identify root cause. Then generate a fix. Then validate the fix.

If this runs as one synchronous chain, every stage inherits every other stage's latency and failure mode.

A slow reproduction attempt blocks root cause work.

A provider retry blocks the entire request.

This is where pipelines with event queues shine.

In FixBugs, the natural stages are:

• analysis: parse and compress the bug report and artifacts
• reproduction: reproduce the bug and write a failing test
• root cause: identify the most likely cause
• fix: generate a patch
• validation: prove the patch fixes the reproduced failure

Those stages do not all have the same workload.

Some are network-heavy. Some are model-heavy. Some are repo-heavy. Some need a sandbox. Some can run with cheaper models. Some need stronger models. Some should retry aggressively. Some should fail fast and ask for human input.

That is why the pipeline should not pretend they are one operation.

memory is a compression layer

Long-context models are useful.

However, they can hide two different problems.

The first is a performance problem. Larger prompts mean more prefill work, more tokens to move through the system, higher latency, and lower throughput.

The second is a reasoning problem. Irrelevant context is not neutral. Old hypotheses, stale summaries, repeated log snippets, and unrelated file notes compete with the evidence that matters for the current step.

A memory layer helps only if it handles both problems: send fewer tokens and preserve the facts the next stage needs.

For one demo, I tested a Mem0-style memory layer to isolate the performance side: extract facts from prior context, store them, and retrieve only the facts relevant to the current step.

Memory is not just for personalization. In agent systems, memory is a compression layer and an evidence ledger.

It decides which facts survive across steps.

model choice is infrastructure

The only way to converge on the best model is to test it on your workload.

Two models with similar benchmark accuracy can behave very differently.

One may stream quickly but produce verbose answers.

One may have great throughput but poor tool-use reliability.

FixBugs treats model choice by the pipeline stage.

Cheap model for broad relevance scans. Stronger model for root cause synthesis. Different prompt shape for reproduction. Different retry policy for fix generation. Different timeout for validation.

The mistake is using the same model, the same timeout, and the same output format everywhere.

The better question is: what does this stage need to be correct about?

A file relevance stage does not need perfect prose. It needs high recall and structured evidence.

A root cause stage needs to reconcile conflicting signals.

A fix stage needs to produce a small patch.

A validation stage needs to be conservative, because false confidence is worse than no answer.

Once you write those constraints down, model selection becomes less mystical.

the checklist

The final checklist from the talk still holds.

• Know your workload.

• Before building the feature, estimate input tokens, output tokens, expected concurrency, and whether the user needs an instant response or can tolerate asynchronous processing.

• Reduce tokens.

• Do not send full context because it is convenient. Compress, retrieve, summarize, and preserve provenance.

• Embrace parallelism.

• If the work is independent, split it.

• Microservices and queues add complexity, but they also let different stages scale, retry, and fail independently. Don't overoptimize.

• Expect failures.

LLM APIs fail. Providers rate-limit. Responses violate schema. Tool calls hang. Sandboxes break. Repos have bad tests. Treat every model call like a network call to a flaky dependency / data source, because that is what it is.

references

• NVIDIA: Mastering LLM Techniques: Inference Optimization
• NVIDIA: LLM Inference Benchmarking: Performance Tuning with TensorRT-LLM
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory