Every AI team eventually hits the same wall. The agent works. The demo impressed stakeholders. Then comes the hard question:
How do we make this good enough, fast enough, and economical enough to actually ship at scale?
This blog documents how we used AI21 Maestro, our agent optimization framework, to systematically navigate the quality–cost–latency tradeoff space of two challenging deep research benchmarks:
- BrowseComp-Plus: Tests retrieval precision and synthesis across deep corpus search tasks
- Deep Research Bench 1: Tests long-form report generation and quality
On BrowseComp-Plus we achieved SOTA performance with 95.18% accuracy.
Before we explain how we did it, here is the final destination. The chart below is a live output from Maestro on BrowseComp-Plus — a full Pareto frontier automatically surfaced from a combination of model and tool configurations, scaling strategies, and execution policies. Every point on the red curve is an achievable operating point: pick your budget, read off your accuracy. The rest of this post explains how each technique contributes, and how Maestro automates the search across all of them.
The challenge of balancing accuracy, cost, and latency in production
Anyone who has deployed an LLM-powered agent knows that the accuracy, cost, and latency of each task are not independently optimizable; improving one almost always comes at the expense of at least one of the others. Call a more powerful model and you get better answers, but your cost-per-query triples. Add a verification loop and accuracy climbs, but latency balloons. Run multiple candidates in parallel and the best one wins, but you’ve multiplied your token spend. And when scaling AI features to large user bases, getting this tradeoff right is not optional; unit economics consistently thwart AI pilots from advancing to production (RSM AI Survey, March 2026).
And yet reaching the right operating point across accuracy, cost, and latency requires searching a space that is, for practical purposes, composed of infinite permutations: model choice, prompt configuration, tool composition, agent harness design, scaling strategy, execution policies. The deeper problem with manual experimentation isn’t just that it’s slow and expensive – it’s that even when you find a configuration that works, you’ve done so by sampling only a handful of points from an enormous space. You don’t know what you missed. And when something changes – budget is narrowed or a new model is released – you have no principled way to know what to adjust or how the change will ripple through the other dimensions.
You’re back to trial and error, with no guarantee of convergence.
We built Maestro to systematically solve this search space optimization problem for any agent task. Below, we first share our experiments applying common agent optimization techniques today to BrowseComp-Plus. We then present Maestro’s solution, demonstrating the value of automatically searching across a compounded search space to identify the optimal execution path.
Optimization technique #1: Model and agent setup
The most natural starting point for any agent optimization effort is configuration: selecting the right LLM, adjusting prompts, picking tools, and tuning the agent harness. Different models carry different cost, latency, and success profiles; different prompt configurations shift which part of the problem space the agent solves well.
To make this concrete, we characterized the BrowseComp-Plus baseline landscape by running a set of representative model and tool variants and measuring their performance on the quality–cost and quality–latency axes.
– Sparse retrieval (BM25) – encodes text as high-dimensional vectors based on token occurrence. This method relies on exact lexical overlap, and is particularly effective when the query and relevant documents share vocabulary.
– Dense retrieval (Qwen2-7b) – utilize a neural encoder to project text into low-dimensional continuous vectors that capture semantic meaning. This enables retrieval based on conceptual similarity, even in the absence of lexical overlap.
– Late-interaction retrieval – using Reason-ModernColbert model which represents text as a set of contextualized token-level dense embeddings rather than a single vector. Retrieval is performed via fine-grained matching between query and document tokens, allowing for more precise alignment.
– + – indicates Get_full_doc tool – Following the commonly used benchmark scaffold, some of the agents were equipped with a tool for retrieving a full document content.
Optimization technique #2: Scaling
When agent setup hits its ceiling, the next lever is scaling: generating multiple candidate solutions and selecting the best one at run time. Scaling comes in several forms, which can be combined and applied on top of any configuration choice.
2a. Single-variant scaling (best-of-N)
The simplest scaling strategy is to run the same agent configuration N times on the same query and select the best output, sometimes called best-of-N.
This technique exploits the fact that LLM agents are not deterministic. The same model, prompted with the same query, will sometimes succeed and sometimes fail. This run-to-run variance has been well documented in the scaling literature and it means that running multiple independent candidates and selecting the best one is a principled way to improve coverage: each rollout explores a different part of the solution space, and the probability of at least one succeeding grows with k. You can see this in the figure below:
But for this to be exploited in practice, you need a way to know which candidate is the best at run time. In a closed evaluation, you can check against ground truth (an oracle). In production, you need a runtime validation signal.
While working on BrowseComp-Plus, we took advantage of the fact that the models’ self-generated confidence scores were a reliable proxy for task accuracy, when properly calibrated. This means we can use the model’s own uncertainty signal as a selection mechanism in production: pick the candidate with the highest expressed confidence, and you’re selecting with a significantly better-than-random success rate.
We show that running variants@k and using the model’s maximum self confidence score to choose the best candidate realizes the potential represented by the “oracle” curves and establishes a new Pareto frontier (the optimal tradeoff curve between accuracy and cost/latency) on BrowseComp-Plus:
2b. Ensemble scaling (model heterogeneity)
Single-variant scaling exploits variance within a single model and configuration. Ensemble scaling goes further: it draws from a portfolio of diverse models and configurations, each solving a different subset of the problem space.
Diverse models and retrievers make different kinds of errors. A query that stumps GPT-5@4 with a dense retriever might be solved by MiniMax@32 with sparse retrieval, and vice versa. The degree to which different variants complement each other – their success covariance – is itself a measurable property of the portfolio, and it directly determines how much the ensemble gains over any individual member.
This also echoes broader research findings: “More Agents is All You Need” (Tencent, 2024) demonstrated that ensemble size and accuracy scale together even when individual agents are smaller and cheaper. It’s also the thesis behind projects like Andrej Karpathy’s LLM Council.
In our BrowseComp-Plus experiments, running ensembles of top-performing configurations shows individual ensemble points sitting meaningfully higher on the accuracy axis than the best single-variant@k Pareto curves. The collective intelligence of the ensemble reaches accuracy levels that no single-model scaling strategy can match.
For example, we found that ensembling Minimax, LateInteraction, + X 8, GPT-5, LateInteraction, + X 2 and GPT-5 ,Dense X 2 can reach higher accuracy level as the best single configuration scaling (GPT-5, LateInteraction, + X 4) at significantly lower cost and latency.
2c. Execution strategies
Given that we’re now running multiple candidates – from a single configuration or an ensemble – the way we execute them becomes another optimization variable on top of the scaling choice. Execution strategies determine how candidates are sequenced, when execution terminates, and how budget is allocated across the run.
Parallel vs. sequential execution. Running all N candidates simultaneously minimizes latency but maximizes token spend. Running them in sequence and stopping when a high-confidence candidate is found, dramatically reduces cost – at the expense of higher latency in the worst case. A hybrid option is running small batches of candidates in parallel, executing batches sequentially, and stopping when any candidate in a batch meets the confidence threshold. This captures much of the latency benefit of parallelism while keeping cost closer to sequential execution.
Early stopping upon candidate selection. Once a candidate is selected – whether from a parallel batch or a sequential run – any rollouts still in progress can be terminated immediately. This avoids paying for completions that will never be used, and on queries that are resolved confidently early.
Cascading. Start with cheap models; if they fail or express low confidence, escalate to more expensive ones. The cost savings from handling easy queries cheaply more than offset the cost of the expensive model on the hard long tail.
The following graph illustrates how different execution strategies on top of the same ensemble occupy a distinct position on the accuracy–cost–latency tradeoff surface.
– Minimax + GPT-5 ensemble: Comprised of <Minimax, LateInteraction, +> X32 + <GPT-5, Dense> X4
– Sequential execution strategy: Produces the candidates in a sequence, starting from the Minimax rollouts and then proceeding to the GPT-5 rollouts, and stopping once a candidate passes the self-confidence threshold.
– Batched execution strategy: Produces 32 Minimax candidates in parallel. If at least one passes the self-confidence threshold, the highest scoring candidate is returned. If not, proceeds to produce 4 GPT-5 candidates in parallel and returns the highest scoring candidate.
Each execution strategy occupies a different position on the tradeoff surface. The optimal one will depend on the specific query distribution, deployment constraints, and customer preferences.
Optimization technique #3: Critique and repair loops
The techniques above select from independent candidate runs. A third class of optimization works differently: it iteratively improves a single answer through critique and repair, taking validation outside the inner agentic loop and making it an explicit, repeated step.
This is particularly powerful for long-form generation tasks where quality is hard to capture in a binary success metric. Here the validation produces a continuous score and structured feedback – not just a signal for stopping, but actionable guidance for the repair step. We applied this on Deep Research Bench 1, a benchmark for deep research report generation.
For each query, the process works as follows:
Step 01. Generate query-specific rubrics. Before the agent produces its report, we generate evaluation criteria specific to the query at hand. For a query like “What are the macroeconomic effects of central bank digital currencies in emerging markets?”, rubrics might include: Does the report cover monetary policy transmission mechanisms? Does it cite at least two empirical studies from different regions? Does it address the distinction between retail and wholesale CBDCs? These criteria become part of the context for the first generation step, giving the agent an explicit quality target from the outset – and they also drive the subsequent validation steps.
Step 02. Generate a research report. The agent produces an initial report using a standard agentic loop with recursive spawning of research agents.
Step 03. Validate against rubrics. A judge model evaluates the report against each rubric, producing a score and specific feedback for each criterion the report fails.
Step 04. Fix. A repair step takes the rubric feedback and improves the report, addressing the specific gaps identified.
Step 05. Repeat. Steps 3–4 run iteratively until the report achieves a satisfactory score or a budget limit is reached.
Our rubric-based scoring framework uses a similar approach to the evaluation metric of the Deep Research Bench 1 benchmark itself, and we were able to calibrate it to correlate with benchmark scores.
Using our iterative critique and repair loop, we show that quality improves meaningfully with each iteration.
Efficiently searching across an infinite solutions space
Step back and consider the space we’ve just described: model selection, prompt configuration, tool composition, scaling strategy, execution policy, ensemble composition, critique loop depth. Each dimension has multiple settings, each interacts with the others, and each combination has different effects depending on the query distribution of the specific use case.
The problem is not just that the space is large. It’s that sampling a few points tells you almost nothing about what you missed, and gives you no leverage when constraints change. You’re back to trial and error, with no map and no guarantee of convergence.
What’s needed is a method that maps the entire achievable tradeoff surface automatically, efficiently, and in a way that’s adaptable to a changing AI stack.
Automatic, efficient, adaptive: Building AI21 Maestro for agent optimization
Maestro is an agent optimization framework that automates the exploration of this space and exposes the full Pareto frontier across quality, cost, and latency for any production agent.
The system has three main components:
- Offline optimization: Action Model training. Given an agent, a problem set, a success metric, and an action portfolio (the set of models, tools, and configurations available), Maestro trains an Action Model: a probabilistic model that predicts the cost, latency, and success probability of each configuration across different queries
- Tradeoff visualization. After offline simulation, Maestro generates a full Pareto frontier across accuracy, cost, and latency for the specific workload. Teams can directly inspect every achievable operating point – not just the handful they happened to test – and select a configuration based on their deployment constraints, without needing to manually build or search for it.
- Budget- and utility-aware runtime. At inference time, the runtime dynamically composes and executes the optimal configuration per request, based on Action Model predictions, the chosen ensemble, and the selected cost/latency settings, with no retraining required. The runtime implements the full range of execution strategies: parallel and sequential execution, hybrid batching, early stopping on validation, and cascading across model tiers. Budget caps can be specified in dollars per run or per action; output token budgets are consumed in streaming chunks with early stopping to limit spillover; prefix caching is accounted for in cost estimates.
AI21 Maestro on BrowseComp-Plus
Using Maestro, we identified the optimal model portfolios and execution strategies for BrowseComp-Plus. Here’s how it worked:
After ingesting a subset of 100 examples from the BrowseComp-Plus benchmark, Maestro learned the distribution of success, cost and latency for each model in the portfolio, along with their correlations. It then simulated many execution strategies that select actions based on this distribution and on observed performance during run time; these simulations formed the basis of the Pareto frontier for our model portfolio, extending beyond what any single-variant baseline or manually constructed ensemble achieves.
Maestro reveals the full tradeoff surface and lets users choose the preferred operating point. A team that needs low latency can read off which configuration achieves the highest quality within their specific constraint. A team with a strict per-query cost budget can do the same. The result is a clean answer to the question every team is really asking: for a given cost or latency budget, what’s the best accuracy we can achieve — and vice versa. As new models release or query distributions shift, the frontier is recalibrated and the available operating points are updated – no manual re-experimentation required.
For example, a user that is looking for the operating point that achieves above 90% accuracy with the lowest possible cost may select the highlighted point on the curve with avg accuracy = 93.9%, avg cost = $0.96, avg latency = 532s. The following graph is presented by Maestro as an example of an execution that ran under this policy:
Informed by the action model’s predictions and the user’s preferred operating point, Maestro chose to run <GTP-5, LateInteraction, +>, <GPT-5, Dense> and <Minimax, LateInteraction, +> X3 in parallel. Since none returned a confidence score above the 90% threshold and the budget was not fully consumed, it decided to continue to generate another batch of rollouts. Runtime telemetry from the first batch was returned to the action model for re-callibration and based on its new predictions Maestro chose to run <Minimax, LateInteraction, +> X5 in parallel. After the third wave, given the budget constraint, Maestro decided to stop the execution and return the highest scoring candidate – from the <GTP-5, LateInteraction, +> rollout in the first wave – which in fact solved the query accurately.
In the figure below, we show that the new pareto frontier generated by Maestro leads to significant accuracy, cost and latency gains compared to the best single-variant scaling <GTP-5, LateInteraction, +>.
Maestro on DeepResearch Bench 1
The budget-aware runtime is especially visible in the Deep Research Bench 1 results, where the critique-and-repair loop must be managed within cost constraints. Maestro’s runtime knows when to stop refining: additional iterations are only executed when the marginal quality gain justifies the marginal cost, given the selected operating point.
A cost-sensitive deployment can operate at lower iteration depth; a premium research product can run deeper critique loops. The operating point is a configuration choice, not an engineering effort.
Agent optimization as a first-class engineering challenge
Agent optimization is not a one-time configuration choice, but an ongoing practice of navigating a high-dimensional space under evolving constraints.
The building blocks are real and powerful: model and agent tuning establishes a baseline frontier; best-of-N scaling with confidence-based selection extends it; ensemble heterogeneity pushes it further; critique-repair loops unlock quality levels that generation-only approaches can’t reach; and execution strategy determines exactly where on the tradeoff surface you land. But these techniques compound on each other into a search space that no team can navigate manually with confidence.
Maestro automates that navigation. By training an Action Model to predict cost, latency, and success probability across the full configuration space, and coupling it with a budget-aware runtime that executes the optimal strategy per user requirements, it turns agent optimization from an art into a repeatable engineering practice.
Want to learn more? Explore Deep Research Preview on AI21 Maestro →