Perhaps the most interesting / ambitious thing that we are working on at tuneloop.io right now is on offering the right level of model intelligence for the right task, seamlessly. We are starting with coding agents / engineering workflows, since that's where companies are spending enough for this question to be pertinent at the moment.
Before we can build it, there's a set of interlocking questions we need to answer. This post is the map. Future posts in this series will cover our experiments, findings and design decisions we make along the way.
Q1. What does success mean?
Operating at the efficiency frontier given the current set of models.
What are we optimizing for?
Generally, task success rate and cost per task. Task success rate is the fuzzier of the two, and needs to be thought out carefully for each dataset.
What is a task?
A unit of work that fits into a taxonomy, is defined by a prompt, and has a success criteria. The task taxonomy is defined by a combination of a few attributes, such as task type, complexity and domain, or learned clusters. This opens up the next set of questions around calculating these attributes / clusters. Keywords / calibrated LLM classifiers / embedding models are a few common options.
How do we define success criteria?
Just use LLM as a judge with the agent trace? Or carefully defined criteria for each task type? For example: should there be a merged PR for each implementation task given to a coding agent? Or a threshold on the number of addressed review comments for a code review agent? Or just use LLM as a judge with all the hard evidence passed in whenever available?
Where do we source tasks from?
Datasets are private and specific to each enterprise. Success metrics are only meaningful if calculated on tasks that are representative of a company's specific workload. Two main sources:
- Generated artifacts: Merged PRs, posted code reviews, documents written. Pros: well-defined success criteria, easy to experiment on public data. Cons: low coverage — not all tasks result in one of these artifacts, not all artifacts can be generated one-shot.
- Agent traces: Agent session transcripts from Claude Code, Codex, Cursor, or custom harnesses. Pros: high resolution data on how tasks get done, 100% coverage by definition. Cons: definition of success is fuzzy.
Q2. What could a solution look like?
Three patterns worth exploring:
- Explicitly designed workflows, built in to harnesses, or otherwise. Claude Code's advisor pattern, and appropriately configured orchestrator/subagents are some examples.
- Model routing at the gateway/proxy layer: start simple, then escalate as needed, or define default models for different tiers. Routing decisions have to be cache aware. A lot of existing options here.
- Fine-tuning / training a domain specific model or set of models? How small can we go here? How many teacher model trajectories do we need?
What is the delta that we see in practice from each of these techniques?
Q3. How do we evaluate these techniques on our task dataset?
Two main components:
Task Runners
The ability to run each task in our dataset with our chosen set of parameters (with/without a harness config, model routing, or different models). Harbor is probably the best available framework for this purpose currently.
- How do we simulate multi-turn / interactive sessions, especially given that trajectories can diverge from the original? Do we have a user simulator agent? Or do we define tasks such that user input is minimal, and treat our tasks as one-shot agent runs?
- How do we prevent data leakage? Especially in cases where the agent accesses enterprise state via MCP / CLI tools? What are the right sandboxing parameters?
Verifiers
These are the graders that look at an agent's output and confirm that requirements are met. There's a variety of choice here:
- Pass-to-pass and fail-to-pass tests for tasks with PRs.
- Scorers that penalize the agents for non-maintainable code
- Scorers that compare the generated artifact to the original for correctness and completeness.
Taken together, we need to be able to create datasets from enterprise repositories and agent traces, evaluate solutions, and see metrics (success rate and cost per task) for each solution. In addition, we need to be able to inspect agent runs to understand the source of errors and take corrective action.
We are working our way through these questions, using a combination of public datasets and our internal data. For context on where large teams are on this problem, see Databricks' internal benchmark post and Uber's efficient software factory writeup. Next in this series: the first experiments and what we learn from them. If you're interested in us running these experiments on your data, reach out at founders@tuneloop.io.