Introducing NeoEval: The Eval Engine for your Production Data › NeoSigma

A user asked one of our customers' agents to make a file of a set length from their data. The agent made a file twice as long. It called the right tool, created the file, saved it and reported success. It was one of 928 sessions that week.

Most agent failures in production look like this. The agent doesn't crash. It does almost the right thing, reports success, and leaves the user to fix the rest. Today, teams find these failures by reading a few sessions with Claude Code or Codex, changing a prompt or a skill, trying the change on a few examples and shipping. Most evals send a prompt to a model and check the answer against a rubric. An agent works across many turns, calls tools, writes files and uses integrations, and the failures happen in those steps.

Your production data is a goldmine of real tasks, failures, and human corrections. It captures how people work, where agents fall short, and what a successful outcome requires. Over time, those corrections accumulate into institutional knowledge: the judgment your team has developed through experience.

NeoEval, our flagship evaluation engine, turns that experience into evaluation environments grounded in your real tasks, workflows, and edge cases. A failure becomes a task you can replay, with a clear check for what went wrong and what success looks like.

These evals provide a test and training bed for your agent. They turn your team’s judgment into reward signals for success, giving you a way to evaluate performance and optimize the agent against the outcomes that matter to your business.

NeoEval finds these failures in your production traces and turns them into evaluation environments. Each eval runs your real agent in the same production environment, with a simulated user-agent interaction and your own verifiers. On one week of a customer's production traffic, the evals NeoEval built covered 42 of the 51 agent failures (82%) from the two weeks before.

One of our customers runs an agent that answers its users' questions about their data and makes reports, charts and other files from it. We ran NeoEval on one week of that agent's production traffic, 928 sessions in all. In 80 of them, the user didn't get what they asked for. 67 of these were the agent's own mistakes, and the other 13 were infrastructure failures outside the agent. The 67 agent failures came down to 8 problems:

• Linked a file that wouldn't open (25 sessions)

• Returned a template instead of the finished file (13)

• Left placeholders where numbers belong (12)

• Made claims the data doesn't support (6)

• Ignored an explicit instruction (4)

• Couldn't explain missing data (3)

• Answered in the wrong language (2)

• Refused a how-to question about the product (2)

Some of these problems showed up in more than one tool, and each tool gets its own eval, so the 8 problems became 16 evals.

Evals only help if the agent repeats its mistakes, so we measure coverage: of the failures your agent makes, how many would the evals catch?

To measure coverage fairly, the failures have to be ones the evals were not built from. So we checked the 16 evals against the 51 agent failures from the two weeks before. A failure counts as covered when one of the 16 evals uses the same tool and checks for the same problem, so the eval would fail if that failure happened again.

The failure from the start of this post is a good example. Its eval covers three of the earlier failures, each a file that came out longer than the user asked for. They came from different users with different requests, but it was the same tool making the same mistake each time.

In all, 42 of the 51 earlier failures were covered.

An eval is a test of whether your agent completes a real user's task. Each one is a task folder with four parts:

• Environment: your repo at a commit, with your agent's databases and third-party integrations booted up in a sandbox.

• Task: the user's original request from a production session.

• Simulated user: a persona with a goal and constraints, who talks to the agent until the task is complete or a turn limit is reached.

• Verifiers: the rubrics you run on production sessions to grade the agent's behavior.

For each session, NeoEval reads three things:

• The trace: every message, tool call and tool result

• The results of your verifiers, the checks you already run on your agent. They can be LLM judges, code or agents.

• Observations: our own analysis of how the agent behaved, which catches problems that no single verifier checks for

Verifiers and observations flag the sessions that may have gone wrong. For this customer they flagged 383 of the 928 sessions, and NeoEval kept the 80 that were real failures. It then groups failures of the same kind, the same problem in the same tool, and builds one eval for each group. The 25 sessions where the agent linked a file that wouldn't open became one eval for each tool that made a broken file.

Each eval gets a fresh sandbox. Your repo is cloned at the commit under test, and your agent's environment is booted up, including its databases and third-party integrations. Your harness starts inside the sandbox with its real tools, integrations and model calls. The original request goes to your harness, and the simulated user replies to the agent the way a real user would, over as many turns as it takes. The conversation ends when the simulated user is done or a turn limit is reached. Then the verifiers grade the whole run, including every message, tool call and tool result. The result is pass or fail. A fail on an eval that passed before is a regression.

Every eval in a run uses the same commit and container, and each one starts in a clean sandbox, so results from two commits can be compared directly.

When you edit a prompt, swap a model or change a tool, run the evals on the new commit. The evals built from past failures show whether those failures are fixed, and the rest show whether anything else regressed. Changes that look like obvious fixes often aren't. Hex found that "prompt changes weren't nearly as impactful as we expected they would be. The vast majority of pure prompt tweaks failed our evals."

In the example above, the change fixes the broken file but makes the report longer than one page. You would revise the change and replay it before a user finds the problem.

1. 1.

   Send your agent's traces to NeoSigma with our SDK.

2. 2.

   Connect your existing verifiers.

3. 3.

   NeoEval reviews each session, finds the real failures and proposes evals.

4. 4.

   Run the evals on any commit to check for regressions before you push to production.

5. 5.

   Use these evaluation environments directly to build your own custom intelligence for model training and harness optimization.