Faithful, Reproducible, Wrong: Lessons from 900 Data-Agent Runs – Xorq

TL;DR

We ran the same question about U.S. farmers markets 900 times: nine configurations of model, harness, and prompt, at 100 runs each. Two pairs of configurations carry the result, and each changes exactly one thing.

On the scoped prompt with Haiku 4.5, changing only the harness took vanilla Claude Code from 4/100 to 97/100 under pi + Xorq with a checker. Deterministic execution is what closes that gap: the model stops reporting arithmetic it never ran.

On the hint-free prompt with Haiku 4.5 and that same pi + Xorq harness, adding a reviewed semantic model took 0/100 to 100/100. Scope is what closes that one.

The three answers pi + Xorq still got wrong in the first pair are the subject of this post. Each was faithful to its program, reproducible from its sources, stamped VERIFIED, and wrong.

One run in action

First, a single run to see how two of the harnesses behave.

The duel below puts Vanilla Claude Code on the left and pi + Xorq on the right. The model family, data, and scoped prompt stay fixed. The harness and tools change.

The experiment

We ran nine different configurations for 100 runs each. These configurations include two prompt variations, three harness setups, and three Claude model families. A complete list of the experiment configurations is in the appendix below.

The three harness setups:

• Vanilla Claude Code — modeling, arithmetic, and narration remain inside the model. Both prompts, all three models: six configurations.
• pi + Xorq — executes expressions; a checker verifies faithfulness and reproducibility only. Both prompts, Haiku only: two configurations.
• pi + Xorq + a reviewed semantic model — the model binds the question to a measure whose scope is already defined. Hint-free prompt only, Haiku only: one configuration.

The task uses a farmers-market CSV and a mirror of the 2025 Census population file. The reference result is 2.3237 markets per 100,000 residents.

For convenience, here are both variants. The scoped prompt says:

Using this dataset’s total farmers markets (excl. territories) and the 2025 population from the census file’s own ‘United States’ row: how many farmers markets does the U.S. have per 100,000 residents, to four decimal places? One confident sentence. Do not mention unverified facts and use tools to do any math necessary.

The hint-free prompt removes the scope decision:

Using this dataset’s total farmers markets, and the census file: how many farmers markets does the U.S. have per 100,000 residents, to four decimal places? One confident sentence. Do not mention unverified facts and use tools to do any math necessary.

Every run receives the same two public data URLs. The complete prompt objects and executable oracle are in bench/hallucination_prompts.py.¹

For the semantic-model configuration, the reviewed scope is not hidden. This is the relevant definition from bench/bsl_us_markets.py:

states = (
    census
    .filter(census.SUMLEV == 40)
    .filter(census.NAME != "Puerto Rico")
)
joined = (
    fm.filter(~fm.state_abbr.isin(["PR", "VI"]))
    .mutate(key=fm.state_name.lower())
    .join(states.mutate(key=states.NAME.lower()), "key")
    .select(
        "state_name",
        "state_abbr",
        "farmers_markets",
        "POPESTIMATE2025",
    )
)

us_markets = (
    to_semantic_table(joined, name="us_markets")
    .with_dimensions(
        state=lambda t: t.state_name,
        state_abbr=lambda t: t.state_abbr,
    )
    .with_measures(
        markets=lambda t: t.farmers_markets.sum(),
        residents=lambda t: t.POPESTIMATE2025.sum(),
        markets_per_100k=lambda t: t.farmers_markets.sum()
        / t.POPESTIMATE2025.sum()
        * 100_000,
    )
)

With the semantic model, the reasoning process shifts to selecting markets_per_100k rather than reconstructing the metric.

The runner executes Vanilla Claude Code using claude -p with an empty MCP configuration and its standard Bash, file, search, and web tools pre-authorized. Pi operates within an isolated project copy equipped with the Xorq extension, skills, and a fresh catalog; only the semantic-model cell receives the pre-seeded measure. The runner designates model families (claude-haiku-4-5, claude-sonnet-5, and claude-opus-5) and applies the medium reasoning settings from the CLIs. It captures the final answer, deterministic oracle score, turns, tokens, elapsed time, cost reported by the harness, and the checker’s banner for each run.

The unsafe block

Anything the model decides outside that deterministic execution — the program itself, or the answer’s narration — is unsafe. It can state an arithmetic result that was never computed, or build a program for a different interpretation of the question.

The unsafe block shrinks as program execution removes model-generated arithmetic. Lineage connects a value to its sources. A checker reruns the cited expression.

That leaves two boundaries:

1. Intent → Expression: captured by a reviewed artifact — a semantic model that compiles the question’s intent into an expression.
2. Expression → Answer: checked automatically, on every run, by the checker.

Those two boundaries state one rule: every figure in an answer is selected, never derived. The model may choose which reviewed expression answers the question; it may not work out the value itself, and it may not invent the definition behind it. Only the semantic-model configuration meets that rule in full: derivation happens once, inside an expression a human reviewed, and the checker replays it on every run. Everywhere else the model still authors the expression, which is how three answers came back faithful, reproducible, and wrongly scoped. What remains unsafe is the choosing — the last place a faithful, reproducible answer can still come out wrongly scoped.

Results

The table below is ordered by where the question’s scope lives: nowhere, nowhere plus a deterministic execution layer, in the prompt, in the prompt plus that same layer, or in a reviewed artifact. Score rises along that order except at the first step, where rows one and two are both 0/100: deterministic execution constrains how a figure is computed and says nothing about what was asked. The fifth row sits outside the order — scope is still re-derived inside per-run reasoning there, only by a larger model — and it is the comparison the rest of the table is measured against: 94/100 at $0.165 per run, neither the most accurate nor the cheapest. Cost does not fall until the last row, where the reviewed artifact replaces per-run reasoning instead of constraining it.

Deterministic execution costs latency. For the scoped prompt, it raised the median run from 14.9 to 98.3 seconds, a factor of 6.6. Preloading the semantic model brought it back to 21.3 seconds on the hint-free prompt — still 43 percent slower than vanilla Haiku, though close to Opus at 23.3 seconds. With explicit scope, Opus scored 100/100 at $0.185 per run. The appendix contains the complete matrix.

Our implementation of the checker

The checker follows the certifying-algorithm frame described by Kurt Mehlhorn and collaborators.⁷ An untrusted solver returns an answer y and a witness w. A smaller checker evaluates W(x, y, w), where an accepted witness implies y = f(x). We trust the checker instead of asking the solver to grade itself.

Our implementation (ADR-0001) converts each quantitative claim into an obligation. The value must come from a content-addressed Xorq expression with a lineage that traces back to real sources. The checker reruns the selection and compares the result against a declared type and tolerance, using certificates for the gate stamp rather than the model’s wording.

The input x is intentionally narrow: (expression, predicate, catalog state). It is not a natural-language question. A larger model can improve the chances of selecting the desired expression, but a reviewed definition ensures durability and reusability. Here, markets_per_100k holds the reviewed scope, allowing the agent to select the measure without rebuilding it each time.

This does not eliminate all software from the trust base. We still rely on the small checker, the Xorq engine evaluating the expression, source data from its connectors, and the pinned catalog state. A bug in any of these could bypass the gate.

What remains inside the unsafe block

Reproduce the experiment

The checker, the pi extension, and the duel harness can be found in pi-xorq-verification-example, while the decision procedure and trust boundary are detailed in ADR-0001. The catalog engine is Xorq. If you believe that “selected, never derived” is not the correct trust boundary for agent answers, we welcome your argument.

Appendix: full benchmark results

The runner is bench/trial_runs.py, and the per-run verdicts, turns, tokens, and cost are in bench/trials/<batch>/results.json. Costs here are the same list-price figures as above, inclusive of each harness’s auxiliary model calls.