GitHub - Thomeras/agent_detective: An eval framework for multi-agent systems that names the culprit: OTEL traces in, execution graph out, blame analysis on top.

Agent Detective

🧪 Beta — the analysis is real, its wording is not frozen

Classifications still move between minor versions (0.2.0 reclassified what 0.1.0 called loop_detected), so gate CI on the exit code — 0 clean, 1 incident, 2 could not run — never on an exact report_type.

An eval framework for multi-agent systems that names the culprit. Ingest standard OpenTelemetry traces, rebuild the execution graph, and find the first node where quality broke — the origin, the propagation path, and the downstream cost. OTEL-native: any OpenInference / OpenLLMetry instrumented agent works with no code change.

pip install agent-detective
detective analyze trace.json

── graph 3f2a91c8  [content-pipeline]
   FAILED  ·  cut_point  ·  confidence 62%

   Origin — where quality broke
     translator

   Defects
     ● Contract breach — translator
       A carried input/output parameter was silently rewritten at translator.
       observation 100% · attribution 92% · channel deterministic

Docs

Quickstart — one run, zero infrastructure

flowchart LR
    AG["your agent"] -- "OTLP/HTTP JSON" --> CAP["detective capture :8900"]
    AG -- "AGENT_DETECTIVE_TRACE_FILE" --> F["run.json"]
    CAP --> AN["detective analyze"]
    F --> AN
    AN --> V["verdict · exit 1 on incident<br/>terminal · --json · --markdown"]

detective capture --once --out run.json # receive a trace straight from your agent
detective analyze run.json              # the verdict; exit 1 on an incident (CI gate as-is)
detective analyze run.json --markdown   # findings brief for a coding agent
detective doctor run.json               # is the trace even worth trusting?

Already instrumented? Point the exporter at it: OTEL_EXPORTER_OTLP_PROTOCOL=http/json OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:8900. Not yet? detective-sdk (pure stdlib, zero dependencies) emits the same spans:

from detective_sdk import run

with run("intel", task=user_request) as r:
    with r.step("write") as s:
        s.output = dossier_markdown            # the work, not {"ok": true}
        s.cost(usd=0.03, tokens_in=8_000, tokens_out=900, model="gpt-4o")

No database, no broker, and by default no LLM: the deterministic evidence channel needs nothing but the trace. Set JUDGE_BASE_URL / JUDGE_MODEL (any OpenAI-compatible endpoint — OpenAI, OpenRouter, local Ollama; concrete configs in docs/usage.md) to also turn on the per-node quality judge — without it, nodes report unscored, never silently "fine".

Example: catch a fault in a diamond topology

examples/diamond_eval.py instruments a diamond — one extractor, two parallel writers, one editor — and gates on the verdict. Each writer declares the facts it was handed as a contract; --inject makes the marketing branch silently rewrite the price:

with r.branch("marketing_writer", input=facts) as s:
    s.contract(price=facts["price"], availability=facts["availability"])
    s.output = write_marketing(facts, inject)    # --inject rewrites the price

$ python examples/diamond_eval.py --inject
graph 0a68bb06: cut_point — culprit: marketing_writer    # exit code 1

The verdict names the branch that rewrote the fact — not the editor who merged it downstream — and verifies the breach actually shipped, all with no LLM:

FAILED  ·  cut_point  ·  confidence 95%

Origin — where quality broke
  marketing_writer

● Contract breach — marketing_writer
  supporting: contract_breach (rule input_contract:price)
  supporting: breach_propagated at terminal (contract-propagation check on the deliverable)

Deterministic signals
  fail contract_violation (marketing_writer) price: $12/user/month → from $5/user/month

Point the same script at the full stack and the graph appears in the web UI — culprit ring on marketing_writer, propagation path into press_editor:

AGENT_DETECTIVE_ENDPOINT=http://localhost:8001 python examples/diamond_eval.py --inject

Full stack — continuous ingest, inbox, history

docker compose up --build                 # infra + ingest + worker + api + web
./demo/run.sh                             # happy-path demo: graph, no incident
./demo/inject_fault.sh && ./demo/run.sh   # a cut_point incident appears

Web UI at :5173, read API at :8000, ingest at :8001; the bundled mock LLM judges, so no external API keys are needed.

The self-hosted stack has no authentication yet. Every host binding is therefore on 127.0.0.1 by default — the UI and API, and Postgres, ClickHouse, Redis and MinIO with them. What sits behind those ports is your traces' payloads: agent inputs and outputs verbatim, which is exactly the material you would not want public. Reaching the stack from elsewhere is one deliberate knob, and only belongs behind something that authenticates:

BIND=0.0.0.0 docker compose up --build

Authentication is not on the roadmap as something this project will grow on its own — put it behind a reverse proxy that already does it.

flowchart LR
    AG["your agents"] -- "OTLP/HTTP" --> ING["ingest :8001"]
    ING --> CH[("ClickHouse<br/>raw spans")]
    ING --> PG[("Postgres<br/>graphs · runs · edges")]
    ING -.->|"payload overflow"| MIO[("MinIO")]
    ING -- "Redis Streams" --> T1["worker tier1<br/>cheap checks + terminal judge"]
    T1 -- "flagged / sampled" --> T2["worker tier2<br/>per-node scoring + judge"]
    T2 --> BE["blame engine<br/>(pure networkx)"]
    BE --> INC[("incidents +<br/>blame reports")]
    INC --> API["read API :8000"]
    API --> UI["web UI :5173"]

The report separates where quality broke (origin) from where it surfaced (manifestation), flags rubber-stamping verifiers, and reports capped, split confidence (observation × attribution). What the trace did not capture, no analysis can manufacture — absent evidence renders unverified, never ok.

Runtime vs. analysis — how the pieces fit

Your agent's process needs only detective-sdk (25 KiB, zero dependencies): it emits the trace and never computes anything. The verdict happens wherever agent-detective is installed — your laptop, CI, or the deployed stack — and that one install brings the whole analysis side with it (otel-mapper span mapping, agent-detective-worker pipeline, blame-engine verdicts, the CLI). detective-ci is a separate, opt-in install because it registers a pytest plugin. Details per distribution: docs/usage.md §4.4.

Repository layout

packages/
  blame_engine/    pure, I/O-free blame analysis (networkx only)
  otel_mapper/     OTLP span -> AgentRun/Edge mapping
  detective_cli/   the `agent-detective` pip distribution: local mode + CLI
  detective_sdk/   zero-dependency instrumentation helpers
  detective_ci/    deterministic golden replay + pytest plugin
services/          ingest · worker (tier1/tier2 + judge) · read API
db/                Alembic migrations        docker/clickhouse/  ClickHouse init
web/               React + Vite + cytoscape UI

Development

uv sync --all-packages --all-groups   # install the workspace
./scripts/test.sh                     # every unit suite (mirrors CI)
uv run pytest tests/e2e               # acceptance test (needs a running stack)

for p in blame-engine otel-mapper detective-sdk detective-ci \
         agent-detective-worker agent-detective; do
  uv build --package "$p"             # the pip distributions
done

License

Apache-2.0 — the whole repository (see LICENSE). Run it, fork it, ship it inside your product, or operate it for other people.

It was Business Source License 1.1 until 2026-07-31, relicensed for better accessibility. There is no paid tier and no hosted edition.

Contact

Questions or collaboration: tomje11@seznam.cz.

A trace that produces a wrong verdict is the most useful thing you can send — see CONTRIBUTING.md.