GitHub - Handshake-AI-Research/gandalf-the-grader: Agent-as-a-Judge grading framework for evaluating AI outputs/deliverables

Your verifier is probably the bottleneck. We built one that isn't.

Read the launch blog post for the motivation, benchmark results, and design rationale behind Gandalf.

Gandalf is a reactive agent-as-judge for rubric-graded agent environments. Given a rubric of binary criteria, it runs inside the rollout environment, uses the same tools as the rollout agent, and decides at inference time which files to open and which tool state to query.

That lets Gandalf grade criteria that depend on artifacts or state — formulas in a workbook, charts in a deck, files on disk, MCP tool state, or whether an email was actually sent — rather than just the final text response.

Gandalf is built around three design choices:

• Environment alignment: Gandalf runs in the same filesystem, Python interpreter, installed packages, and tool environment as the rollout agent, using the OpenHands SDK as the agent harness.

• Reactive verification: Gandalf chooses what evidence to inspect while grading, instead of relying on a precomputed transcript or serialized snapshot.

• Swappable domain guidance: Domain knowledge enters as natural-language guidance at runtime, making the same verifier portable across domains.

In our evaluation, this design beat text-only, snapshot-based, and workflow-based agentic verifiers at a fraction of the cost — see the blog post for the full meta-eval.

Examples and integrations: BankerToolBench is a public agentic RL benchmark environment that uses Gandalf as the verifier. rle-pkg is a reference runtime that integrates Gandalf. Both run under the Harbor framework, but Gandalf's design and implementation are framework-agnostic.

Installation

Gandalf is published on PyPI.

uv tool install gandalf-the-grader

For production use, we recommend that you pin a specific version of Gandalf, and furthermore use the [pinned] version to pin all transitive dependencies.

uv tool install 'gandalf-the-grader[pinned]==1.0.0'

Runtime dependencies

Important: Gandalf is built on top of OpenHands, which works best when tmux is installed. The judge refuses to run when tmux is not on PATH rather than silently falling back to a less stable subprocess-based terminal.

Quick start

The repo ships a runnable example under examples/quickstart/ that grades a pre-staged workspace + ATIF trajectory against a 3-criterion rubric. Two criteria are designed to be met and one is designed to fail, so you can see Gandalf's partial-credit grading and per-criterion reasoning in one run. From a fresh clone:

# 1. Install
uv tool install gandalf-the-grader

# 2. Provide a Gemini API key (any litellm-compatible model works; see Configuration)
export LLM_API_KEY="<your-gemini-api-key>"

# 3. Run from the repo root
gandalf-the-grader --config examples/quickstart/grader.toml

# 4. Inspect the result
cat examples/quickstart/output/reward.json   # -> {"reward": 0.75}
cat examples/quickstart/output/info.json     # per-criterion verdicts + reasoning

Expected verdicts: the welcome.txt file exists (met), the message mentions Gandalf (met), and the message is not longer than 50 words (unmet, by design). Raw score 3.0 of a possible 4.0, for a reward of 0.75.

The example uses gemini/gemini-2.5-flash and runs the inner judge as the current user (no sandbox_user, no sudo). To adapt it to your own setup, edit examples/quickstart/grader.toml. See the Configuration section below for the full field reference.

Configuration

grader.toml

MCP servers can be configured as TOML array of tables:

[[mcp_servers]]
name = "magic-server"
transport = "stdio"
command = "/usr/bin/mcp-server"
args = ["--verbose"]

Custom Judge Prompt

By default, the grader uses a built-in prompt template to kick off each judge session. judge_prompt / judge_prompt_path let you replace it entirely with a custom Jinja2 template.

Note: This prompt is sent as the opening user message to the judge agent, not the LLM system prompt. The underlying agent framework (OpenHands) has its own immutable system message with coding and tool-use instructions that we never modify. Our prompt sits on top of that as the first user turn, setting up the grading task.

For most use cases, judge_guidance / judge_guidance_path is all you need: it injects extra instructions into the built-in prompt without replacing it. Fully overriding the judge prompt is an uncommon escape hatch for situations where the built-in prompt structure itself is unsuitable.

The template receives these variables:

Individual and batch modes use separate built-in templates. In a custom template, use {% if criterion is defined %} vs {% if criteria is defined %} if you need to distinguish modes. In batch mode, use loop.index0 for the criterion index (e.g., {% for c in criteria %}[{{ loop.index0 }}] {{ c }}{% endfor %}).

Rubric JSON

A JSON array of objects with criterion (string) and weight (float). Weights can be negative to penalise undesired outcomes:

[
  {"criterion": "The output file exists", "weight": 2.0},
  {"criterion": "The output contains correct totals", "weight": 3.0},
  {"criterion": "The agent used hardcoded values instead of computing", "weight": -1.0}
]

• Positive weight: adds to the raw score when the criterion's condition is met
• Negative weight: deducts from the raw score when the criterion's condition is met (the bad thing happened)
• The judge evaluates each criterion on its own merits; it never sees weights

Trajectory Format (ATIF)

The grader reads agent trajectories in Agent Trajectory Interchange Format (ATIF). An ATIF file is a JSON object with a steps array:

{
  "steps": [
    {"source": "user", "message": "Build a hello world web app"},
    {"source": "agent", "message": "I'll create the file now", "tool_calls": [...]},
    {"source": "agent", "message": "Done! I created index.html with a Hello World page."}
  ]
}

The grader extracts the final agent message (last "source": "agent" step with a non-empty message and no tool_calls) and passes it to the judge as context.

Environment Variables

Tracing / Observability

Gandalf builds on top of OpenHands, which has built-in OpenTelemetry tracing that automatically instruments LLM calls, tool executions, and agent steps. Set the OTEL_EXPORTER_OTLP_* variables above to export traces to any OTEL-compatible backend with no code changes required.

Example: Langfuse

# Encode your Langfuse keys
echo -n "pk-lf-...:sk-lf-..." | base64

# Export the variables
export OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel/v1/traces
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<base64-encoded-keys>"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf

Output

The grader writes to output_dir:

• reward.json: Reward file (e.g., {"reward": 0.75}) (always in [0, 1]). Only written when all criteria are successfully evaluated. If any criteria still have errors after retries, the grader writes info.json but skips reward.json and exits with code 1.
• info.json: Always written. Per-criterion results with met/not-met, reasoning, evidence, LLM usage, plus reward, raw_score, minimum_score, maximum_score, errored_criterion_count, and evaluated_criteria_pct.
• judge_trace_*.txt: stdout/stderr capture for each judge invocation. Naming varies by mode: judge_trace_{i}.txt (individual), judge_trace_batch.txt (batch), judge_trace_batch_split{i}.txt (batch with splits). Retries append a _retry{N} suffix.

The reward in reward.json is clip(0, 1, raw_score / sum_of_positive_weights), always in [0, 1]. info.json additionally includes raw_score (the raw sum of weights for met criteria, which can be negative) and minimum_score/maximum_score bounds for reference.

Next steps

• Try the benchmark environment. BankerToolBench on Hugging Face is the public RL environment that Gandalf was originally evaluated against. Clone it, run rollouts, and grade them with Gandalf.
• Adapt Gandalf to a new rollout environment. Edit examples/quickstart/grader.toml to point at your workspace, trajectory, and rubric. See the Configuration and Custom Judge Prompt sections for the full reference, including domain-specific judge guidance.

License

Copyright (c) Handshake. Released under the Apache-2.0 license. See LICENSE.txt for details.