Analyzing Jev Outcomes with Validator

Validator logo embroidered on a fabric label

Find the errors. Measure what changes.

System 1 decision models like Jev run inside a workflow. Think of them as cognitive cogs in a deterministic machine of source code. How do we know where and when these decision model outputs are wrong, and how can we improve them?

I built Validator for that job. It is a CLI that compares reviewed expected labels with saved classifier outcomes. It works with single-label and multi-label classifiers, including Jev workflows.

Bring your data. Pair expected labels with your workflow’s saved predictions.
Find the mistakes. See missed labels, extra flags, and the cases behind them.
Measure what changes. Compare runs after adjusting the workflow.

Worked example: Evaluating a code-review workflow

87.1% micro-F1. Half the security findings missed. These 20 synthetic reviews show how Validator exposes a weakness behind the average.

Validator's synthetic code-review example: 87.1% micro-F1, but only 50% security recall, with three of six expected security findings missed.

Synthetic example, not measured Jev performance. Figure based on Validator’s JSON results.

Command-line interface

Run your classifier, save its predictions, and evaluate them against your reviewed labels:

validator evaluate \
  --dataset data/golden.json \
  --predictions data/predictions.json \
  --config validator.json \
  --out runs/current

After a change, compare the saved reports:

validator compare \
  runs/baseline/report.json \
  runs/current/report.json

See the Validator README for the input formats and full example.