Infrastructure @EvalEval
A unified, open data format and public dataset for AI evaluation results. We are collecting all evaluation results in a standardized schema to enable rigorous research and broader impact.
Developed by the EvalEval Coalition
with feedback from researchers at
Why this Schema?
Addressing the fragmentation in AI evaluation to enable trust and comparability.
Unifying the Ecosystem
Evaluation results are currently siloed by framework. This schema creates a common interchange format, allowing results from HELM, EleutherAI, Inspect, and custom scripts to co-exist and be compared directly without complex mapping.
Trust through Provenance
A score without configuration is just noise. We go beyond the metric to capture the full experimental context—prompt templates, inference parameters, and system states—making every result traceable, transparent, and reproducible.
Actionable Meta-Science
Liberating evaluation results from static PDFs and closed leaderboards. We transform scattered metrics into a structured, queryable global dataset, powering the next generation of meta-analysis and automated leaderboard construction.
The Schema
A granular, line-by-line breakdown of the standardized format.
Design Decisions
Built for scale, reproducibility, and scientific rigor.
Conflict-Free Identity
We assign every evaluation a unique UUID. This prevents filename collisions and allows multiple runs of the same model—from different dates or configurations—to coexist safely.
Temporal Versioning
Models change silently. We mandate retrieved_timestamp to capture the exact moment of inference, enabling precise studies on API drift and model versioning.
Full Stack Provenance
Performance depends on the runner. We explicitly separate platform (provider) from engine (inference system), isolating hardware and software variables in your analysis.
Unified Metrics
From simple accuracy to complex LLM-as-a-Judge scores, our schema standardizes all outputs. Compare results across different evaluation libraries without writing custom parsers.
Analysis-First Design
Built for data science. Flat, structured JSON files mean you can ingest millions of results into Pandas or SQL in seconds, slicing by architecture, date, or task immediately.
Reproducibility Standard
Science requires receipts. By linking every score to its exact prompt template and generation parameters, we ensure that every result in the dataset is fully reproducible.
The Public Dataset
We are collecting all evaluation results in our schema in a public dataset. This repository serves as a standardized metadata store for results from various leaderboards, research papers, and local evaluations. If you are an eval provider or leaderboard maintainer, we are looking for your generous data contibutions via pull requests!
Contributor Guide
How we organize and validate the data.
Repository Structure
data/
└── {benchmark_name}/
└── {developer_name}/
└── {model_name}/
├── {uuid}.json
└── {uuid}.jsonl
Data is split by individual model. Each evaluation consists of an aggregate result (JSON) and a detailed results file (JSONL). Both share the same UUID to ensure clean organization.
Cold Storage Option: For massive datasets, the detailed JSONL file can be stored externally (e.g., S3, Hugging Face Dataset) and referenced via URL in the aggregate JSON. This keeps the repository lightweight while preserving data access.
Data Validation & Converters
We provide tools to easily adapt your existing workflows.
🚀 Automatic Converters
We have ready-made converters for popular frameworks:
Inspect AI HELM lm-eval-harness
# Run validation locally
uv run pre-commit run --all-files
How to add data for a new eval?
- Add a new folder under
/datawith the name of the benchmark. - Create a 2-tier folder structure:
developer_name/model_name. - Add your results: one aggregate JSON file and one detailed JSONL file. (For large files, host the JSONL externally and provide the URL).
- Run the validation script to check against the schema.
Citation
Cite this work
If Every Eval Ever informs your research, please cite the paper:
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura, et al. 2026. Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results. arXiv:2606.14516. Read it here
@misc{batzner2026evaleverunifyingschema,
title={Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results},
author={Jan Batzner and Sree Harsha Nelaturu and Damian Stachura and Anastassia Kornilova and Jon Crall and Tommaso Cerruti and Yanan Long and Yifan Mai and Sanchit Ahuja and Asaf Yehudai and Marek Šuppa and John P. Lalor and Oluwagbemike Olowe and Jatin Ganhotra and Brian H. Hu and Eliya Habba and Andrew M. Bean and Chang Liu and Sander Land and Steven Dillmann and Aniketh Garikaparthi and Elron Bandel and Saki Imai and James Edgell and Wm. Matthew Kennedy and Jenny Chim and Patrick Meusling and Asteria Kaeberlein and Venkata Ramachandra Karthik Chundi and Manasi Patwardhan and Martin Ku and Austin Meek and Leon Knauer and Brian Wingenroth and Srishti Yadav and Usman Gohar and Felix Friedrich and Michelle Lin and Jennifer Mickel and Arman Cohan and Stella Biderman and Irene Solaiman and Zeerak Talat and Anka Reuel and Mubashara Akhtar and Gjergji Kasneci and Avijit Ghosh and Leshem Choshen},
year={2026},
eprint={2606.14516},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2606.14516},
}