Introducing HLE-Diamond

Scale AI · Humanity's Last Exam ·

3 min read Original article ↗

We are releasing HLE-Diamond, a refined subset of Humanity’s Last Exam (HLE) question collection, following a year-long process of cleaning and refinement with input from research communities.

HLE-Diamond consists of 1,000 questions.

Main results. We compare the performance of current models on HLE-Diamond without tools.

All models are evaluated with reasoning high.

HLE-Diamond

Accuracy (%)0255075100GPT-6 Astra: Accuracy 59.9%59.9%GPT-6AstraGemini 4 Argon: Accuracy 56.9%56.9%Gemini4 ArgonClaude Opus 5.5: Accuracy 54.6%54.6%ClaudeOpus 5.5GPT-6.1 Sol: Accuracy 53.2%53.2%GPT-6.1SolClaude Fable 5.1: Accuracy 51.6%51.6%ClaudeFable 5.1Muse Wonder: Accuracy 34.7%34.7%MuseWonderClaude Sonnet 5.5: Accuracy 33.1%33.1%ClaudeSonnet 5.5Grok 4.7: Accuracy 22.8%22.8%Grok4.7

Token Usage

Reasoning high

Mean output tokens (log₂)4k8k16k32k64kGPT-6.1 Sol: Token Usage 4,491.6 output tokens4.5kGPT-6.1SolGPT-6 Astra: Token Usage 5,171.7 output tokens5.2kGPT-6AstraClaude Sonnet 5.5: Token Usage 8,583.8 output tokens8.6kClaudeSonnet 5.5Claude Opus 5.5: Token Usage 9,614.2 output tokens9.6kClaudeOpus 5.5Claude Fable 5.1: Token Usage 14,816.8 output tokens14.8kClaudeFable 5.1Muse Wonder: Token Usage 16,124.6 output tokens16.1kMuseWonderGemini 4 Argon: Token Usage 26,086.5 output tokens26.1kGemini4 ArgonGrok 4.7: Token Usage 43,244.4 output tokens43.2kGrok4.7

Dataset. HLE-Diamond consists of 500 reasoning and 500 knowledge questions, measuring reasoning and expert knowledge, respectively.

Reasoning and Knowledge Partitions

ReasoningKnowledge

Accuracy (%)0255075100GPT-6 Astra: Reasoning 74.2%, Knowledge 45.6%74.2%45.6%GPT-6AstraGemini 4 Argon: Reasoning 61.8%, Knowledge 52.0%61.8%52.0%Gemini4 ArgonClaude Opus 5.5: Reasoning 62.4%, Knowledge 46.8%62.4%46.8%ClaudeOpus 5.5GPT-6.1 Sol: Reasoning 67.4%, Knowledge 39.0%67.4%39.0%GPT-6.1SolClaude Fable 5.1: Reasoning 61.8%, Knowledge 41.4%61.8%41.4%ClaudeFable 5.1Muse Wonder: Reasoning 43.2%, Knowledge 26.2%43.2%26.2%MuseWonderClaude Sonnet 5.5: Reasoning 42.4%, Knowledge 23.8%42.4%23.8%ClaudeSonnet 5.5Grok 4.7: Reasoning 31.2%, Knowledge 14.4%31.2%14.4%Grok4.7

All models are evaluated with reasoning high.

Evaluation with tools. HLE-Diamond questions are designed to be answerable in a closed-book setting, testing both reasoning and expert knowledge. Since HLE is also used to evaluate the capabilities of agentic systems, we outline our recommended settings for evaluating HLE-Diamond with tools here.

HLE-Diamond with tools

Without toolsWith tools (web+code)

Accuracy (%)0255075100GPT-6 Astra: Without tools 59.9%, With tools (web+code) 82.9%59.9%82.9%GPT-6AstraClaude Opus 5.5: Without tools 54.6%, With tools (web+code) 73.9%54.6%73.9%ClaudeOpus 5.5GPT-6.1 Sol: Without tools 53.2%, With tools (web+code) 69.1%53.2%69.1%GPT-6.1SolClaude Fable 5.1: Without tools 51.6%, With tools (web+code) 72.3%51.6%72.3%ClaudeFable 5.1Claude Opus 5: Without tools 38.1%, With tools (web+code) 68.9%38.1%68.9%ClaudeOpus 5GPT-6 Sol: Without tools 32.8%, With tools (web+code) 65.0%32.8%65.0%GPT-6SolMuse Spark 1.3: Without tools 24.6%, With tools (web+code) 55.4%24.6%55.4%MuseSpark 1.3

All models are evaluated with reasoning high.

Acknowledgement. We extend our deepest gratitude to all participating question contributors and experts involved in creating and refining the dataset, and to the researchers whose inputs across HLE-Rolling shaped HLE-Diamond.

For any inquiries, please contact agibenchmark@safe.ai.

Citation

@article{phan2025lastexam,
      title = {A benchmark of expert-level academic questions to assess {AI} capabilities},
      author = {{Center for AI Safety} and {Scale AI} and {HLE Contributors Consortium}},
      journal = {Nature},
      volume = {649},
      pages = {1139--1146},
      year = {2026},
      doi = {10.1038/s41586-025-09962-4},
      eprint = {2501.14249},
      archivePrefix = {arXiv},
      primaryClass = {cs.LG},
      url = {https://arxiv.org/abs/2501.14249}
}