GitHub - Steel-predictor-project/steel-llm-eval: Open benchmark: how well can LLMs predict knife-steel properties (edge retention, toughness) from chemical composition, scored against CATRA/Charpy lab measurements.

How well can large language models predict knife-steel properties from chemical composition alone?

An open, reproducible benchmark that gives an LLM only a steel's composition (e.g. C=1.45%, Cr=20%, V=4%, Mo=1%, powder-metallurgy: yes) and asks it to rate two properties on a 1–10 scale, then scores those ratings against objective laboratory measurements:

• Edge retention ← CATRA standardized machine-cutting test (total card stock cut, mm) — 48 steels
• Toughness ← Charpy impact energy (ft-lbs) — 12 steels

Scoring is scale-free (rank correlation + pairwise ranking accuracy), so a model is judged purely on whether it orders steels correctly, not on how it calibrates the 1–10 scale.

📊 Site with charts & analysis: https://steel-predictor-project.github.io/steel-llm-eval/ · Docs: methodology · results & analysis

Leaderboard

Ranked by mean Spearman rank correlation (ρ) vs. the measurements. Higher is better; 1.0 = perfect ordering, 0.0 = random.

Edge retention n=48 (CATRA), toughness n=12 (Charpy). Zero-shot, temperature 0, one sample per steel.

† Important fairness caveat: the reference ML model (Steel-predictor) was trained on these same CATRA/Charpy measurements, so its scores here are largely in-sample and are shown as an upper-reference bar, not as a fair head-to-head with the zero-shot LLMs. The model's honest out-of-sample performance is its LOOCV MAE (0.391), reported in that repo. The LLMs, by contrast, have never seen this labeled set.

What the numbers say

• LLMs are genuinely good at ranking edge retention (ρ ≈ 0.85–0.92). Wear resistance is strongly and legibly encoded in composition (carbide-forming elements — C, V, Cr, W, Mo), and frontier models clearly "know" that chemistry.
• Toughness is where they struggle (ρ 0.38–0.84). It depends on subtler factors (carbide size/distribution, powder-metallurgy processing, matrix state) that aren't obvious from a composition string, and the spread across models is large.
• Frontier > small. Claude Sonnet and Gemini lead; the smaller/cheaper models drop off sharply on toughness while staying competitive on edge retention.

Reproduce

git clone https://github.com/Steel-predictor-project/steel-llm-eval.git
cd steel-llm-eval

export OPENROUTER_API_KEY=sk-or-...   # one key → OpenAI, Anthropic, Google, Meta, DeepSeek, ...
./run_benchmark.sh                    # runs every model and rebuilds the leaderboard

Run a single model, or a quick offline sanity check with no API key:

python harness/run_eval.py --model anthropic/claude-sonnet-5
python harness/run_eval.py --provider mock     # deterministic heuristic, no key needed
python harness/score.py

Raw per-steel responses are written to results/raw_<model>.csv; scores to results/scores.csv and results/leaderboard.md.

How it works

1. Prompt (harness/prompts.py) — a fixed system + user prompt gives the model the composition, PM flag, and test hardness (when known) and asks for JSON: {"edge_retention": n, "toughness": n}. Identical for every model.
2. Run (harness/run_eval.py) — queries a model for all 51 steels via OpenRouter and parses the JSON.
3. Score (harness/score.py) — vs. the measurements:
   • Spearman ρ and Kendall τ rank correlation (headline; scale-free).
   • Pairwise accuracy — over all steel pairs, how often the model orders them the same way the measurement does (ties excluded).
   • Normalized MAE — a calibration sanity check after min-max scaling the measurement to 1–10 (secondary; see scores.csv).
4. Baselines — the purpose-built ML model (upper reference, in-sample caveat above) and a constant predictor (floor).

Methodology notes & limitations

• Ground truth is objective measurement only (CATRA, Charpy). No subjective 1–10 expert ratings are used anywhere in scoring.
• Composition-only. Models are not told heat-treat protocol or geometry (only hardness where recorded), so this measures what chemistry alone implies — the same constraint the reference model operates under.
• Small toughness set (n=12). Treat toughness ρ as indicative, not definitive; single sample per steel at temperature 0 (no self-consistency / multi-sample averaging yet).
• Rank metrics are primary precisely because different models calibrate the 1–10 scale differently; ranking is what's comparable and decision-relevant.

Data sources & attribution

The benchmark (data/benchmark.csv) is derived from the processed dataset of the Steel-predictor project. The two ground-truth measurements come from:

• Edge retention (CATRA, 48 steels) — Larrin Thomas, "Testing the Edge Retention of 48 Knife Steels" (2020), KnifeSteelNerds.com. This benchmark's entire edge-retention ground truth is built on Larrin Thomas's published CATRA measurements — full credit to him.
• Toughness (Charpy, 12 steels) — Crucible Industries published datasheets (CPM series).
• Compositions & test hardness — manufacturer datasheets (Crucible, Böhler-Uddeholm/voestalpine, Carpenter, Alleima, Hitachi/Proterial) plus published literature.

Every individual source is enumerated with links in the Steel-predictor repo's DATA_SOURCES.md. Underlying factual measurements remain the property of their original publishers; this repo redistributes only its own normalized compilation and derived features.

License

Code: Apache-2.0 (LICENSE). Curated benchmark data + reference model outputs: CC BY 4.0 (data/LICENSE), covering only this project's compilation/derived features. Attribution requested: "Steel Property Predictor Project" with a link to this repo.

Contributing

PRs welcome to add models (extend the list in run_benchmark.sh), prompt variants (few-shot, chain-of-thought, self-consistency), or additional measured steels (with cited public sources). Please don't add subjective-rating datasets as ground truth.