Terminal Bench 4 Is Saturated
preferencemodel.com
1 thread
I think ambiguously worded questions have their place, but you have to give up automatic scoring on such tasks.
You don't have to give it up entirely, you could grade on the interpretation they chose. Personally I'm a fan of models which seek clarification on critical ambiguities rather than barging ahead "autonomously" though.
But wow, they found a lot of issues with the benchmark.
I really like the benchmarks where frontier models score each other's solutions to a problem.