Zsolt Ero (@hyperknot) on X

I believe this started with Sonnet 3.7. Sonnet 3.5 was and still is a much more reliable or 'accurate' model compared to 3.7 or Gemini 2.5 Pro. Interestingly, Sonnet 3.5 was only in 13th place on the

leaderboard when 3.7 was released.

Which is interesting, because at that point, it was by far the best coding model.

reported that Sonnet 3.5 was so much ahead of any other model for coding that GitHub Copilot probably lost market share by not offering it sooner. Basically, Microsoft, the main

So we had a model which was beating anything else in real-life coding tasks, yet had a terrible place on

. With Sonnet 3.7, it changed; Anthropic finally started to optimize for the

race! And with it came a model which became a "drunk professor". This

This can be seen in the samples revealed in the Llama 4 fiasco, for example.

OpenAI was the first lab to optimize for Arena; they turned up Markdown decorations to 11. To this day, Markdown decorations are still so burned into OpenAI models that it's almost impossible to get an answer without over-decorated Markdown. Try using chatgpt-4o-latest in an

At this point, I agree with the opinion that optimizing for LMArena is damaging the entire AI community:

Interestingly,

made a benchmark for "accuracy", where, not surprisingly, Sonnet 3.5 beats Sonnet 3.7 and Gemini 2.0 Flash absolutely beats Gemini 2.5 Pro

I believe "accuracy" or "reliability" benchmarks like this should be way more important for the next wave of model releases. Now, a collection of opinions I've bookmarked during the last few weeks.