Show HN: Redactle LLM Leaderboard
redactle.netI got nerd sniped and ran a bunch of evaluations to see how well models could play Redactle. It turns out Gemini flash is in a league of it's own in both performance and cost. It's able to one-shot every eval in a just a few seconds.
1 thread
What's interesting is how well the models know the exact wiki text. If the wiki excerpt is rewritten the pattern matching doesn't work quite as well. Gemini is only able to one-shot about 1/3rd of the time which is still very impressive considering not long ago LLMs couldn't count the letter 'r' in strawberry.