TLDR: We benchmarked 18 frontier models on how much AI slop they write, scored against real human writing from before ChatGPT plus a live blind human vote. We found GPT 5.4 Mini to be the least slop, while Fable was the most. You can vote yourself and influence the rankings: https://theslopindex.com/arena
[Time of writing of post was 12:00 AM PT July 21st]
At this point, I think it’s impossible to not acknowledge that a significant portion all of our days is spent consuming LLM product content, a lot of which is “slop”.
And @Slashy we spend a lot of our time figuring out how to make AI that sounds like you not slop (specifically in the arena of email).
But we actually surprisingly have never spent a lot of time figuring out what models are the best at not producing slop.
Partly because I think benchmarks are mostly silly for writing: there's no quantitative verifier. Even "did the email get a reply" isn't ground truth, since plenty of emails only get sent because an AI made them (for example a common one being the pass notes Slashy drafts for the VCs who use us haha).
But anyways that’s just a tangent, onto what you’re actually reading for the real benchmark :)
1) The Baseline:
Slop can only be measured compared to stuff that already existed. So we got a corpus of data for various areas of writing (email, social, chat, and essays) so that each has a human baseline.
2) Tasks
I then hand-wrote 112 written scenarios for the models to egenerate outputs to across email, Slack, social media posts, and essays (a cold email, a schedule change, a launch tweet, an argumentative essay, etc). Every model gets the identical scenarios at default settings, several samples each: and you can see all the exact outputs in my Github repo.
3) Axes
Now for how to decide to measure slop we settled with 5 dimensions.
- Conciseness (one of the most annoying parts of AI writing is how it takes 6 paragraphs to say 2 sentences)
- Templating (AI often reuses the same sentences/styles across unrelated scenarios)
- Rhythm (Variance in sentence/paragaphs, humans often switch this up while models stay p similar)
- Tells (Over used vocab and construction for stuff like "delve", "it's not just X, it's Y")
- Human Preference (I think this is most important as everything else are just heuristics for this)
Note how we DELIBERATIVELY don't have any LLM judging, I think it'd be pretty stupid to have LLMs judge LLMs
Now for the results
Frontier seems to be more slop, not less: it’s clear that all frontier development is focused on code-gen, and I think it’s why the cheaper smaller models actually generate less slop nowadays.
Human pref and mechanical structure can disagree: Muse Spark did great on our mechanical axes, but flopped on the human benchmark. (As an aside, we found a good chunk of making high quality drafts is just getting the em dash and the signatrues right, so models that are more likely to use em dashes tend to do a lot worse in human pref, even if everything else is good).
AI Slop Detectors are pretty good: We actually tried @pangram as it’s own axis, but had to scrap it as it found all models to be 100% AI for all outputs 💀
And I think this is indicative that as the models more recently have become more benchmark optimized, they've actually produced more slop than less. Which is where good prompting, harness, memory and more matter. As I don’t think the models have gotten dumber, I think they’re just made with the assumption that at the agent/harness layer slop will be taken care of instead of the model layer.
But either way would love to hear all of your thoughts 🙂
And if you’d like to vote on slop can vote here https://theslopindex.com/arena
Everything is open: method at theslopindex.com/methodology, outputs and code linked from there.