Finding Miscompiles for Fun, Not Profit
newsletter.semianalysis.comGiven the $10k price tag for tokens and high rate of bugs (several per minute) they mention, it'd be very interesting to see this experiment run with cheaper models too.
I wonder if we get to a world where a full repo sweep like this is a default Github action after commit.
Most C/C++ projects I know don't even run tests with ASan/TSan/UBSan before each commit/merge.
and in the meantime, just a sweep of the committed code (or the to-be-committed code for lots of us) and the code it interacts with, is increasingly catching lots of problems.
Author here; I'm happy to answer questions, take criticism, etc etc.
Thank you for posting this.
I had heard LLMs were finding a lot of bugs very quickly and now I can see what that looks like from a user perspective.
> Codex and I collaboratively wrote a fuzzer.
Why are you using phrasing that equates AI and humans? You used Codex to write a fuzzer. It didn't decide to join you.
Why are you using phrasing that equates AI and humans? Codex isn't in a position to decide whether to do work.
I wonder how much damage all those countless bugs caused in real life.
Does MRI low level code produce wrong images? Do some kind of unexpected http connection quirks happen? Does (LL)M inference produce randomly wrong and non reproduceable output? Graphical artifacts in video games? Application crashes that happen once every billionth request? Security vulnerabilities? Race conditions?
Boy, I told DeepSeek V4 Flash Free to find compiler bugs, esp. with the gcc torture test suite, and it did find plenty. For free. To fix them in my compiler it did cost about $40 or so. Cooperate guys just have too much budget in their hands to throw at the rich.