Coping with the onslaught of kernel security bugs

Benefits for LWN subscribers

The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

Kroah-Hartman started with a plot of the number of kernel CVE numbers that were fixed with each release; a version of that plot appears to the right. The trend is clear: after running at a steady rate of about 500 fixes per release, that number has suddenly quadrupled. Not that long ago, he said, the old rate, being about 50 fixes per week, seemed incredible. Now the kernel project is fixing 33 CVEs per day. Something has changed; that something is, of course, the advent of LLMs. The good news is that the preparation work done in the kernel community has paid off, and the tooling is able to scale to this level of activity. Other projects are in a panic, but "we're doing fine, and could go faster".

Nobody should panic, he emphasized, this flood is not a big problem. There is a lot of coverage in the press, and a lot of "doom marketing", but we should just ignore it. As an example of how this marketing works, he recounted an episode where he got a phone call informing him that the Mythos LLM had found 79 Linux kernel vulnerabilities. After being concerned initially, he obtained access to the raw data, which he is now able to talk about. This set of alleged vulnerabilities was found by looking at old bug fixes and finding other places with similar problems, mirroring work by people like Julia Lawall and Sasha Levin in the past.

Of those 79 bugs, 24 had no detail beyond "something crashed". Another 14 were simply not bugs, and three were the result of made-up data and were fake from the beginning. There were 15 that were real bugs — but they had already been fixed in the current kernel release. That last set, he said, is typical of these tools; they are sycophantic and will work hard to find bugs if asked. They will even read the mailing lists to find and report bugs that have been fixed by others. This pattern of "finding" bugs that others have already reported is quite common.

That leaves 26 that were actual bugs needing attention — almost. Six of those were duplicates. And, he noted, the total doesn't add up to 79; LLMs are not always good at basic math. So 20 bugs needed fixes; of those, seven were only exploitable by way of a malicious filesystem image, something that the kernel community has long documented as not actually being a security issue. Two bugs required the ability to inject a malicious packet into the middle of the network stack, something that can only be done by root, so these were not real bugs. Two others only affected NOMMU systems (those without a memory-management unit); one of those was in io_uring. Nobody, he suspects, uses io_uring on NOMMU systems. He has fixed both bugs, but the fixes are not upstream yet, nobody really cares much about security on NOMMU systems. There were two minor IPv6 issues, and one that allowed a malicious local user to compromise the system via the GPU — but that user would already be capable of doing far worse.

The end result is that this exercise resulted in ten real bug fixes, in a project that is merging a little over ten patches every hour. So addressing it represented one hour of the kernel community's work.

The real problem

The problem that we should be worried about, he said, is the mean time to exploit — the time between the discovery of a bug and the appearance of exploits for it. In 2018, that time was 63 days; in 2026 it is 7 days. So this problem is real, but the biggest problem is that people are just not updating their systems. Running old software is a luxury many of us do not have anymore.

The bill is finally coming due for many years of underinvestment; our software simply was not designed for perfect security. Now we're going to have to grind through this phase of bug discovery. But this is something we have seen before; the advent of fuzzers years ago also resulted in a flood of bugs needing fixes. We did the work, dealt with those bugs, and the flow of fuzzer-discovered bugs eventually dropped considerably. When you fix all the bugs, he said, the reports stop.

As an example, consider rsync. Andrew Tridgell worked his way through all of the reported bugs (taking some grief along the way), and now all of the code scanners declare that rsync is clean.

He paused for a moment to reflect on the recent reports of communications within Microsoft suggesting that at least some executives there were concerned that their models were stealing much of the data they absorbed. A report from LG Research, he said, suggested that only 20% of the data used at LG for model training was legally obtained. We may not like how these tools were created, but we can make use of them to help make our software better.

That said, not everybody is working toward that goal. There are people flooding the kernel lists with patches, often hundreds per day; these patches are untested at best. We should just ignore them. The best models, he said, are still wrong 50% of the time, so we should just assume that 50% of LLM-generated patches are wrong. We do not have to just accept them. There are some patterns that can often be seen in LLM-generated code. For example, the models often want to replace, incorrectly, calls to mutex_unlock() with mutex_destroy(). They generate highly verbose and useless changelogs, and add far too many comments to the code.

When people have tried to create bots that are better at creating kernel code, they have often started by training those bots on the kernel's development history. But that history demonstrates a lot of patterns that the community has long since left behind; it is full of bad examples. What those bots produced, he said, was "bad code that cursed". Maintainers should not fear to push back when they see this kind of code. Often the best question to ask is "how did you test it?". One should also bear in mind that these systems leak; anything they tell you will be told to others as well. Anything you upload into them will be handed out to others. That is how researchers continually "discover" bugs that were publicly reported a day earlier.

The future for these tools, he said, is not bright. For years, Coverity developed a code-scanning tool that produced many high-quality reports, and it was deterministic as well. But the company never managed to make money; people are not willing to pay for this kind of tool. The current tools, he predicted, will meet the same fate. Nobody wants to pay for code-quality tools when they work well, they certainly won't pay for tools with a 50% error rate.

Adaptation

So how is the kernel community adapting to this situation? One useful effort has been improving the documentation of the kernel's threat model; sometimes, the bots actually read that material, and patch submitters can be pointed to it in the remaining cases. The security-related documentation now requires that maintainers be copied on security reports; any sort of attempt to keep a lid on this information is doomed to fail. Reporters are also requested to submit an initial fix with their report; they can always ask their LLM to create that patch, and it helps kernel developers come up with a proper fix. He said that there is also now funding for a full-time developer to work on reports sent to security@kernel.org.

Developers and maintainers can push back on both bug reporters and patch submitters; if they don't respond to questions, developers do not have to respond to them. For their own testing, he said, developers should run local models; they are good enough for the job, and non-public information should never be uploaded to a non-local system. And, of course, "fix the bugs". A useful tool for developers wanting to run models on their own code is Chris Mason's kres; he also suggested reading this article by Willy Tarreau on how to use local models to find code bugs.

For managers, he said, the "Securing Open Source in the Age of AI" document from the OpenSSF should be required reading.

He concluded by saying that the flow of bug reports does not look like it is slowing down in the near future; it is going to be "a rough 18 months". But, when the dust settles, our code will be better than it was before, and that is a good thing.

The video of this talk is available on YouTube.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my travel to this event.]