Development statistics for the 7.3 kernel

· LWN.net

9 min read Original article ↗

[LWN subscriber-only content]

Welcome to LWN.net

The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!

Linus Torvalds released the 7.3 kernel, at the end of the usual nine-week development cycle, on October 18. This cycle, abundantly fueled by LLM-assisted contributions, broke just about every record imaginable. As usual, we have compiled some statistics showing how this release, which is expected to be the next long-term-support (LTS) kernel, came together. Also included is an attempt to determine whether automated patch review is actually reducing the number of bugs introduced into the kernel.

This development cycle was the first in the project's history to exceed 20,000 commits overall, 18,765 of those were non-merge commits. These commits added nearly 920,000 lines of code while removing just over 270,000, for a net growth of about 650,000 lines; that is a lot of new code to absorb. This code was contributed by 3,021 developers, smashing through the short-lived record of 2,652 set by the 7.2 cycle. To give a sense for what is happening with contributor counts, here is a plot of contributors per release over the entire git history:

[Plot: developers per
  release]

The most active developers this time around were:

Most active 7.3 developers
By changesets
Christian Brauner 2161.2%
Bui Duc Phuc 2061.1%
Pengpeng Hou 1650.9%
Sean Christopherson 1610.9%
Pan Chuang 1500.8%
Tejun Heo 1420.8%
Maxime Ripard 1340.7%
Krzysztof Kozlowski 1310.7%
SJ Park 1230.7%
Alex Hung 1220.7%
Namjae Jeon 1200.6%
Uwe Kleine-König 1150.6%
Alex Deucher 1110.6%
Takashi Iwai 1040.6%
Breno Leitao 990.5%
Rosen Penev 960.5%
Imre Deak 950.5%
Kumar Kartikeya Dwivedi 910.5%
Lorenzo Stoakes890.5%
Thomas Weißschuh 850.5%
By changed lines
Aurabindo Pillai 12646412.5%
Alex Hung 400824.0%
Jeff Chen 358273.5%
Dongdong Hao 170461.7%
Christian Brauner 148731.5%
Lachlan Hodges 139491.4%
Dimitri Sivanich 135801.3%
Tejun Heo 120131.2%
Mickaël Salaün 100811.0%
Eric Biggers 93490.9%
Ching-Te Ku 85760.8%
Bhawanpreet Lakha 82990.8%
Sean Christopherson 81080.8%
Ce Sun 74920.7%
Mikhail Kshevetskiy 73470.7%
David Christensen 70510.7%
Chun-Tse Shao 64110.6%
Taniya Das 63990.6%
Kumar Kartikeya Dwivedi 63770.6%
Ian Rogers 63060.6%

The biggest contributor of non-merge changesets this time around was Christian Brauner [KSDB], who made significant improvements to the binfmt_misc subsystem and contributed fixes throughout the virtual filesystem layer. Bui Duc Phuc [KSDB], a contributor who was first seen in the 7.1 development cycle, was a close second with work that focused mostly on fixes within the sound-driver subsystem. Pengpeng Hou [KSDB], first seen in 7.0, made fixes throughout the driver subsystem. Sean Christopherson [KSDB] worked extensively with the KVM hypervisor. Pan Chuang [KSDB], who had three commits in the kernel (starting with 6.17) until now, contributed a pile of driver-subsystem cleanups.

In the lines-changed column, Aurabindo Pillai [KSDB] contributed five commits containing, inevitably, a large set of machine-generated amdgpu register definitions. Alex Hung [KSDB] added numerous AMD graphics tests. Jeff Chen [KSDB] contributed three commits, one of which was a driver for NXP IW61x WiFi interfaces. Dongdong Hao [KSDB] added the leapraid SCSI driver and associated documentation.

The 7.2 development statistics included a note that the percentage of commits containing Reviewed-by tags had been dropping for a few development cycles; that trend has reversed in 7.3, where 1,399 commits (7.5%) had Tested-by tags and 9,545 (50.9%) had Reviewed-by tags. The top testers and reviewers were:

Test and review credits in 7.3
Tested-by
Dan Wheeler 26116.5%
Luca Ceresoli 855.4%
Mykola Marzhan 291.8%
Tommaso Merciai 241.5%
Victor Nogueira 201.3%
Michael Kelley 191.2%
Mostafa Saleh 191.2%
Deborah Brouwer 191.2%
Wolfram Sang 181.1%
Nathan Chancellor 181.1%
Randy Dunlap 181.1%
Charlie Jenkins 161.0%
Heiko Stuebner 150.9%
K Prateek Nayak 140.9%
Samuel Salin 140.9%
Reviewed-by
Konrad Dybcio 4743.7%
Dmitry Baryshkov 3052.4%
Christoph Hellwig 2461.9%
Andy Shevchenko 2061.6%
Frank Li 1981.6%
Krzysztof Kozlowski 1931.5%
Simon Horman 1701.3%
Ilpo Järvinen 1641.3%
Luca Ceresoli 1391.1%
Thomas Zimmermann 1231.0%
Andrea Righi 1170.9%
Bhawanpreet Lakha 1110.9%
Hannes Reinecke 1060.8%
Linus Walleij 1050.8%
Cezary Rojewski 1000.8%

Dan Wheeler [KSDB] remains the most prolific tester, trying out about four changes to the AMD graphics drivers every day of this 63-day development cycle; nobody else comes close. Meanwhile, Konrad Dybcio [KSDB] somehow managed to review 474 Qualcomm driver and devicetree changes — over seven per day. Dmitry Baryshkov [KSDB] (nearly five per day) and Christoph Hellwig [KSDB] (just under four per day) were slackers by comparison.

It is also worth taking a quick look at the Reported-by tags in this release. These tags appeared in 1,411 commits, or 7.5% of the total, and the numbers look like this:

Top bug reporters for 7.3
Sashiko 30419.4%
Syzbot23915.3%
kernel test robot 895.7%
Vega 835.3%
Nicholas Carlini 412.6%
Konrad Dybcio 301.9%
Xiang Mei 201.3%
Dan Carpenter 171.1%
Chris Mason 140.9%
TencentOS Corvus AI 130.8%
Yuhao Jiang 130.8%
Weiming Shi 110.7%
Martin Hodo 110.7%
Mark Brown 90.6%
autonomouscodesecurity@microsoft.com90.6%

The Sashiko review tool [KSDB], which did not exist one year ago, is now responsible for nearly 20% of the Reported-by tags in this release. Overall, automated systems are the source of nearly half of the credited bug reports.

See this KSDB page for more information on the use of all of these tags in the 7.3 kernel.

Work on 7.3 was supported by 232 employers that we were able to identify; the most active of those were:

Most active 7.3 employers
By changesets
(Unknown)385120.5%
AMD11866.3%
Intel11796.3%
Google11386.1%
Qualcomm11155.9%
Red Hat8374.5%
(None)6793.6%
Kylinsoft6243.3%
Meta5242.8%
IBM4722.5%
NVIDIA4002.1%
SUSE3631.9%
(Consultant)3031.6%
Oracle2711.4%
(Academia)2681.4%
Renesas Electronics2401.3%
NXP Semiconductors2371.3%
Arm2341.2%
Amutable2161.2%
Huawei Technologies1740.9%
By lines changed
AMD22517322.3%
(Unknown)14438714.3%
Qualcomm617716.1%
Google569775.6%
NXP Semiconductors455064.5%
Intel447744.4%
Red Hat384203.8%
(None)320753.2%
Meta277682.7%
IBM204412.0%
NVIDIA167371.7%
Realtek157311.6%
Amutable148731.5%
Morse Micro139491.4%
HP Enterprise136981.4%
Renesas Electronics117561.2%
(Consultant)114051.1%
Cloudflare100701.0%
Collabora84050.8%
Samsung83990.8%

The continuing flood of new developers is behind the inexorable growth in the "unknown" category; the bulk of them probably are working on their own time, but there is no easy way to know for sure. An interesting trend to note is the increase in contributions from Kylinsoft, which develops the Chinese Kylin OS distribution. There are 92 developers affiliated with this company who contributed to 7.3, most of whom contributed only a handful of commits. As a group, they appear to be working through lists of LLM-identified bugs.

First-time developers and young bugs

Quite a few of the above-mentioned Kylinsoft employees made their first kernel contributions in the 7.3 cycle. All told, there were 807 new developers seen this time around, continuing the trend we have been seeing over the course of 2026:

[First-time developers by
release]

Well over a quarter of the developers contributing to 7.3 were people who had never been seen in the community before. The growth trend in new contributors clearly cannot continue forever; maintainers are already losing their ability to keep up with this flood. But it also shows no signs of slowing down.

These new contributors are making changes all over the kernel. Many of them are trivial — typo fixes and the like — but many others are significant changes to important kernel subsystems. It is clear that many of these contributors do not really understand the code they are using LLMs to modify, leading to concerns about the quality of their patches and the long-term maintainability of a kernel that is being flooded with those patches.

One claim that has been heard recently is that the advent of automated patch review has increased the quality of the incoming code, making it safer to accept more of it with less human intervention. Determining whether that is really true will take years, because it takes that long for many bugs to surface. One can try to get an early indication of which way the wind is blowing, though, by looking at the bugs that did show up early on:

[Short-term fixes, by release]

Here, the lower (blue) bar is the number of commits that included a Fixes tag indicating that the bug had been introduced in the previous release. The blue bar for 7.3, on the right-hand side, is thus the number of 7.3 commits fixing bugs that were introduced in 7.2. That value appears to be relatively stable over the recent release history, with no obvious trend. That said, bear in mind that the overall patch volume has increased with recent kernels, so a steady rate of previous-release fixes suggests a lower bug rate overall.

The upper (orange) bar is the number of fixes to commits merged during the same release cycle; these are bugs that never find their way into a released kernel. That number does appear to be falling with the most recent releases, especially when considering the increased patch volume.

Rather than trying to imagine how the change in patch volume affects these numbers, one can simply plot them as a percentage of the total number of commits in the relevant release:

[Short-term fixes, by
release, percentages]

Here, the change seems clearer. The number of short-term-buggy commits as a percentage of the total has remained roughly constant over recent years. But that percentage began to fall in 7.1, and has fallen more sharply since. Specifically, bugs that turn up and must be fixed within the same release cycle seem to be declining. That suggests that automated patch review may, indeed, be filtering out the sorts of bugs that become immediately apparent once the affected code starts seeing wider testing. Of course, another explanation could be that, while bugs are being introduced at the usual rate, they are now being found and fixed at a lower rate. Given the accelerated rate of bug-finding in general, though, that explanation seems unlikely. The real story will take years to play out, but the preliminary indications are that the quality of code going into the kernel, with regard to obvious bugs at least, might just be improving.

As the development community heads into the 7.4 merge window, there are nearly 16,000 non-merge changesets sitting in linux-next. That is significantly more than the patches that were staged for 7.3, suggesting that the records set by 7.3 will, once again, be short-lived. The nature of the kernel-development community is changing, and what we will end up with is far from clear. It is certain to be an interesting ride, though; stay tuned and we will see where it goes together.

Index entries for this article
KernelReleases/7.3