Linux 6.15 Git Tanked Nginx HTTPS Web Server Performance - Here's The Bisect Review

With the Linux 6.15 kernel settling down nicely, I've been testing out the current Linux Git state on more systems in looking for any performance changes. Unfortunately this week I ran into a large performance regression affecting the Nginx HTTP(S) web server. Here's a look at that problem currently affecting Linux Git.

Of the systems I frequently test out new Linux development kernel builds on is the custom 4U server build with the Supermicro H13SSL-N and AMD EPYC 9655(P) cooled by SilverStone XE360-SP5 liquid cooling. The AMD EPYC 9655 with 96 cores / 192 threads while clocking up to 4.5GHz and boasting a 384MB cache and just a 400 Watt TDP makes for light work of kernel builds when resorting to bisecting any interesting regressions...

Linux 6.15 AMD EPYC Benchmarks

First up was running the Linux 6.14 stable and Linux 6.15-rc3 kernel builds using the Ubuntu Mainline Kernel PPA when I began this particular round of kernel testing.

Llama.cpp benchmark with settings of Backend: CPU BLAS, Model: granite-3.0-3b-a800m-instruct-Q8_0, Test: Prompt Processing 512. Linux 6.15-rc3 was the fastest.

Llama.cpp benchmark with settings of Backend: CPU BLAS, Model: Mistral-7B-Instruct-v0.3-Q8_0, Test: Prompt Processing 512. Linux 6.15-rc3 was the fastest.

Llama.cpp benchmark with settings of Backend: CPU BLAS, Model: Llama-3.1-Tulu-3-8B-Q8_0, Test: Prompt Processing 512. Linux 6.15-rc3 was the fastest.

RocksDB benchmark with settings of Test: Read While Writing. Linux 6.15-rc3 was the fastest.

Schbench benchmark with settings of Message Threads: 64. Linux 6.15-rc3 was the fastest.

Running Linux 6.15 on this Supermicro EPYC Turin server build showed some nice improvements with the Linux 6.15 kernel with the likes of Llama.cpp and RocksDB and then synthetics like Schbench for testing the kernel scheduler improvements. And then dozens of unchanged results omitted for this Linux 6.14 vs. 6.15 Git benchmarking bout.

nginx benchmark with settings of Connections: 500. Linux 6.14 was the fastest.

DaCapo Benchmark benchmark with settings of Java Test: Apache Lucene Search Engine. Linux 6.14 was the fastest.

DaCapo Benchmark benchmark with settings of Java Test: FOP Print Formatter. Linux 6.14 was the fastest.

PostgreSQL benchmark with settings of Scaling Factor: 100, Clients: 800, Mode: Read Only. Linux 6.14 was the fastest.

DaCapo Benchmark benchmark with settings of Java Test: PMD Source Code Analyzer. Linux 6.14 was the fastest.

DaCapo Benchmark benchmark with settings of Java Test: Spring Boot. Linux 6.14 was the fastest.

OpenFOAM benchmark with settings of Input: drivaerFastback, Small Mesh Size, Execution Time. Linux 6.14 was the fastest.

But I also hit some unexpected performance regressions... Nginx in particular saw a very brutal performance hit going to the Linux 6.15 development kernel. There were also some regressions noted for the OpenJDK Java benchmark DaCapo as well as for the OpenFOAM computational fluid dynamics performance. But the biggest regression by far was with Nginx in the simple single-server HTTPS web server test case. For this article today I am just focusing on the big Nginx regression. I haven't yet confirmed if it's the same regression or not affecting those other workloads.

So then thanks to the 96-core / 192-thread AMD EPYC Zen 5 server, it was off to bisect this hard-hitting regression.