Bug #2089789 “malloc performance degradation with CPU affinity m...” : Bugs : glibc package : Ubuntu

3 min read Original article ↗

[Impact]

 * Jammy has a malloc() performance degradation
   if CPU affinity masks are used (not default).

 * The maximum number of arenas for malloc() is
   calculated based on the number of processors.

   However, glibc 2.34 changed that to be based
   on sched_getaffinity(), which is the number
   of processors available _to the process_
   (i.e., based on CPU affinity masks). [0]

   Previously, glibc 2.33 instead used the
   of processors available _in the system_
   (i.e., based on sysfs and procfs files).

 * This is not an issue by default, as without
   CPU affinity masks, the returned number of
   processors is the same as sysfs and procfs.

   But it _is_ an issue if CPU affinity masks
   are set, as it can increase lock contention
   (less arenas), and thus degrade performance.

 * CPU affinity can be set at the process-level
   (e.g., taskset, numactl, sched_setaffinity())
   or at the system-level (kernel boot options).

   The latter is common in hypervisor and/or DPDK
   deployments, where CPU partitioning is applied
   with isolcpus, cpusets, systemd's CPUAffinity.

[Test Plan]

 * The upstream bug report [1] has a reproducer,
   used in comment #5 to reproduce the problem,
   and in comment #6 to validate the fix patch.

   It is copied/attached to this bug as backup
   (test-glibc-malloc.c).

   The expected behavior is that these 2 steps
   (measuring the average time taken by 50.000
   malloc+free calls, with one thread per CPU)
   take similar amounts of time with & without
   CPU affinity masks (parameter 2: true/false),
   in a system with a great number of CPUs.

   $ ./test-glibc-malloc $(nproc) false false
   $ ./test-glibc-malloc $(nproc) true false

 * glibc has a build-time test suite.

 * glibc has autopkgtests (rebuild, ie, above)
   and triggers autopkgtests in a great number
   of reverse test dependencies.

[Regression Potential]

 * Theoretically, any fallout should be contained
   in malloc() and be related only to performance,
   not to functional errors.

 * This happens because this malloc() patch [2] changes
   only which method to get the number of processors.

 * The method it changes to is what has been already
   used by previous versions of glibc (up to 2.33),
   which has been adopted back (2.39) and backported
   to all glibc releases after that version (2.34-2.38),
   which includes the version in Jammy (2.35 [3]).

 * The method it changes to is also exercised in other
   code paths (not just malloc()), thus it is already
   used and tested in Jammy -- it is not something new.

[Other Info]

 * For details and analysis of (no) required
   dependencies, see comments #1, #2, and #3.

 * Upstream bug report [1]

 * Build-tested in PPA with supported archs
   and only -security (ppa:mfo/lp2089789) [4],
   with successful build & test-suite results.

[0]

glibc 2.33:
$ git log --oneline origin/release/2.33/master -- sysdeps/unix/sysv/linux/getsysstats.c | grep 'misc: Add __get_nprocs_sched'
$

glibc 2.34:
$ git log --oneline origin/release/2.34/master -- sysdeps/unix/sysv/linux/getsysstats.c | grep 'misc: Add __get_nprocs_sched'
e870aac8974c misc: Add __get_nprocs_sched

glibc 2.35:
$ git log --oneline origin/release/2.35/master -- sysdeps/unix/sysv/linux/getsysstats.c | grep 'misc: Add __get_nprocs_sched'
11a02b035b46 misc: Add __get_nprocs_sched

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=30945

[2] https://sourceware.org/git/?p=glibc.git;a=commit;h=472894d2cfee5751b44c0aaa71ed87df81c8e62e

[3] https://sourceware.org/git/?p=glibc.git;a=commit;h=d47c5e4db7924bb10efe14b787c4bd868b604e48

[4] https://launchpad.net/~mfo/+archive/ubuntu/lp2089789