Vendor hooks in the Android kernel

Benefits for LWN subscribers

The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

The 6.6 kernel is the one used on the largest number of Android systems currently, he began; 6.18 can be found mostly on flagship devices at this point. The Android 6.6 kernel carries a large number (just over 3,100) of out-of-tree patches, nearly half of which (1,531) are backports of features from newer kernel releases. There is a relatively small set (287 patches) of "from list" changes that are being discussed on mailing lists but which have not yet found their way into the mainline. The remaining 1,306 patches, about 42%, are "pure Android" changes that are not upstream, and are not currently on their way upstream. Of those, something over a third are vendor hooks (much of the rest is build-system and configuration changes).

A vendor hook, at the lowest level, is simply a special-purpose tracepoint that has been added to the Android kernel. When an Android vendor wants to make a change to the generic kernel, a vendor hook is placed in the appropriate location; the code tends to look like:

    trace_android_vh_feature_name(parameters);

That vendor's device loads a special kernel module that attaches to the vendor-hook tracepoint in order to intercept the control flow at that point. The attached hook can change data structures, inject new code, short-circuit mainline code, and more; it is kernel code with few limits. These hooks allow vendors to add their special code to their kernel without interfering with others, he said.

Finally, 15% of the Android-specific patches are exports of symbols for consumption by the vendor hooks. Android attempts to maintain a stable ABI for these hooks which, Kjos said, often constrains the changes that can be made.

Nearly half of the vendor hooks in the Android kernel are located in the memory-management subsystem and the CPU scheduler; that is where vendors are trying hardest to differentiate their products. Some of these changes are on their way upstream, many others are not.

The chief reason for the use of vendor hooks, he said, is urgency; changes are needed for current products, and the upstreaming process is far too slow. Vendors often have "good intentions", but lack the patience to do the work to mainline their changes. Sometimes they eventually follow through, often they do not. Another reason for vendor hooks is to support hardware that has no upstream drivers.

In general, the policy for the generic kernel image is that the only way to add features to the kernel (other than contributing them upstream, of course) is by way of a vendor hook. The Android maintainers do not understand all of the systems out there and are not able to determine which proposed changes are actually important. As a result, Kjos said, the policy toward vendor hooks is permissive.

The most complex vendor hook in use currently is Qualcomm's Window Assisted Load Tracking (WALT) feature, which is a replacement for the per-entity load tracking (PELT) mechanism used in the mainline. The load tracker informs the CPU scheduler about the resources actually being used by each task in the system, allowing for better scheduling and placement decisions; WALT (which was posted upstream a decade ago) is said to be better for mobile workloads than PELT is. Some of the ideas from WALT, he said, have worked their way into PELT over the years.

Another common use for vendor hooks in the scheduler is avoidance of priority-inversion problems. Some vendors tweak the kernel's locking primitives; others maintain a special run queue in the scheduler for important processes. The objective is always to ensure quick response to user action and smooth displays without dropped frames.

There are about 800 vendor hooks in the kernel, which is quite a few. Kjos concluded by saying that the Android developers would like to reduce that number, and asked how the situation can be improved.

Noting that many of the tweaks were focused on the scheduler, I asked whether thought had been put into directing vendors toward the extensible scheduler class (sched_ext) as an alternative to vendor hooks. Kjos answered that sched_ext is supported in the Android kernel now; that has, he said somewhat ruefully, inevitably resulted in the addition of vendor hooks to the sched_ext infrastructure as well. He added that, in all honesty, he thought it would be better to remain with vendor hooks rather than moving to scheduler implementations that are entirely out of tree. Later in the Android track, Aiqun Yu talked about an effort to move the scheduling of realtime tasks into sched_ext as a way of removing some vendor hooks.

Another audience member observed that one way to reduce the number of vendor hooks would be to simply push back harder on their addition. That is, of course, easier said than done. Kjos responded by repeating that the Android group lacks the expertise to judge all of these hooks, so it is hard to say that any given addition is not valid. At that point, the session ran out of time and the discussion moved on.

The slides from this talk are available.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my travel to this event.]

Note: you can avoid this step in the future by logging into your LWN account.