Fedora's modularity mess

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Fedora's Modularity initiative has been no stranger to controversy since its inception in 2016. Among other things, there were enough problems with the original design that Modularity went back to the drawing board in early 2018. Modularity has since been integrated with both the Fedora and Red Hat Enterprise Linux (RHEL) distributions, but the controversy continues, with some developers asking whether it's time for yet another redesign — or to abandon the idea altogether. Over the last month or so, several lengthy, detailed, and heated threads have explored this issue; read on for your editor's attempt to integrate what was said.

The core idea behind Modularity is to split the distribution into multiple "streams", each of which allows a user to follow a specific project (or set of projects) at a pace that suits them. A Fedora user might appreciate getting toolchain updates as soon as they are released upstream while sticking with a long-term stable release of LibreOffice, for example. By installing the appropriate streams, this sort of behavior should be achievable, allowing a fair degree of customization.

Much of the impetus — and development resources — behind Modularity come from the RHEL side of Red Hat, which has integrated Modularity into the RHEL 8 release as "Application Streams". This feature makes some sense in that setting; RHEL is famously slow-moving, to the point that RHEL 7 did not even support useful features like Python 3. Application Streams allow Red Hat (or others) to make additional options available with support periods that differ from that of the underlying distribution, making RHEL a bit less musty and old, but only for the applications a specific user cares about.

The use case for Modularity in Fedora is arguably less clear. A given Fedora release has a support lifetime of 13 months, so there are limits to the level of stability that it can provide. But there is still clearly an appetite for Modularity; Fedora leader Matthew Miller articulated three goals during one of the threads:

Thus far, it is far from clear that any of those goals have been met. In particular, there are few modules in Fedora with multiple streams, and the lifecycles of modules are required, by policy, to line up with a Fedora release. But that is just the beginning of the problems for Modularity in Fedora.

The trouble with Modularity

The list of complaints being raised against Modularity is long; some of them were described by Modularity developer Stephen Gallagher in a thread dedicated to that purpose. Covering them all would create a long article indeed, so only a few of them will be discussed here.

The first of those, and the immediate cause of at least one of the longer mailing list threads, has to do with how modules interact with upgrades of the distribution itself. One of the policy decisions behind Modularity states that once users pick a particular stream, they will not be moved to a different one. That plays poorly with upgrades, though, where some streams are discontinued and others pick up; the result may be blocked upgrade operations and irritated users.

Consider the case of libgit2, which is packaged as a module in Fedora 30. Some module maintainers also make non-modular packages available (in an unbearable bit of jargon, these are sometimes called "ursine packages" because they are "bare"), but that is not the case for libgit2. When the DNF tool is given a request to install one of these module-only packages (perhaps as a dependency for the package the user really wants), it will silently install the module, giving users a system with Modularity without their explicit wish and, probably, knowledge.

In this case, a Fedora 30 user installing a package with a dependency on libgit2 would end up getting the libgit2 module, on the 0.28 stream. That stream, however, does not exist in Fedora 31, so any dependencies on libgit2 will not be satisfiable and the upgrade fails. There is an ursine libgit2 package in Fedora 31, but once a module is installed, there is no provision for moving a system back to a non-modular version, so it could not be used.

This was a rather late and unpleasant surprise as the project was trying to get the Fedora 31 release together. The initial solution that was arrived at was to have the upgrade process print out a URL at the beginning to indicate where affected users could find information on how to fix their systems. This struck some developers as being less than fully user-friendly so, after further discussion, another solution was adopted: a rather inelegant hack to DNF that would forcibly reset the stream for libgit2 when upgrading to Fedora 31. This violates the "no stream changes" policy and it clearly is not a sustainable solution going forward, but it got the project past the problem for now.

One of the promises that came with Modularity was that users would not have to deal with it if they didn't want to. It has become increasingly clear, though, that this promise cannot be kept, for a number of reasons. The existence of module-only packages is one of those, along with the inability to revert back to a non-modular version once a given module is installed. Once a package goes module-only, any other package that depends on it must also make that transition, forcing some developers to create and maintain modules whether they want to or not. These problems have led some community members, such as Kevin Kofler, to suggest that module-only packages should be banned, as should modules for anything but leaf packages that no others depend on.

Then, there is the issue of buildroot-only modules. Fedora's use of "buildroot" does not refer to the Buildroot buildsystem; instead, it describes the environment in which packages and modules are built. A buildroot-only module is one that is created solely for the purpose of building another module in the buildroot environment; it is not shipped as part of the distribution itself. The idea behind these modules is to make life easier for packagers to deal with dependencies that are not available in Fedora; they can use a buildroot-only module to embed that dependency in their module without having to support it for other users of the distribution.

The problem with buildroot-only modules is that they make it difficult (if not impossible) for others to rebuild a module themselves. The process of rebuilding a module in general is rather more involved than building from a source RPM package. Buildroot-only modules make things far worse by hiding away the dependencies that were used to build the original module. The result, according to Kofler, "contradicts both the transparency expected from a community-developed project and the self-hosting expectations". Or, as Stephen Smoogen put it:

I find however that modularity is being used as a tool to weld parts of the engine away and it drives me bonkers. I can't just take a bunch of rpms from download.fedoraproject.org and rebuild them with some options to get what I want.

For an extreme example, see this post from Neal Gompa, describing why it is essentially impossible to rebuild Rust modules in Fedora 30. Among other things, those modules build with a private module containing a special version of RPM, so the source packages they create don't work on the target distribution.

Is it all worth it?

Given the difficulties, it is not entirely surprising that there have been some calls for Fedora to drop the Modularity initiative entirely. Simo Sorce, for example, asked whether it would be better just to use containers for the use cases targeted by Modularity. Gallagher responded that "one of the main points of Modularity is to provide a trusted source of software to install into containers". Daniel Mach, one of the Modularity developers, argued that containers can't solve all of the problems, and that Modularity is needed for the reliable provisioning of containers in the first place. He also worried that some of the problems with Modularity cannot be fixed without massive changes, though.

While others might like to see the end of Modularity, its total removal is not something many developers are actively arguing for. That may be because they realize that it is unlikely to go away regardless of what they think. As Gompa put it: "because it's in RHEL now, no one can afford to let it fail". Gompa also said, though, that "while it is a hard problem to solve, it's a worthy one" and lamented that the Fedora project lacks the infrastructure resources it needs to implement this idea properly. Miller said that the problem is "a fundamental one we need to solve in order to continue to be relevant not just as an upstream for RHEL but in general". So the project is unlikely to walk away from Modularity at this time.

Modularity 3.0?

Over the course of these discussions, a number of approaches for addressing at least some of these problems have been raised. One of those is to do what the project has already done once: drop the current Modularity implementation and design a new one. Adam Williamson, for example, stated that it was a mistake to deploy this version of Modularity into RHEL 8, but that "inventing new stuff is hard" and that is how we learn:

We just need to hold our noses and fix the icky problems, and then sit down and think about the design issues that have become apparent in Modularity v2 through our actually implementing it and using it (which is what Fedora is for, remember!) and figure out how to address them in Modularity v3.

Robbie Harwood asserted that "starting from scratch should be an option". Gallagher replied that there would be no fundamental redesign of Modularity, though. He pointed out that development on Modularity is funded by Red Hat, and the developers so funded are committed to supporting the current Modularity implementation in RHEL 8. "A full redesign in Fedora is not realistically possible with the people and resources we have available to us while also maintaining the current implementation for ten years". Instead, he said, the focus will be fixing problems in the current implementation.

Miller also stated his support for the current Modularity effort:

But the team in Fedora actually working on Modularity today includes some pretty smart, very invested Fedora people and I don't feel bad at all about standing up for their wanting to continue to refine the path they've chosen and are working on.

He suggested that if others want to see a fundamental redesign of Modularity, they should work on creating it and the results could be evaluated once a prototype exists.

Fixing Modularity 2.0

With regard to the upgrade issue, there are a few ideas in circulation. One of those is, as mentioned before, to disallow module-only packages from Fedora. That seems unlikely to fly, though, for a couple of reasons. Gallagher pointed out that converting module-only packages back would be difficult at best, especially for those that rely on buildroot-only modules. Forcing packagers to create both modules and ursine packages would add to their workload, which could cause some of them to leave the project. Smoogen noted that the number of packagers is already in decline; developers will be leery of changes that could accelerate that trend.

Miro Hrončok suggested that the solution to upgrades is to make the default stream for modules behave like an ordinary package. He quickly followed up with a grumpy note after the Modularity maintainers voted not to pursue that idea, saying instead that they would "implement a mechanism of following default streams to give people the experience they want". In other words, users will be left dealing with modules whether they want them or not, and the proposed solution leaves many feeling less than entirely happy.

The current round of discussions was actually touched off by this proposal from Gallagher on how to handle the update problem. It involved a complex set of module states meant to record the user's "intention" for a module, along with rules for state transitions. In short, it would allow a module that was installed as a dependency to switch to a new stream if the depending module changed.

Later, Gallagher posted an alternative proposal involving a new "upgrades:" attribute on modules that would specify a stream name. A module tagged in this way would, during a system upgrade, replace the current stream if it matches the given name (and if a few other conditions are met). Neither proposal was received all that well; to quote Zbigniew Jędrzejewski-Szmek:

In short, the project does not yet have any sort of convincing answer for the upgrade problem. That, in turn, suggests that these discussions are far from done.

With regard to the problem of the one-way transition to Modularity, Gallagher said that the problem is being looked into. Reverting to an ursine package is something that would be useful in a number of situations, he said. "We haven't figured this one out yet, but it's on the queue."

With regard to modular dependencies forcing other packages to turn into modules: the current proposal is to change the rules so that the non-modular buildroot can contain module streams. This plan, dubbed "Ursa Prime", would make dependencies available for other packages to build on without those packages having to be modules themselves. The November 11 meeting of the Fedora Engineering Steering Committee approved Ursa Prime for the Fedora 32 release, though it will start with only two modules. Not everybody is at ease with this plan, but this test will at least show how it will work in practice.

Buildroot-only modules are another outstanding problem. They are included in a set of requirements for Modularity posted by Gallagher: "Build-time only dependencies for an alternative version may be excluded from the installable output artifacts". Many developers would like to ban them as being fundamentally incompatible with how Fedora is supposed to work, but there is a tradeoff: as Smoogen pointed out, an alternative to modules using buildroot-only dependencies may be the removal of those modules altogether. Still, that may well be a price that many in the Fedora project are willing to pay.

Inventing new stuff is hard

Established Linux distributions tend to have a following of longtime users who are often hostile to fundamental changes. Those who have adopted a solution because it works, and who have since found ways of dealing with the parts that don't work as well, tend not to react well if others want to come in and shake things up. That is one reason why any big change in the free-software world tends to be accompanied by interminable mailing-list threads.

Distributors, though, have reason to worry about their relevance in a changing world. Software distribution has changed considerably since the 1990s, when many of the organizing principles behind most Linux distributions were laid down. A failure to change along with the wider industry and provide features that new users want is not likely to lead to good results in the long run. So it is not surprising that distributions like Fedora are experimenting with ideas like Modularity; they have to do that to have a chance of remaining relevant in the future.

Incorporating such changes can create pain for users, but that does not always mean that the changes are fundamentally bad. Those of us who lived through the ELF transition might have been happy to live with a.out forever, but that would not have been good for Linux as a whole. One of the truths of software development is that it is often impossible to see all the consequences of a change before implementing it. The value of free software is that we can implement those changes, see where things don't work, and fix them. The downside is that we have to live through the "see where things don't work" phase of the process.

Fedora is deeply within that stage when it comes to Modularity. Since it's a free-software project, all of the difficulties with Modularity are being exposed in a highly public way. But, for the same reasons, those problems are being noted and effort is going into proposing solutions. Eventually, the Fedora project will figure out how Modularity fits into its distribution and how to make it all work well. Users may wonder someday how they ever did without it. But there will be a lot of emails produced between now and then.

---