Abandoning Scientific Linux Was a Mistake

12 min read Original article ↗

A recent CERN announcement caught my attention. By the end of 2026, more than 2,200 computers and embedded systems around CERN's accelerator complex are expected to be running Debian 13.

Just to be clear, this is a very narrow migration. Debian support currently covers only the accelerator front-end systems, but AlmaLinux and RHEL will remain the main supported distributions across most of CERN. Still, it this took me back to the decision by CERN and Fermilab to abandon their RHEL based distribution called Scientific Linux, which started more than a decade ago. In my opinion, I think that was a mistake, and this post is my attempt to explain why.

Scientific Linux did not need to be technically better than CentOS to be worth keeping. It was never free or low effort to maintain, and it was never a lever over whatever Red Hat (or IBM, whose acquisition came later) might decide to do. Its value was that the scientific community had the people and the infrastructure to run its own Linux distribution. Too much of that work got written off as duplication.

Scientific Linux came out of a practical problem in high-energy physics. A large experiment spans many laboratories, universities, and computing centers around the world. If one site builds against one version of glibc, another has something slightly different, and a third runs a different packaging environment altogether, things get messy and painful fast.

The community needed a distribution that stayed stable for years, worked with enterprise software, could be freely redistributed, and ran across institutions without a commercial license for every machine (which would have been prohibitively expensive).

Red Hat Enterprise Linux had the stability and the long lifecycle, and Red Hat published the source needed to rebuild it. Scientific Linux turned that source into a community resource. Fermilab announced the distribution in 2003 and CERN joined development soon after. Universities, research institutions, experiments, companies, and even systems aboard the International Space Station ended up running Scientific Linux or something derived from it.

People who called it "RHEL with a different wallpaper" missed the point. The scientific community had its own implementation of Enterprise Linux, under its own institutional control.

When Red Hat and CentOS joined forces in 2014, moving to CentOS looked reasonable. CentOS already gave most Scientific Linux users what they wanted: a freely available Enterprise Linux rebuild with a much bigger general-purpose community, and now Red Hat itself behind it.

CERN started building its next major release on CentOS instead of Scientific Linux CERN, as CERN CentOS 7. Scientific Linux 5 and 6 stayed supported, but the future platform was going to be CentOS.

The engineering argument was simple. CERN and Fermilab had limited staff and more pressing priorities, and CentOS was already doing most of the same work. Converging on a bigger shared platform would free that time for other things. Red Hat's backing also made CentOS look more sustainable, and probably more reliable.

In 2019 Fermilab took the argument to its literal end. There would be no Scientific Linux 8. The lab would deploy CentOS 8 and work with CERN and other Labs to improve CentOS for HEP.

I understand that decision. What I think it undervalued was the ability to walk away if CentOS stopped meeting the Labs' needs, which is exactly what happened about a year later.

The cost of Scientific Linux was easy to see. Someone had to rebuild packages, test updates, maintain repositories, cut releases, ship security fixes, and support users. Probably a few dozen people worked on it over the years, and none of it was trivial. That cost showed up immediately, and it showed up every year.

This benifit was harder to quantify while CentOS was doing exactly what CERN and Fermilab expected and were doing with Scientific Linux. So under those conditions, Scientific Linux looked like a redundant effort that is hard to justify and easy to look at as obvious waste reduction target.

But the idea here is that by keeping the distribution alive, it also kept the people, processes, infrastructure, and institutional knowledge needed to produce a RHEL-compatible environment. That capability had option value. The community could go years without needing an alternative and still benefit from having one ready when its main platform changed or the license terms became unacceptable. You don't see what losing that costs until you need it.

And those assumptions changed surprisingly quickly.

[!NOTE]

CERN makes a similar argument for the FCC. The knowledge needed to build large machines like accelerators and detectors has to be preserved. If Europe doesn't build the next machine after the LHC, that expertise may be lost, the next big machine may be built in China (and now the US is in that conversation too), and Europe may lose its lead in the field. The same logic applies to software infrastructure.

In December 2020 Red Hat changed what CentOS was. CentOS Linux, the traditional downstream rebuild of released RHEL versions, was discontinued in favor of CentOS Stream, which sits ahead of RHEL instead of behind it.

People had picked CentOS because they wanted a freely distributable approximation of the current RHEL release. A new major version would have been one thing. Here, the product they had standardized on basically stopped existing.

CERN and Fermilab had to rethink their Linux strategy. CERN pointed out that Stream's shorter lifecycle didn't work for some scientific use cases. The two Labs looked at Stream, RHEL licensing arrangements, and the Enterprise Linux rebuilds that were appearing. By 2022 they were recommending AlmaLinux as the standard distribution for experiments (while also negotiating a RHEL site license with Red Hat).

So a few years after retiring Scientific Linux, partly because another RHEL rebuild seemed unnecessary, CERN and Fermilab adopted another independently governed RHEL-compatible distribution.

AlmaLinux is a good project and adopting it made sense. But the need it filled was one Scientific Linux had already been filling.

In 2023 Red Hat changed how RHEL-related sources were published. CentOS Stream became the only public repository for that material, replacing the old git.centos.org model.

That didn't make RHEL "closed source" in any real sense. It did make it harder to ignore how much the rebuilds depend on Red Hat. If you're compatible with a vendor's product, the vendor's decisions reach you, even when you maintain your own distribution. At the very least, carrying your own changes gets much harder.

Scientific Linux still had value inside that constraint. CERN, Fermilab, and other institutions could reproduce a critical piece of the ecosystem they relied on. Keeping that capability alive would have been easier than rebuilding it after the people had moved on and the infrastructure was gone.

CERN's accelerator group first tried to stay inside the Red Hat ecosystem. According to their talk at MiniDebConf Winterthur 2026, the original plan was CentOS Stream, with Debian prepared as a fallback because the team couldn't afford any more surprises.

Then hardware compatibility became the problem. RHEL 9 raised its x86-64 baseline to x86-64-v2, and RHEL 10 moved it to x86-64-v3.

For an ordinary server fleet, retiring old hardware is annoying but manageable. An accelerator control computer can be wired to custom electronics, specialized boards, legacy buses, and real-time systems, and equipment installed around the accelerator complex can stay in service for decades. Replacing one computer might mean redesigning the electronics attached to it. Now do that for thousands of computers across the complex.

CERN's 2023 risk analysis put the cost of staying in the Red Hat ecosystem at around 5.4 million CHF (~6.5 million USD for friends on the other side of the Atlantic). It could also mean redesigning about eleven boards, hiring more engineers and technicians, reorganizing racks, recabling, and recommissioning equipment.

Changing the operating system was a way to keep working hardware running. CERN (the Accelerator Department, at least) chose Debian, and the migration to Debian 13 should cover more than 2,200 computers and embedded systems by the end of 2026.

The accelerator team specifically praised Debian for continuing to support older and less common architectures, and for being a community-led distribution that can compete with Red Hat. There's a reason Debian is the distribution people think of first when they think of stability and long-term reliability. It also shows that the scientific community has options beyond RHEL and its rebuilds.

CERN is also sponsoring Freexian to strengthen Debian's long-term support ecosystem. That's a practical admission that community infrastructure costs money, and that paying some of that cost can be worth it. Consolidating on one vendor's ecosystem may save engineering time now and leave an institution with a much bigger bill later, which is part of the case for keeping Scientific Linux too. This is not your typical economy of scale problem.

The Scientific Linux project history lists only a handful of major developers in its later years. Fermilab provided the main sponsorship, the build infrastructure, bandwidth, and the website. I don't think that cost the lab much. It wasn't free, but it was a small fraction of what the lab spends running the rest of its computing.

A small team doesn't mean the work was cheap or easy. Release engineering, security updates, testing, package rebuilds, infrastructure, and user support all take real time. But the size of the team is one reason I doubt that ending the project saved as much as we assumed.

Scientific Linux gave an enormous scientific ecosystem a stable platform for roughly two decades. The engineer-hours saved by moving to CentOS were one side of the ledger. The other side was what it might cost to lose the ability to maintain that platform if the ecosystem around it changed.

The migrations that followed give at least a partial answer.

There's a stronger version of this argument: independent rebuilds constrained Red Hat, and losing one of the major ones helped concentrate power around CentOS and RHEL. I think there's something to it, but nobody can establish the counterfactual. Scientific Linux probably would not have stopped Red Hat from changing CentOS.

The timeline matters too. IBM completed its acquisition of Red Hat in July 2019, after Fermilab had already announced there would be no Scientific Linux 8. It would be misleading to turn this into a story about IBM killing something CERN and Fermilab should somehow have seen coming.

What I do believe is that a credible alternative changes incentives. A vendor has more reason to accommodate institutions that can leave than institutions that would first have to rebuild years of infrastructure (or spend months migrating to something different, like a Debian-based system). The possibility matters even if nobody switches and even if the alternative is small. It tells the people making decisions that switching would be cheap if the vendor changed direction. That kind of leverage is hard to quantify, but it's real.

Scientific Linux gave the Labs and other scientific institutions that option. And because scientific institutions maintained it, its incentives lined up unusually well with infrastructure that is expected to stay in service for a very long time. This is a dependency-management problem, and it goes well beyond Red Hat.

HEP can't run efficiently if every university and lab invents its own incompatible platform. CERN and Fermilab were right to want common interfaces, compatible binaries, common packaging, and predictable environments.

Scientific Linux could have served those goals while staying independently maintained. That's where I think the strategy went wrong: we assumed that adopting a common platform meant giving up our own implementation of it.

Scientific Linux could have stayed boring (it was certainly boring as a desktop). It didn't need to compete with Fedora on new ideas or with Ubuntu on desktop adoption. It needed to remain a reproducible, institutionally controlled Enterprise Linux that the scientific community could keep running when commercial priorities drifted away from, or ran against, the needs of scientific computing.

Most of its value would have been invisible during the years when everything else worked. It's a bit like keeping iodine tablets around in case of a nuclear accident: you hope you never need them, and you're glad they're there if you do. That's not a great analogy, because iodine tablets get used up, and Scientific Linux was a capability you could use again and again. It was also in active use the whole time. What carries over is that its value only becomes obvious once you need it. The rest of the time it sits quietly in the background.

I don't think the answer today is to bring Scientific Linux back. AlmaLinux and Rocky Linux exist (I like AlmaLinux's approach and have a personal preference for it, so viva AlmaLinux). Debian is turning out to be more and more useful at CERN. Containers and modern software distribution have also made the host OS matter less for many workloads.

I do think research institutions should change how they account for the infrastructure they maintain. We're good at calculating what it costs right now. We're much less comfortable putting a value on the ability to keep maintaining it.

Any decision to retire institutional open-source infrastructure should weigh governance, concentration risk, the cost of migrating if an upstream project changes direction, and the institutional knowledge that walks out the door with it. It should also count the value of an alternative that might sit unused for years.

[!NOTE]

I've heard a story that it was actually a US funding agency (probably the DOE Office of Science) that looked at this, asked why the Labs were duplicating effort with Scientific Linux when a working solution already existed, and pushed the Labs to move to CentOS and shut Scientific Linux down. I don't know if that's true, but the Labs aren't the only stakeholders in these decisions.

The CERN accelerator team closed its Debian talk with this advice:

"Distribution portability is a very good thing."

More than a decade after the move away from Scientific Linux began, the Labs have been through several Enterprise Linux strategies, adopted a new community rebuild, and started moving thousands of accelerator computers to Debian. I wish we had given more weight to the independence we already had.