CERN's migration path from CentOS Linux to Debian

Please consider subscribing to LWN

Subscriptions are the lifeblood of LWN.net. If you appreciate this content and would like to see more of it, your subscription will help to ensure that LWN continues to thrive. Please visit this page to join up and keep LWN on the net.

The European Laboratory for Particle Physics, usually just called CERN, is not only the birthplace of the World Wide Web, it is home to the Large Hadron Collider (LHC), the world's largest and highest-energy particle accelerator. As such, its computing environment is both truly unique and of great interest to people outside of CERN who hope to find lessons applicable to their own computing needs. The upcoming migration of some of CERN's systems from CentOS Linux to Debian, which was the topic of a talk at the recent MiniDebConf Winterthur 2026, is of particular interest.

Federico Vaga and Nikos Tsipinakis, both CERN employees, delivered the talk "Controlling CERN's Accelerators with Debian" (video and slides) on August 30. I was not fortunate enough to attend in person, but watched the video of the talk shortly after.

Powering up

Vaga has worked at CERN since 2014; he described himself as a software engineer and FOSS enthusiast "with little free time to make concrete contributions". Tsipinakis has been with CERN for about five years as a system engineer. He said that he is an open-source maintainer for "a few things", has been a Debian user for about ten years, and a Debian package maintainer for about seven years. He said that he hoped to become a full Debian developer at some point in the future.

CERN's main goal was not to invent things like the Web, he said, but "to operate the biggest accelerator complex on the planet", and to host experiments involving the LHC. He showed a slide with pictures of people standing near some of the collider's machinery: "You can judge yourself the size by looking at the people walking there. These are really huge systems to make discoveries about things that are more than invisible, [they are] subatomic." The systems are generating a large amount of data, which is then streamed to CERN's data centers: "We store just part of what gets actually fetched from the sensors because otherwise it would be too much". In the end, petabytes of data are stored for analysis.

However, Vaga said that he would not be talking about CERN's data center, which primarily uses Red Hat Enterprise Linux (RHEL) or the RHEL derivative, AlmaLinux. Instead, he would be focusing on the computers and operating systems that actually run the chain of accelerators and then smash particles "to see what's inside". The accelerator complex is distributed in an area of more than 43 square kilometers in size, and is 100 meters underground at its deepest point. The bulk of the accelerator is underground since it would be challenging to have to travel from one side to the other using surface streets: "We would need crossroads for beam particles and cars not to collide among themselves".

He put up a slide, shown below, with a map of the area where the complex is located. A large blue circle highlights the collision track of the LHC, the Geneva International Airport is highlighted in a red rectangle for comparison. The smaller blue circle is the second-largest machine in CERN's complex, the Super Proton Synchroton (SPS).

Since this is a "very special system" there is no factory that churns out parts that are suitable; CERN has a lot of custom equipment to operate the complex that is designed in-house and then produced in collaboration with industry partners. There are more than 36 kilometers of beam pipes, with 17,000 devices scattered throughout the complex in total and about 70,000 cables connecting those devices to about 2,200 computers in order to operate the devices remotely. Those numbers continue to grow, Vaga said, because CERN always has new needs. As with the parts that make up CERN's accelerator equipment, the PCI cards and such that control all of the devices are usually designed at CERN as well. CERN is more than its hardware, cables, and pipes, he said. It is also made of people.

We have a very heterogeneous population. We have software engineers, like we are, system administrators, we have mechanical engineers, chemical engineers, we have civil engineers to design the caves for the tunnels, but also our own buildings. So we need to do a bit of everything. And of course, scientists, physicists, eventually they need to analyze all this data and tell us the meaning of life.

Because CERN has such a diversity of people with various competencies, and such a big machine with specialized needs, it strives for stability. "So we need a very stable system, and it's sometimes challenging to keep it stable."

CERN and Linux

CERN began using Linux in 2008, "with the realtime patches on top"; prior to that, it used Microware OS-9 and LynxOS, both real-time operating systems. At first, CERN started with its own Linux distribution based on RHEL 5 and made in collaboration with Fermilab, which is a 6,000-acre particle physics and accelerator laboratory based in Batavia, Illinois. That became Scientific Linux based on RHEL 6.

Eventually, Vaga said, it was decided that it would be better to contribute to an upstream project like CentOS Linux rather than rebuilding their own, "because there was nothing very special about it". Pat Riehecky of Fermilab announced the end of Scientific Linux in April 2022, in favor of CentOS Linux 8. "We will collaborate with CERN and other labs to help make CentOS an even better platform for high-energy physics computing."

After that, Vaga said, "we were more or less on autopilot with RHEL derivatives". Unfortunately, the life of CentOS Linux 8 was cut short: Red Hat announced that it was ending CentOS Linux in favor of CentOS Stream. That raised some questions about what to do next. There are thousands of distributions, he said, but which one would fit CERN's needs? "It was important to judge what is the level of control and maintenance that we can afford in operating and configuring an operating system and how well it integrates in the current infrastructure".

CERN first considered RHEL-based systems. CentOS Stream was the first obvious candidate; it didn't guarantee the same stability, but it was not as cutting edge as Fedora. It was decided to start with CentOS Stream 9, with the idea of moving to 10. "However, we knew that we could not afford to have more surprises, so the first thing we did is to make our infrastructure more portable to other distributions and having Debian as a plan B solution."

RHEL to Debian

Debian quickly became plan A, however. The final straw that broke the camel's back, he said, was "a simple compiler flag". Specifically, Red Hat's decision to optimize RHEL 9 for the x86-64-v2 microarchitecture, and then x86-64-v3 for RHEL 10. Turning on gcc's -march=x86-64-v2 flag would have major consequences for CERN.

Vaga's analysis of the situation found that 47% of CERN's systems, which have older CPUs such as the Intel Core Duo line, would be unable to run the v2-optimized binaries, and the systems would not boot. "You get a message from glibc, 'I cannot go further than telling you I cannot go further'." Another 17% of CERN's systems were unable to boot using v3-optimized binaries. That meant that nearly 65% of its systems would not work.

Simply upgrading to newer systems is not really an option for CERN. PCs made before 2010 were generous with PCI slots, but today's are not. "You are lucky if you get one on your desktop" systems, he said. Industrial systems might have two slots, "if you pay a bit more", but that's still not enough. "This means that some systems that must be reconverted, one system might need three computers, and this occupies [more] rack space than it was occupying before."

CERN also uses VMEbus single-board computers that would be rendered obsolete by CentOS. "This is our workhorse for the control system". CERN has designed a new generation, "but the production might be slower because of a war somewhere, because of AI somewhere else, and the price increases, and so on and so forth". As Tsipinakis explained later in the presentation, the specifications for CERN's electronics are public, "but I'm not sure someone else would have a use for them other than if they want to make a particle accelerator". It also means that CERN has a number of custom kernel drivers for its hardware, which makes maintaining the stability of its systems "much more difficult" than if it only had custom user-space applications, since any bug in the drivers could affect the overall stability of its systems.

So CERN would be faced with upgrading a huge number of systems, and with much greater logistical problems than most organizations face when swapping out old equipment for new. CERN's systems, he reminded the audience, are distributed over 43 square kilometers, often underground. After loading a vehicle with the new computers, "you go underground, you change, you re-cable, and perhaps you [mistakenly] swap two cables, and you will discover this only later at commissioning time, and this might happen more often" than it would through normal maintenance.

Even worse, adding new systems might require additional rack space, which might require actually excavating additional space to make room. It would also require two electronic engineers to redesign control boards, two software engineers to develop new drivers, and two additional technicians to physically replace systems. "It starts creating a mess, and meanwhile all these people must be kept happy, because otherwise they cannot do their job. So it's very challenging." Vaga said that the estimated total cost of upgrades would be about 5.4 million Swiss Francs, or about $6.67 million.

In the end, it made more sense to migrate away from RHEL-based systems to Debian, "a universal operating system that tries to work on as much hardware as possible", for computers controlling CERN's accelerator. CERN will now use Debian, but build its own kernel from kernel.org sources and with all of the needed drivers built in.

The long shutdown

The change to Debian-based systems also requires CERN to revise the way it provisions systems. CERN's systems are diskless, with the GRUB bootloader delivered over FTP and other components delivered over HTTP, he said. All of the artifacts are stored in a GitLab package registry, and there is a configuration service that indicates the operating system version, kernel version, and packages that a system needs.

Tsipinakis said that, once the decision to switch to Debian was made, the first thing they had to do was to look at the release schedule. The accelerator is running 24 hours a day, seven days a week, and no updates can be provisioned during that time. There are short "technical stops" a few times a year, he said, but those gaps in the schedule are not long enough to accommodate a major distribution upgrade. "So you might have a five-year plan", he told the audience, "But at CERN, we have a minimum of a 15-year plan, so we know the state of the machine all the way up to 2041". He displayed a slide, shown below, with the long-term schedule for the CERN accelerator complex.

The green portions of the graph indicate when the accelerator is in operation. The blue portions are short-term technical stops for maintenance. Red portions are "what we call the long shutdown, which is when we perform the big maintenance of replacing hardware, major software updates, and so on". Until now, CERN has been running CentOS Linux 7, and the plan is to deploy Debian in early 2027 and keep it as stable as possible until the end of 2033 when the accelerator will again enter a long shutdown.

He then went into a discussion of Debian's release cycle, its long-term-support (LTS) cycle, and the extended LTS (ELTS) project managed by Freexian. The regular lifetime of a release is three years, with an additional two years as an LTS release, and five years beyond that as an ELTS. That meant that CERN could stick with Debian 13 ("trixie") through 2035 if needed. "So this was really valuable for us because if we find out that we cannot do major updates during the run, this is a very valuable backup plan, so we can stay with the same distribution for the entire period."

CERN's layered infrastructure

Now that CERN had mapped out a plan that aligned Debian's releases with its own schedule, it had to figure out how to actually build and deploy Debian on its infrastructure. Until now, since CERN had a homogeneous environment with its infrastructure entirely based on RPM-based systems, most of the people at CERN had little or no experience with Debian. Tsipinakis said they approached Freexian and asked for help. "So they sent us Seb (Sébastien Delafond), who did a four-day-long Debian training" that included package building, hosting APT repositories, managing build repositories, and more.

Tsipinakis noted that CERN's infrastructure is split into three layers and not all of those systems are being moved to Debian. The first layer is what it calls "consoles", which are "basically desktop computers in various rooms about CERN". These are used by physicists or the accelerator's operators, and have applications written in Java or Python used to send commands to the accelerator. "It can be, 'I need to ramp up the magnet strength to this, or increase the intensity of the beam to that'". Those systems are running AlmaLinux 10, with the Xfce desktop. "They will stay as is."

The next layer lies within CERN's data center; after an operator sends a command to the accelerator it is logged, authenticated, and analyzed in the data center to ensure that it's within the safety parameters of the system. He said these systems are currently running RHEL 9, and will likely be migrated to AlmaLinux 10.

What is being migrated to Debian is the third layer, Tsipinakis said: the systems that lie 100 meters underground.

They are paradoxically called the frontend computers. However, if you are an electronic engineer, the computer that you plug your magnet into is the frontend, and everything else is the back-end. So these frontend computers, being migrated to Debian, they are responsible for receiving the commands from the data center and then implementing those commands with hardware signals.

So you receive a command, okay, ramp the magnet strength to this value. This means they will set the command to a power supply to ramp the amperage to this value that will increase the magnet strength. So they're basically the interface between the electronics world and the software world.

Build and deploy

Before introducing Debian, CERN's "legacy" infrastructure—designed in 2005, before the first run of the LHC—had a single bare-metal hardware server operating as its boot server. It provided GRUB and kernel-boot configurations over FTP, and the root directory and applications over NFS. He said that this model had some advantages, such as knowing that every system has the same root directory. It also had a big disadvantage.

This server is the Holy Grail of the data center. You do not approach this server, you do not breathe on this server, because if this server goes down, a lot of people at CERN will have a very long day, because it means all the 2,000 embedded systems suddenly lost their root drive

Tsipinakis noted that CERN's infrastructure was "kept simple on purpose"; it was designed to have as few moving parts as possible. Now, "we have new technologies, we can do something better". Since most of CERN's infrastructure will remain on RHEL-like systems, they had to figure out how to compile Debian software on RHEL. He said that the Embedded Linux Build Environment (ELBE) stood out, "you run the script, and now your entire environment, GCC, and all the other build tools you need are coming from the Debian image, so you can basically compile Debian software on Red Hat".

It also meant introducing a Kubernetes-based, multi-server boot system to replace the "big bare-metal server we cannot touch", and introducing Debian package repositories for applications served via GitLab package registry.

Challenges

There were a few challenges along the way, Tsipinakis said. The first thing CERN ran into was that there is no standard tooling for automating the building and publishing of packages. Yes, he said, Debian has software it uses for building and publishing packages, such as buildd and the Debian Archive Kit (dak): "but these are very specialized Debian [tools], and I think people know that if you want to run dak on your systems, it is not easy to set up or easy to use".

He thought that Debusine, a continuous-integration platform for packaging and distribution-related workflows for Debian-based distributions, looked very promising. Sadly, he said, "it has some blockers that at this point we cannot support in our infrastructure" though he did not elaborate on what the blockers were.

He also complained that many of the tools do not support having multiple versions of the same packages in a repository. CERN wants to be able to distribute different versions of the same packages to different systems. Tsipinakis noted that there is a branch of reprepro, which is a tool for managing APT repositories, that supports multiple package versions; however, it is still in Debian's experimental repository.

CERN already has an extensive infrastructure around building RPM packages, based on Fedora's Koji build-system software. Since Debian is not currently the largest part of CERN's package-building infrastructure, he said, "why don't we bring Debian to the RPM infrastructure?" That is, in the end, what they did. Tsipinakis gave a quick shout-out to Erick Andrei Vilcica; he did the work to create a Koji plugin to allow it to build Debian packages and create repositories for them.

The final takeaway, Tsipinakis said, is that it is a mistake to assume that if you have a working system on one version of a distribution that it will keep working if you update to the next major version. "This is exactly the reason that we are here." Not being bound to a specific distribution, and having the ability to switch, is important. He added that the accelerator would shut down on September 1, a few days after the talk was given, and that all 2,200 of CERN's embedded industrial computers would be migrated to Debian trixie after that.

Q&A

The first question came from former Debian Project Leader Andreas Tille, who wanted to know if it would be possible to hold the next MiniDebConf Switzerland at CERN. Vaga said that it would "definitely be an option", but it would require going through the bureaucracy first. He said he would "come back with news" at a later date.

Another attendee wondered if they had considered rebuilding CentOS Stream without the v2 and v3 compiler flags. Vaga said that was discussed with the AlmaLinux team, but "no one wanted to become responsible for it" at the time. "It's not that you remove the flag and it's all fine; you need to go through a validation process, and our main goal at CERN is doing particle physics". AlmaLinux has since announced the availability of a version of AlmaLinux 10 built with the v2 compiler flags, though it does not offer a version without v2 or v3 flags.

One person wanted to know why CERN was concerned with keeping its frontend servers up-to-date; couldn't it run a "vintage" version of an operating system? "Is there some security concern that you need to address?" Tsipinakis confirmed that the systems were "not on the public Internet and not even on the CERN network". They were in an entirely different network domain, which was more secure because "you have to go through two layers" to get to them. Nevertheless, he said it was still good to keep patch levels up-to-date, though not "at the same level as the bleeding edge". Security updates for those systems happen about once a year during a technical stop.

Vaga added that updating systems was also important so that CERN could use more modern technology. Up until a few months ago, he said, everything was running CentOS 7, which meant that they were stuck with GCC version 4. "You are in an environment that is frozen in the past." Some people had begun to maintain their own compiler in order to move on, and so forth.

The final remark came from Bastian Germann, the maintainer of reprepro. He said that the experimental branch of reprepro still needed more testing, and that he had received a flood of emails complaining that it broke people's repositories the last time he uploaded it to Debian's unstable repository. He said he would love to have CERN's input if it could switch to the experimental branch. Tsipinakis said that was the plan and that they would get back to him after they had done so.