This article brought to you by LWN subscribersSubscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.
Compromised accounts are one of the most common ways that attackers can sneak malware into the open-source supply chain. One way to reduce account compromise is for projects to require two-factor authentication (2FA) or multi-factor authentication (MFA), but that is easier said than done. However, Fedora is currently discussing putting 2FA requirements in place soon, following an an alleged account compromise that led to an AI agent causing a number of problems for the project. After some discussion, Fedora will begin by requiring packagers in the "provenpackager" group to enable 2FA within the next three months or so.
"Rather embarrassing"
Fedora took notice of the agent's activities in May, but it seems
to have had access much earlier; how much earlier is unclear. Its
contributions appear to have been benign, if unhelpful, but there is
no reason to expect that Fedora's luck will hold if another account is
compromised. On June 11, Daniel P. Berrangé replied
on Fedora's development mailing list in the thread about the
unsupervised AI agent. He pointed out that Fedora had considered
mandating 2FA two years ago, after the XZ backdoor, but had yet
to do so. If the episode was truly a case of password compromise
leading to account takeover, "then this is rather an embarrassing
situation for Fedora
".
Of course, Fedora is not alone; none of the other major Linux distributions with community contributors have 2FA requirements, and it does not seem that there is widespread infrastructure support for it, either. Debian's Salsa collaboration platform does not support enabling 2FA as far as I can tell, nor does openSUSE's Open Build Service (OBS). Ubuntu's single-sign-on service, which is used for its Launchpad collaboration platform, does allow users to enable 2FA, but it is unclear whether there is any requirement for Ubuntu contributors to actually do so. However, Debian and Ubuntu require signing packages with an OpenPGP signature before upload. (Update: Salsa apparently does support 2FA. Apologies for the error.)
Michael Catanzaro said
that he uses 2FA for "basically everything *except* Fedora
",
even though his Fedora account would be of high value to an
attacker. "Compromise a Fedora packager and you can push malware
more or less directly to users.
" However, he argued that Fedora is
not ready for 2FA.
His complaint was that he did not want to use 2FA until GNOME's online
accounts feature supported Kerberos ticket renewals; Fedora uses
Kerberos
authentication for some infrastructure, such as its koji
build system. Catanzaro followed
up to say that he was pretty sure he'd made the same objection
before, perhaps several years ago, and no progress had been made
since. "Unfortunately we're all busy with our usual work, and
nobody has been prioritizing these problems. So: basically the usual
explanation for how things happen in open source projects.
"
Support for Kerberos 2FA in GNOME's online accounts feature seems to have been implemented years ago, but has not yet been accepted. Alexander Bokovoy said that he had submitted a merge request to enable MFA Kerberos authentication, but it was stuck on the GNOME side. The feature was submitted in June 2024, and it has been reviewed extensively since, but not yet merged. Catanzaro pointed out that the request's status was still set as "Draft"; perhaps it will see the light of day soon once Bokovoy clicks the proper button.
Make it mandatory first
The problem with waiting until Fedora was fully ready for 2FA, Berrangé said, was
that no one was motivated to prioritize 2FA improvements because its
use was not mandatory; if it were mandatory, someone would make it
a priority to fix the remaining problems. "Or we could continue
ignoring the problem until another Fedora account's credentials are
compromised and does greater damage that causes Fedora significant
reputational harm.
"
Stephen Smoogen replied that he would love it if that would fix things, but the reality is more complicated:
Anything which slows down builds or makes an already complicated system worse, gets pushed down the queue over higher priority items. Pretty much every time it is said "This time will be different and you have this top priority to work on this over other items" gets a week later "I know we said that, but we really need to make sure that this compose gets out for X reasons so drop whatever you are doing and help".
He noted that the Fedora infrastructure team had been trying to
replace its Nagios monitoring with "anything else
" since 2009,
and other critical infrastructure almost as long, but the priority of
those projects does not take precedence over producing the builds for
Fedora's twice-yearly releases.
Adam Williamson thought
that Fedora should make people use 2FA even if some things wouldn't
work immediately; the prevalence of real-world attacks making use of
account compromises outweighed the user experience. In addition, he
argued that making people use an open-source project is a way to
ensure that it gets better. "We've been sitting on our hands saying
'yeah, well, mandatory 2FA would be great but somebody needs to fix
<long wishlist> first' for years now. It clearly isn't
working. We need to do something else.
" He added that he had been
using 2FA for Fedora work for years, and thought it was fine. The
Fedora wiki's instructions
for Kerberos authentication cover using 2FA.
At a minimum, he wanted to mandate 2FA be used by members of Fedora's provenpackager group; anyone in that group can commit changes to any package, not only the ones that they own. Thus, a compromised account belonging to someone in that group could do a great deal of damage. He said that Fedora also needed to enable SSH authentication for the project's Forgejo-based collaboration platform, Fedora Forge, require use of SSH keys by proven packagers, then extend that requirement to all packagers.
The forge is hosted on an OpenShift instance, and it is not
currently possible to access Git repositories on the platform via
SSH because the OpenShift cluster is behind
a proxy that is not directly connected to the internet. On
June 13, Kevin Fenzi said
that he had been working on enabling SSH access, "slowly in the
background
", and had made some progress but there was yet more
work to be done.
Simo Sorce objected
to requiring 2FA, on the basis that some people would use the same
application for their password and to generate 2FA tokens. "Sadly
whenever you improve security you find someone that self-defeats it
better.
" That did not deter Williamson, though; he replied
that 2FA could still be useful in many situations.
Convinced
On June 12, Catanzaro said that he had been convinced it was time for Fedora to require 2FA. Gary Buhrmaster wondered what the process was to make 2FA a requirement. Berrangé said that the last time it was proposed it had been submitted to the Fedora Engineering Steering Committee (FESCo).
That was in March 2024, when Miroslav Suchý submitted a proposal to make 2FA mandatory for all Fedora packagers. FESCo accepted the proposal, but watered it down; instead of requiring that all packagers enable 2FA, FESCo changed the policy to say that those in the provenpackagers group "SHOULD" enroll in 2FA. It would consider changing the policy to "MUST" once the user-experience for 2FA was improved.
Little has changed since then to improve the 2FA
experience. Williamson followed
up the current thread on June 15; he said that he had spoken
to Fenzi at Flock
and learned that "it's technically difficult to really implement
this
" at the moment. It would be possible to enforce use of 2FA by
monitoring whether actions, such as commits to Fedora's DistGit
system, had made use of 2FA. "Then we can go yell at people who
didn't follow the rule
".
That practice would not scale well for large groups, such as
Fedora's provenpackagers group, he said. "So I didn't file
the ticket yet for that reason. I'll talk more with Kevin about what
might be a way to move forward here.
" FESCo member "Maxwell G",
however, said
that he had filed a ticket with FESCo to
require 2FA for proven packagers. He acknowledged in the ticket
that it was an open question how to implement it, so he was not yet
proposing that all packagers be required to use 2FA.
In the proposal's discussion, Fenzi noted that there were 114 people in the group, and 59 of those people did not have 2FA enabled. Maxwell G said that it would be possible to check for 2FA enrollment when adding new people to the group, and set a date to remove people from the group if they had not enrolled in 2FA by the flag date.
FESCo decision
The topic was discussed during the FESCo meeting on June 23 (summary, log). One sticking point during the discussion was Fedora's lack of a recovery mechanism if a user loses the device used for 2FA. Many services offer backup codes that can be used if a user does not have access to their one-time password (OTP) application or hardware.
Both Neal Gompa and Michel Lind said that they used 2FA for other services—but avoided it for Fedora because there was no backup recovery method. Fenzi said that, if all else failed, a person who lost access to their OTP device could email Fedora's admin group and recover their account that way. That is not an ideal solution, but one would hope it would not be necessary often.
After discussion, FESCo decided
to require members of the provenpackager group to enable 2FA. The vote
was seven in favor, none against; Gompa cast
a "+0" vote to indicate his unhappiness with the current 2FA
recovery support. "I wish I could be more supportive of it but I
feel we're setting people up for failure
".
There will be an email to the Fedora development list (draft) as well as an email directly to the members of the group to notify them of the new requirement. There will be a grace period of three months before the requirement would be enforced. After that, users will be removed from the group if 2FA is not enabled. Once it is enabled, they would need to file a request with the infrastructure team to be added to the group again.
The addition of 2FA for Fedora's 114 proven packagers is a small step in adding layers of security to the project, but it is progress nonetheless. By deciding not to make perfect the enemy of the good, the project might save itself and its users some headaches down the road. It may also ensure that the project is able to work out the rough spots and eventually require 2FA for all packagers.