Systemd, ten years later: a historical and technical retrospective
blog.darknedgy.netWhen writing software, it’s comparatively easy to come up with grand new ideas and turn them into lines of code and files filled with modules.
What’s much harder is (1) to expand ones code without coupling everything together (2) present it in a persuasive way that naturally builds a user base.
SystemD feels like the new grad hire who decided the first thing the would do after joining — the opening power play — is to convince the PHB we should rewrite everything in Haskell.
At this point the gentler folk start quietly honing their CVs and sneaking their personal belongings off their desks, one night at a time, in the hope no one notices they are all about to quit.
It felt like Unix systems administration was a haven from this sort of politics — I don’t remember tmux, git, or python trying to push behavior changes on upstream components of the OS[1]. I’m sure it’s why Unix attracted a certain type of personality for so many years and I’m sad to see that changing.
[1] If you are thinking “but those are command line utilities, whereas the init process is a much more specialized case” then I’d encourage you to revisit the Unix principles of every component being a small and simple program! Even init, cron, and login! It’s not some grand ideal to be zealously adhered to: it’s the principles of small components that mean the same OS can run on a quad core Xeon as runs on a $45 network switch.
I think you underestimate how badly the old sysvinit sucked in the world with many good distribution-provided packages.
It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages:
Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script.
This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate.
Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.
What you are missing is that it is possible for two things to be true at once. It is possible for sysvinit to suck and for systemd to suck in entirely different and potentially much worse ways.
It's as though someone is complaining about having to live under the USSR and how bad things are, and then someone responds that the German rocket program has been dismantled and the British never really had much of one and the Americans are doing okay but the fact is that the USSR was the first to put a man into space. As if that justifies everything else they did.
I hear that a lot.
It's not my experience.
Setuping services has been a pain for 15 years for me. Then with systemd it was suddenly easy.
Detractors keep repeating systemd sucks, but that this point, Debian and Fedora apparently decided it was good enough. And I never heard an end user complain.
I've stopped to be open to convincing.
I don't know if I qualify as an "end user" to you, but I definitely have complaints. I started with an open mind about systemd, because I'm not so attached to any init system in general. So my issue is not that systemd is a monolith or breaks with tradition (sysv init scripts were also a hot mess). My problem is that systemd looks and acts like a freshly written proof of concept that needs to be polished up, but when you dig in to debug a problem you're having, you inevitably end up at bug reports that condescendingly defend the current state of affairs as desired behavior.
"Your issue is user error, because systemd doesn't provide the logical guarantee you want, and adding a hook is outside the scope of systemd." I've run into several instances of this, the worse one being around unit ordering and network interfaces. I eventually solved it with a hack service that put 'sleep 90' between network.target and network-online.target. So much for that startup parallelism.
Another issue is that a dependency loop will cause it to just randomly drop units. There's no insight into this until you notice it happening and go digging. In the abstract, yes there is nothing else systemd could do apart from refuse to boot. Except the problem has only been dragged into the abstract by using a configuration model that punts on real world concerns.
Both of these are exacerbated by magic processes that half-parse crypttab/fstab/interfaces to create runtime units. I don't know if these are from Debian or the upstream. But if the systemd way is better, why haven't those been fully moved to native systemd?
> Debian and Fedora apparently decided
That's not how I remember it. Fedora was pushed to adoption (thread starter characterises it as an internal "power play"), and Debian was presented a fait accompli (article summarises Russ Allbery with "[not] systemd-vs-the-alternatives, but how-much-of-systemd").
> And I never heard an end user complain.
This is ridiculous! I can also claim the world is always dark when I shut my eyes.
Linux mailing lists are full with complaints. A recurring one was an error message that suggests running a command as remedy, only for that command to also error out. (The remedy was formulated wrong.) By then, systemd already had a version number in the hundreds.
It is even easier with daemontools, which is much more lightweight and follows the Unix philosophy.
But somehow distro maintainers can never integrate djbware for political reasons. In the case of RedHat, also NIH syndrome.
The article did a good job articulating that the "end users" of an init system aren't sysadmins, they're distro maintainers. If most distros have switched then it must make things better for them.
The problem, of course, being that for most init systems the "end users" are the distro maintainers but for systemd, because it spreads itself into so many other things, it also impacts the sysadmins and the upstream third party package developers. Who then complain about it and are met with "it was good enough for Debian and Fedora" even though they aren't the ones complaining.
Exactly, my biggest problem with the systemd complainers is that it's all kind of abstract things, like it's not the Unix philosophy, which in practice doesn't mean much and certainly doesn't automatically mean it's a bad thing.
SysVinit was a horrible, inconsistent mess of bash scripts of per-individual-developer variations in quality and not insignificant variations between distros, or even individual services within the same disto, I wouldn't hold that as the cornerstone of the Unix philosophy, that is if one's intention is to paint it in a positive light.
> What you are missing is that it is possible for two things to be true at once.
Actually, it's possible for more than two things to be true at once. It could be:
1. It could be true sysv init sucked. (Actually, it wasn't that bad as a modular version of local.rc, but parallel and delayed startup was always just a kludge).
2. It could be true that systemd is a good init system. (For me it's a step change improvement sysv init for parallel and delayed start up. About my only complaint is you don't see failures on "systemctl start xxx"; you have to run journalctl.)
3. It could be that adding another 1M lines of code into the systemd source unrelated to starting stuff was an unsound engineering decision - almost inexplicable coming as it does from a senior engineer. Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud.
4. It could be that moving sending state changes via an RPC interface (dbus) so you have a million places you have to inspect to understand the state of the system and how it got there rather than writing the desired state of the system to disk with a journal where everyone can inspect it was a bad idea.
> Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud.
I don't think this invalidates the thrust of your point, and I don't necessarily subscribe to the concept of monorepos, but Google is a monorepo, so senior engineers are definitely signing off on this sort of thing. Is it a ball of mud though? Actually does seem true more and more as time goes on.
Probably the biggest difference is that well-disciplined engineers utilizing monorepo do sustain those well-defined interfaces. The example of systemd-logind having no interface guarantees (cited in the linked article) is a sign of trouble.
Sysvinit is still around, you can always go back to it if you really want. The package is even still in debian.
This is the kind of false argument being thrown around often in OSS discourse that ignores the structural power differential.
Don't like it? Write your own/Leave.
The fact is you can't go back to it as an individual, because the system has changed and as an individual you're powerless to change the situation at all, especially against an army of developers paid full-time. The latest news from Debian is sysvinit support is no longer guaranteed.
If you can't pull your own weight and don't want to try anymore then yes, you should leave and come back when you can. Sorry if that sounds rude but this is the way it is. OSS (and software in general) has always been driven by people who are fortunate enough to have access to expensive computers and who have the free time and motivation to spend programming. You can't let what they do bother you, and there's nothing wrong with taking a break and coming back later when you're ready. I'm sure the companies who employ these armies of developers are hiring so you could probably work there too if you really wanted.
If anything the power differential has gotten much smaller in recent years with things like github, and last time I checked systemd was accepting pull requests. And even though they won't guarantee it, it sounds like Debian also will continue to accept contributions from those who want to spend time trying to support sysvinit. What exactly is the barrier you're having?
Before I comment here: I am a fan of systemd in many ways, and I happily use it on my systems. That being said, you do really have to think about the systemic factors at play here. As noted in the original article, Debian found itself in a position where the amount of work to sustain a non-systemd path in a systemd-dominated ecosystem would be too much work _for them_. If you personally have more development time then the entirety of the Debian project, then by all means. Let's just not assume that this is a feasible thing for a single person to handle, and really for all intents and purposes, trying to maintain a Linux distribution (or even just a personal Linux system) without using systemd at this point is folly, no matter how much you dislike it.
That being said, SysV init is (in fact) terrible. I'd say put the effort into something that can supercede systemd some day. That part of the problem is tractable, though success is quite a long-shot given its entrenchment.
The person who wrote the headlined article also wrote commentary back in 2014 on discussions of systemd, which highlighted the false dichotomy that people propound that the decision is between van Smoorenburg init+rc and systemd. You are doing that very thing, years later.
That was never the case, especially so for Debian that you mention. In the Debian Hoo-Hah, the choices were van Smoorenburg init+rc, OpenRC, Upstart and systemd; the latter three being the main contenders, as was acknowledged partway through the affair. In Fedora and Ubuntu, the choice was between Upstart and systemd, Upstart having been what they used for some years before systemd.
* https://web.archive.org/web/20141222234706/http://uselessd.d...
I do want to say that despite my comment here, seems like projects like Devuan, Void, and Arch are doing OK with this, in concert with projects like elogind and eudev etc that forego systemd while providing compatible replacements.
> Debian found itself in a position where the amount of work to sustain a non-systemd path in a systemd-dominated ecosystem would be too much work _for them_
Just as a side note for the interested, there is a project named Devuan that launched to keep alive a Debian sans systemd.
(I've never run it myself; I just happen to know it exists.)
Lennart and the systemd team did exactly that 10 years ago. Why can't you?
The code is there for someone with enough skill to show how bad systemd is right?
> Lennart and the systemd team did exactly that 10 years ago. Why can't you?
It would be impressive if Lennart managed that by himself. Red Hat wanted it done, Ret Hat also has its hands in various open source projects that suddenly started to sprout hard dependencies to systemd. Projects like Gnome were Red Hat is by far the biggest contributor.
Not much an individual programmer can do compared to a corporation throwing its weight around to break things.
> Red Hat wanted it done
Red Hat had zero interest into a new init system when Lennart started systemd. They had just moved to upstart and Red Hat customers don't really care about init systems. Red Hat customers pay for their systems to be stable and to have someone to call when something goes wrong.
Systemd gives Red Hat no competitive advantage whatsoever.
I'm probably not going to make myself a lot of friends by saying this but I don't really understand the systemd opponents.
The components systemd is slowly replacing or sitting on top, most of the low level userspace of linux, is mostly garbage and poorly documented garbage at that. PAM is aweful. I don't even want to talk about ConsoleKit. Cron has one of the worst configuration format ever and I'm certainly not going to miss fstab. I would find journald a step in the right direction even if the only thing it brought to the table was the ability to configure logging from the service file and not have to fiddle with logrotate.
Networkd gives you a nice, uniform and well documented way to configure both interfaces, rules, custom routing tables and vpn tunnels from declarative files. Who in their right mind can miss the hodgepodge of scripts that was there before ?
> I'm probably not going to make myself a lot of friends by saying this but I don't really understand the systemd opponents.
I think sections 3.5 and 3.6 (all of them, including subsections outlines current problems pretty clearly. Summarized, systemd is complex to reason about, ambiguous in how it functions, and poorly documented in many cases. All the very similar directives with slight nuances paint a picture of people that didn't really understand the problem space as well as they thought they did (and to be fair, nobody did, and they probably had the best understanding, as woefully inadequate as it was). For a sysadmin, who rarely (but not never!) has to dive into the intricacies of unit files and the myriad directives and their nuanced behavior for creating a server but less rarely is affected by odd behavior introduced by systemd, it's a liability.
What we have now is a chance to learn from those missteps and build something even better. Systemd as a catalyst for rapid chance was wildly successful. As an init system that capitalizes on those changes in a way that users (that is, system/distro packagers) can take advantage of, not so much, since it's so wildly complex. The article posits that BPF will be used to create programs to take over some of these tasks, which may be right. Alternatively, we could start pulling out the components that systemd subsumed, and formalizing them into new standards that others could develop for. Whether that means actually pulling out the component (e.g. logind) to a new repo, or just formalizing API is up for debate.
> Cron has one of the worst configuration format ever and I'm certainly not going to miss fstab. I would find journald a step in the right direction even if the only thing it brought to the table was the ability to configure logging from the service file and not have to fiddle with logrotate.
What you don't like about cron format? It is nice, consistent. Same for fstab, the only issue I have with it that e.g. Debian doesn't handle mounting network filesystems after the network is up (there are a couple of those and it shouldn't be hard to just grep the file for those).
As for journald, as long as they keep my log files in /var/log in pure text format it can stay.
The issue I have with systemd is that it tries to fiddle with my files, last year I noticed that /etc/resolv.conf contains some strange line with local resolver, WTF? I didn't install one for sure so why (and it was breaking my network).
> It would be impressive if Lennart managed that by himself. Red Hat wanted it done
RedHat wanted it done once Lennart convinced them it is the right thing to do and Lennart put in the effort to be at RedHat at the right time to have the ability to convince them.
In fact Lennart is one of the few people willing to put in the effort to touch the fundamental building blocks that otherwise rot, but nobody is willing to touch.
> Projects like Gnome were Red Hat is by far the biggest contributor
Yeah, I guess it's not that bad that in FLOSS, the people willing to ultimately sit down and put in the time and effort to actually write the code, even the non-sexy bits, get to steer the project over Hacker News commenters.
That's not the worst outcome out there if you ask me.
> The fact is you can't go back to it as an individual, because the system has changed and as an individual you're powerless to change the situation at all...
Use Void Linux, help Devuan etc. You're not alone, but you're clearly in small minority.
My personal favorite is Alpine Linux. It ditches a ton of other unnecessary cruft too. It feels lean and modern.
The problem is not that sysvinit doesn't exist, it's that other packages now have dependencies on systemd, so it doesn't just swap out.
Exactly, and to prevent this from happening is what Devuan is all about (as I'm sure you know): to keep all the maintenance work going into Debian, while remaining mostly bullshit-free.
It's what I'm about to be installing after a decade on the Ubuntu desktop. The recent snap fiasco is just the final nail in the coffin. The trifecta of SystemD, Gnome3, and SnapD means that I can just use Windows with WSL if I wanted a distro that is maximally intransparent and pushing things onto me (saying this without satire). Actually, gnome3 is really the worst regression here, and it being so strongly tied in to SystemD makes it an easy target to get rid of.
Other contenders I've been looking into: Void Linux, Slackware, and the BSDs (I used to run FreeBSD in the late 1990s/early 2000s). At one point there was even a variant of Debian (userspace) running on FreeBSD, which however was abandonded due to SystemD and cgroups/namespaces invading too much of Debian.
Good luck with Devuan (I have no experience with it).
If it doesn't work out I can recommend Void Linux. I've been using it for years as my daily driver since Debian switched to SystemD, both at work and at home.
I don't see how that is the problem. It's exactly what you asked for. If you're replacing a low-level component with something else that hasn't been updated in years and doesn't implement features that other packages do then yeah, you have to be prepared to accept that some newer things will break, and you'll have to revert a lot of other packages to older versions too. What else can we do? This isn't even a systemd-specific thing.
Systemd deliberately refused to conform to standardised interfaces and got other packages to be changed to explicitly depend on it instead (e.g. RedHat made changes to Gnome to make it break under non-systemd). This was not necessary to achieve the technical things that systemd wanted to achieve - competing alternatives (e.g. runit) didn't need that. (Though it probably was necessary to force adoption).
This is not true in any way. Gnome still works on non-systemd systems but you have to install elogind. And, there were valid technical reasons to have gnome depend on a login manager, in particular supporting multi-seat securely is very difficult without one. I should also mention that the previous "standardised interface" for this was ConsoleKit which was also written by the same developers as gnome and systemd. So this is not a matter of some developers going around and raiding projects and "forcing adoption", it's all stuff that happened under the same umbrella anyway.
If the "competing alternatives" don't need it then I would advise those projects to put their money where their mouth is and either start making improvements to elogind, or develop a new login manager that better fulfills the needs of gnome. In particular the solution used by elogind to deal with cgroups is to just disable them entirely, or at least it was last time I checked. I don't know what runit does for cgroups but a good step would be to make elogind compatible with that.
> This is not true in any way. Gnome still works on non-systemd systems but you have to install elogind.
elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility. It's like saying that windows programs work on Linux, you just have to use wine - it's sort of true for now, but it's not something you can rely on.
> And, there were valid technical reasons to have gnome depend on a login manager, in particular supporting multi-seat securely is very difficult without one. I should also mention that the previous "standardised interface" for this was ConsoleKit which was also written by the same developers as gnome and systemd.
I remember multi-seat working fine long before either systemd or ConsoleKit, so that seems pretty questionable.
> I would advise those projects to put their money where their mouth is and either start making improvements to elogind, or develop a new login manager that better fulfills the needs of gnome.
"The needs of gnome" are a constantly moving set of goalposts under the control of RedHat. Gnome worked fine long before systemd and systemd has not noticeably improved it (if anything the opposite); there was no technical consensus that the current hard-dependency was necessary, it was forced through because RedHat wanted it that way. So they're not going to accept patches to offer compatibility with non-systemd, and if they do then they'll just find a different way to hard-depend on a different part of systemd.
>elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility
Isn't that the whole reason for using another init? Because you see full compatibility with systemd as being undesirable and you have the means yourself to maintain things yourself using other solutions? I don't get it, it sounds to me like you're now saying the opposite of what you did before.
>I remember multi-seat working fine long before either systemd or ConsoleKit, so that seems pretty questionable.
Multi-seat might have worked but it did not do so securely. The issue is that you need a daemon to multiplex the display and input devices, and programs that want to request logins must be able to speak a specialized IPC protocol to that daemon. Otherwise all you have is a bunch of setuid root programs that do not communicate and can easily clobber each other at any time. Remember that the kernel does not have any built-in support for this, these are just device nodes in /dev like anything else, and in the old days the X server just used to run as setuid root at all times and sessions running in the background could read the keyboard whenever they wanted or take control of the display. The issue doesn't just affect multi-seat either. Even on your laptop you basically need a login manager to do secure screen-locking. There are an absurd number of ways that pure X- or Wayland-based screenlockers can be escaped. In an ideal world the kernel would probably prevent this stuff, but it doesn't right now, so login managers remain necessary.
>"The needs of gnome" are a constantly moving set of goalposts under the control of RedHat.
1. GNOME is an open source project not in control of any one company or organization, and anyone can contribute or fork the project if they so desire.
2. Nothing was "forced through" as the decision was agreed upon by the current and previous maintainers. If you disagree, you can revert to an old version as previously stated. This may be unpalatable to you but it is not realistically possible for anyone besides you to write code that you will agree with 100% of the time. It's up to you to make the tradeoffs you want on your system.
3. Also as previously stated, there is no hard-dependency on systemd. You can get it working with elogind. If you refuse to because you fear it will break in the future, that's on you. The fact is that it works now.
> Isn't that the whole reason for using another init? Because you see full compatibility with systemd as being undesirable and you have the means yourself to maintain things yourself using other solutions?
Compatibility is the only way it becomes possible to make those changes. E.g. in the pre-systemd days there were at least four different syslog systems and at least three different cron systems. But because they conformed to standard interfaces and were loosely coupled to the rest of the system, users were free to choose any of them, or to make their own. Same with desktop environments: you can run KDE or Gnome or Xmonad or whatever, but since they all follow common standards, any program works with any environment.
If the systemd people want to make a shiny new init with cool features, great. It's when they break compatibility with alternatives that I have a problem with it.
> 1. GNOME is an open source project not in control of any one company or organization, and anyone can contribute or fork the project if they so desire.
> 2. Nothing was "forced through" as the decision was agreed upon by the current and previous maintainers. If you disagree, you can revert to an old version as previously stated. This may be unpalatable to you but it is not realistically possible for anyone besides you to write code that you will agree with 100% of the time. It's up to you to make the tradeoffs you want on your system.
The fact is that most contributions are made by RedHat employees on RedHat time, so anything that can be decided by 51% of maintainers is under RedHat's control (and I can't do anything to change that since I can't outspend RedHat). There was a decision to adopt systemd but it was very much not a consensus; there was bitter disagreement at the time (and still is).
Forking by a minority is only practical if the system is made up of small components with standardised interfaces between them. Good open-source citizens follow those standards for the sake of allowing their users to do that (e.g. KDE migrated from DCOP to the technically inferior DBUS for the sake of being able to have common standards and compatibility with GNOME). If the whole system is a big ball of mud with no way to replace or decouple individual pieces, then you can't ever change or fix one part of it without forking the whole thing, and you don't have the Four Freedoms in any practical sense.
>Since they all follow common standards, any program works with any environment.... If the systemd people want to make a shiny new init with cool features, great. It's when they break compatibility with alternatives that I have a problem with it.
That is not relevant because the "common standard" in this case is the logind D-Bus API. There are no other active and viable alternatives to login management. It has two implementations, systemd-logind and elogind. You can choose either one, and only the first one requires systemd. Or if you really want you can choose to write another implementation. Or you can choose to hack GNOME and remove the logind bits and just run your display manager and shell insecurely as setuid root. Or you can revert it to some old version when consolekit was still supported. Or you can just do nothing and wait for someone else to guess what it is you want. You have options. If none of these will please you then I think you need to adjust your expectations because you can't have it all.
>The fact is that most contributions are made by RedHat employees on RedHat time, so anything that can be decided by 51% of maintainers is under RedHat's control (and I can't do anything to change that since I can't outspend RedHat).
I don't see what this has to do with anything. It's open source. If you want to collaborate, you can submit a pull request, work with the maintainers and get them to spend their paid time reviewing and merging your code. You don't need to outspend them, the code is already written and every bit of it lands on github ready for you to use it. Them getting paid more only benefits you.
>There was a decision to adopt systemd but it was very much not a consensus; there was bitter disagreement at the time (and still is).
I can't speak for other distros but looking at the actual Debian votes on systemd (not flamewars on mailing lists), your statement does not seem to reflect reality. Systemd, as well as support for alternatives like elogind, was voted in with strong consensus. https://www.debian.org/vote/2019/vote_002
>Forking by a minority is only practical if the system is made up of small components with standardised interfaces between them.
Systemd is composed of several small components, and its interfaces have become standard across many Linux distros, so this shouldn't be a problem.
>you don't have the Four Freedoms in any practical sense.
It is not reasonable to expect that every single person will have the time and expertise to contribute to any given project, or that every project will be structured in a way that lets them accept maximum contributions. I don't mean to dismiss your frustration with being unable to contribute. I get that. But you continue to have the opportunity to do so and you will for as long as these projects remain alive. The code is not going to magically disappear one day and it's up to you to find the time to actually dive in. Good luck.
> That is not relevant because the "common standard" in this case is the logind D-Bus API. There are no other active and viable alternatives to login management. It has two implementations, systemd-logind and elogind. You can choose either one, and only the first one requires systemd.
Is there any commitment to treating those as stable interfaces, with a decent deprecation cycle around any incompatible changes? If so then that's a big improvement; there certainly didn't used to be.
> I don't see what this has to do with anything. It's open source. If you want to collaborate, you can submit a pull request, work with the maintainers and get them to spend their paid time reviewing and merging your code.
I saw patches submitted and rejected on the grounds that the project leadership had no interest in supporting non-systemd (and/or non-Linux). Even if the leadership does want to accept a patch, that can take months; in my experience it's not worth making a change if you can't run it on your own systems without upstream getting involved. So if a project is large and not cleanly factored into smaller pieces with stable interfaces between them (or at least open to being factored that way in principle), then it's just not worth trying to contribute unless you can work on it full-time.
> It is not reasonable to expect that every single person will have the time and expertise to contribute to any given project, or that every project will be structured in a way that lets them accept maximum contributions. I don't mean to dismiss your frustration with being unable to contribute. I get that. But you continue to have the opportunity to do so and you will for as long as these projects remain alive. The code is not going to magically disappear one day and it's up to you to find the time to actually dive in. Good luck.
I believe in open source not because it gives me a chance to write code but because it lets me fix bugs or change behaviour that's bothering me. It's the "RMS printer driver" situation; if something goes wrong in Linux these days I feel like I'm no better off than I would be on windows. For the moment I'm using FreeBSD (and KDE) and I can still patch stuff without my patches getting broken every other month, but as things integrate more deeply with systemd it seems like a matter of time before the programs I use go Linux-only.
> I saw patches submitted and rejected on the grounds that the project leadership had no interest in supporting non-systemd
What do you mean with project leadership. If someone submits a patch, the maintainer(s) of the git module can obviously reject it if they don't want to maintain it. Unless they're persuaded because it's better for the project as a whole.
However, you talk about "project leadership". That doesn't really exist. There's loads of maintainers. There's a GNOME foundation, which mostly handles admin stuff, sysadmin stuff, etc. There's a release team. They ensure the various maintainers make releases, plus verify stuff actually builds.
If you help out in GNOME you're just a person in a big group of people. There's not really any "leadership", just loads of people who often agree, sometimes disagree, etc.
It appears they have marked the Logind API as being stable because it is intended to be used by the display server and by the DEs. I don't have any secret information here, I just saw this chart on their website. https://systemd.io/PORTABILITY_AND_STABILITY/
About BSD and non-systemd systems: I actually think it is possible for there to be better compatibility here eventually if someone builds more shims. BSD using Mesa for example is a really good thing for everyone. However FreeBSD and Linux have both refused to budge for quite a long time on various things and cgroups seems to have been the final nail in the coffin for any kind of expectation of modern cross-platform tooling. Everything now is container this, container that. The only non-systemd init that hasn't dragged their feet on this has been OpenRC, and even they still would not make any effort to be systemd-compatible because of objections over other things. So what is the solution here? Do you see how hard it even is to get people to run something like elogind on top of another init with another cgroup implementation? I don't blame the systemd developers for not bothering with that. It is not feasible to expect them to maintain everyone else's init as well as their own.
I checked the systemd contribution graph and it seems they have a large number of smaller contributions from outside developers, so the data does not really support your assertion about it not being worth it. https://github.com/systemd/systemd/graphs/contributors
Also, just going by the data, the repos for KDE (collectively) and FreeBSD (a monorepo, like systemd) are much, much bigger, just as tightly-coupled if not moreso, and get a lot more activity than the repo for systemd. It seems strange that you would pick those projects to contribute to.
> The only non-systemd init that hasn't dragged their feet on this has been OpenRC, and even they still would not make any effort to be systemd-compatible because of objections over other things. So what is the solution here?
You either hammer out a common interface with compromises from both sides, or you implement the parts you want as progressive enhancement from a lowest-common-denominator baseline. It's not easy, it's not glamourous, but it's the right way to do it. Look at how e.g. web standards stuff happens when there's disagreement between different browser makers.
> Also, just going by the data, the repos for KDE (collectively) and FreeBSD (a monorepo, like systemd) are much, much bigger, just as tightly-coupled if not moreso, and get a lot more activity than the repo for systemd. It seems strange that you would pick those projects to contribute to.
I'd disagree with the idea that KDE is more tightly coupled; in my (limited) experience it's very cleanly factored and it's clear where the interface boundaries lie. It's a big project but if you want to fork and patch one part of it you can do that.
FreeBSD is theoretically more tightly coupled than GNU/Linux, but in practice I've found a lot less of the kind of interface churn that you see in Linux. Linux went through two or three rounds of different audio APIs where on FreeBSD you still just use OSS. I've lost count of how many times the way you do network config on Linux changed - I genuinely can't set up a network on Linux any more - whereas in FreeBSD it's all just ifconfig. Linux forced a quick migration to HAL followed by an equally quick deprecation in favour of udev; FreeBSD is still using HAL. There's just so much more commitment to stable interfaces, both in theory - a deprecation cycle for even the kernel ABI - and in practical experience. Or maybe it's just that the project doesn't have enough manpower to churn as much as Linux does, but either way the end result is that I can patch my system and expect my patch to keep working.
> elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility.
That's because elogind is architecturally all wrong - instead of sticking to the defined stable public DBus interfaces of systemd, and implementing those independently from scratch, what they did was rip the implementation of systemd-logind out of the systemd package and hoping that the intentionally unstable internal interfaces in libsystemd that it relies on will continue to be provided.
> Gnome worked fine long before systemd and systemd has not noticeably improved it (if anything the opposite); there was no technical consensus that the current hard-dependency was necessary
There was a very, very simple reason for why Gnome depends on systemd-logind: nobody volunteered to maintain the ConsoleKit backend, or ConsoleKit itself for that matter, so it bitrotted and was eventually removed.
> elogind is a stub piece of systemd and systemd refuses to make any commitment to maintain compatibility.
You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?
That's kind of an explicit choice to allow the maintainers to move forward with the limited resources they have without too much red tape.
There's always *BSD, if you want compatibility back to the Sun days in some cases, (Win32 is another option). Linux is about being as modern as possible without having to worry much about backwards-compatibility and all the cruft that comes with having to maintain that.
> You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?
It does for userspace APIs. Linus is notorious for chewing people out for breaking them (example: https://lkml.org/lkml/2012/12/23/75).
Yes, I should have clarified I meant kernel-level APIs, not userspace.
> You do realize even the Linux kernel doesn't make any commitments regarding API stability, right?
You say "even" the Linux kernel; actually most projects don't and Linux is a rare exception, and I do think it's a bad idea for both technical and nontechnical reasons. (Compare with GCC's refusal to offer a stable API between frontend and backend, which I'd argue is a big reason why LLVM is overtaking it).
> There's always BSD, if you want compatibility back to the Sun days in some cases, (Win32 is another option).
I actually do run FreeBSD these days; one of the reasons this issue still bothers me is that compatibility with BSD is likely to be collateral damage from the systemd push (it's getting harder and harder to run Gnome on BSD, for example). It was particularly frustrating that systemd basically killed off Debian GNU/kFreeBSD just as it was becoming a first-class architecture; that could have been the best of all worlds.
I noticed the ConsoleKit->logind change at the time being largely justified by support for multiseat but had forgotten about it. It seems like a rather extreme case of the tail wagging the dog but perhaps I'm missing something. How many people use multi-seat in large deployments? Where and why?
Virtually nobody is using multi-seat and none should.
For most cases you would get better results by networking n cheap boxes rather than trying to buy a single beefy box to support n users on every dimension including perhaps even power usage if you used low power devices.
Contention for io is a problem. Everyone wanting to say decode video simultaneously is a problem. Browsers high ram usage is a problem.
> rather than trying to buy a single beefy box to support n users
This sentiment has always felt weird to me, as I have always seen UNIX as an OS that has been designed - from the ground up (in late 1960s) to support multiple concurrent users - whether it is via a serial terminal, and now multi-seat, VNC or an X terminal. By today's standards it's not even such a "beefy box" that can easily do that.
Blame browsers as soon as 3 users wanted to use facebook youtube on your 1200 box each user will probably have a worse experience than 3 users running 400 boxes plus hardware failure would reduce capacity to 0% instead of 1 box failing at a time.
Except you seem to have forgotten the other big reason was that basically nobody was actually maintaining ConsoleKit at the time at it was a horrible, insecure mess when logind replaced it.
They didn't used to break if you wanted to switch from one init system to another.
What you could do is to reduce unnecessary interdependencies. And where there is inherently interaction between separate packages, use a stable interface and publish the standard so that alternative implementations can replace a single package in a standard way without breaking several others.
Yes, they did. It they didn't break it was because those inits were compatible with sysvinit scripts. You lose that once you start porting your init scripts over to rc or whatever it is you replaced your init with, and now your system will break if you try to switch to something else.
Your init scripts would have to be replaced, that's to be expected, and when multiple ordinary init systems are popular, packages could easily support more than one since an init script is generally only a few lines.
I'm talking about all the things that now break not related to the normal function of init at all.
If those inits supported newer features, the init scripts would easily grow longer than a few lines and become incompatible once they use those features. I don't see how that's any different from what's going on here.
If your complaint is that the scope of init systems has been expanded, I don't know what to tell you other than that it was inevitable. Systemd is not the first project to try and do this.
I think the point is that a software platform that imposes these requirements is inherently brittle and prone to ossification.
And why do you think that is?
It's because systemd offers convenient functionality that those dependent packages want to hook into, because previously they were either maintaining all that themselves and seriosly lacking the manpower to do so or it wasn't being maintained at all and just rotting, (i.e. consolekit)
That's exactly my view.
We had a chance to replace sysvinit with something better. Instead we managed to replace it with something that's horrible in an entirely different set of ways.
The most devastating thing I can say about it is that it feels "enterprise." It's unnecessarily obtuse, complex, and clunky. It feels like a port from some other operating system that's been shoehorned into Linux, or like something designed to sell consulting services and training courses by wrapping very simple things in needless complexity.
We could have ended up with something more straightforward, clean, and simple. It's a tragedy.
> Debian, Ubuntu, Arch all chose the systemd voluntarily.
My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to if they wanted a reasonable default GNOME experience for their users. If they hadn't, GNOME would have been missing some features that would have been obvious and jarring.
You can certainly argue that they made the wrong call there, and that they should have pushed back (though arguably by the time Debian got there, that ship had already sailed), but "chose systemd voluntarily" is an incorrect simplification.
> My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to
This (and much of the article) is incorrect. Debian went through an extensive evaluation process at the time, over the course of 4 months, considering several different init systems, and chose systemd based on this detailed technical evaluation. That evaluation also considered sysvinit, openrc, and upstart. All of that evaluation occurred publicly, and is archived on Debian's mailing lists and bug-tracking system.
Debian also ran a poll a few months ago in order to reevaluate that choice, at least partially. Quite the hint that all is not jimmy-dandy.
As a programmer in embedded, Systemd feels like it solves problems we didn't have, and introduces problems we didn't have. I what point is it more conservatism than "if it ain't broken, don't fix it"? God only knows.
Well, every time the Systemd topic pops on HN or Reddit (and even other places where you wouldn't expect it [1]), I go fetch some popcorn... It seems that different people have different needs and different experiences, hence the flamewars.
Can you reference that poll? There was a GR, which is entirely different than a poll. The outcome of that was also pretty clear, Debian as a whole is ok to depend a bit more on systemd. I don't get why you use it as an example of the opposite.
See https://lwn.net/Articles/808217/ for the outcome of the GR.
It is indeed this GR.
I might be misinterpreting the result, but "Systemd but we support exploring alternatives" means, to me, that there is some unhappiness here. To begin with, I believe the GR would not have happened if there was no issue to be discussed?
The GR happened because it wasn't clear what maintainers should do regarding alternatives init systems. Were they required to support them? Should they just focus on Systemd? The result of the vote was maintainers are required to support Systemd, but should accept patches to support other units.
The unhappiness was that the existing status quo made people afraid to actually use features of systemd, lest they invite flames from people about not supporting alternatives (which don't have those features). The GR changes the status quo to allow hard dependencies on systemd to actually use its features. The "but we support exploring alternatives" ensures that the GR does not preclude the possibility that people might create something better than systemd in the future.
All of my views come straight from primary sources which I meticulously uncovered, including Debian mailing lists.
I realize that you have been one of the more visible members of the pro-systemd side for years now, so I'd definitely like a more coherent rebuttal to both the historical and technical sections.
Of course, you'd probably consider any kind of response to be undignified, and I understand.
I participated in those discussions first-hand, and recall them rather well. At no point was "we don't have a choice" taken seriously as an argument; even most of the pro-systemd folks didn't consider that a valid argument for systemd, and many of the most judicious and respected voices in the discussion called out such arguments as being unhelpful. Debian is large enough and stubborn enough that they're more than willing (occasionally entirely too willing) to do their own thing if they think they have a better path.
As for the rest, I consider https://news.ycombinator.com/item?id=23062725 a good start. systemd was successful because it provided working code that people (including distribution maintainers) wanted to use, for a variety of reasons. This is the type of problem where no amount of architecture-in-a-vacuum discussions make up for actually doing the work and handling all the myriad cases that come up when people use it in practice. systemd did that; the next init system after systemd will need to do that as well.
This is the real point.
Debian did _not_ choose to use Systemd. Debian was forced to choose between adopting Systemd despite going against the very principals of Debian, or loosing Gnome. They were put it a very difficult position.
Ubuntu did _not_ choose to use Systemd. Ubuntu, using Debian upstream, decided not to break from Debian. Shuttleworth, the founder of Ubuntu, has a very sublime message about it at the time.
I'm personally of the opinion that when forced into a corner, do not side with whoever pushed you into that corner. The Debian technical committee did not have that resolve, they took the easy way out. Lots of Debian-derived distros followed suit out of necessity.
And Ubuntu followed Debian. Mark Shuttleworth called it "losing graciously".[1]
A lot of init problems you describe are very real, but it felt like by the late 00s they had mostly been solved with efforts like linting and standards, as well as LSB init shell functions making init a much more standardized process.
PAM was and still is quite hard to understand but, again, it had enough power and flexibility with @ based includes to allow individual distributions to lay things out sensibly and allow new packages to follow a coherent standard that emerged rather than being forced.
Concurrent init systems helped drive a lot of this, back when there was a sweet spot of Linux complexity increasing faster than processing power could handle. (Nowadays I feel like boot speed is less important, given how many more resources a modern machine has, though it’s still a concern for low resource platforms.)
I guess I’ve just never seen revolution as a solution to hacks and untidiness. Linting, guidelines and policy, with technically compelling reasons to encourage people to follow them have seen much more success, where I’ve seen them deployed, than centralized authority.
> A lot of init problems you describe are very real, but it felt like by the late 00s they had mostly been solved with efforts like linting and standards, as well as LSB init shell functions making init a much more standardized process.
These improvements weren't real. There was a long tail of scripts that didn't follow the standard and never would. Every distribution still had their own customizations on how sysvinit scripts worked (e.g. debian with start-stop-daemon, suse with insserv, etc.)
And rc was different on almost every major distro family, which made life even more difficult for no particularly good reason.
And there's a certain point where all these things you are doing are ultimately to paper over the "hacks and untidiness" of everything. The introduction of systemd had two major effects:
1. It broke all non-LSB init scripts (because LSB is all it supported)
2. It introduced a straightforward declarative syntax for system resources (units for services, timers, mounts, etc.)
I would argue that the above two things have done more to standardize the Linux plumbing layer and help people take fuller advantage of it than literally all the linting, guidelines, and policywork we did for the decade prior.
And if that's all it did, we wouldn't be having this discussion. Instead it is becoming some sort of self-appointed "userland middleware", which nobody asked for. People resent "better init" turning into "henceforth home directories will behave differently".
That's a dishonest argument and you know it. The entire home directories thing is aimed at enterprises, and nobody wants to enable it for individuals' machines. Poettering said as much in the talk where he presented it. It would be much more reasonable to respond to people who dislike systemd if they wouldn't constantly make bad-faith dishonest claims about it.
The question remains, what do home directories have to do with init, at all? Okay so Poettering wants to "fix" home directories - is there any reason to couple that with systemd, beyond the fact that systemd is his baby (and point of leverage over the rest of the system)? There are dozens of other examples - network configuration? syslog? cron? This is what I mean by 'OS middleware' - systemd is basically a distro, and a radical one at that. It is aiming to replace huge swathes of stuff that no one except seemingly Poettering believes is broken in the first place. I think distros would have been more hesitant to adopt systemd as an "init system" if this roadmap had been clear upfront.
<conspiracy>
I have heard the opinion ventured that systemd is actually a power play by Red Hat to introduce an API layer between userland and the kernel - one controlled by Red Hat. It's not implausible - as Red Hat is funded by selling consultancy services, they have a financial incentive to convert as much of Linux as possible into Red Hat Linux.
</conspiracy>
That's one shitty conspiracy. If this were Red Hat's play, why would they do it with a project that other distros can adopt and have adopted? If anything, providing APIs in systemd makes it easier to migrate to other distros as long as they use systemd. (All relevant enterprise distributions use systemd, i.e. RHEL and SLES and Ubuntu Server.)
Up front, let me state that I have no desire to die on this particular hill, which is why I weasel-worded the hell out of my <conspiracy> block.
However, in this hypothetical scenario, the idea isn't to make it difficult to migrate away from "official" Fedora - it's to make it increasingly difficult to build a distro that is not, fundamentally, Fedora. And if everyone is building distros around Red Hat's quirky middleware, which distro is going to work best? Red Hat's, of course. Who are you going to hire to tell you how to manage the thing? Red Hat, of course.
(The logical extreme of this is the situation with Darwin. OSX's kernel+"middleware" is technically open source, but it's basically impossible to build an OS around it.)
That also makes no sense. Using systemd does not fundamentally turn your distro into Fedora. There are several distros that aren't Fedora that have established use of systemd, and there are several companies selling support offerings around those other distros. In reality, all these distros have come together to work on it jointly as an open source package, which is just a small part of your overall Linux distro typically made up of thousands of packages.
By comparison, I don't believe any companies are selling support offerings for open source Darwin, not even Apple.
Correct, Red Hat will decide the future of Linux userland, not the distros, they will just have to follow. This is a way of building one distro with many flavors instead of many separate distros.
"One distro to rule them all".
> Remember, even if you can claim for Fedora that someone "convinced the PHB"
If you looked at the discussions that happened during that time (I was there as a Fedora contributor back then, and wow that was a long time ago!), you'll see that we mostly did the change out of weariness of trying to use Upstart features well and failing. And even with that, we delayed it by one release because it wasn't ready during the Fedora 14 development timeframe (it was supposed to be in Fedora 14, we actually launched it in Fedora 15).
Exactly. I was immensely frustrated with sysvinit for over 15 years. Jumped over to systemd as soon as it was available on Debian, never looked back, only one had a problem due to symlinking.
The parent comment is criticizing systemd, not upholding the virtues of sysvinit. There are more than two init systems out there, and it is possible for sysvinit and systemd to both be bad in their own ways
Sounds more like the problem is with the packages and a general lack of quality control with respect to the userland than with GNU init. Perhaps that is why BSD projects, which do have some quality control over userland, can still use BSD init.
You are describing the downside of bazaar-style development. Init had very little to do with it. The real problem was a non-standard way of writing init scripts. That is the _only_ part that systemd was supposed to solve.
The problem with this straightforward thesis is twofold: firstly, distribution developers did not have any coherent position on the disadvantages of sysvinit or initscripts; secondly, the problem was mostly an iatrogenic one, as the very attempts to 'solve' or 'reform' it made things even less sustainable.
When it comes to Upstart, it was almost never used except as a wrapper around initscripts. Effectively you booted from Upstart which greedily synthesized all of its 'events' on startup that in turn triggered initscripts, which brought some degree of parallelism and that was about it. The reverse-dependency 'start/stop on' model was too esoteric, and Upstart's state machine brittle. Hilariously enough, the only system to actually use Upstart to its fullest (still does) is ChromeOS.
Prior to systemd and Upstart, as I discuss in chapter 1, distro developers either simply added various preprocessors for initscripts to introduce dependency headers, or they had these vague hunches of 'waiting for Godot' that we could either replace init entirely with D-Bus, put D-Bus interfaces in initscripts, or some other roundabout way that involved grafting D-Bus. Much of this was owing to the 'irrational exuberance' of Linux developers wanting to hotplug the world after 2003-4, but it always remains an on-and-off effort with the people behind it never certain what they want.
Indeed, besides grunt work of optimizing initscripts, I point out that as late as 2009 distros like Debian were still very concerned about LSB-compliance, and that even when they were willing to adopt a new fancy event-driven init like Upstart, it was conditional on reusing tools like insserv to parse LSB headers, and on incorporating initscripts for service management. Also the entire bizarre diversion that the Debian group had of using a Perl script to generate initscripts.
So far as I know, the daemontools approach was never influential. The available dichotomy was between putting lipstick on a pig, and utopian dreaming.
The paradox remains: how can we trust that the same people who did the Wrong Thing so persistently then suddenly turned out and did the Right Thing like they hit jackpot? If anything, 'jackpot' is the right analogy -- it was a fluke. And the same integration failures that were had with sysvinit have now been shifted into wrestling systemd's job model.
"Unification" was a folly, as the distro is still the place that gets its finger pointed at if upstream can't deliver, and upstream insists it's "just" offering a bunch of "modular tools," which ironically was supposed to be the problem to be addressed! Besides, there is still a whole world of Linux devices out there running Busybox, musl, OpenWrt, etc. that are outside the systemd bubble. Or think of all the container images running Alpine (which is also the basis of postmarketOS). Linux is still the same bazaar it always logically must be, but with a new class distinction added to the mix.
The distro has historically always been the place that gets pointed at because they are the ones who distribute to end users. No system-level component is going to change this, ever.
I am also confused why you're complaining about the job model and all its churn and conceptual problems when the whole point is that systemd was offering to handle that for the distros, which is exactly what they did and are continuing to do now. You can claim they are doing a bad job (which is probably true in some sense because of the impossibly large scope of the task) but the previous alternative was that nothing was being done about it at all. Which you did acknowledge, but then you went back to the same criticism as before. Why? It seems obvious to me that the gritty technical aspects and/or perceptions and fears about some kind of false dichotomy are not what could have ever influenced this kind of decision. Nobody else is invested enough in it for that to have happened.
The other reality I've seen is that there is no standard way to actually implement a daemon. Nobody does it in quite the same way and using things like daemon(3) don't help. So you can choose to A. patch all your daemons, or B. you can attempt to simplify the task by writing more tooling. Every single Linux distro I've seen, when given the choice, has chosen option B. So really, I don't see anything here that is surprising or farcical at all.
The 'job model' is the wrong abstraction, as is the unit concept which tries to reduce things that have irreducible differences. I'm also rather amused how you're conceding that they're doing a "bad job," but still shrug it off as better than nothing. If you're offering a mediocre solution to an "impossibly large scope" of a task, then you ought to rethink things. Even then, a poorly implemented feature can be worse than none at all.
So far as prior art is concerned, see my old post: https://blog.darknedgy.net/technology/2015/09/05/0/
I would also point out OpenWrt's procd as an example of how to have hotplug-capable service management at a fraction of the complexity: https://openwrt.org/docs/techref/procd
Now, this rethinking did indeed come with HAL, when the idea of a master daemon and ad-hoc service manager reacting to hardware state changes was unanimously rejected by the Linux patriciate as wrong. Perhaps with systemd they will one day reach the same epiphany for system and user services. Either way, I'm not terribly personally invested in this anymore -- I just figured I'd wrap my thoughts up on an issue that occupied a good deal of my time years ago.
EDIT: As to B, the problem of process supervision and having an idiomatic way of daemonizing has been solved for a long time: https://jdebp.eu/FGA/unix-daemon-design-mistakes-to-avoid.ht...
The entire purpose of designing models is to reduce things that have irreducible differences and then tell people to shut up and deal with it. There is nothing else there. What is it exactly that you think IT gets paid to do other than this? We have finite time on this planet.
Also, things are constantly being rethought. That is quite literally what the churn encompasses. You can't complain about this while asking for it at the same time. I have seen all these lists of other init systems and while they are interesting, they are completely missing where the actual discussions are happening and what is actually being discussed. The core problem has already moved to a different area and that's why it really doesn't matter if they are doing a bad job. I can see you hinting at this in your article but it's not good to dwell so much on things that happened in the past and are already history. I am very grateful that you actually quoted all your sources though, so thank you for that.
The comparison of init to HAL seems not relevant to me; if anything that has made it obvious that they had the right idea but it was in the wrong place, and that init was the right place to put this type of logic after all. Adding to that it also seems very unlikely that the kernel developers will change their position on the responsibilities of an init any time soon.
And about the idiomatic way to daemonize, nobody follows those guides. Nobody. Seriously. It's a complete clusterfuck and writing more guides doesn't help either. There are an absurd number of programs out there that still use pidfiles. Go and do a search on github for "pidfile" and be horrified.
All models are wrong, some are useful, others not so much. There's no need to treat me with these banalities. The job model ought to have died with Upstart. It did not. Oh well.
> The core problem has already moved to a different area and that's why it really doesn't matter if they are doing a bad job.
Yes, I agree.
Still, the losers of history deserve their voice, too.
"Nobody" is a significant overstatement. A lot of (certainly not all) daemons that used to only support double-forking and PID files were fixed in the wake of daemontools. systemd and other init systems benefited from this work when they appeared. Still, PID files are subject to TOCTTOU regardless of what you use, so that's a given.
Sorry for the glibness I am just really bored with these complaints, it really feels like the same old bikeshedding. As far as I know there is no way to make a transactional state machine without having the concept of a single, reordable work unit. Daemontools did not solve this or present a different model that was more workable. For every developer that rewrote their daemon to get rid of forking, we got more new developers with outdated learning material coming over who expected services to work like they did on some other crazy environment like Microsoft Windows.
I don't particularly care about who was a "winner" or a "loser" either. In general you cannot get away from TOCTOU on Unix without taking extreme care and avoiding things that would seem reasonable, like storing filenames in other files or using the "kill" command. In practical use I have seen no inits that make any concerned effort to solve these issues at all. That is a situation in which we all lose.
> As far as I know there is no way to make a transactional state machine without having the concept of a single, reordable work unit.
You're basically taking systemd's architecture as an axiomatic premise, so no surprise your conclusion is obvious. I reject the premise, and the fruits of it I go into detail in chapter 3.
For the record I am mostly a Unix-hater, so your complaints about Unix are things I agree with. systemd, however, is simply more New Jersey design with unwarranted pretensions.
systemd, in a way, is exactly the init system that a community as infantile and disordered as Linux rightfully deserves. May they enjoy it.
I'll leave things at that.
No I am not commenting on systemd or Linux at all. The fact is that if you implement a dependency graph, you must track and resolve state. There is no way around this. Add in support for event listeners and one-shot syscalls out to hardware (because remember, udev or some equivalent is still a critical piece in all this and there is no HAL to handle this for us) and now you have to deal with transactions. Try to parallelize it and now you have work units. That's the way it goes. Even in microkernels you will find all these things scattered around and they aren't going to be less painful just because you wrote them in eBPF. I liked the third chapter but it left me a bit disappointed because it didn't mention anything other than a critique of the wrong way to do it.
> User A likes to leave their executables on their machine after logout, so we need a cleanup script.
Of all examples you gave, this one is the least convincing reason to introduce this Behemoth on our systems.
> SystemD feels like the new grad hire who decided the first thing the would do after joining — the opening power play — is to convince the PHB we should rewrite everything in Haskell.
I've seen this scenario played out, and it always fails. The product never ships, of ships in such a bad shape the users have a terrible user experience.
That's not what happened with systemd. It shipped. And after a few iterations, it actually provided a better user experience.
I don't know about sysadmins, but as a simple end user of Ubuntu, systemd is the best user experience I ever had for what matters to me: setuping my services.
It's quite easy to create your unit files, declare service dependencies, stop, restart, and so on. It's declarative, conf is reasonably straighforward to read an understand, documentation is there, and my system boot is stable on laptop and on my servers.
Maybe for "advanced case X", there is problem. Apparently, a lot of power users say there is. But I also know the FOSS community has some people advocating purity over practicality, don't care much about the people with less skills than them, don't understand user friendliness or will fight for their ego. So I have no way to be sure if the problem is real, or just another free software drama.
But at this point I don't care: for the average Joe like me, it's great, and I'm very happy it has been chosen by Centos and Debian as the default.
Apparently nobody in the entire history of init system have managed to provide something decent for, you know, initializing the system. I never heard a herd of devs claiming that one is the fantastic, and they were all terrible to use when I tried.
Well, I don't care for choice in my init system. I want it to work.
And systemd works fine for me.
> Well, I don't care for choice in my init system. I want it to work.
We Agree.
> And systemd works fine for me.
But it doesn't for me, and I am left with no choice but to suffer with it.
so what should be done, in your opinion, if it works very well for 51% of people, good enough for 48% and breaks the work flow of 1%?
easiest answer is to say get rid of the inter-dependencies, but that's throwing money and time at a problem affecting 1% of users, at no benefit to the rest. Maybe it's still the right answer, but who's to pick up the slack?
The other component here is "where you came from." When I joined Sun there were two very distinct programmer/hacker cultures; the PC and its spawn and "real" computers (mini and mainframe computers) and their OSes.
The cultures were very different, I straddled them with my hobby working being CP/M->PC->Amiga, and my professional work being RSX-11M->TENEX->TOPS20->UNIX.
Linux (and Minix) started out very much being a PC version of UNIX and PC-BSD jumped in there too. But Linus ceded userland to the community and as the community's membership has grown the people who wanted Linux to be the perfect UNIX lost out to the people who wanted Linux to be an open source version of Microsoft Windows.
It appeared to me that the folks who liked UNIX semantics fled to MacOS, although the complaints on that front are getting louder as well. So here we are.
> SystemD
I don't know why, but it bothers me when someone's trying to criticize systemd, without even bothering to get the case right. It smells of not having looked at it any more than to form the run of the mill opinion on systemd, while looking at what came before with healthy dose of romantic nostalgia.
I don't want my init system to be a series of garage-quality bash scripts with slightly different error rate tolerances etc, systemd solves that. I don't want my system to just blindly shut down, even if it's not necessarily safe to do so immediately, systemd solves that.
What systemd critics often miss is that it's also a rather valuable resource manager in general. I know that many would say this should not be part of systemd, but the thing is while that sounds nice, nobody built anything remotely as convenient for Linux as systemd in this area.
It's nice to say what systemd got wrong, but I haven't seen anything actually built that can seriously compete with it.
I now actively avoid distros without systemd, because the admin experience of those distros has generally been much less consistent.
We can all talk about the right way to do things, but unless there's code that's clearly superior to what systemd offers me today, am not interested.
Why do systemd detractors keep styling it as SystemD? I'm honestly curious.
You know it's written all lowercase, right?
Revelling in their rejection of systemd. It's childish nonsense and a pretty reliable indicator that a person is bitter about it and their comment most likely not worth reading.
> rewrite everything in Haskell.
Yes please!
Pure functional languages are very well suited to declarative configuration, especially where you need significant flexibility not afforded by purely static "data" formats such as YAML or JSON.
My beef with systemd is not its reinventing things. That part may actually be good.
One problem is that a number of reinventions were poorly made.
E.g. the log format. Yes, unstructured logs have a ton of drawbacks. Can we take some ridiculously well-tested, reliable embedded database or serialization format (like sqlite) and use it as log storage? Alas.
Another problem is the "I know better" attitude.
Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior.
No wonder I don't run systemd on my desktops / laptops.
You don't want to use sqlite for logs -- it is way too complex, and in particular, it may roll back (=lose log entries) if it is low on space, and permanently damage data when writing onto disks with errors. It also makes wrong performance tradeoffs for logging -- your choices are either sync() all the time, or roll back on unexpected termination.
I don't like the that journald goes backwards in space and updates indices -- I like "append-only" attribute -- but at least it writes data sequentially one after another, which makes it pretty tolerant of data loss.
As for "without escape hatch", I don't remember any functionality like that? In fact, systemd is full of escape hatches which are documented and supported. Just recently, I had to modify "is computer online" check -- it was amazingly easy, all I had to do was to drop-in a fragment file for the right service. I remember doing the same in upstart -- it was a total pain, requiring reading the source code spread over dozens of undocumented files.
No local logging will work without disk space, and if the disk is bad anything can get corrupted.
If you are low on disk space you want something predictable to happen: the oldest logs are deleted and the newest logs are written. This would be a nightmare to achieve with SQLite, as said above it's just way too complicated.
Does logrotate or plain old write() actually handle those error cases by default?
It sounds like you're arguing to use an even more sophisticated logging system than the status quota of blindly writing bytes to disk.
If you actually care about your logs you should log to a centralized service anyway, and if any of your computers actually ran out of disk space you should fire your sysadmins/sres for incompetence.
This comment denotes, strongly, the dissonance that occurs when talking about something so general.
It’s my opinion (as an infrastructure type) that centralised binary logging is an absolute necessity above 20 instances.
But that leaves out every person with a laptop, or those people with a very low volume of little instances.
Structured logging is difficult, that’s why logstash is so flexible about input and conversions. And I think that was the impetus for systemd’s journals; to log structurally.
However. The parent is wrong if he thinks decisions should be made under low disk conditions. Having a rotate rule to keep your sliding window of logs to a certain size is fine. But your servers (and laptops, and desktops) should not be taking arbitrary decisions based on this kind of condition.
> This comment denotes, strongly, the dissonance that occurs when talking about something so general.
> It’s my opinion (as an infrastructure type) that centralised binary logging is an absolute necessity above 20 instances.
> But that leaves out every person with a laptop, or those people with a very low volume of little instances.
Yep, it also leaves out people on embedded hardware, and on devices that are in remote locations without access to the Internet, or in a car, or on metered Internet, etc. It's hard to hold in your head just how broadly Linux is used these days.
Sqlite is battle-tested though, it has one of the most comprehensive test suite and is used on billions of devices already.
What systemd is doing with this log storage system is reinventing sqlite, because systemd has a fairly new and not battle-tested database implementation, it has shortcomings.
I would argue that either they use plain text logs like before or use sqlite but they should not reinvent a new db.
> Sqlite is battle-tested though, it has one of the most comprehensive test suite and is used on billions of devices already.
That's great, but you've completely ignored theamk's argument that sqlite is the wrong tool for the job.
I don't agree with that argument, sqlite is totally the right tool for the job, it provides consistency, reliability, an api you can query anything upon, transactions, indexing, tools... And fits into files.
You currently can't trust systemd log storage at all, even on my own laptop it manages to have some corruption... Querying is very poor and inefficient, indexing is unreliable... And that's exactly what I expect, building a database is hard.
SQLite doesn't work reliably on NFS, which rules it out for any PXE-booted system with an NFS root.
The issue with NFS and SQLite is that posix advisory locks do not work, or do not work well, on many NFS installations. As long as advisory locks work over NFS, or as long as there is only one client trying to access the database at a time, so that locks are not really needed, SQLite works fine over NFS. NFS is a lot slower, but that's just the nature of NFS and SQLite can't do anything about that.
You can ask SQLite to use dot-file locking instead of posix advisory locking. Or you can ask it to simply ignore locking all together. In both cases, SQLite will work on even a broken NFS system, though with the corresponding concurrency limitations.
Fair. In my experience, the fact that NFS is both pervasive and often broken, for something like a system-wide log manager you can't always assume that your root fs is reliable.
Is dot-file locking a compile- or run-time option?
sqlite is a dirty hack. just because people like hacks doesn't mean it's ready for the serious stuff. it is not. https://research.checkpoint.com/2019/select-code_execution-f...
It's certainly more ready than a custom hacked-up database which has data corruption even on my own laptop.
It's astounding to me the issues people have with this. systemd's existence doesn't preclude anyone from using the old, pre-systemd ways of doing things. There are thousands of linux distributions, and afaik there are at least a dozen which are primarily defined as never-using-systemd.
Face it: the linux userspace has always sucked. Linus punted on it way-back-when (to be fair, he was a solo dev working on a kernel, he had his hands full) and opted for the gnu tools, but those (other than the compiler and libc and a few other spots like grep) have never been best-in-class. It was always just sort of "good enough" to avoid swimming upstream (no pun intended) and trying to swap them out.
Someone decided to make an attempt at making the situation better in one way, doing things differently. Perhaps it was a mediocre attempt, even. (I'm just speaking for sake of discussion here; I haven't read the systemd code, just making a point.). Still, nobody was forced to use it, and indeed many options for avoiding it exist.
Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.
I'm all for experimentation in this space. The old way of doing things was super janky, and I think systemd is an improvement in a lot of ways. I (thanks to a lot more time stuck in my house) have two recent Gentoo installs, one with systemd and one without. They both work, but one works better.
"This is the way we've always done it" seems in opposition to the hacker ethos, to me. Doing things differently to see if there might be a better way seems like it's always valuable, considering that nobody's deleting /etc/init.d/* from your system if you don't participate.
Traditions are not always inherently valuable. Try new things. Discard the ones that don't work for you.
> Face it: the linux userspace has always sucked. Linus punted on it way-back-when (to be fair, he was a solo dev working on a kernel, he had his hands full) and opted for the gnu tools, but those (other than the compiler and libc and a few other spots like grep) have never been best-in-class.
Which userspace are you claiming is better then? Linux used to offer the best userland experience in the world, because you had control over every part of it, and since things were loosely coupled you could mix and match at will (e.g. prefer BSD grep or Solaris tar? No problem, install them).
> Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.
They didn't build a better alternative and let users choose to adopt it organically. They made a wilfully incompatible, tightly coupled thing, and then pressured other projects to hard-depend on it (e.g. Gnome).
This is quite literally the antithesis of everything that Linux and Unix stands for. The whole reason people were able to use the GNU tools with the Linux kernel is because they are loosely coupled and communicate through standardised interfaces. Systemd by contrast doesn't run anywhere but linux, and you need closely matched versions of everything: your linux has to match your udev has to match your systemd has to match your dbus has to match your gnome has to match your...
If systemd had been around when GNU and Linux were starting out, the whole project would have been literally impossible. It's a betrayal of everything those projects stand for.
> pressured other projects to hard-depend on it (e.g. Gnome).
GNOME wasn't pressured in depending on it. There deciding to depend on logind over ConsoleKit. Initially logind could be used without systemd, that changed after a systemd version due to cgroups v2. There's quite a bit of overlap between the people maintaining ConsoleKit and the people wanting to make use of logind, so there was a push by some GNOME maintainers to use logind.
The arguments against were often quite weak. When pointed out that ConsoleKit was unmaintained for years the answer could often be summarized as "why does it need development".
> It's a betrayal of everything those projects stand for.
Stuff like this makes maintainers of things just ignore you. There might be a valid point in your argumentation, but why bother.
> Traditions are not always inherently valuable. Try new things. Discard the ones that don't work for you.
But that's the trouble. The people who make systemd keep integrating it with everything else they make and vice versa, which makes it harder to try new things because instead of things coming in pieces you might replace, it comes as one big interdependent blob.
That's their business, and their decision if they're the ones doing the work. You can always simply not use that big interdependent blob, same as ever. I'm not sure what the problem is.
I recently installed a Gentoo system with ConsoleKit and KDE Plasma, and it works great, not a trace of systemd anywhere. I also understand the original Gentoo guy forked Gentoo and made a new distribution (Funtoo) where systemd isn't a configurable option even if you want it - it just doesn't exist in that distribution at all, as far as I know. In normal Gentoo you can use OpenRC (default) or systemd.
AFAIK Gentoo also supports Gnome with just logind, too. Haven't tried out that config.
Let me try this a different way.
People say that if you don't like systemd then don't use it -- and that's what the people who don't like systemd want to do. But not all of them want to compile everything from source like Gentoo.
So use Devuan, you say, or some other esoteric distribution with six thousand users instead of a million. Which is presumably what they are doing.
But they still want Debian/Ubuntu to use something other than systemd by default, because you need a lot of people to use something in order to get good bug reports and get people interested in fixing those bugs and making improvements to it. Use in popular distributions is the only way for the alternative to get those things, which makes people who don't want to use systemd unhappy when the most popular distributions do.
One big interdependent blob? Sounds like the Linux kernel to me.
I mostly agree with you. I see two sad things:
* Systemd was not exactly great when it had to be considered "the" replaecment for SysV init (which does suck), and happened to be much more than an init system;
* The big distros like CentOS and Debian (and then many smaller) had little choice but to adopt systemd, because Gnome had it as a hard dependency.
It's unfortunate that of all SysV init replacements, the evolution chose systemd. By its spirit, it's very anti-Linux: it's often seen as "monolithic" even though it technically isn't, because it tries to control nearly everything, and it does not mind breaking the old ways when it sees fit (and not when the old ways are broken).
Quite likely, we're stuck with it as a mainstream solution for another decade, or two, much as we're stuck with X. (And in two decades, Linux the kernel may become irrelevant, due to hardware and OS research progress.)
Linux types I've noticed tend to be more holy war orientated. Where did the classic vim vs emacs holy war come from also?
You can say that about Apple users too, but the zealotry has somewhat declined as Apple products has become more mainstream.
Historically Linux users where people that actively choose to use Linux based on a combination of technical & philosophical reasons, thus they are more passionate about it.
Most common Windows users get Windows preinstalled from the store where they bought the computer. It is not really an active choice based on reason, therefore no reason to have a holy war.
The more Linux becomes mainstream, the holy wars will become more & more fringe. Systemd is pushing Linux more into the mainstream.
It's tribalism and jealousy, and becomes loudest when one side perceives they have "lost" and don't have the skills to do anything about it.
And that was one of the sillier holy wars, since text editors have near-zero network effects and it's trivial for distros to ship both.
(FWIW I use nano.)
> And that was one of the sillier holy wars, since text editors have near-zero network effects and it's trivial for distros to ship both.
Text editors like vi may have near-zero network effects, but operating systems like emacs and vim have plenty of network effects. If I spend months building an amazing interface to git in Emacs but my colleague uses vim, our organisation is hurt.
That's (I think) why people care: they want to collaborate.
> (FWIW I use nano.)
boggle
I’m mostly kidding about nano, I only use that when I have to. My main editor is HomeSite+ 5.5 running on Windows XP in a virtual machine on my Mac. And I’m not kidding.
Oh and by the way it starts faster and uses less RAM than Eclipse. That's fucked up.
> Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior.
All those examples you gave are possible to disable in systemd, and in the case of DNS and user process after logout, they are not even the default.
: Killing of user process are enabled by default in upstream but disabled by default in every distro that I know. And systemd-resolved, even if it is build by default, is not enabled by default even in upstream.
Good thing sanity prevails in distro maintainers! But not in upstream, as you duly note.
I also remember that even 2-3 years ago the situation was not as nice yet.
I think what became systemd could have been great software, if not for the, mmm, cavalier attitude of its creators (not necessarily personal; it might be amended by RH corporate deadlines).
The fact that `KillUserProcess` is default to on is not something that systemd forces to your distro, there is even a flag to disable it at build time. It does makes sense in many situations (like IoT, embedded, etc.), that is why it is on, but distro mantainers also needs to know what is better for its users (that anyway is a really small part of Linux users, embedded is much bigger).
"Not in upstream" doesn't seem to correspond with the comment you replied to.
Quoting: "Killing of user process are enabled by default in upstream but disabled by default in every distro that I know."
I really shouldn't write HN comments right after waking up.
resolved has been enabled in Ubuntu for years and in the next Fedora.
This is a decision of your distro mantainers, they could also default to unbound, dnsmasq, bind, or anything else. They use systemd-resolved because it probably solves some problem for them, even if this problem is reduced burden of maintenance.
I had to build something on top of systemd-journald recently for work. I was prepared for the worst after hearing all of the gripes people have with systemd.
And you know what? It’s fucking awesome. Structured logging is just baked in. I don’t have to parse fields back out that have been smashed into a string. I get full timestamps, machine IDs, pid, uid, gid, and all the rest. I get a cursor that I can use to durably resume consumption of logs between restarts. Shit, the cursor even lets me ensure that I’ve consumed logs even when they’re being aggregated from multiple machines, without a single change to my code.
Sure there are a few warts here and there. Everything has them. But it took less than a week for me to be convinced that it’s past time for plaintext logs to die.
This, exactly this - I remember one of the most bandied about reasons for the binary log format was that it made replication over the network easier, or something.
Fast forward 10 years and I still cannot find a reliable way to do this with journald, apart from forcing it to output to syslogd and then shipping the syslog logs.
Linux desktop security was (and largely still is) in shambles. Logind at least tried to force some sanity.
Was is executed well? No, not at all. In typical FOSS fashion it just changed from one release to the other, and that's it.
Yes, there were probably notes about it somewhere in a long forgotten changelog deeplinked 4-5 clicks below the surface of Debian's and Ubuntu's homepage.
Should have they introduced a deprecation period? I don't know, maybe, probably. (But who are "they"? systemd maintainers? Debian/Ubuntu?)
> Another problem is the "I know better" attitude.
All good software is based on this attitude. Wait, I hear you interject. You have examples to disprove this. Well whatever it is you might think isn't the "I know better" attitude is probably just the same "I know better" attitude with one layer of abstraction.
As a somewhat dilettante and casual home sysadmin (currently) I was messing around with screen on a box downstairs and ran into this:
https://www.reddit.com/r/programming/comments/4ldewx/systemd...
Namely that systemd doesn't allow persistent processes started from the shell by default, preferring to terminate them when the user logs out.
This would include processes like "screen" whose entire raison d'etre is to persist after the user logs out. (Well, it has other uses, but this is the main one IMO.)
The stated workarounds - fiddling with some options like "KillUserProcesses=no" in logind.conf &co. - have so far failed.
I don't know whether this situation is a problem with systemd or the distro, but it seems very much a problem with the culture summarised by the top commenter in the above thread, of (paraphrasing) glibly breaking existing workflows then casually brushing away criticism with arguments often boiling down to: "this is the right way, I don't care about tradition or protecting 'incorrect' usage."
This is yet another example of pointless incidental complexity in systemd. The whole point of "nohup"-based tools like tmux and screen is to cleanly separate the management of user sessions from the incidental mechanism of whether a remote connection is being closed (the 'HUP' in nohup is short for "hang up" i.e. close a [possibly remote] connection). Systemd should simply acknowledge this fact and keep the user session going when a program has been launched under nohup, instead it tightly couples its own "session" concept to the remote connection and then adds a totally ad-hoc, hacked-together feature called EnableLinger to somehow make nohup work anyway. It's amazing.
The trouble is that 'nohup' is not a particular state that specifically marks processes that want to survive logout. Instead, the rule is that processes running inside a particular terminal are killed when that terminal closes, and processes running outside any terminal run as they please. If you SSH to a server, or telnet, or login on the text console, or via a serial port, you get a terminal and everything you run runs inside it, and the system works.
However, if you login to a graphical desktop, that is not a text terminal, and therefore everything is effectively nohup'd: your browser, your chat program, your media player, all the miscellaneous helper processes they spawn, etc. Most people do want that stuff to be automatically shut down when they log out.
Since the kernel's idea of "a login session" is wrong (it only includes text-terminals), and it can't easily be changed for compatibility reasons, systemd implements its own idea of "a login session" that works the way most people (who aren't experts on POSIX job control) expect.
> However, if you login to a graphical desktop, that is not a text terminal, and therefore everything is effectively nohup'd
Systemd's revamped notion of "session" includes graphical ones. Yes, that's different from the legacy sense of a mere text-terminal window. But since they've had to reimplement the whole concept of graphical sessions anyway, they could have made them work like textual ones. If you're not running a session manager as part of your windowed environment (almost everyone is these days) the whole thing might break, but then you can just enable the hacky "lingering" mode to make it work and you're no worse off than before.
> Since the kernel's idea of "a login session" is wrong (it only includes text-terminals), and it can't easily be changed for compatibility reasons
This is a CADT attitude. Improving existing interfaces while maintaining compatibility is hard; it's also what makes the difference between a serious software professional and an incompetent vandal.
I agree it would be nice to fix this in the kernel instead, but as the article said, it might not be possible due to compatibility constraints.
The fact that the logind developers came to a different solution than you, after spending much more time thinking about it and actually implemented it, doesn't exactly imply logind developers are the ones with an attention deficit, or that logind is bad. ("CADT" apparently means "Cascade of Attention-Deficit Teenagers")
Also, never call out others as "incompetent vandals" if you think of yourself as a "serious software professional". This is the kind of toxic behavior that makes communities non-inclusive and leads to impostor syndrome.
>The fact that the logind developers came to a different solution than you, after spending much more time thinking about it and actually implemented it, doesn't exactly imply logind developers are the ones with an attention deficit, or that logind is bad. ("CADT" apparently means "Cascade of Attention-Deficit Teenagers")
The logind developers went against the established wisdom of experience (that big rewrites are generally a bad idea) with the predictable result: high costs (both in migration and in handling outright bugs) for nebulous benefits, with the result that desktop linux is flakier and (understandably) less popular than ten years ago.
> This is the kind of toxic behavior that makes communities non-inclusive and leads to impostor syndrome.
Is that supposed to be a bad thing? We should be less "inclusive" of people who want to rewrite everything. They should feel like impostors. You can't produce good quality if you're not willing to call out bad quality; Linux succeeded (for a time) because Torvalds had high standards and was willing to maintain them.
>We should be less "inclusive" of people who want to rewrite everything. They should feel like impostors. You can't produce good quality if you're not willing to call out bad quality; Linux succeeded (for a time) because Torvalds had high standards and was willing to maintain them.
I'm making a humble request, please do not bring this attitude in open source projects. Really I mean it. It's not helpful and it only makes people angry. You are also misinterpreting the behavior of Mr. Torvalds and confusing things. The kernel developers have actually been some of the most adamant about rewriting major parts of the kernel and breaking internal APIs over and over again (not syscalls) because it's known that the only way to thoroughly improve on the code is to aggressively iterate on it like this. This is actually a major strength of open source: anyone who wants to try to rewrite something can pick up the code and just do it. If it's bad then you throw it away and forget about. If it's good then you keep it. This is precisely how the "high standard" even gets maintained.
> I'm making a humble request, please do not bring this attitude in open source projects. Really I mean it. It's not helpful and it only makes people angry.
I respect your position but I disagree. Projects shying away from criticism in the name of inclusiveness has gone hand in hand with a drop in quality, not just in terms of unwise rewrites but in terms of plain old bad/buggy code - which should not be surprising.
> You are also misinterpreting the behavior of Mr. Torvalds and confusing things.
My point is that Torvalds - historically - used language on the level of "incompetent vandals" freely where appropriate. In a serious software project people are, and should be, willing to state those kind of views very clearly and directly.
> The kernel developers have actually been some of the most adamant about rewriting major parts of the kernel and breaking internal APIs over and over again (not syscalls)
They have; at the same time they've been adamant about the need to avoid regressions, both in terms of maintaining external interfaces and in not ripping things out before a replacement offers feature parity and there's a reasonable migration plan in place. The bottom line is that they caused nowhere near the level of user-facing breakage that the systemd/gnome folks have, and that speaks to higher standards and better judgement.
> Also, never call out others as "incompetent vandals" if you think of yourself as a "serious software professional". This is the kind of toxic behavior that makes communities non-inclusive and leads to impostor syndrome.
I want to exclude incompetent vandals from the software community; they are impostors.
Some people consistently make bad decisions. Some of them can change, but others cannot. I do not want the ones who cannot or do not learn to make good decisions to make decisions which affect me (I acknowledge that I myself may be one of those people!).
Quality matters. Reckless vandalism matters too. Breaking nohup was and is indefensible.
> However, if you login to a graphical desktop, that is not a text terminal, and therefore everything is effectively nohup'd
No, it's not.
I think you're looking at this in the wrong way. The concept here is one of sessions and the parent-child relationship between them, as well as the decisions parent processes make when they create and manage (or don't manage) child processes.
If you log in to a text console, you end up with a process (like bash) controlling the terminal. If you run normal programs in the foreground, or even in the background (as long as you don't disconnect them from the controlling terminal), they are all children of that bash process. When you quit bash, all its children get terminated as well.
When you log into a graphical session, the login manager (or whatever) will start your desktop's session (which might just be a plain shell script, maybe just your window manager, or might be a full-blown session manager, or something else entirely). Whatever that is, it starts other applications (say your window manager, a panel or dock, desktop manager, etc.), which then can start other applications (browser, chat, media player). If you log out of your desktop, ultimately what happens is that original session-starter (script, WM, session manager, whatever) quits, and it takes all its children with it.
And regardless, the graphical session still runs in a TTY, just not in text mode!
> Since the kernel's idea of "a login session" is wrong
The kernel has no concept of login sessions at all (it does have the concept of "sessions", but they are unrelated to user login). It just starts something as root (init, as PID 1), and from there userspace takes over and does whatever it wants, including the possibility of starting a getty, which can run login, which can setuid() and launch your shell if you put in the right password. (Or run a display manager that does something analogous with graphical sessions.)
So back to the beginning:
> The trouble is that 'nohup' is not a particular state that specifically marks processes that want to survive logout. Instead, the rule is that processes running inside a particular terminal are killed when that terminal closes, and processes running outside any terminal run as they please.
Those two sentences would seem to be contradictory, no? "nohup" is of course not a particular state, though it is responsible for putting a process into a particular state: that of not having a controlling TTY. Which is (greatly simplified) what determines whether or not a process keeps running once the TTY's controlling process exits.
> If you log in to a text console, you end up with a process (like bash) controlling the terminal. If you run normal programs in the foreground, or even in the background (as long as you don't disconnect them from the controlling terminal), they are all children of that bash process. When you quit bash, all its children get terminated as well.
> When you log into a graphical session, the login manager (or whatever) will start your desktop's session (which might just be a plain shell script, maybe just your window manager, or might be a full-blown session manager, or something else entirely). Whatever that is, it starts other applications (say your window manager, a panel or dock, desktop manager, etc.), which then can start other applications (browser, chat, media player). If you log out of your desktop, ultimately what happens is that original session-starter (script, WM, session manager, whatever) quits, and it takes all its children with it.
This isn't how it works though: killing a parent does not kill its child, which is probably one of the bigger design flaws in the original unix (and probably persists because through another design flaw it's the only way to reparent processes, as the insane double-fork 'daemonisation' routine demonstrates). This is one of the things systemd aims to fix: by keeping track of process relationships with cgroups it can kill all the of the processes spawned in a session.
And I'm not sure about your assertion about graphical sessions: pretty much all processes on my PC have no controlling terminal.
> and it takes all its children with it.
Now what happens if it crashes? Repeatedly? Or if a user intentionally manipulates their session to keep running?
Should any user on a shared university computer be able to spawn processes to run for all eternity?
Should your desktop environment crashing lead to all software continuing to run, for all eternity, leaking memory like there’s no tomorrow?
> Should any user on a shared university computer be able to spawn processes to run for all eternity?
That's for the university IT to decide, of course. Software should focus on providing general mechanism, not policy.
> Should your desktop environment crashing lead to all software continuing to run, for all eternity, leaking memory like there’s no tomorrow?
AIUI, that's pretty much what might happen if you're forced to enable the lingering option to make nohup work. Systemd does improve session management under *ix-like systems, but the fact that it doesn't manage to interoperate cleanly with the likes of tmux and screen is a pretty blatant papercut.
> the fact that it doesn't manage to interoperate cleanly with the likes of tmux and screen is a pretty blatant papercut.
Interoperating with broken hacky solutions is nothing admirable. At some point, you have to introduce a new API, which handles this properly, and introduce a way to give/take permissions for these things (so e.g. zoom doesn’t just run their dataminer forever through the same mechanism).
I've never understood this argument. What keeps, in your example, Zoom from just setting the Linger state via the new API?
You can grant or remove the linger state permission from users and applications.
Default is usually that either any user can have any software lingering, or that any user can authenticate (as you'd do with e.g. sudo) to set this state. This would work via PolicyKit and elevated permissions, similar to the UA prompt on Windows.
You can also white- or blacklist individual applications and services from this :)
That's what I meant with "proper" API in contrast to old tmux/nohupd.
And as service management is the init system's task, it's clear this is something where you have to interface with the init system (as the init system is actually even supposed to reap any process reparented to it to reduce zombie processes).
I'm not seeing a lot of hope here. There is an incredible investment in systemd by distributions, there is no turning back.
Help me..
Slackware doesn't use it, and Patrick Volkerding recently got a Patreon account set up so Slackware 15.0 or 14.3 will hopefully be out soonish with updated packages. I'm writing this on 14.2 and have been running it since it came out with basically no issues. Also have it on a backup server, runs great.
I love slack.
There's something extremely beautiful in launching htop and seeing less than 10 processes (including those of root).
But it’s 2020, and we need package management.
Use Alpine or something then. I use Alpine as well on a server, OpenBSD on a server, slackware on a server, slackware on my laptop... There's some options. We even use Devuan to host OP's essay, although we'll eventually move to openbsd probably for other reasons.
In terms of package management... why do you need it? I have no problem maintaining everything with slackpkg and sbopkg along with slackbuilds.org. Sometimes it takes a while to find all the requirements for an application and add them to a queue, but once it's set up it's just sbopkg, click on update, upgrade, and you're good. It's pretty much rock solid once I get everything installed and I haven't missed package management much at all. My main gripe is the old packages in 14.2 but -current has a lot newer stuff. I don't mind waiting though.
It's 2020, and nix and guix both run on top of Slackware just fine.
My bet is that it depends on whether NixOS or Guix takes over the world. (And my bet within a bet is that it will be NixOS.)
Void Linux, amazingly awesome distro.
Second this, an incredible distro with an impressive package manager.
Except its maintainer recently quit.
That's not accurate. There are many competent maintainers.
OpenRC, it's available on some Linux distros.
Gentoo is reaching out to you.
Or if you don't want to go source-based, maybe Void or Alpine
Devuan.org
I still don't entirely get the point of Devuan. I put Debian buster on a low-RAM embedded box, and since systemd eats up precious RAM I'd rather keep, I just switched to OpenRC with
If I needed it, Debian packages elogind as well.apt install openrc && apt purge systemdRebooted and it worked perfectly. Now, I get that Debian doesn't really support OpenRC[0] (or sysvinit), and it could break in horrible ways when bullseye goes stable, or get removed entirely, but... I don't see why we need a fork before that happens? It seems like it's a lot of work to maintain a distro fork, when I feel like that effort could be more productively redirected to stronger maintenance and advocacy of OpenRC and/or sysvinit in Debian itself?
[0] Debian's openrc package hasn't been updated in a little over a year, which is indeed concerning. sysvinit does seem to be more actively maintained, though.
Late reply, sorry.
The problem isn't systemd's init. It's probably fine.
The problem is the systemd project is taking over the userspace with mutually cross-linked modules.
Devuan was started by a group of Debian maintainers who disagreed with the direction systemd was taking Debian.
> there is no turning back.
Iron grip, aggression, pushiness are needed to lobby systemd out.
Another important initiative is to not to let Poettering, Sievers and co. continue throwing new systemds onto distributives. A preemptive action is needed.
No, it has nothing to do with remote and such.
systemd uses cgroups (kernel namespaces), and if you want something to remain after you log out, then start it in a separate cgroup.
cgroups are control groups, you know, to control processes. Which was simply missing for decades in Linux.
nohup should be enhanced to support systemd, or systemd should provide a nohup wrapper, and just start a new scope (cgroup) for whatever the user launches with nohup.
...
Now, that said, I have no idea why distros and systemd did what they did without much communication, but ... that's usually the Linux way :/
> systemd uses cgroups (kernel namespaces), and if you want something to remain after you log out, then start it in a separate cgroup.
Why should the user be concerned with how systemd happens to implement its session management. If you start a screen or tmux instance, it's unambiguous that you want that part of the session to persist after logoff and to be reachable upon logging on to the system again. That's what screen and tmux were designed to do.
The user shouldn't be concerned. That's why the distro should solve this. But as we all know, abstractions are leaky, and encapsulation only goes so far.
How should anyone know what the user really intends. When I start a tmux in a graphical session and click log out I want the system to stop my things completely, (almost) like I haven't even logged in. After all, my session never was set up to be an always-on server session thingie. The distro never advertised this, etc.
Of course users who grew up on UNIX/POSIX/nohup and on distros that worked in a specific way (ie. without cleaning up processes after log out) expected this to continue.
Debian made systemd default in Jessie, which was released in 2015, and it took 2 years for this subsection to appear there: https://wiki.debian.org/systemd#Orphaned_processes
It either did not bother that many people or no one really did anything to document/address it.
Distros did not care apparently. systemd maintainers were aware that it breaks all and every double-forking self-backgrounding resident stuff, but they accepted the trade off, and even documented it: https://github.com/systemd/systemd/commit/65eb37f8fcf0c82db0...
> When I start a tmux in a graphical session and click log out I want the system to stop my things completely.[] The distro never advertised this, etc.
The graphical environment could advertise that you have launched something that will persist upon disconnecting the session, and give you an option to terminate it entirely instead. (It's after all easier for graphical environments to offer these sorts of "user friendly" hints than for terminal-based workflows.)
Have you tried `sudo loginctl enable-linger [username]`? This used to be well-documented, but I don't see much about it now. It's possible things have changed since the last time I had to deal with this?
What do you mean? I still see it in `man loginctl`.
> enable-linger [USER...], disable-linger [USER...]
> Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller. See also KillUserProcesses= setting in logind.conf(5).
Mainly I meant that I no longer see it recommended on the Arch wiki, which I think is where I first learned it. But it looks like that might be because Arch now compiles systemd with lingering on by default (or something similar to that, I'm probably using the wrong term).
Linger does more than just keeping the session open after logout. It also means that the session gets started automatically on boot, which is necessary if you want to have systemd units for that user autostart. Case in point, the PulseAudio setup on my homeserver: https://github.com/majewsky/system-configuration/blob/2f93ff...
> The stated workarounds - fiddling with some options like "KillUserProcesses=no" in logind.conf &co. - have so far failed.
Strange, KillUserProcess=no worked fine for me
In addition I'm pretty sure every distribution ships with it disabled anyway.
Not Archlinux. But That's not surprising given arch's policy of keeping things as close to upstream as possible.
Given the insanely vitriolic level of personal criticism leveled at the systemd people on an ongoing basis, some degree of glibness is an unfortunate byproduct.
How about we give them some benefit of the doubt, that they are mostly trying to make things better, and instead ask them "what is the technically correct way, in the systemd world, of implementing this functionality?"
I suspect the answer, for screen, would probably require some degree of messing with the user manager daemon.
> "what is the technically correct way, in the systemd world, of implementing this functionality?"
In the case of terminating programs left running in an ssh shell by a logged out user, it's a pretty simple answer IMO: just leave it alone - if the admin of the ssh server wants to terminate user processes on logout I think that's where the burden should lie and there's no good reason for systemd to get involved, that I can see.
That thread also mentions the solution that will work: https://askubuntu.com/a/904125
I.e.,
systemd-run --scope --user screenNope:
[user]@[host]:~$ systemd-run --scope --user screen
Job for run-r0b2a43d19a564e2c9f36b5c9b934420f.scope failed.
See "systemctl status run-r0b2a43d19a564e2c9f36b5c9b934420f.scope" and "journalctl -xe" for details.
[user]@[host]$ journalctl -xe | cat
May 04 03:02:54 [host] systemd[918]: Failed to start /usr/bin/screen.
-- Subject: Unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit UNIT has failed.
--
-- The result is RESULT.
May 04 03:02:54 [host] polkitd(authority=local)[879]:
Unregistered Authentication Agent for unix-process:1504:595444 (system bus name :1.25, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
----------------------------
I changed my user and hostname to "[user]" and "[host]" respectively, the rest is verbatim.
It's interesting to note that to even see the error report (such as it is) I'm instructed to invoke a special command rather than just look at logs where I might expect to find it, and that when I do so I am locked into some dysfunctional pager requiring me to pipe through "useless cat" in order to cut and paste the uninformative details on long lines.
While the consistent boorishness of my systemd install continues to amuse, the details of this issue and the fact of its resolvability or otherwise by invoking recondite special systemd functionality are beside the point.
Why should I need to know how to run normal and trusted software in special systemd compatible ways that I've previously had no issues with for decades?
As I opine below, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents. I've run into this in so many ways since ceding defeat and allowing it onto my systems. This is just the latest in a long line of examples.
> As I opine below, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents.
Ha, if only that behavior was confined to systemd... I am forced to consider it a defining feature of much of the OSS/Unix community.
Systemd is mild compared to many projects and its adherents in this regard.
It might be this issue https://bbs.archlinux.org/viewtopic.php?pid=1724323#p1724323
I know that there is a separate command that can be used to tell systemd to allow a program to live. I know that there are systemd libraries that an executable can link against in order to opt out of the new behavior. These do not matter, because they shows that systemd is willing to break existing programs, and to break specified conventions. Systemd developers cannot be trusted to provide a foundation to build upon.
I see this kind of talking about stuff like the Unix way and existing conventions and programs. I almost have to ask myself when did we end up in a mausoleum. The Unix way is just a guideline from a time some folks wrote some code. A seminal and important time. Its just one(perhaps of many) piece of anecdata. Old code is old code. It's useful but the patterns and conventions it was built on may no longer be relevant. It is not obvious apriori that it is worthwhile to preserve existing conventions and not break existing code.
The conventions "Do one thing and do it well", and "Don't break the user experience" are not threadbare or shabby. Not UNIX nor its derivatives did spring fully formed from the foreheads of K̶n̶u̶t̶h̶ Thompson or Ritchie; it was written as a series of counterpoints to the prevailing designs and implementations of the day.
To us the ashes of our ancestors are sacred and their resting place is hallowed ground. You wander far from the graves of your ancestors and seemingly without regret.
Donald Knuth was not a designer of Unix, and he is still alive.
Also, Linux is not Unix and was never intended to be Unix or follow its conventions.
The way Linux works now seems sane. You might very well have seen a change in the way Linux works. But when you've seen it change, multiple times, and each time 50% of what you know has to be tossed out and relearned, it gets tiresome. At times it just seems like it's change just for change's sake.
I want to have fun with the computer, not run madly just to stay in place like the Red Queen. Remember to re-read your own comment 30 years from now and see if you feel the same way.
It's a good question and infects all of our software "sacred cows."
The underlying thing is that we keep driving ourselves to "forward progress" in the sense of a collaborative hegemony, and only in those terms. Either a business wants to own the platform, or the developer wants to build that platform. To do that they have to achieve buy-in from existing stakeholders, but simultaneously reinvent incompatible things. Thus through repeated application of this approach the world of professional software development has aggregated itself into conformance to standards that barely make sense, are poorly specified, and have limited proof of concept, but tick whatever buzzword boxes are relevant to the immediate climate.
If you want to take a real stand, invest yourself in "dead" technologies. Then you can choose whatever you want, and if other people want to follow you on it it's implicit that they are working on a similar problem, and not trying to play the platforms game(else they would be looking for an angle to "modernize")
Healthy skepticism is necessary for any technologist. However, you do have to use it in both directions. I know my biases are that I prefer new technology to old. So I pay attention to that. However, it is just as bad to reflexively consign something as premature or ill-considered. Sometimes the collective hegemony gets it right. They may have missed on microservices but they didn't miss on the cloud really. I do try to be eclectic in my technology choices if only for cognitive reasons.
I'm actually very happy they chose to break this particular behaviour, based on their explanations about security and not allowing long-running services to hog up a machine when its user isn't logged in.
This is severe scope creep for an init replacement. I don't exactly know what "the service manager of the calling user" is meant to actually do (that's way the hell outside POSIX), but if a user wants one they should be able to choose, and it shouldn't matter to them how the sysadmin chose to create routing tables and mount filesystems.
Why should I be forced to prepend all of that to every command I want to survive terminal hang ups? The old mechanism was more user friendly, and better documented.
It’s just another example of systemd making everything on the system worse.
You could alias the command. You distro could do it for you. Or your ops person could. There are plenty of points where this could have happened.
I read this comment, thought after having recently installed Debian replacing Slackware I'd wipe it again because that's unacceptable. So I ssh'd into the machine, started a screen session, detached it, killed ssh session, ssh back in and the session was still there. Is this an older problem? So far, after coming from Slackware, systemd hasn't been in the way so much, thankfully.
Does nohup work? That's what I would expect to be the standard solution, as the Unix tool that "does this one thing and does it well".
No. SystemD defaults to breaking nohup, tmux, screen, emacs server, etc.
I'm not sure why you're getting downvoted when it's the stated position of the systemd developers that they default to breaking those things. They've acknowledged it publicly. Hell, there was even a thread on the exact issue with tmux 4 years ago: https://news.ycombinator.com/item?id=11797075
Top comment on that link:
Or Nicholas Marriott's 9 year old as-yet unanswered questions to the systemd developers, that I copy from a post by hn user JdeBP:"""Salient comment: "Or somebody could go find the actual problem @keszybz saw here - systemd/systemd#3005 - which is: In particular, for my gnome session, if I log out, without KillUserProcesses=yes I get some processes which are obviously mistakes. Even if I log in again, I'm much better starting those again cleanly. fix that, and stop trying to make systemd break the world because somebody's gnome session doesn't currently exit cleanly."""
https://news.ycombinator.com/item?id=11798515> "Shouldn't this code be part of glibc instead of tmux?" > -- Nicholas Marriott, 2011 > If you want to change how processes daemonize, why don't you change how > daemon() is implemented [...] ? > -- Nicholas Marriott, 2016 > * https://news.ycombinator.com/item?id=11798173 > * https://news.ycombinator.com/item?id=11798328You shouldn't be downvoted, this is correct. Sane distros will change the default.
> Sane distros will change the default.
And sane developers don't break multi-decade behaviour in the first place.
Screen still works. I use it all the time.
Do you have decent examples of it failing to work? If so, then that sounds like a bug against screen. The link you provide is absolutely huge and from three years ago.
Please give me a simple "stages to reproduce ..." style report. Please keep it simple and short and I'll fill in the blanks if I can and only trouble you for stuff I'm too daft to work out.
To answer your question:
remote login, start screen, ctl-A d (detach), log out, log back in, screen -r (resume) - there is no screen session because it was killed by systemd immediately after log out.
(syslog entry: May 3 09:01:25 $HOSTNAME systemd[1]: session-6.scope: Killing process 3290 (screen) with signal SIGTERM. )
But that's hardly the point, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents. I've run into this in so many ways since ceding defeat and allowing it onto my systems. This is just the latest in a long line of examples.
Based on one of the sibling comments, if you modify screen to add a runtime dependency on systemd, the problem will go away.
I agree this is totally unacceptable, and that the inevitable tone deaf response to such complaints is also unacceptable.
Is it that weird that if you want to run a job in the background outside your session that ask your service manager to spawn the job?
I mean all of this is just a difference of opinion between, "I want a process supervisor to manage all my background jobs" and "I just want to double fork and throw processes at init."
> remote login, start screen, ctl-A d (detach), log out, log back in, screen -r (resume)
I just tried it on a fully up-to-date CentOS 8 box (which uses systemd), and it worked perfectly fine: after logging back in through ssh, `screen -r` restored the screen session as expected.
From what I have read, that screen session would only be killed if I had set KillUserProcesses=yes on /etc/systemd/logind.conf, which is not the default.
It is the default.
This behavior is controlled by the KillUserProcesses= setting in logind.conf, and the previous default of "no" is now changed to "yes".
https://github.com/systemd/systemd/blob/2d4f8cf467b6825c9127...
This is one of those fun situations where everyone's right.
The upstream default has changed, but CentOS still uses the old default.
Almost every distro I know turns this option off by default, including Arch, NixOS and Debian (the last one I am not sure).
It defaults to "no" on Debian 10. Maybe Fedora changed the default?
Fedora does not have this feature on: https://src.fedoraproject.org/rpms/systemd/blob/265d91aff516...
That's most interesting, given that both systemd and Fedora are Red Hat products. If there's one distro that would use the systemd defaults, I would expect it to be Fedora. Who exactly are these defaults for, then?
I've mainly seen it switched on for appliances and things that don't really have a legacy of this behavior.
It's not switched on in RHEL or Fedora. It's likely to never be so.
Switching to CentOS in order to use screen is not an appealing option, and the config appears not to work on my current distro (that might be my poor testing, but it's certainly not the default anyhow.)
I understand Slackware still lacks systemd, so maybe that's a better choice for me :)
Please submit a bug report if the option doesn't work for you.
I'm not getting this at all - so this is a can't duplicate. What distro are you using?
The box in question is in use as a media server, but reloading logind logs out the X session, so it wasn't really possible to test properly until today. Having the opportunity to restart the machine (probably I just needed to systemctl daemon-reload?) the "don't kill my processes bro" options are working.
The distro is kde neon.
The point isn't that I can't (or the distro maintainers can't) get it to work, the point is that the aggressive systemd default breaks established practice, and that this is typical systemd behaviour: arrogant and uncaring. IOW user-hostile in the fullest sense of the term. It's not like this is an isolated example.
Which distro is this?
> The link you provide is absolutely huge
The first message in the bugtracker already has the context of whats happening.
Maybe you should offer to move this bug report conversation somewhere more appropriate? It seems that this is better suited to a github issues thread
found the project manager!
> Do you have decent examples of it failing to work?
Poettering:
> Note that in some cases it might be a good thing to kill screen sessions when the user otherwise logs out […]
* https://lists.freedesktop.org/archives/systemd-devel/2011-Ju...
This is such a classic example of the type of argument I see from most anti-systemd proponents that it made me laugh out loud when I clicked your link.
For the lazy, here's the context of that cherry-picked sentence:
> In order not to break screen we currently do not set kill-user=1 or kill-session=1.
> Note that in some cases it might be a good thing to kill screen sessions when the user otherwise logs out (think university, where students not logged in on a workstation should not be able to waste CPU), in other cases its a bad thing however (i.e. in yours). That means it must be configurable whether screen is considered an independent session or not.
My argument is remarkably simple.
1. Systemd did not solve any problems I actually had.
2. Systemd introduced problems that I did not have previously.
3. Systemd did not provide me any net benefit, that is, the few benefits it did provide over Upstart/SysV/etc. (easier service configuration and ordering) did not overshadow the issues it caused. Its introduction into my home and professional computing life has been a net negative.
This, exactly.
My first experience with SystemD (Ubuntu 16.04 IIRC) turned into a frustrating experience, mostly around trying to mount a NFS volume in /etc/fstab, defined with 'soft' and 'bg' mount options. SystemD would often fail to mount the volume, and as a result would never fully boot to login prompt. Additionally, on shutdown SystemD would hang again forever trying to unmount that volume. No amount of mount option flag tweaking seemed to make much of a difference, so I simply removed the NFS volume from fstab and manually mounted when I needed it.
Happily running MX Linux today and back to mounting NFS in /etc/fstab.
systemd did make package maintainer's lives a lot easier. We don't often see the shift to systemd from their perspective. They are the ones making the distros so I think they are the ones who really get to say what goes and doesn't go in a distro. They don't have to keep maintaining long init scripts for the hundreds of packages that they are responsible for. Unit files are so much easier to deal with.
Translation: We broke long-standing POSIX/Unix behaviour of noHUP and think that's a good thing.
I mean that is their position yes. It’s a good default too since it’s weird that “logging out” doesn’t imply “and end all my programs.” This is basically the same behavior as Windows where anything not going through the task scheduler ends when you log out.
This is a feature that sysadmins have been asking for. On any multi-user system you run into this crap where background processes for users who are long-since gone just hang around forever because they do weird things or hang and ignored the signal. We implemented it ourselves with PAM but systemd’s solution is a lot cleaner.
And they’re not breaking any POSIX behavior. Nowhere does it say when the system isn’t allowed to kill a process.
> it’s weird that “logging out” doesn’t imply “and end all my programs.”
Weirdness is in the eye of the beholder. I regularly run programs whose lifecycles are not in sync with my login session. Why do I need to stick around to see a batch job complete?
I can understand that different folks have different backgrounds which changes expectations... but come on... somebody bringing up tmux, screen, etc. should simply end the conversation. "Oh, that is a common and historical use case that I have not considered, today I learned something."
>> it’s weird that “logging out” doesn’t imply “and end all my programs.”
> Weirdness is in the eye of the beholder
It's a windows acolyte mindset. Many people don't imagine that a login is just another program, rather than the parent process.
Yes but that model conflicts with the human notion of being “logged in” to a system.
And the current behavior is literally what you describe. It’s just the quirk of subreapers being implemented recently that daemonizing a process wasn’t local to your session.
It makes zero sense that a process that double forks is reparented by init instead of your session leader. If how things worked currently was proposed today it would sound crazy.
>It’s a good default too since it’s weird that “logging out” doesn’t imply “and end all my programs.”
I have to disagree, I don't think I've ever run something where what I intend is "kill this mid-run if my connection drops".
>This is basically the same behavior as Windows
My recollection from when I used Windows Server (admittedly some time ago) is that the default behavior on connection loss or closing an RDP window is indeed the tmux-like behavior. The user has to click a separate "Log out" button, and even then, has to manually confirm that it's okay to kill anything left running.
> It’s a good default too since it’s weird that “logging out” doesn’t imply “and end all my programs.”
It may be a good default if you're starting from scratch, but GNU Screen has came out in 1987. (Poettering was 7 years old then.)
> This is basically the same behavior as Windows […]
Comparing it to Windows' behaviour is not going to win you any positive points with me. :)
> This is a feature that sysadmins have been asking for.
As a sysadmin this breaks my own daily workflow. I have 400 VMs that are treated as pets and I probably screen sessions on at least a third of them. (I have another 500 that are cattle-like.)
> And they’re not breaking any POSIX behavior. Nowhere does it say when the system isn’t allowed to kill a process.
Aka, malicious compliance. See also: nowhere does it say that the US President cannot fire the FBI Director. Also: nowhere does it say that the US Senate has to hold hearings when a US President nominates a judge for the US Supreme Court.
> This is basically the same behavior as Windows where anything not going through the task scheduler ends when you log out.
But when using remote connections, ie RDP, the default is that your session is disconnected. You have to actively choose to log out of an RDP session.
When using SSH I typically want the same. If I put my laptop to sleep or a networking issue causes the TCP connection to end, I want my terminal session to be restored when I reconnect via SSH.
”ain't no rule says a dog can't play basketball”
Why exactly should students not be able to use CPU cycles while not logged in? I certainly did this as a student and would probably not have complained if it decided to kill my processes when doing so.
I don’t know if these even exist anymore, but I’m imagining a class that meets in a room of workstations. Students can login to any workstation in the room. In this kind of shared environment, you may not want to have a student start a screen session and use the resources of a workstation when they logout. They could be sucking up cpu time from the next person to log in (from the next class).
It’s a scenario that might make sense here. But then again, having a watchdog script just kill leftover processes has been doing this same job for forever. Including this use case in systemd seems like overreach.
Yeah, these still exist, though they're becoming more rare. I think what you actually want though is for processes to be killed at the end of class/lab period (even then, I think this should be done by students manually to avoid losing unsaved work). For example, say a student is running some big batch job as part of a lab. Now, if he wants to, say, go to the bathroom or go to the whiteboard to consult with other students, he has to either leave the computer unlocked (possibly leaving neighbors access to private student account data) or kill the running job.
> It’s a scenario that might make sense here.
Perhaps. I have worked half my career in academia & research, going back to when Solaris (SPARC) workstations were a thing, and this was never a problem.
You don’t deserve the downvotes. This is precisely the use-case this feature is good for. Multi-user systems where persistent background tasks for unprivileged users doesn’t make sense and is more indicative of a misbehaving program than a user trying to run a job.
Oh my god the number of times we had to bounce systems because the unintended background processes of long gone users overloaded the system.
While it's a valid argument, it's also a prime example of a special case that should not be the default - especially so because in my university, people were instructed to use the Linux workstations remotely for long-running tasks.
Sounds like you had junior admins at best leading your university's *NIX department.
A mid-level admin in 1995 could have easily made a 100 line script to kill background processes from non-logged in users that were running over three hours (or whatever).
systemd brought nothing to this arena.
I don't understand this angle. You acknowledge the functionality could have been useful even 25 years ago; it makes perfect sense for a daemon developer to integrate the functionality into a session manager and put it behind a config flag which is exactly what they did.
The functionality existed 25 years ago. systemd coming along and poorly implementing it provides nothing.
The functionality of implementing it yourself?
But that’s such a useless definition because then all software “provides nothing” since it’s implementation proves that you could have written it yourself.
logind is far more robust than any of the janky shell scripts I’ve seen over the years to accomplish this.
The functionality to gracefully end long running background processes has been a part of standard killall, pkill, etc. for quite some time.
Even then, yes, the functionality of using ps and kill together has existed for 20+ years. Those tools are already implemented, providing the functionality for decades.
In no universe is logind considered robust, and basic scripts from middling UNIX admins have provided this functionality for decades, untouched. Even the "janky" ones.
See my comment below. These tools are fundamentally broken for this use case and have never provided the functionality needed. Bash scripts are not a process manager and it is incredibly wrong to try and make it work like one.
I responded to the comments below. These tools are not fundamentally broken, you're just ignorant in this space. They have always provided the functionality needed. Scripting can easily manage processes at a higher level, and it's basic functionality to make it work like that.
If you already had that feature then you are free to use your 25-year-old solution instead of systemd.
You have to, since systemd implemented the feature incorrectly. You're missing the forest for the trees.
I've already been in a forest of bash scripts and I would not go back there again. I have no comment on systemd's implementation but the implementation you're talking about is also incorrect. It has never been safe to kill random processes using a bash script running in the background, on most Unixes (and Linux) it is 100% impossible to do that without race conditions due to the limitations of procfs. Doing "ps | grep" and "killall" is a footgun. You would need to implement this in C for it to have a chance of being safe at all, and even then, you would need to rely on system-specific functionality because there is no portable way in POSIX to actually do this.
>I've already been in a forest of bash scripts and I would not go back there again. I have no comment on systemd's implementation but the implementation you're talking about is also incorrect.
In certain cases I don't disagree, but systemd does not implement this feature correctly, so using functionality that's easily reviewable from decades past makes sense. If systemd could properly implement the feature, there would be no need for the scripting.
>It has never been safe to kill random processes using a bash script running in the background, on most Unixes (and Linux) it is 100% impossible to do that without race conditions due to the limitations of procfs.
This is hilariously wrong. Lots of UNIX OSes don't even implement the ps suite of tools by using procfs.
Even then, there were no race conditions in this use case anyway.
>You would need to implement this in C for it to have a chance of being safe at all, and even then, you would need to rely on system-specific functionality because there is no portable way in POSIX to actually do this.
It's already implemented this way on plenty of UNIXen. ps and kill is the POSIX portable way to approach this, so you're wrong about that as well.
Almost everything you said above is incorrect, or misunderstanding basic UNIX.
If you really want systemd to add this then I'm sure they will look at your feature request or PR.
I would urge you to actually check with your system instead of blindly dismissing me as wrong just because your bash script happened to work without error. In my experience BSD-based Unixes get it right and don't use procfs for ps or pkill. They don't have the problem because they use special syscalls for this. But SysV-based systems have historically used procfs to implement ps. Linux also still does. Try unmounting proc and running ps or kill and see what happens. If you can't do it then your system suffers from the problem, which is that you can't safely send a signal to the process after reading it because there is no guarantee that the actual PID will persist in between calls to read() and kill(). POSIX says nothing about this because it doesn't specify procfs, or how pkill should actually be implemented. This is all fair game as far as compatibility is concerned.
There is also the other more obvious race condition in your bash script which can also be pre-empted in between the calls to ps and kill. This can happen on any Unix and is not some big mystery either. PID reuse has been a known problem for decades and Linux finally got a solution to it a couple years ago with pidfd_send_signal. There is also the matter of cgroups but I am not going to get into that because I doubt you will want to hear about it.
>If you really want systemd to add this then I'm sure they will look at your feature request or PR.
The less systems touches the better.
>I would urge you to actually check with your system instead of blindly dismissing me as wrong just because your bash script happened to work without error.
Who said this was bash? The script from all those years ago worked perfectly without race conditions.
>They don't have the problem because they use special syscalls for this. But SysV-based systems have historically used procfs to implement ps.
Makes no difference, procfs works fine for this.
>Try unmounting proc and running ps or kill and see what happens.
You're effectively never unmounting procfs. Also, if you managed to, systemd would crash!
>If you can't do it then your system suffers from the problem, which is that you can't safely send a signal to the process after reading it because there is no guarantee that the actual PID will persist in between calls to read() and kill().
There's maybe 10 lines of code to ensure that logic. Again, middling sysadmin work.
>POSIX says nothing about this because it doesn't specify procfs, or how pkill should actually be implemented. This is all fair game as far as compatibility is concerned.
POSIX does say something about this; read about signals. You stop the process before killing it, and ensure that the start time is the same for the stopped process before the kill. That completely eliminates the race condition, using basic POSIX signals.
>There is also the other more obvious race condition in your bash script which can also be pre-empted in between the calls to ps and kill.
That race condition is eliminated with the logic above. That eliminates the PID reuse race condition, even if it's very rare.
>reuse has been a known problem for decades and Linux finally got a solution to it a couple years ago with pidfd_send_signal.
That functionality is where it belongs now, so instead of coding a logic every time you have to ensure PIDs, it's now handled for you.
>There is also the matter of cgroups but I am not going to get into that because I doubt you will want to hear about it.
Correct, I don't want to hear it from someone who doesn't understand POSIX and basic/intermediate sysadmin work.
at university I remember working around the auto-killing of logged out users processes by running screen, then inside that screen ssh'ing into localhost
Beautiful.
I think this really sums this article up:
> One thing I’m certain of is that this shift cannot emerge from dilettantes, outsiders and proverbial basement hackers. One does not unseat a platform without already being part of the patriciate that calls the shots on what gets integrated where across the largest nodes in the ecosystem.
It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work. An init system is harder than it might seem at first and requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators. If you’re not going to invest that level of engineering time, I don’t see how it’s reasonable to expect an equal voting share with those who do.
> An init system is harder than it might seem at first […]
systemd-as-init-system is not the problem. systemd-as-kitchen-sink is the problem.
It's the tight coupling that annoys many people.
Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?)
What does systemd 245's systemd-homed have to do with an init replacement?
* https://www.techrepublic.com/article/linux-home-directory-ma...
> So, for the simple act of logging in, three mechanisms are required (systemd, /etc/shadow, /etc/passwd). This is inefficient, and Poettering has decided to make a drastic change. That change is homed. With homed, all information will be placed in a cryptographically signed JSON record for each user.
10 years ago this would have been considered satire
homed is a purely optional, independent tool, for use in situations where you’d use roaming home directories on windows, or similar technology.
No one is forcing you to change your broken buggy 30-year old shell scripts, you can always continue using them.
> No one is forcing you to change your broken buggy 30-year old shell scripts, you can always continue using them.
How about usernames that start with a digit? Am I still allowed to use those?
* https://ma.ttias.be/giving-perspective-systemds-usernames-st...
POSIX says that any tool should always translate usernames into userids at the earliest possible point, and pass UIDs to children it spawns.
Now assuming I have a user called 1000 with UID 2000, and a user called 2000 with UID 1000.
Now what do you think is gonna happen if multiple POSIX-compliant tools call each other?
This is clearly broken, and the reason why BSDs don’t actually implement POSIX cleanly, instead allowing all their tools to take unambiguous identifiers on the CLI, with the # prefix to mark a UID, and have all their tools call all their children with this format.
The POSIX standard is clearly suboptimal here, and the set of valid usernames, and the set of valid UIDs should never have been allowed to overlap.
The weblog post has an user called '0day', i.e., "zero day". This breaks systemd as outlined in the post. This is a WONTFIX per the systemd folks.
That’s one of the possible options, the other would be replacing chown, chmod, ls, etc (as the BSDs do)
This must be satire, not? Because soon my .forward won't work, my .procmailrc won't work, my crontab scripts won't (etc pp) unless my encrypted home directory (planned to be decrypted on login) is decrypted. Or all these are moved to other locations.
Or you opt to not encrypt your home separately.
But how large a proportion of machines do you think have mail delivery rely on .forward or .procmailrc in $HOME these days?
In any case, procmail can be executed directly rather than from .forward in most cases, and can be configured to take the rc file to be a file in /etc/procmail-rcs/ as an alternative to $HOME/.procmailrc (at least the version of procmail in Ubuntu; unlike /etc/procmailrc, the files in /etc/procmail-rcs/ will take on the user id of the user the file is named for, like $HOME/.procmailrc).
User crontabs on my system at least (Ubuntu) are in /var/spool/cron/crontabs/[username], not in $HOME anyway. I can't remember the last system I had with user crontabs in $HOME.
People have had systems where relying on accessing the home directory of a logged out user has been undesirable or impossible for a very long time.
> But how large a proportion of machines do you think have mail delivery rely on .forward or .procmailrc in $HOME these days?
I help run a 500 node HPC cluster that leverages procmail. Do I count?
> User crontabs on my system at least (Ubuntu) are in /var/spool/cron/crontabs/[username], not in $HOME anyway.
The crontab(5) may be there, but where is the script that is actually executed? Probably in $HOME/scripts/ or some such.
> People have had systems where relying on accessing the home directory of a logged out user has been undesirable or impossible for a very long time.
I have daily experiences indicating otherwise. I would hazard to guess that many other syadins in academia & research would agree. Even assuming $HOME is in /home breaks a bunch of stuff:
* https://utcc.utoronto.ca/~cks/space/blog/linux/Ubuntu2004Sna...
> I help run a 500 node HPC cluster that leverages procmail. Do I count?
You do, but you're a tiny minority, and as noted /etc/procmail-rcs/ provides an alternative to putting them in /home. If that doesn't work for you, all you need is to not configure encrypted home directions and nothing changes.
> The crontab(5) may be there, but where is the script that is actually executed? Probably in $HOME/scripts/ or some such.
So don't configure encrypted home directories, then. Nobody is forcing you to enable it. Or you can provide an alternative location, and let your users protect their data better and still be able to use cron jobs. It does not take any flexibility away from you, but it adds capabilities for those who want them.
> I have daily experiences indicating otherwise. I would hazard to guess that many other syadins in academia & research would agree. Even assuming $HOME is in /home breaks a bunch of stuff:
Your experiences with systems that are fine with accessing $HOME does not mean that there aren't plenty of systems where it has been an issue, hence why plenty of software supports alternatives.
Your point of not assuming $HOME is in /home is irrelevant to this issue. The point is that software have had to deal with a subset of users with setups where a users home directory is not necessarily accessible when the user is not logged in for many years. As such most software has options to avoid relying on a users home directory being present already, which reduces this to a relatively simple choice of whether or not you consider the ability to encrypt home directories to be worth making those changes for.
Maybe my point wasn't stated clear enough: the crontab itself is in spool, obviously, but the scripts started from crontab entries reside in my home directory. And while a procmailrc can live in some spool too, the files to be delivered into are in my home directory too.
So then you have two choices: Move them elsewhere, and be able to encrypt the rest of $HOME, or don't encrypt $HOME. This gives you additional options for protecting your home directory if you want it. Noone is forcing you.
Yeah, don't get me started on the fragile glass test tube called systemd-resolved that fails in mysterious ways when I use wicd on Ubuntu and the WiFi gets momentarily disconnected. I don't see the sense behind treating name resolution as something so special that it has to be part of an init system.
I like the overall theoretical concept of systemd but it has ugly implementation details like this.
But systemd-resolved is such an odd complaint since it’s only connection to systemd is the name and that it’s in the same repo. It’s not connected to the init system at all.
The entire repo is a big ball of mud. No one knows which parts can be removed without breaking things.
Heck, the author of this article couldn’t even find a design doc or adequate manual for the init part of the train wreck, and it’s been in development for ten years.
Just set ENABLE_RESOLVE=0 when compiling and resolved is gone?
You use the word "just" as if this were a simple thing to understand and know. Is this magic incantation communicates in a way that the usual people who compile this package can understand to use it without being intimately familiar with the code?
I ask not rhetorically, just out of curiosity. It's not a typical type of suggestion for other packages I've encountered with open source, and it feels far more difficult to discover than typical decisions at compile time.
> this magic incantation communicates in a way that the usual people who compile this package can understand to use it without being intimately familiar with the code?
Define “usual people who compile this package”. In my books those are distro maintainers who indeed should be familiar with options provided by the init system they choose. If you decide to include oddballs who compile everything themselves (Linux from scratch?) then again, yes, they are expected to know a thing or two or dig into options or even code from time to time.
I mean the parent was talking about the source code being a ball of mud and it being difficult to determine what components can be removed without breaking things.
In this case it should be assumed we’re talking about someone who’s compiling systemd and familiar with the source. And yes, these are flags that anyone who builds systemd from source would be familiar with and know how to set.
It’s the same as when you run a configure script and pass —-disable-feature.
So common with open source - flags that once worked are now tangled up with subsequent code/flags and won't even compile if not part of some common configuration. Because its not fun to exhaustively test all combinations, it doesn't get done.
It seems exactly as difficult as with any other stuff written in C. Let's say uWSGI. systemd uses meson, and there's a file listing all the options, and the first match for "resolve" seems to be that flat: https://github.com/systemd/systemd/blob/master/meson_options...
Furthermore, even if you compile it, you don't have to use it. Even if you install it, you don't have to use it. (After all it has 2-3 APIs, one of them is the /etc/resolve.conf symlink, if you instead just put 1.1.1.1 in that file, libc will not go through resolved. And similarly for the other entry points.)
It's possible to accidentally compile, install and have systemd-resolved set up on your computer, but it still seems rare that someone can compile and install systemd, yet not know what they're doing.
Yes it is.
> Just set ENABLE_RESOLVE=0 when compiling and resolved is gone?
From HHGTTG:
"But the plans were on display …"
"On display? I eventually had to go down to the cellar to find them."
"That's the display department."
"With a flashlight."
"Ah, well, the lights had probably gone."
"So had the stairs."
"But look, you found the notice, didn't you?"
"Yes," said Arthur, "yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.'"
This is a config option that you pass when compiling systemd in the same was as --enable-feature in a ./configure script. If you are familiar with the systemd source you would be aware of these options. They’re not hidden.
> [...] and that it’s in the same repo. It’s not connected to the init system at all.
And that's exactly the problem.
For all the complaining I do about systemd, I'd be much less annoyed with it if all it did was PID 1. It's all the other stuff that gets tightly coupled for no discernibly-good reason that poisons the well for me.
Funny you mentioned systemd-resolved, it generated so many coredumps across the machines I use at home (Arch, Manjaro, Fedora), its stability / reliability improved over time but still not up to standard.
Unfortunately, currently systemd-resolved is the only OS level DNS-Over-TLS solution that works for my use cases: roaming laptops and desktops running Linux. At home it's fine because the core router runs AsusWRT-Merlin which does DNS over TLS for all clients using it as resolver.
I don't have a strong opinion against systemd (initially I disliked it moving from Arch's BSD style rc.d). Trend is that systemd is taking over more and more *ctl, seem like its goal is the entire user-space?. Most distros choose to accept and adopt it, and overall (at some point) it eliminated distro differences and lowered the barrier for new Linux users (desktop) and sysadmins.
Personally I started systemd when it rolled into Arch Linux, after which point, rolling upgrade has never gone broke (workstation use case - Desktop Environment / WM), could be skill levelled up after gaining experience though lol...
BTW: I watched `reinventing home directories` by Lennart Poettering (recording @ All System Go! 2019), interesting one. Personally I like to see reinventing wheels to solve legacy problems (people get used to them, sometimes confused, question but don't know how to solve) others don't even consider problems. These helped open my mind and look at things from a different PoV.
[1]: notes on systemd-homed https://sites.google.com/site/imterry/aboutme/profiles/skill...
systemd isn't just an init system. It's a suite of tools that provides a system and service manager. It's not handling everything in one monolithic process. This has been said thousands of times before. I don't know why people don't get this.
systemd is not an init system. That it has one, it out of necessity. To manage a system you have to manage what runs on it, to manage that you have to manage what gets started and stopped, and so you have to manage boot and shutdown. And now, here's your nametag, there's the counter, don't forget to smile, you're the new init system!
udev doesn't have to be there, but when a handful of people maintain critical parts of the Linux ecosystem and they saw ways to make their maintenance work more effective, they went ahead and made their work more efficient.
I mean, criticizing systemd/udev/Debian/Ubuntu/Arch on how they did stuff, and the technical merits of how things are is completely fair, but criticizing "why" is strange. They did because they saw that it will help them, their projects, their users. (And they probably understood very well that it won't please all of their users.)
Voting share ? There is no vote unless you work for RH. And even then.
An init system is harder then it seems ? You and Jonnax seem to think that unless you made it you have no right to say anything. Have you ever made an init system ? Researched what exactly goes into it ? Sure, there's more then just starting programs. But not much more. The only really complicated thing about getting a [linux] computer from kernel boot to firefox is setting up devices. And even that [udev,"hotplug"] is not complicated as much as it is.. obscure. I know about these stuff, do you ?
I'l defend a newbies right to say anything. Nobody was born with all the knowledge of the world. I will not have compassion for your kind of ignorance.
SystemD is about as much an init as my foot is me. That is the problem that most of those you so casually dismiss have with it. That and that it wants to become mandatory. Could they have just extended udev instead of integrating it ? An i wrong to say that ? Is my opinion wrong because i don't have "Red Hat senior engineer" next to my name ? Programming is just a hobby for me. I don't even have a degree in it. So i'm ignorant on the subject ?
Funny thing.. most of those on teh internets who are vehement in favor of systemD don't know much about how a computer system boots and what goes into that.
edit: Never mind. I see feelings and fancy words are still more valuable then knowledge or thinking. Maybe one day, when computing is not as young of a science as it is now.
> An init system is harder then it seems ? You and Jonnax seem to think that unless you made it you have no right to say anything. Have you ever made an init system ? Researched what exactly goes into it ? Sure, there's more then just starting programs. But not much more. The only really complicated thing about getting a [linux] computer from kernel boot to firefox is setting up devices. And even that [udev,"hotplug"] is not complicated as much as it is.. obscure. I know about these stuff, do you ?
I know enough to know that you left out networking and authentication, network service dependencies, filesystems, volume encryption, etc. – not to mention automatic restarts, conditional behavior based on the network or other devices (stuff needs to happen when I leave the office), and the various edge cases which I’ve seen in three decades running Unix systems. Every other operating system Linux is competing with has a much better model than SysV init (Windows has since the early 1990s!), and the world changed from static servers which rarely changed to most systems needing dynamic behavior – what worked in an old server room isn’t a good fit for a laptop!
Again, I’m not saying that systemd is perfect but that most of the complaints tend not to acknowledge the many problems it solved and a decade of angry forum threads doesn’t seem to have resulted in any concerted effort to build a serious competitor. The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
There is no competitor because there was no problem. For most of use cases SysV, rc.d worked. Just like ifconfig.
Arch Linux switched to systemd earlier than Debian. And I have not noticed. Arch community accepts breaking changes. Debian? I believe they value stability.
Systemd should have matured, proved its citizenship and qualities. And it would have been accepted. Instead it usurped. People had no choice but to abandon their platforms. Decade of angry forum thread shows that there is a problem. Just like with python3, it's better to learn. And it is especially painful that there was PulseAudio just before. Same pointless change. Removed it and never looked back. ALSA works for me, still.
Usurped how? Who and how forced Debian and Arch to switch to systemd?
You're missing a point. It's about feelings. Not my feelings at all. But empathy.
Replacement for "no problem" should run without problems.
python2 had no problems - there was a clear transition path:
ALSA had no problems - just wait till PulseAudio is ready.1. "str" => b"str" 2. wait some time 3. "str" is u"str"SysV, rc.d had no problems - just wait till systmed is ready.
If SysV was as bad there would be same amount of outcry as with systmed
Was it really necessary to make it painful?
Who? Somehow systemd defenders has no empathy and Lennart Poettering is a prime example. No matter right them or wrong they are perceived as threat.
Just be nice. It may hinder innovation but it helps community.
GNOME, essentially. If you want to run a fully-functioning GNOME on your distro, you have to be running systemd. Or have the time and patience to maintain a large patch set to work around the systemd dependencies.
This became this way only once systemd has won. At least during Debian systemd debate, both sysvinit and upstart could run Gnome just fine.
(also, what were the alternatives? I don't run GNOME myself, but given what GNOME wants do to, it does need a solid session management. I have seen ConsoleKit, it was worse than systemd.)
Debian's tech committee had their infamous vote about the default init system for Jessie in February 2014. Earlier, in October 2013, Josselin Mouette, a GNOME packager for Debian, wrote this:
"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."
Debian used to value stability. Nowadays - not so much.
> you left out networking and authentication, network service dependencies, filesystems, volume encryption, etc. – not to mention automatic restarts, conditional behavior based on the network or other devices (stuff needs to happen when I leave the office), and the various edge cases which I’ve seen in three decades running Unix systems.
I’ve been running Unix systems for 25 years. All of the things you mentioned were already solved adequately in the Linux 0.9 days, and the BSD’s still use old fashioned shell based inits, and work fine in my experience.
In fact, the only time I’ve had issues with any of this stuff has been in the last ten years, and only on systemd-based distros.
The genius of init systems is that they factor all this stuff out, and just handle starting and stopping daemons.
> All of the things you mentioned were already solved adequately
really?
Let's say (not entirely hypothetically) that you have a daemon. This daemon has a config file, which is large and complex. This daemon's /etc/init.d file builds command-line dynamically, then passes it to start-stop-daemon as usual.
Puzzle level 1)
You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
Puzzle level 2)
Turns out this daemon used LC_ALL (PATH? HOME?) when parsing config, so it works fine when started or restarted from command line, but fails when started as an part of startup process. How do you debug this?
Puzzle level 3)
One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?
The problem of all the init systems is they could be great, but their development just stopped at some point -- and instead of fixing the real annoyances in a general way, they were just saying "this is all fine". Until systemd came and ate their lunch.
Not sure if joking (since, sysvinit is very old and was in need of replacement) so I'll answer in good faith.
> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
"oh my process failed; let's add `-x` to the invocation."
> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?
as above.
> One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?
`dmesg` or /var/log/messages (or /var/log/dmesg)
Puzzle level 1) ini file:
you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?
I would not call this "easy" or "solved". Possible, yes. Pleasant, no.
Puzzle level 2) inherited env -- I think you made copy-paste error? You just copied the same paragraph again.
Puzzle level 3)
Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.
The sysvinit recommended way is to call things like "log_failure_msg" and "log_warning_msg", and those only print to stdout, which is the console. That's it. Gentoo's OpenRC does claim to have boot console logger, but it was broken every time I tried to use it.
Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.
> you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?
I mean, yes? Granted I'm not a developer but printing out exactly what's happening line by line with all variables expanded and all logical conditions printed is awesome, it's something we try to reverse engineer into programs with tools like bpftrace and strace.
Conversely an opaque C application which fails has much less tooling in the way of debugging failures. Especially if your package maintainers have stripped symbols.
> Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.
Actually, that's the opposite. A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.
> Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.
Console scrollback is "nice to have", personally I'm very thankful that it does not use memory keeping things there because there is the odd regression when you configure something (like when you tell IPtables to log deny's: that also gets printed to console.)
re----gardless. I think we're talking passed each other. I had no real issue as a sysadmin with a fully fledged bash, because I can debug it. I get that there's issues with that and most people do not like it- I'm not advocating for it, just saying that what you presume to be "impossible" was somehow solved for decades, and quite competently. It may not be pleasant to you but the alternative is not pleasant for me.
The larger issue I have is that "the alternative" is embedded and entangled itself to everything- there can never be anything better produced.
With old init's, it was a small process with well defined functionality so replacing it was easy. (as evidenced by the many init's among unices.).
Anyway, I'm getting ahead of myself.
If you want to see what systemd could have been, I invite you to look at Solaris' SMF. Please ignore the XML-ness of it, it's "very old" in comparison to systemd. :)
Are you perhaps confusing something?
> A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.
The daemons never print to dmesg -- they may print to syslog, but not all do. The only daemon that I know of that prints to dmesg is systemd :)
To be specific, let's do a simple example. Here is an actual sysvinit file, /etc/init.d/hwclock.sh . It contains:
This version has been around since at least 1998 [0]. Old school enough for you?if /sbin/hwclock --rtc=/dev/$HCTOSYS_DEVICE --hctosys $HWCLOCKPARS $BADYEAR; then # Announce the local time. verbose_log_action_msg "System Clock set to: `date $UTC`" else log_warning_msg "Unable to set System Clock to: `date $UTC`" fiSo here is a simple question: let's say my hwclock chip/driver is unreliable and it did not work on startup for some reason, so that "hwclock" call is failing. My kernel timezone is not set right, the clock is off, and the system does log_warning_msg to warn you about it. How do I find out about this warning?
To make your job simpler, here is how the how this function is defined on my system:
Note no dmesg here, no syslog, no other loggers. You can say "bootlog", but it has been broken every time I tried to use it.if log_use_fancy_output; then YELLOW=`$TPUT setaf 3` NORMAL=`$TPUT op` echo " $YELLOW*$NORMAL $@" || true else echo " * $@" || true fiNow, nothing is impossible! I can hack this script so it writes to a temporary file. I can re-execute it again, and hope it fails again. But this gets old, you know? I can spend all day debugging early boot shell, but I did it for a long time already, so now I just want to type "systemctl show --failed", see all the failed services, fix it quickly, and move on to more interesting stuff.
[0] https://www.mail-archive.com/debian-bugs-dist@lists.debian.o...
log_warning_msg is a function which is wrapping 'echo'.
if you 'echo' it will go to the console.
Your distro can/does log everything that goes to the boot console.
The function 'log_warning_msg' is provided by the sourced bash file: /lib/lsb/init-funtions ; you can find a copy here: http://www.linuxfromscratch.org/lfs/view/7.0/scripts/apds02....
It's just doing echo.
If you add "echo" it will print. I promise.
And anything printed goes to whatever your distro is using to log tty0 console output. (usually /var/log/dmesg or /var/log/messages)
Which distro logs tty0 console output from echo ? Debian does not. Fedora does not. Gentoo used to, but it broke.
Both /var/log/dmesg and /var/log/messages log printk output from inside the kernel. They do not log echo output from userspace.
Are you sure you are an admin?
> Which distro logs tty0 console output from echo ? Debian does not. Fedora does not. Gentoo used to, but it broke.
None, anymore, that's a little bit the point.
Devuan does, so does Debian 7 and RHEL6.
I just tested it on all three in a VM.
yes, I'm sure I'm an admin.
udev was the biggest step towards "dynamic systems". Or should i say "hotplug" was, since it is more or less the same thing. I completely agree that the init should be an event based system. I'm not arguing that. I never argued that. I did not in fact argue much other then the statement that nobody should say their opinion about anything.
You say i left out stuff. I did. A lot more stuff then you wrote here. It would be a looooooong list of things, all of them simple (including "networking" and encrypted filesystems). I mentioned udev as an example of something slightly more complex, but even it is not.
> .. much better model than SysV init ..
You talk as a user, as in an admin. I talk as someone who was once interested in how all these stuff actually work. What i played with (read about, coded, and researched in general) includes init, udev, network hardware management (how ip and iw actually do stuff, excluding wpa), and more. What have i learned ? That almost all of it is done by the kernel. SystemD is not the only.. program that responds to events and goes over a graph and such. In fact, it is not even that hard to do what it does. Remember how i asked why udev doesn't have a generic event.. protocol ? You can scrap udev if you don't use systemD. And why ? For better integration ? The kernel tells udev when a device appears. So why doesn't udev just send a message of some sort ? It could even have a "mode" where it doesn't do anything unless it receives a message telling it to. So why not ? Oh, i don't think it's malice.
> The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
The debian debate was a clusterfuck of grand proportions. There was one (1!) post on the whole debate that addressed the technical aspects. In the end the conclusion was that it is easier to just use systemD as stuff depends on it more and more (GNOME, notably, funny). RH is a huge company, they are not altruistic in their goals. I wouldn't say malice, but i wouldn't say innocence either.
And again, nothing of substance was said. Just fancy words and feelings. Brainwashing..
PS Fun fact: Xorg depends on udev to send it events over dbus when you plug something related to it. It still works without it, until you re-plug your mouse.
Wasn’t the debian vote stacked by a bunch of Canonical employees that voted on party lines, since Ubuntu had already decided on systemd?
I thought it was exactly the opposite, that Canonical employees were fighting for Upstart and Ubuntu only switched to systemd after losing the war in Debian-land.
Weren't Debian maintainers affected by the fact that it was... problematic to run Gnome without systemd? (Which might be entirety not systemd's fault.)
Correct. By the time the question of default init system was being decided by the Technical Committee, the tentacles of systemd were so entrenched that its relative merits or detriments as an init system were of little importance. As Russ Allbery (a committee member) pointed out at the time:
"We’re not really talking about choosing between two competing ecosystems. Rather, we’re talking about whether or not to swap out a core component of an existing integrated ecosystem with a component that we like better."
Also (to try holding back the tide of historical revisionism) let me highlight that Debian's decision was in no way unanimous and the process was not without its own controversy. In reality, the Tech Committee was divided down the middle when it came to their vote on the default init system, and the tie was only decided by the double vote of Bdale Garbee, the chairman of the committee.
GNOME didn't have any significant systemd integration until well after Debian had decided to go ahead with it.
I remember this being a strong supporter, among others: https://lists.debian.org/debian-ctte/2014/01/msg00287.html
Then why did Josselin Mouette, a GNOME packager for Debian, write this in October 2013:
"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."
> Again, I’m not saying that systemd is perfect but that most of the complaints tend not to acknowledge the many problems it solved
It solved nothing but the problems made up to justify its existence. And the ones those faux solutions created downstream. Boot ordering problem? Feh. What a gaslight fest. All caused by chasing 1.2s faster boots. What a colossal waste of everyone's time. This is the same cancer that Shuttleworth tried to foist on everyone with that UI abomination. Servers are desktops are laptops are tablets are phones!
> Funny thing.. most of those on teh internets who are vehement in favor of systemD don't know much about how a computer system boots and what goes into that.
Why can't you have a discussion without childish insults?
As a FreeBSD and Mercurial user, I get more than a little schadenfreude when Linux users complain about something being foisted on them by someone with a more dominant mindshare or marketshare.
Linux users spoke and set the system up to do exactly what it does--use a dominant position to drive out alternatives. They only complain when they suddenly wind up on the receiving end.
Perhaps if you don't like getting the pointy end of the stick, maybe you shouldn't sharpen it in the first place?
> It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work.
I've read it more as a quirky history writup with some technical and idiology discussions.
FWIW I still think the technical criticism in the article has a lot of merit and what seems "turgid" to you is a nice link collection to learn about systemd minutiae...
Syntax wars are harder than it might seem at first. If you're not going to invest that amount of time, I don't see how it’s reasonable to expect an equal voting share with those who do.
Sometimes vocal minority is just that.
> how open-source projects work
they work differently depending on the project.
so this statement:
> requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators.
is only true in some cases.
c.f. rcng, which was developed by a very small few people, and is a generally sane improvement on previous BSD init that i've really never heard any complaints against vs the previous method.
same could have been true here. it was not - and the need for 'consensus' was largely sidestepped by the 'market pressure' of gnome being dependent on systemd features, and the leading core desktop environment that people had already attached themselves to, esp in the enterprise market.
Debian voted on the issue, and there was a very lengthy discussion. I don't think the Gnome dependency was the linchpin, at all. Debian developers wanted to get rid of thousands of silent but deadly shell scripts shipped in every package doing service management (rc scripts), they wanted nice cgroup integration, and that's it. At least this was my impression.
>systemd still remains poorly understood and understudied from both a technical and social level despite paradoxically having disproportionate levels of attention focused on it.
This statement makes no sense. It is well understood and has been studied and critiqued by many, including me. Also, I don't know what "social level" has to do with anything here. It was just created for the commercial aspect of Linux and forced into the relevant distro's by commercial interests... it's just that simple.
Pretty much. If it was just some guys project nobody would give a shit until it was actually well tested and proven.
But it was made by guy hired by Red Hat so instead of maturing and having to take actual user feedback before being used in anything significant it was pushed by a guy that vision is limited to "It works fine on my laptop".
Who then decided to reinvent everything along the way like log storage that still takes over hundred of file opens (and tens of seconds) to get "the last 5 log lines of a command" ( https://github.com/systemd/systemd/issues/2460 ), because why the fuck use say SQLite when you can have fun evening inventing your own shitty binary journal format.
I do find a lot of stuff behind systemd useful but the ever-present half-assness of its solutions is annoying. Half-assed logging here, half-assed dhcp client there, quarter-assed ntp client elsewhere, all while basic functionality is still riddled with edge cases.
Still a net positive over having to fix yet another fucked up init script...
This has not been my experience at all. The systemd project seems to take extreme care to adhere to the spec when implementing features. They don’t adhere to “this is how it used to work” which is where the bulk of the issue seems to be. The big controversy was when systemd-resolved implemented the spec to the letter and broke people’s assumption that listing DNS servers in order means they’ll be queried in that order.
> The systemd project seems to take extreme care to adhere to the spec when implementing features. They don’t adhere to “this is how it used to work” which is where the bulk of the issue seems to be.
Really, the issue here is that "how it used to work" is mostly unambiguous, but I have no clue what you mean by "the spec", and I suspect you're actually referring to a multitude of specs, several of which were only created in the process of writing the relevant systemd components.
> you can have fun evening inventing your own shitty binary journal format.
I thought the reasoning was to make logs harder (or impossible) to tamper with.
Something something about trading freedom for security.
> https://lwn.net/Articles/512895/
the binary logs handled by the systemd journal can be "sealed" at regular time intervals. That seal is a cryptographic operation on the log data such that any tampering prior to the seal can be detected.
This might be news to some people.
I think on the social level there's a lot of people that go "urgh systemd" because that's the popular opinion but overestimate their actual technical knowledge to comment on the matter.
I remember reading some advice about not doing rm -rf / on UEFI systems and looking that up, it looked like some MSI laptops would brick if certain UEFI variables were wiped.
I found an issue in systemd's issue tracker. And from what I could tell the ability to write to these variables was required to perform certain functions like rebooting into the UEFI system.
But it looked like that some forum or another had linked to the issue. And plenty of people were up in arms about it without anything substantial to say.
This is a hard point to refute because it's basically "some unknown number of people who don't like systemd are possibly technically illiterate" which is always going to be true but to an unknown extent.
If we're going on technical credentials, I have a few minor ones. I'm not writing dozens and dozens of lines of C every day, and I'm definitely not writing kernel components- but I've been doing low level technology stuff for 15 years now.
Everything from watching access patterns on drives and recombining data to writing/patching memory allocators and aggressively tuning system internals. I'm currently responsible for automating the deployment of the largest always online game my publisher has ever known. I'm relatively technical and highly regarded for my technical competence.
When it comes to systemd; you could argue that I'm "stuck in my ways"; but I've used systemd since fedora 21, I saw bugs you couldn't imagine, I read the code, I saw the hostility of the developers (maybe it's just defensiveness?) and I was a little worried, but ok, it's just fedora, it'll be better in RedHat when it releases... and while a lot of the lower hanging bugs went away for most use-cases; many remained in the edges. Instead of them being fixed though there's been a propensity to build a software eco-system and "system layer" to the OS.
My argument has never really been that "systemd totally sucks" it's that the conversation surrounding it is very hostile.. If you say that you don't like journald (because it's very easy to corrupt and has access drawbacks, encoding issues, tooling is sub-par etc) then people shout at you: "YOU CAN USE OTHERS", but you can never turn journald off.. it's wasted IOPs, IOPs are not free.
The biggest issues I had with systemd is the fact that (unlike Solaris' SMF, which is great btw) it will hide failure from you, and failure can happen relatively randomly because nothing is deterministic. This is not a holdover from being multi-threaded, there is actually a dependency graph that gets generated. But I've never been able to access that. (Maybe that's changed?)
My argument has always been, instead: "systemd's tight coupling of so many components means that we will never be able to have an alternative or successor to systemd". Because re-implementing single components is next to impossible and the sheer scope of the "system layer" is many times more lines of code than the kernel itself.
Anyway the conversation is steeped in some level of tribalism.. Which I truly hate.
You can't talk about systemd without being "pro" or "against" and then people take extreme stances, like "systemd saved my baby" and "sysvinit was trash" (and on my side: "unix philosophy!", and "systemd ate my utility!")
Well, my point is that (for instance) the specifics of its dependency model are still fairly obscure and undocumented knowledge. So I would say the statement is quite justified. And it also has a fascinating history that often gets reduced to culture war-related soundbites, which too deserved to be told.
I'd say software doesn't always win out on it's merits, there's almost definitely some social aspects regarding what does and does not get adopted. I do agree the commercial interests probably played the largest role, though. I think beyond the initial knee-jerk one might have towards this it's pretty comprehensive and there's interesting points here that are at least worth exploring.
Aren’t those social aspects part of their merits? A big part of why systemd won widespread adoption was by stepping up – the number of people working on the others wasn’t enough to be more competitive. I lost track of the number of Upstart bugs we avoided by switching to systemd, and that had the backing of one of the most popular Linux distributions.
> Aren’t those social aspects part of their merits? A big part of why systemd won widespread adoption was by stepping up – the number of people working on the others wasn’t enough to be more competitive.
They had the people because they had the money. The debate about systemd is only partly technical; it's mostly a culture war about how Red Hat is hostilely taking over the free software ecosystem, and how that invokes memories of embrace/extend/extinguish.
I suppose so ya, can't sustain without a community or company unless you want to do all the work yourself.
Another factor I was thinking is that almost everything in systemd is there because someone saw a need, even if it’s a niche – things like mounts aren’t a big deal for many people but the people who had something which wasn’t well served before wanted something better in a new system. Trying to replace a core component will flush out a lot of those smaller communities.
> It is well understood and has been studied and critiqued by many, including me.
It seems a lot of people simply don’t want to understand systemd, and go out of their way to avoid learning anything about it.
Could that be considered a social factor?
Systemd is an amazingly clear example of the second-system effect, as described in The Mythical Man Month. I fully expect it to be replaced down the line by something dramatically simpler and more intuitive, but that might take some time. Nonetheless, it does seem to solve some real problems with the earlier, rc-scripts approach.
It's dramatically simpler and more intuitive than what it replaced, so I'm super excited when someone comes up with something even better!
> It's dramatically simpler and more intuitive than what it replaced
I hope you're being deeply ironic.
It's several orders of magnitude more complex. It's more difficult to understand and reason about because its internal state is a black box, and the number of interfaces and file formats to understand is again orders of magnitude more complex.
sysvinit had a socket interface with a single message type. It had a single and very simple configuration file (inittab). Everything else was delegated to higher levels. Simple, flexible, and extensible. But above all, understandable in its entirety, and completely predictable.
> It's several orders of magnitude more complex. It's more difficult to understand and reason about because its internal state is a black box, and the number of interfaces and file formats to understand is again orders of magnitude more complex.
As a user I 100% disagree. Before systemd I had to learn a few dozen of different config formats for various things * various distros, now all my systems are configured in a reliable & simple way, for init, network, timers, etc etc
You're not disagreeing with anything I said. You're making a completely different point.
systemd might be more convenient from the point of view of an end user. And unit files might seem superficially simpler than a script.
But systemd is not simpler. It provides significantly more complexity, while also providing a lot of features.
systemd is also not more reliable. The previous system, even with LSB dependencies, was usually run in a linear mode with scripts run in a strict sequence. System boot and service startup was deterministic.
If you read the part of the linked article about the sheer quantity and complexity of the internal state managed by systemd, and how it is not deterministic, or even fully understandable even by its authors due to the combinatorial complexity of all the options and how the same settings work differently in different contexts, I'm afraid that I can't do anything further to convince you about the significant design problems it has which directly impact its reliability. Never had it hang irrecoverably at boot, for no discernible reason?
There is a reason people such as myself are appalled by its design. It doesn't even have a comprehensive specification for such a critical piece of functionality. It's defined solely by a single implementation.
The systemd state is a massive hairball. It's effectively a black box, and understanding why the system exhibits certain behaviours is difficult.
The sysvinit state consists of a current runlevel and the state associated with inittab tasks, which is pretty trivial. It doesn't attempt to maintain a mirror image of the state of the whole system, which is largely a constructed fiction.
Systemd is not only more convenient, it's also more simple.
Proof? sysvinit's surface is anything you can call. It's internal state is everything on the system.
Systemd's surface is the (well documented) commands in the systemd unit files. Each unit is pretty much independent, the dependencies are specified and well defined. Their internal state can be examined and explored.
It's interesting how many people have such wildly different experiences with systemd.
For me systemd has hidden some really horrible things behind it's dependency graph (which is a black box) and socket activation issues (for instance, cockpit "listens" on a port, but it's really systemd's socket activation (PID1) and there was a RCE against it).
The configuration file format is, in my mind, mysterious, with random keys in the unit file which have random meaning, and behaviour that is anything but deterministic.
But I've been using systemd for a long time (since Fedora 21). Maybe we're all coloured by not only when we first encountered it but by our distro and how simple they make things? FWIW I'm using Debian and Arch these days and still struggle to "enjoy" systemd, but it definitely functions better on my arch machine.
> For me systemd has hidden some really horrible things behind it's dependency graph (which is a black box) and socket activation issues (for instance, cockpit "listens" on a port, but it's really systemd's socket activation (PID1) and there was a RCE against it).
I don't understand how this is an horrible thing. Thanks to it you can do stuff as
and have an interactive representation of your boot flow graph, where you can click on any node to highlight any other dependent node. This is super nice ! systemd-analyze critical-chain was also very good to try to find out which service was keeping my NAS stuck for 5 minutes on boot.systemd-analyze dot > /tmp/foo.dot ; xdot /tmp/foo.dot
To give you my experience I was never able to do anything with older systems without reading tutorials on the internet while with systemd I'm able to solve my problems with man systemd-whatever most of the time, e.g. just look at `man systemd.service` or `man systemd.network` with actual configuration examples for common use casesThe configuration file format is, in my mind, mysterious, with random keys in the unit file which have random meaning, and behaviour that is anything but deterministic.
> Everything else was delegated to higher levels
Read: A confusing mess of different daemons, scripts, and configuration files, which neither worked together elegantly, were simple to configure, nor were easy to debug. Before systemd I would avoid writing custom daemon configurations at all costs because when I did I would need to learn how this version of this distro did it, fiddle with the byzantine mess of scripts, and inevitibly debug multiple subtle issues, now it's basically trivial to get a new daemon running, and I extremely rarely need to debug issues (And I have had to track down systemd bugs due to obscure use cases like one systemd unit modifying other systemd unit configs at the same time another systemd unit with multiple execstart lines is running).
Systemd has many pluses, but simplicity and intuitiveness are most certainly not among them.
Simple... for what?
I’d like to setup a service, which depends on another service, and which must always be running, and if it goes down, it needs to have all forked processes killed, must be restarted, and it needs to run as a specific user.
With systemd? 5 lines or so of boilerplate, independent of distro.
I don’t know about you, but I call that simple.
Technically (and I think this is what this thread's misunderstanding is coming from), this is not systemd being simple. It's systemd being easy.
Simplicity is an intrinsic property of a product, describing that it solves a problem without needing many parts. Easiness is a property of the product's user experience, describing that it can be used without much training.
Systemd reflects the underlying system it is meant to support: the Linux kernel. And the Linux kernel is neither easy nor complex.
That Systemd can make the mania of Linux knobs usable is, IMHO, a huge achievement.
I've run across a couple of cases where the "depends on" part seems to intermittently fail. That is, the implementation seems to end up with a race. That presumably means that the dependencies were specified incorrectly, but getting it right seems to be quite difficult. Or, conceivably, it's a bug in systemd, but debugging that is a lot harder than debugging a couple of tiny shell scripts.
I'm not necessarily a "remainer" on the subject of systemd, but I think it's not correct to sell it as a huge win on simplicity. Maybe it is simple when everything goes right. When it doesn't, though, systemd is fairly nasty to deal with.
I think this subthread is talking about the complexity of a system's implementation, not that of its user interface.
As a user, systemd wins for me hands down, because it is simple to use. That it masks complexity away from me is a feature, not a bug. This is also why distros is implementing it almost everywhere: It makes their job easier.
Now if you are measuring the complexity of the system's (full) implementation, you can't really compare systemd to to SysV-init. You need to compare a systemd-based system to a SysV-init based system.
And then you need to account for the 100s of inconsistently written shell-scripts, all the code distributed in the mess of third party dependencies (sh, bash, Perl, Awk, Python, su, supervisord, cron, etc) to ensure that the functionality of the systems you are comparing is really somewhat equal, not to mention the added complexity of the glue between all those components, and how it may fail.
And when you do that comparison, I do believe you will end up concluding that 1. full systems are somewhat complex, for both systems, and 2. that systemd-based systems are not fundamentally more complex, and 3. systemd-based systems have the benefit of a single, centralized implementation for all these core functions, so that developers don't have to reimplement them inconsistently, and possibly buggy.
With systemd, for better or worse, either your entire init-chain is broken or it isn't, so when it works, you know it works. With SysV-init you have no such guarantees.
Downvoted for truth, I guess?
The implementation of a correct version of the above in sysvinit is no less complex, especially in its emergent behaviour.
... which is as I recall, one of the main differences between the New Jersey Style and the MIT Style (per Gabriel).
Depending how you look? Here is a great example: disable the service before installing -- i.e. make sure that even if someone installs the service, it won't get activated.
On systemd, it is "systemctl mask NAME". On sysvinit? Well, there maybe a file in /etc/default/ -- but each package has its own option name... and some of them cannot be disabled at all.
i hope you are correct.
it is so easy to deprecate someone else's skills and knowledge with massive change like this. would it be that hard to respect some of the users?
Can I just say, from a non-sysadmin perspective, how happy am I that these days I don't have to install something like Supervisord and just use the systemd + journald toolchain. Setup script is just one cp and few systemctl commands (start/stop/enable).
Yeah, that's nice.
I think there's a lot of people who get caught up in the merits of systemd and assume that there's nothing better. Or that the alternative is the old sysvinit which was in dire need of being replaced- but that's not true.
Process supervision is totally feasible in init (just look at runit, Solaris' SMF and MacOS's launchd) even Windows has one.
A key quote:
> As it turns out, there was a little-known Freedesktop-affiliated project called xdg-hostname in 2009 which was a start towards creating such D-Bus “utility daemons” for use in desktop widgets and others, all as a standalone project. Had this been followed through instead of having the utility daemons end up being part of the systemd source tree, a good deal of political acrimony could have been avoided – though at the cost of reducing systemd’s leverage.
That is, if they used xdg-hostname instead, SystemD wouldn't be tightly coupled with Gnome, so users could choose between SystemD and InitV, and all this mess could have been avoided.
God I wish more people used VoidLinux and packaged more and more stuff without needing Systemd. Systemd may be good for some, bad for some. But, it certainly should not be the only way to get things done. We need alternatives.
It’s as if web developers were told now that there is IE, that is prevalent, we can write web apps that only work there ;-)
One thing I’m surprised about in the init wars of past decade(s?) is that in almost half a century of Unix nobody got up and said “why don’t we, instead of bickering about implementation details, come up with single format to declare desired system startup behavior, and then users can pick whichever implementation they like to read and execute that format”?
Can’t be impossible to distill some basics about service startup configuration and dependencies, can it?
Then make it extensible and cry about differences in extension support like we do about per-browser css options, but at least converge on some basics.
If you don't like something, put your energy into implementing it the way you do. With any luck, the ensuing distraction will be more effective at weakening the original than simply complaining about it.
Sadly, systemd has corporate backing, which is a very hard fight to win for a ragtag bunch of nerds with dayjobs who have the completely unreasonable expectation of not having their shit broken, regardless of how much energy we have. Money and full-time devs beats energy every day of the week.
My point was that there is nothing (no competing implementation) to be judged against, so it rather wins de facto.
This is like declaring McDonalds burgers the best because small burger joints can't offer Burgers _and also_ ice creams, salads, chicken and toys for children.
If we're talking about init; there are actually better inits already available but you'll never hear about them (runit, for instance) precisely because you don't just leave systemd.
You leave the systemd ecosystem.
This is the main point of the article: systemd as a "job system" is actually quite bad, but the ecosystem has some value in some places and you have to buy the shit with the pig.
I don't fully understand systemd, but the technical critique looks interesting and seems to indicate systemd is poorly designed (but I don't understand it all). Anyone else commenting on that?
Enjoy https://blog.darknedgy.net/technology/2015/10/11/0/ , by the same author.
10 years ago I never had shutdowns hang with "Stop job waiting for PID 10834: 30 seconds... 90 seconds..."
I do now. Regressions are a bad thing.
vezzy-fnord! I have always enjoyed your commentary and perspective of operating system, it's nice to see you back again!
> it's nice to see you back again!
Not for long, probably. I drifted out of this sphere years ago, and came back specifically for systemd's 10 year anniversary, as I felt obligated to at least do that.
Judging by the tone of the comments, 10 years later is still too soon to discuss systemd dispassionately, but when I come back in 10 more years I'll see if things have changed.
Would you like to tell me what OS are you currently using? Are you still keeping pace on OS research?
Honest question: Which relevant technology is ever discussed dispassionately?
If we can't ever perfectly adhere to an ideal, then I guess we ought to go full on with the connivery and deceit, then.
Regardless, I did have at least some expectation that people would actually focus on the original post and not brawl in the comment section over tangents that often aren't even part of it.
Oh, well. And hey, if other people down the road read this and it makes something click for them, I'll be happy with that.
Thank you for the article! It's too bad that your conclusion, that systemd doesn't/shouldn't matter, wasn't able to be grasped by the community.
I do really appreciate this type of long form, researched blog post. I have no strong opinions on systemd, I've made simple service units and it has worked well enough. Having the history behind the design is a great aid to understanding.
My worst gripe with systemd is that I cannot find reliable docs for older versions.
Your manpages should match the installed version.
Right. All we need now is systemd-selfupdated, so that it constantly keeps itself up to date, even through egress filtering firewalls.
My biggest gripe with it is that its name is not ‘sysd.’
The author sounds distressingly prescient with their prediction that BPF will be the next venue for this sort of farce.
Oh good, the monthly HN systemd 5 minute hate is upon us once again.
It's worth considering why the topic makes such a regular appearance. I suggest it might be because that so many users of this website find it to be a remarkable example of not only how something so widely unpopular can become so widely established, but also being something that they have to contend with frequently.
Is it actually unpopular or is there just a vocal minority constantly complaining about it while a much larger part of the user base is happily using it?
Mostly just unpopular here from what I've seen in industry as a programmer with lots of sysadmin, ops, and infra engineering experience.
The usual arguments are it is not "Unixy" and then fall back on original designer intent and philosophical attacks, or worse ad hominems against Red Hat or Linus Pottering.
It's unpopular. It has funding. No other project in the space has funding, so it wins by default.
Exactly. This is the measure of the situation. Well put.
Sounds like a backdoor to me.
It would appear from distribution adoption that systemd is a lot more popular than it is unpopular.
One of the reasons distributions appreciate systemd because systemd unit files are easy to write and can easily be written and maintained upstream and used with very few if any modifications downstream.
Your average HN user doesn't see or particularly care that it makes things easier for distro maintainers, so it's much easier to push the corporate conspiracy angle than to accept that a lot of the hand-wringing takes huge amounts not-very-fun volunteer labor for granted.
This is one thing which is often claimed, but to my mind is poorly justified.
As a (Debian) package maintainer for over a decade, this used to be a non-issue. They were write and forget. The average package was just some simple boilerplate.
I love not having to maintain half a dozen init scripts for three distributions but just one unit file for a well defined service manager.
As a Slackware and OpenBSD user I've really never understood this complaint. The init scripts they ship with are neither difficult to understand nor hard to read, and new ones are very easy to write.
Maybe the LSB stuff was awful; I never used it, and for that matter I can't think of a reason I would want to. It's the old FreeDesktop.org issue: it's a probably-interesting solution to a problem I just don't have.
Red Hat made systemd a hard dependency for gnome, forcing all distros that have no manpower to patch gnome to use systemd.
Seems to me there are several options:
1) run the last version of gnome that works without systemd
2) run a patched later version of gnome from another distro that works without systemd
3) don't use gnome, use a different desktop environment
4) fork/patch gnome and remove the dependency yourself
systemd can't take any of those away from you.
If the people actually working on gnome want to add another dependency to gnome, that's their business, is it not? It doesn't break the existing, pre-systemd versions of gnome.
Please read the article before commenting.
> Indeed, one ought to observe that the most important consumers of an init system are not sysadmins and ops people as commonly believed, but rather distro maintainers. It is their laziness that determines the balance.
I had. :(
Users choose their distribution, ultimately. I don’t agree with the author’s conclusion there.
That's an oversimplification.
The article goes into detail about why there was a desire from some distro maintainers to switch to something else. (clean slate).
The reason I accused you of not reading the article is because it's very clear why systemd was adopted.
1) RH Made it, that doesn't automatically make it good, but there was a lot of posturing from the RH devs to get more things tied in to it under the guise of "all distros will use us eventually, they're already in the process of adopting us!"
2) There was a strong desire to get off of sysvinit, many alternatives had come and gone, though most had the desire to be backwards compatible, none could get any real traction, mostly because it was hacks on hacks. Upstart was promising but people didn't use it properly.
3) GNOME added a hard dependency. Small in isolation but then.
4) UDEV was made systemd-only. UDEV being something that underpins a lot of software. A fork was made (eudev) but that was only the beginning because krdb was next to be systemd only.
5) when it came time for Debian to talk about systemd as the last major upstream distribution (IE: one that sysadmins actually deploy and one that other distros are essentially variants of): it was not "systemd or alternatives" it was "how much systemd do we have to take to be able to support most software."
6) When your distro can't take a major desktop environment, that's a lot of points off of your distro.
--
People choose their distro, but their distro often chooses a distro.
If you're a business building your company, you don't trust Gentoo- you run it on RHEL, Debian or potentially Ubuntu these days.
Oh God no. Alpine is the most-used Linux distro out there (containers count, remember) and it's mostly used specifically to avoid the opaque mass of spaghetti unit files that is systemd service management.
Interesting comparisons of a GNOME volunteer to Stalin. Also found it very interesting that the author put inclusivity in quotes.
Oh, but it is a good comparison.
Many people have compared free software to communism. I always used to dismiss it. But, in a way, they were right. Both are movements with a self-conceited historicist endeavor to eliminate exploiting classes ('it is inevitable that the tendency for the rate of profit to fall to produce immiseration as capitalists can no longer profitably invest, leading to a proletarian seizure of power' versus 'it is inevitable that proprietary walled garden development models stagnate as they're overtaken by decentralized communities pooling their knowledge and labor into the most optimal end' and create a perfectly equitable free association of producers no longer having their surplus value be seized by exploiters.
One degenerates into a repressive bureaucratic collectivist (I'm not a Trotskyist, but it's a good term) mechanism for reproducing surplus through either direct requisition or coercive planning. The other degenerates into a way for large-scale cloud providers to appropriate the free labor of volunteers so as to reinvest it back in their walled gardens.
Stalin represented the Leninist wing against ultra-leftists, anarchists and other factions who were against tight top-down organization in favor of 'spontaneous' organizing, and the GNOME/Freedesktop/Red Hat nexus is against the faction in free software that insists on a loosely coordinated and loosely coupled bazaar without a central vision. But in no sense has systemd abolished this, it has only reduced a few nodes at best.
EDIT: Actually I should point out that 'free software' per se is a deontological argument, and it's 'open source' specifically which specifically uses these historicist arguments to justify itself. But then open source was unabashedly an effort for corporate sanitization of free software, which most people forgot after it mostly crowded out 'free software.'
TLDR: down with sYstemd :]
The dependency problems he's describing sound like they could be relatively easily fixed with some new systemd keywords?
Of course it would take some time to migrate existing unit files.
It doesn't sound like a fundamental critique. Would be nice to turn this into a constructive proposal to fix systemd.
It feels like a lot of the existing keywords were added in order to paper over previously-discovered dependency issues. And the result has been a) confusion, and b) more dependency issues. I don't think adding more is going to help the matter.
PulseAudio is the only program I've seen really stress my Ryzen. I recommend fixing the default sampling config if you have similar issues.
This is a really entertaining writeup, given the subject matter.
I wish systemd haters would make a public petition to remove systemd from this world. I would absolutely use such a list to make sure I never accidentally hire any one of them for any system administration positions.
As a person who does not work as a system administrator professionally but does use self administered linux machines as workstations systemd has not brought any new productivity gains but it sure has increased the learning curve.
As a person for whom Linux and BSDs have been a hobby since 1996, I have no idea what you're talking about.
One small example that i've struggled with is logging. It used to be I could reuse my editor knowledge to search and read log files. Now every time I want to look at a log I have to re-read the man pages to figure out the proper incantation.
Learning new tricks gets harder when we get older. Is that the software’s fault?
You could make several arguments here, but they're all small until you aggregate them.
1) Breaking working systems should be dissuaded, especially when it comes to human interaction. There's a reason that Microsoft still has a start menu and "File/Edit" menu's
2) There is a certain level of arrogance to the notion that a developer of core infrastructure for literally millions of users can just change things and expect everyone to fall in line without giving credible reason for people to do that.
3) Adding an additional tool to everyones cognitive load should be done with care and delicacy, as it stood there was one set of tools for _all_ file accesses, and there is now a new one.
PS; on a personal note, I'm glad I do not work with you, I find your attitude in the grandparent comment deplorable.
Yeah, you’re right. I was exaggerating for some reason I don’t recall anymore, a bad day perhaps, too much or too little coffee or something. Who knows.
Of course I don’t discriminate at hiring time against people who disagree with me on such a trivial technical detail. Or hunt for them so I can maintain secret lists of undesirables. That would be unethical, probably illegal, and definitely stupid.
A hobbyist making hiring decisions and maintaining a blacklist of professionals who bitch about a change they had no control over? You're not helping.