Systemd through the eyes of a musl distribution maintainer
catfox.lifeI use much functionality in systemd that was not present in SysV init, and I really appreciate it. It has never crashed any OS that I have run.
However, there are a few aspects of it that are inconvenient.
Automount units use an unintuitive naming scheme, and you are not free to name them as you wish (as you might for a socket unit). If you are mounting an NFS volume immediately below the root directory, you don't see the problem, but if the mount is several directories deep and/or uses ASCII symbols (non-alphanumeric), it is not pretty.
Socket units require two files per port. When I am moving complex inetd.conf setups to Linux, it's far easier to implement them with busybox inetd than convert dozens/hundreds of services to unit files, despite the increased functionality. Somebody has probably written some scripting to do this.
I am not aware of any include directive for my own directories, so I don't have to place everything in /etc/systemd/system. There probably is a way to do this, and I am betraying my ignorance.
And my, things can get messy in a hurry in /etc/systemd/system.
I don't know how to configure users to be able to maintain their own (personal) units.
And lastly, it's so seductive that I have no idea how to do many things in other operating systems that I easily do in Linux. I wish this itself was not a walled garden (but I'm not leaving).
> I am not aware of any include directive for my own directories, so I don't have to place everything in /etc/systemd/system. There probably is a way to do this, and I am betraying my ignorance.
Systemd by default looks at a bunch of directories depending on the context.
$ systemd-analyze unit-paths (or --user / --global)
https://www.freedesktop.org/software/systemd/man/latest/syst...
> I don't know how to configure users to be able to maintain their own (personal) units.
This is something I use all the time! Just put unit files in ~/.config/systemd/user and use systemctl --user to start/enable them
Just a note that this requires the admin to enable linger if the units need to be started before the user logs in and run after they logout.
Nice article, after years of emotional flaming finally something fair and balanced about systemd.
The point about there existing no competition and no respect for glibc alternatives is very valid and to that I can only say; if you build it, they will come.
Systemd had the advantage of Red Hat backing so they got a great headstart. But in general people will use whatever works. So roll your sleeves up and get cracking.
People will use what works. But there is a limit. If you have two things, one of which does everything under the sun, and the other only does mostly everything, people tend to choose the first. The more people do this, the more "weight" the first carries. Until one day, it makes no sense at all to use anything but the first. Even if it causes you tons of headaches. The more different the two are, the less and less likely that you could or would ever switch. It's too big not to use. And so, building an alternative becomes a fool's errand. Nobody would use it, because there's just too many reasons why they shouldn't or can't.
(In case you don't believe me: it is now impossible to run a desktop without a half dozen Systemd shims, because the entire ecosystem is now hard coded to use Systemd. I know because my desktop is Alpine Linux, a musl distro)
> it is now impossible to run a desktop
I find that hard to believe, unless what you mean is "it is impossible to run a desktop with all of the features that I want, many of which are coincidentally easier to implement/maintain using systemd ecosystem features"
No kde?
KDE really, really needs logind - it can't really use ConsoleKit any more. Same with SDDM, though I think the CK2 backend is still writhing in agony somewhere in its codebase.
And KDE's touchpad integration relies on libinput, which relies on udevd. I think libinput might work with mdev if you really try hard enough?
I am not aware of KDE having dependencies on any other bits of systemd, at least Plasma 5 LTS doesn't.
I was under the impression Plasma 5 (supposedly even Plasma 6 Beta) runs at least "fine" on the BSDs, no?
They have ConsoleKit2 support patched back in.
libudev-devd - a stub implementing UDev APIs atop FreeBSD devd: https://www.freshports.org/devel/libudev-devd/
I always appreciated this presentation for being a reasonable and balanced take, if you're interested in other examples: https://youtu.be/o_AIw9bGogo
If someone built the vision in this article I would use it. I love declarative config.
The article is such a good take it makes me want to try out the author's Adélie project.
Red Hat backing SystemD was hardly an advantage beyond the fact that it had financial muscle.
So for example, Red Hat backing meant that Canonical would have to be pulled in kicking and screaming to use SystemD and Ubuntu is one of, if not the most influential distro in the Linux world.
Ubuntu created and used upstart until it basically lost out to SystemD and had to move over.
What I heard and I'm not sure how much truth is in it -- but it's fun gossip so enjoy -- is that upstart needed some work done, and Poettering went off for 6 months and came back with a replacement (instead of a fixed upstart). If anyone knows the full/real story of systemd origin I'd be curious
jrb explained the origin story to me once in passing at a gnome meetup in SF irl ages ago. But the only part I recall is it involved Lennart and Kay collaborating on a new init system on their own... without rh even initiating the project. I got the impression that this was a sort of rogue effort from these guys.
Folks direct a lot of systemd hate at Lennart, but just take a look at the journald header[0]: uint8_t signature[8]; /* "LPKSHHRH" */
== Lennart Poettering, Kay Siver, Harald Hoyer, Red Hat
I don't know the details but I heard at some point Kay and Lennart had a falling out within the project. By the time I got involved in journald development @ CoreOS, neither Kay nor Harald were visibly participating anymore... It was kind of annoying, as it left just Lennart to review any journald PRs, who was obviously busy, but eventually got around to it. I think it's worth noting that despite being the person who receives all the systemd hate, Lennart didn't promptly abandon maintenance of the project after getting installed everywhere.
[0] https://github.com/systemd/systemd/blob/v255/src/libsystemd/...
RHEL6 did use upstart, but when RH tried to submit patches/work with upstream like they do with most projects they ran into issues with Canonical's CLA and other submission policies. So when Poettering came back with systemd they went "what the hell" and switched to that.
Reading the section about resolved reminds me of musl having a "broken" stub resolver implementation for years (whenever the dns payload was larger than 512 bytes). "Broken" because the TCP fallback was supposedly intentionally not added for complexity reasons.
DNS and all the additions seem to be really not easy to handle.
Yes - that lower level networking stuff is just a minefield of edge cases which most people never need but are really annoying if you’re unlucky enough to hit them, and Linux has always had this problem where if something works for Windows the people who run those broken networks and many users will say it’s a Linux problem. A lot of the systemd decisions are probably explained by someone getting a ticket saying their Red Hat laptop takes much longer than Mac to get on the corporate network or was unable to deal with the captive portal on some hotel Wi-Fi, and they realized fixing it was going to require work across 5 different code bases.
I think this is the most likely explanation for why systemd includes everything but the kitchen sink. For consistency, and ease of maintenance, having it all under one umbrella is certainly a benefit (for the developers of systemd.)
Also, NIH is a hell of a drug.
Systemd's catalogue of kitchen sinks is getting to be truly impressive. Here's an NVMe-over-TCP server, for some reason: https://github.com/systemd/systemd/blob/main/src/storagetm/s...
This is a great example of an edge case which many people might never need but is really useful in a pinch: that’s what provides the equivalent of the Mac target disk mode, which is really handy if you need to repair a no longer-bootable system.
https://github.com/systemd/systemd/commit/1761066b135f1a322c...
That's really not a great argument for why it belongs in systemd.
Kitchen sinks are generally useful, but 17 kitchen sinks in a helicopter is not a good idea.
Where would you put it instead of something started by pid 1? It needs to boot the kernel and unlike macOS they have to run on a ton of different architectures so they can’t count on the firmware.
Huh? It's just another userspace program serving TCP.
Have you ever setup an iscsi shared device in linux? It’s quite complex and a bit messy.
If systemd can make something similar (i still get a block device from another host in the network) in a simpler manner that would be just awesome, truly a huge win for everybody.
Yeah musl enthusiasts strangely never acknowledge this enormous bugs that musl libc has, i wonder why.
I was a big skeptic earlier on, but I have come to appreciate systemd in recent years. That said, the section on resolved is a big one–it has been one of the only things that consistently causes me annoyance over the three years since I started primarily using Linux on the desktop.
After one particularly deep rabbit hole where I never actually solved a problem where local network mDNS lookups always took exactly 6 seconds, I wound up disabling it entirely in favor of unbound, which works great but revealed that a handful of other things seem to expect resolved to be present and won’t work properly without it.
Systemd is Godsent for people who have to administer diverse Linux boxes.
I can see how people who want to run their own tight ship on their machine would balk at it. I also sort of hate the systemd-resolv and the fact that it instantly rendered worthless 99% of online guides of how to unfuck your local DNS resolution.
I had no opinion on systemd until I found out I could configure the mount points in fstab to magically mount themselves whenever they were needed.
I have a lot of network shares and some of them caused real trouble when they failed for whatever reason.
Now I really appreciate it.
Various automount tools have been available since the 90s.
Indeed, why use Dropbox, when FTP existed for decades?
Except these automount tools provided exactly the same functionality that is currently provided by systemd. And did it better
I contest 'better'. What's the difference? It's mounted... no?
Allowing 'systemd' to be aware of mounts lets you create dependencies between your mounts and services.
For example: 'Requires=mnt-myfs.mount' (and 'After').
Use 'PartOf' if you want the service to get restarted during an unmount/re-mount.
This integration is mint! Anything not systemd [when it's already managing your services] trying to orchestrate is, by nature, second-rate
One may note the removal of the fstab generator. It's overblown. Any distribution including systemd vendors it.
You obviously haven't used any of them and I have a feeling that in fact you haven't used anything but systemd because your remarks about systemd ability to create dependencies between mounts and services as something amazing is just hilarious.
Sigh.
I was hoping to be a bit more productive. I haven't used them, obviously, I asked - 'what's the difference'.
They haven't even been named. Is this a troll?
You obviously miss my point. In this hypothetical I'm picturing a systemd system with services and mounts.
The recommendation appeared to be... have another service [or something] try to provide something that 'systemd', which is already present, can handle completely fine.
So, I repeat, what's the difference? What am I missing? What's better than having the thing managing your services also manage your mounts?
Between the lines I'm saying this isn't fancy work. I want these mounted and useful relationships and that's it. They serve a purpose; I need a compelling reason to replace init.
Agree, but tou had to
> Systemd is Godsent for people who have to administer diverse Linux boxes.
I disagree whole hearty. With 15 years of system administratorship, me and systemd don't get on.
Services timing out on start-up/shutdown, "waiting x/y -- x/y retries and hijacking resolv.conf are my common two.
> Services timing out on start-up, "waiting x/y -- x/y retries
What would you prefer happen instead? Having seen plenty of downtime caused by not having retries or hanging indefinitely weren’t exactly improvements in my experience.
Leave the service dead, or just kill -9 the process.
"service didn't start, oh well"
"service didn't stop, force killing it"
Is just fine.
It's my responsibility to ensure the service is operating correctly, not all-for-one systemd. Redhat bloatware dictating how Linux should be.
I am aware that I can change the timeout of retries, but that's documentation overhead.
> Is just fine.
It’s prolonged downtime, that’s what it is, and it may make the system unusable. I saw that several times during the transition period where a network outage or data center shutdown meant that some important daemons failed on startup (one of them had the concept of retrying a connection error, but not a DNS resolution failure) and the admins of the SysV boxes had to manually restart everything while the Upstart & systemd boxes recovered almost immediately. One fun case had the developer almost getting this right: they had implemented retries but without a time delay or back off so their init script maxed out its retry count in a second of “host not found” errors and then exited permanently.
When you leave things like retries, logging, or dropping permissions up to each daemon you end up with a hodgepodge of incomplete implementations and things like error handling are where corners tend to get cut the most since the situations are infrequent and often hard to simulate.
> It’s prolonged downtime, that’s what it is, and it may make the system unusable
IMHO, that's how I feel about systemd --- if a startup task is stalled, the system is non-interactive, unless something has changed since I last experienced this issue.
If some other operator access has started, you can use that, but the console is useless.
> It's my responsibility to ensure the service is operating correctly
Really, it's only on you to fix it if it does not work, which in my experience is far less likely with systemd.
> Really, it's only on you to fix it if it does not work, which in my experience is far less likely with systemd.
That's just lazy if your waiting for systemd to report you a failure. It's on you to maintain, ensure it's all operating correctly; including the daemon starting and stopping manually correctly.
Makes sense that on machines I've inherited that such checks are not done.
I've been running Arch continuously without a single complete reinstall for a decade now. I don't think it's fair to claim I wouldn't know how to maintain my systems correctly.
I'm not advertently claiming you cannot. But working with teams and systems where they wear the badge of "I'm a DevOp who can SysAdmin" and try to prove themselves in such as a manner throws myself a sigh.
If your unable to compile your own kernel, you don't pass in my books, it's a rite of passage.
> If your unable to compile your own kernel, you don't pass in my books, it's a rite of passage.
Pulling the PKGBUILD (or similar for other distros) and throwing in a couple of patches before `makepkg`-ing it is the easy part. Of course you could do it manually, but then what's even the point besides bragging rights. Very few people have a reason build a completely custom system LFS style, and those that do will.
I'm not talking about LFS.
The Linux kernel comes with thousands of different drivers for all kinds of devices which adds weight and such that even slows bootup times. For the home user, sure, default works best.
Why use a kernel with bluetooth modules, wifi modules applied when the actual server has none of that hardware? Not forgetting added security risks. It's all clutter and bloat that's not needed at all.
Do you really need to support a 1998 3Com IPX network card on your production server?
It's about understanding the system as a whole not just the userland or /etc/. Sysctls parameters and fine tuning the cogs that drive the system; the Linux IP stack is abystmal without tuning.
The default kernel works, but your not going to get near best performance without fine tuning and minimalizing the kernel. It's not a bragging excercise, it's a real excercise to ensure the system is secure and to ensure smooth operatation. You'll find these excerised in enterprises where such performance are a requirement.
Knowing how to compile your kernel is key piece of important knowledge to know and you'll find that someone with that knowledge will end up getting hired than someone without.
The saying is true: SysAdmins treat servers like pets; developers treat servers like cattle.
Sure you can dive into that rabbit hole and fine tune your kernel, but then you're on the hook for security patches when you could've just enabled livepatch on Ubuntu Server and been done with it.
Ubuntu live update probably works with your own kernel too as long you compile the live patching module within.
Your probably needing less of the kernel patches anyway because your not using the affected drivers.
What do you do when there's no outside internet access for "live patches" or where such services are restricted?
Say, banking?
Either way now we are getting in to pedantic territory. I marked my reasons, you do you.
At least let me cancel it from the console easily: control-c and continue. Waiting on retries that are never going to succeed is incredibly annoying.
Shouldn’t you need a password to log in? Seems bad to just let a some rando interrupt your boot process. It’s pretty easy to modify the timeouts that bother you.
The timeout issue is often a temporary issue after an upgrade. I need to get into the machine to debug and fix it, but I'm delayed waiting for the timeout.
If they have access to the physical console, they can already permanently interrupt your boot process with an alt-sysreq, ctrl-alt-delete, and likely several other methods (power switch, or pulling the plug.) I'd say being able to gracefully cancel a timeout/retry would be the least of your worries.
If "some rando" has access to the console keyboard while the system is being booted and some service is timing out... You've got a bigger problems than the fact they can cancel the start-up of that service.
I often run into these sorts of issues when upgrading Ubuntu. A couple times, with major upgrades, it decided to rename network interfaces, resulting in various systemd services taking minutes to time out and continue. It would be nice if I could at least control-c it from the console. Super annoying when you're debugging things and have to wait minutes between cycles.
> I prefer my init scripts to handcrafted by local artisans. Each time the computer boots up is an objet d’art.
In all seriousness this is a wonderful article. Has the author/op seen this talk[0] by a FreeBSD developer on systemd?
What I like about the talk is asking about what is going to happen when we need to move on to newer ways of doing things that will have advantages and disadvantages over the old way. We can see a similar story with X11 and Wayland. Systemd does do some things better than the old way but does have flaws.
I generally embrace systemd and have been pretty happy with it but there's one component which simply doesn't work correctly and that's systemd-resolved in combination with DNSSEC. I eventually had to replace it with Knot Resolver which works flawlessly on the same machine / network.
Maybe a good way to get competition is to give someone a possibly dubious (but fun!) goal of writing a systemd compatible alternative in rust.
Reimplementing such a massive piece of infrastructure is pretty daunting. I don't think many people who are not being paid to would embark on it and then keep up the momentum to actually cover all those edge cases and maintain a community around it.
Rewriting things in rust seems to be pretty motivating though, so maybe it could be a force for good here
There's a willingness from the systemd devs to start incorporating Rust into it, possibly quite soon: https://github.com/systemd/systemd/pull/19598
Ah, because while systemd is divisive, couldn't it be _more_ divisive?
This choice literally only affects contributors and packagers for distros. If you even notice it at all as a user, it should only be through a very slight reduction in crashes.
From 2021?
The original PR is from 2021, but there's been activity on it that's less than a month old indicating that it's still wanted, especially as the kernel starts to integrate Rust.
If you're just looking for init and service management, the S6 system (and its wrapper 66suite) are pretty good. It's the only alternative that seems to match Systemd for feeling like a modern tool in my opinion, rather than a relic of the old days.
Don't want s6 as it brings with it a binary configuration database (Windows registry). systemd at least has text files for config.
Sorry, where's the binary configuration database in S6? The config looked like a collection of directories with scripts to run to start things.
EDIT: Ah, you might mean https://skarnet.org/software/s6-rc/s6-rc-compile.html -- that indeed is getting to be weirdly complex.
My sole resistance to s6, binary config AND default-all privilege for spawned daemon.
Better to stick with INI v1.4 format.
Been watching that for a couple years now. At one point there were plans to integrate with alpine Linux, but it wasn’t ready yet. Has there been progress towards that?
Yes, there is some progress it seems: https://skarnet.com/projects/service-manager.html
I'm Just going to plug myself here:
https://github.com/KillingSpark/rustysd
Rustysd is a reimplantation of the core features of the systemd service manager with a deliberate focus on less features so it can work on more platforms (e.g. BSD)
It's been a while since I actively progressed on the project but it works and would probably be a good starting point for anyone wanting to do further work in this direction.
I've found systemd-nspawn to be a great alternative to the olde chroot method for fixing systems in a rescue shell. systemd-nspawn -D /mnt mounts the system like chroot but the real value comes with systemd-nspawn -b -D /mnt/ that actually boots the system in a container.
I use systemd-nspawn as a built-in alternative to LXC/LXD.
Thanks! I hadn't considered that use case.
I'm enjoying artix linux with dinit. I tried a couple of other alternatives - s6 and openrc - although I wouldn't say I was that fair on openrc.
dinit may lack many things - I don't know - I haven't had any reason to be unhappy with it and it seems to work in a fairly understandable way. I defined a new service (for minidlnad) and that appears to be straightforward.
More structured metadata in journald is nice and I don't have any qualms with binary logging formats, but the author does have a point on log shipping. Additionally, reading journal files has always seemed painfully slow to me (30 seconds of 100% CPU usage)
I'm not overly found of timer units, either. They seem feature rich but much more complicated to setup than a single line in a crontab.
On the other hand, real dependency management between daemons, mounts, and sockets is a huge win.
I can't quite remember the right terminology but afaik systemd-resolvd supports routing different domains to different DNS servers on different network interfaces which can be auto configured via DHCP. The practical implication being, you can connect to a split tunnel VPN and domains accessed over the VPN get routed to the VPN DNS server.
It’s called split DNS. Can be implemented using dnsmasq
It can be implemented with dnsmasq but dnsmasq doesn't integrate with DHCP (edit, client) out of the box and requires a restart to apply configuration updates which invalidates query cache.
Edit-
I think split dns more commonly refers to serving different internal/private and external/public zones.
In this case, the DNS servers may return the same results but if I'm connecting to a VPN running some distance away, I don't want to route all DNS requests to that server--only the ones for domains on that network. The Domains directive allows configuring this https://man.archlinux.org/man/systemd.network.5#%5BNETWORK%5...
You can add a dhclient hook to reconfigure dnsmasq with `server=` directives to achieve the same thing, but, as mentioned, that still requires restarting the dnsmasq daemon. That usually manifests as: the VPN connection times out, the hook runs, dnsmasq is restarted, but a random web page or request will fail while the daemon is restarting. Sure, you could do SO_REUSEADDR or something to prevent the disruption, but, once again, this all just works out of the box with systemd-*
> dnsmasq doesn't integrate with DHCP
DNSMasq is a combined DNS and DHCP server! Of course it's integrated!
I'm talking about a local caching resolver, not a server running somewhere else on the network. dnsmasq doesn't ship with a DHCP client
I don't get your point. DNSMasq doesn't ship with a client, because it's a server. It's a cacheing resolver and a DHCP server that is usually run locally. I don't know of any system that doesn't come with its own built-in (or standard) DHCP client.
Can you clarify your objection to DNSMasq?
His objection is that it doesn't replace the use case that resolvd is supposed to solve. Which is to provide a local service that can redirect DNS queries dynamically as local interfaces go up and down, which is where the integration with a DHCP client comes in.
Oh, OK. Thanks.
Makes sense; a lot of the systemd ecosphere seems to be about hotdesking, plugging things in and unplugging them, moving from one wifi AP to another, hibernating and so on. I don't encounter these use-cases, on the whole, which is probably a large part of why I find systemd so annoying.
Yeah, and I have some sympathies with what systemd is trying to do here, but the implementation is often backwards and/or half-arsed.
Still no DHCPRELEASE.
Dnsmasq and resolvd is a beast to tame in homelab, and split network.
> I'm not overly found of timer units, either. They seem feature rich but much more complicated to setup than a single line in a crontab.
I’ve twice tried to use them. Closest I got to having one work was that it reported it ran, and didn’t error, but it did not run. Extremely simple cases that were basically copy-paste an example and barely modify it, too.
Oh well, add a line in cron and forget about it.
I like the honesty of the article. I never got the grunge others have against it. I am a software engineer and work in office automation (a.k.a. the boring stuff). On Linux I am a poweruser. If Ubuntu used initd I use it. They switched to systemd so I use that. I liked it. Making an application start on reboot was easy to do. And easier than with the rc files 20 years back.
I think the problem of systemd is that it is complex. It is fine when it works correctly, but when you eventually have to dig into any sort of problem, it is hell.
For example, all of my boxes spew many logs per minute about "Failed to set up mount unit". This is apparently a bug with the generated name of some internal unit related to mountpoints (too long). This one is not as bad as others, because it does have a bug reported for it, but is still something I'll have to deal with until I upgrade the distro on all the servers. Many similar bugs I can't track down at all.
Systemd makes a lot of things easy, but not simple, and that is a big problem in practice.
This is it. For example, systemd (a) allows you to run services as a normal user, not as root, (b) treats child processes of a service process as part of that service, and (c) collects logs from services. Except if you try to do all three, that just doesn't work, because they've half-arsed the log collection process:
https://serverfault.com/a/1143851/92104
There's no reason for this not to work! Taking a very straightforward and traditional approach to running processes (as in my demo code in that question), you don't have this problem. But rather than that, they did something clever which doesn't work properly.
I can't make heads or tails of either of those answers, and they seem to contradict each other. The output capturing is all to the same file descriptors, this is the way resources are inherited on fork (and there's no redirection here). That it works with unbuffer is weird because the subshell and subprocess both end up flushing their output when they exit, so there should be no need to flush before exiting.
If the first answer is correct, then the unbuffer works because the process introspection as the source of the output is unified to be the unbuffer process. But why bother introspecting the exact process that originated a write to the output? Did we have a problem with file descriptors being highjacked? And even if we did, losing the output (or not associating it with the parent process) is worse.
AIUI, what happens is that no logging is ever lost, it is just not associated with the right service. This happens when a child process spawns, writes something, then dies before systemd is able to find the child's PID and map it to a service.
This implies that systemd does not just set up a pipe for each service and read everything from that. Rather, it does something like set up a unix domain socket, and then use SO_PEERCRED to find out who is at the other end for each write, then map that to a service by walking the process hierarchy or something. Maybe it's something less nuts than that, but i am not optimistic.
Yeah, that's what I gathered too. But if you can't find the output, output that is implicitly associated with a process tree as in this example, it is effectively lost. And even if you could find it, that it's disassociated from the origin reduces the utility of having captured the output at all.
SO_PEERCRED is on a per-socket basis, not per-write basis. From unix(7):
Child processes that inherit the file descriptor like in the example would not influence the SO_PEERCRED result. So it must be something else. For vague reasons.This read-only socket option returns the credentials of the peer process connected to this socket. The returned credentials are those that were in effect at the time of the call to connect(2) or socketpair(2).
I think the author is right on just about everything.
But the author is especially right on the need for competition. So I am working on it.
I am building a build system, and I am making it usable as a library. When I have made it so, I will implement an near drop-in [1] replacement for systemd.
You'll be able to use systemd unit files, and there will be an option to have binaries with the same name (off by default to not interfere, though) so that users don't have to learn new stuff right away.
[1]: I am not going to implement journald for example; logs will be text, even if they are structured internally.
Besides, implementing things to be perfectly compatible would only strengthen the monoculture, not weaken it. The plan is to be compatible on the things that matter for distro integration, so that distros can ship both with little work, but have other things different, so that users can choose what works best for them between the two.
> I think the author is right on just about everything.
The author is very wrong on the systemd-resolved bit.
Not understanding the ability to have per-interface specific zones is ok; it is a thing for desktops with multiple interfaces that come and go (like VPNs, for example). There is no other resolver on Linux capable of doing it and integrating with NetworkManager. You can kinda-sorta make dnsmasq do it, but with some limitations.
But the crown is taken by the "mDNS is better taken care of by a dedicated package like Avahi." Uh, oh. It is like saying, that you don't need Chrome, it's duties are better taken care by nginx. Similarly, you can use Avahi for _advertising_ mDNS services, but not for _resolving_. Which, as an user, you probably are interested in.
You do realise avahi-client is a standalone CLI resolver, and Avahi is what both Gnome (gvfs) and KDE (kio) use to find mDNS clients on the network?
There's also nss-mdns at https://github.com/avahi/nss-mdns but it is mostly unmaintained.
In a similar vein, but already existing, nosh lets you use systemd unit files and commands via a shim layer:
Does anyone use it? It there a distribution (BSD or Linux) using nosh?
Have you considered integrating this with OpenRC?
I am not sure what you mean.
Linux isn't what it was - it's contributed to via a lot of companies with commercial interests. The open source developers who started doing it all for love are often doing it for money now. Certain features are desired, there's a way to do it, they do it.
The actual "customers" of linux now - the ones providing the money that employs people - aren't Linux enthusiasts but big companies. So it's absolutely obvious that their needs are "what matter".
It's ok. The rest of us can fork and "be irrelevant" in the same way Linux was irrelevant before it became popular.
I would in fact love a portable user-level process manager that can speak systemd unit files and is portable.
Something like supervisord, but you use systemd unit files, and it tries to do as much as it can within limitations (process tracking sure can be wonky, no dbus everywhere, no cgroups everywhere, no absolute freedom in resource limitations if you’re not PID 1).
Process tracking without cgroups seems like an absolute nightmare w.r.t. race conditions
I originally wrote this comment for reddit, but I feel like hackernews might appreciate it. If you think it's too long or detracting from the article or something feel free to downvote, I won't be offended.
Here are some of my criticisms, although I do still use systemd daily on my personal devices and servers.
* Bad security
Systemd is architected in a way that has a lot of code running as root, it's also written in a language that isn't memory safe. This means it has a large attack surface (a lot of code you need to make sure is bug free to be secure) and it's harder to make sure your code doesn't have really severe security related bugs (The [NSA recommends](https://www.nsa.gov/Press-Room/Press-Releases-Statements/Pre...) using memory safe languages for critical stuff like this).
There are certainly other critical projects (like this linux kernel) that are similarly important and written in memory unsafe languages, but systemd has had some [pretty critical vulnerabilities](https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=systemd) discovered that do not inspire confidence in their ability to use these kinds of dangerous languages safely, and they don't have nearly the same budget as the linux kernel for detecting and preventing these kinds of issues.
This is less of a problem if you're a large enterprise customer running up to date SELinux, but it should still have been possible to write systemd in a way that limited the pid1 attack surface while retaining all current functionality.
* Journalctl is a pain for desktop users and smaller teams
Journalctl is how systemd manages logs. By default it saves logs in a binary format with additional metadata. This makes it easier for large teams to ingest the logs into a centralized log-collection daemon, like the ones offered by redhat for enterprise deployments, but it breaks a lot of workflows that older sysadmins probably used. Things like just rsync-ing a bunch of logs to one place, or using tools like grep and find to inspect logs. Systemd does of course provide replacements for those tools, instead of using grep you can use journalctl to search through your logs, but you could use grep to search through any text file. Config files, source code, or logs. Now I need to memorize all the flags for one more tool, and change a bunch of stuff about how I collect logs.
This also presents a challenge for people doing embedded work, as you can't just grab the SD card out of a system can look at it's logs. You need a working journalctl CLI on your host machine. They've add in a --directory flag, but in the past this was much harder, requiring you to actually chroot into the embedded system (that may be broken in strange ways) in order to read logs.
Journalctl has advantages, but they're mostly enjoyed by large enterprises.
Yes I know it's actually systemd-journal or just "journal" or whatever they call it. Journalctl is the command most users will be familiar with though.
* Poor [Locality of behavior](https://htmx.org/essays/locality-of-behaviour/) makes it harder to reason about
When you're trying to understand how a system works it's nice to be able to see everything in one places. There are like 9 different places a systemd unit file can live, you can apply an over-ride to a unit file, unit files can depend on other files like socket files.
This is good for large teams as it makes it easier for specific groups in a company to claim ownership over parts of the system, but it means as a desktop user or sysadmin working with a small business you've added a lot of complexity. You can't just type `ls /etc/init.d` to get a rough overview of what services exist, you need to memorize more systemd-specific commands. If you want to edit a service you can't just edit a service, you need to create an over-ride using another systemd specific command, make sure you have the EDITOR environment variable set up, and then open the original service in another editor so you can compare the two.
It creates some more work and complexity and encourages you to use a bunch of systemd-specific tools (and presumably get red-hat certified training).
* People use systemd stuff before it's ready
I'm not sure this is something I can blame systemd or redhat for, but the official stance of redhat is that systemd-resolved is not ready for production, and yet it's used all over the place. That can give people a poor impression of systemd after the 9th time they try to do something even slightly different with their networking setups and systemd-resolvd breaks, not to mention the numerous security issues in systemd-resolvd.
* Unix philosophy
A lot of people say that either unix philosophy doesn't matter, or that systemd does embrace unix philosophy. That stuff about logging and systemd-specific tools I mentioned above? That's what people actually mean when they talk about unix philosophy, they mean being able to grep through their logs and rsync logs to a remote server. They mean using standard text files and not needing to have a special command that wraps your text editor to "properly" edit a unit file.
* Doesn't run in a chroot
As someone who splits my time between embedded linux and server linux this is just a personal pet peeve of mine. It makes it very hard to debug embedded systems that use systemd if you're not using systemd on your workstation. It does feel like I'm being forced to use systemd sometimes, and while I've largely gotten over it I'm still a bit bitter. It's also made some small personal projects, like getting a full linux distro running on a KoBo e-reader, much much more difficult than they had to be.
It's a mess under docker, and why I need to use alternative OCI-runtimes like nestybox to do a bunch of testing for embedded systems. Thankfully I wasn't an early adopter to docker and didn't have those problems until there were already mature solutions. But there's really no reason why it should have to run as pid1, other than them wanting you to use docker-alternatives that are deeply integrated with systemd, like their podman tool or systemd-nspawn. This was just such a blatant attempt to abuse their near-monopoly position that it bears some extra whining.
* OpenRc does everything systemd does, but better
Unfortunately other red-hat influenced projects like Gnome won't support or test on non-systemd init systems, let alone providing default services files for them, meaning that any distro that wants to be compatible with gnome will need to do a bunch of extra work to write and test service files. For one project that's potentially reasonable, and certainly there are distros that do that extra work, but for projects like Arch who have the explicit goal of sticking as close to upstream sources as possible it makes it more or less impossible.
Systemd survives not because it's a good solution to the problem, but because it has a large corporate backer, is widely deployed, and is a safe thing to code against. There's a very old IT saying, "No one ever got fired for buying IBM". If you pick a safe industry-standard options no one can blame you if it goes wrong, even if it's the technologically inferior option.
As much as I'm a systemd-hater I still do use it on my personal devices and servers, because it's by far the path of least resistance.
I hope we see a similar situation like with pulseaudio and pipewire, where the pulseaudio rewrite was much much nicer than the original. I don't think that's going to happen until systemd slows down though, right now if you tried to re-implement systemd I suspect you'd get the rug pulled out from under you as they changed standards and behaviors (I've seriously thought about doing it myself, at least for relatively simple unit files). I'd still prefer to be using OpenRc, as I don't know how a rewrite would deal with the locality-of-behavior issues, but systemd has been getting better and more reliable over time.
> Journalctl has advantages, but they're mostly enjoyed by large enterprises.
I don't agree with this. As a single administrator, being able to do `journalctl -u <service>` and get all output from a unit (and its subprocesses), including anything that it erroneously send to stdout or stderr instead of its logfile, is a godsend. Back in the SysV days I had to manually step through initscripts to figure out what's wrong when a daemon immediately exited without logging anything.
Sure, but couldn't you get the same advantages is systemd logged to a text file instead of a custom binary format? The advantages is that it's handling stdout, stderr, log creation, log rotation, etc.
I'm not comparing it to sysvinit here, I'm comparing it to any other modern init system.
My biggest issue with that binary log format is that they are just reinventing a DB without being aware of it. And as they discovered, building a DB from scratch is hard, there's corrupted logs even in my own personal machine now.
> You can't just type `ls /etc/init.d` to get a rough overview of what services exist
You never really got that on svinit systems either, since there was still inetd, and cron had at least three different places you had to look as well. With systemd, you have systemctl list-units and you're looking at everything -- to say nothing of being able to filter by status. rc files make you implement service control yourself from scratch every time (save for a small set of primitive utility functions), and every one is a snowflake.
> Systemd is architected in a way that has a lot of code running as root
What would you want to move to a different user specifically?
Systemd is the first one that actually makes modern security features first-class. Most big services run with NoNewPrivileges set, with specific paths allowed, with PrivateTmp, and a bunch of other features. Then there's DynamicUser and other fun bits.
Not only did systemd make it simpler and more common to run bits as different users, it also makes the root ones more restricted.
> Systemd is architected in a way that has a lot of code running as root
That's just objectively wrong.
systemd uses less privileged accounts for many parts, for example systemd-networkd runs as the systemd-network user. This is different from the classic UNIX way of running everything as root (ISC DHCP Client, NetworkManager, ifupdown, ... all just run as root).
This is quite a balanced take on the matter. Appreciated!
Before this topic boils down to: SystemD gave me cancer/SystemD cured my aids as it always seems to. Please permit me get my opinion across (as it is more nuanced) before I continue.
I sincerely believe that systemd solved a problem that nobody was willing to solve, and it has every right to solve those problems any way it wants (when you ask for help you don't get to choose how you are helped after all) - my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture; and a monoculture that almost certainly will stifle innovation since a replacement will need to be bug-for-bug compatible. Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year.
That said; and with the knowledge that while I am afraid of systemd as monoculture (and a relatively opaque one); this line "systemd, as a service manager, is not actually a bad piece of software by itself. The fact it can act as both a service manager and an inetd(8) replacement is really cool."
Is something I vehemently disagree with.
For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this.
"But", I hear you thinking already: "systemd runs services as independent users, it solves that!", and I would agree with you, except now pid 0 is listening to the network instead.. That doesn't strike me as much better.
If there's a bug/backdoor in your binary distributed version of systemd then you are SOL and your whole system is owned as root, but at least a bug in your application might not expose your entire inetd user. :\
It's also a common issue that inetd's architecture can lend itself to getting DoS'd harder than other more-standard daemons, except now it's your pid 0 being DoS'd; not sure how you recover from that honestly.
EDIT: if you are going to downvote, please provide reasons. Sick of this holy war, lets just end it reasonably please.
The monoculture problem is one that I really tried to cover in the article, indeed.
While I do agree that a "super server" listening to the network could be a new (or old, with (x)inetd) concern, a well-written and well-designed pid1 doing this isn't much more risky than having the services manage themselves. The listener could/should be just as unprivileged as the target daemon, nominally using the same uid/groups and root directory as the target as well. And indeed, systemd's .socket files support the same environment control variables (User=, Group=, SupplementaryGroups=, RootDirectory=, WorkingDirectory=, et al) that services do.
So that boils down to "are people writing socket activation units correctly", which is probably "no", but could be "yes".
> The monoculture problem is one that I really tried to cover in the article, indeed.
The issue is that the previous "multi-culture" really sucked real bad. It was a shit show, really.
systemd is rising the bar by A LOT in terms of systems management, and most alternatives are simply not keeping up.
But can you really blame it on systemd making a stellar job on its own?
If anything, we could blame it on the "alternative projects" doing a fairly poor job and delivering very little.
edit: systemd is so good that FreeBSD people have already started pondering if they should build something similar for themselves: https://www.youtube.com/watch?v=o_AIw9bGogo
Fuuuuuck I hate that talk by Benno.
Mostly it strawmans the opposition, says its because they dont like change primarily.
Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration.
Its a talk that comes up often in these threads but it completely betrays the idea of having a fruitful dialog about the pros and cons of this design and the situations where it can be beneficial.
> Then he indicates that macos’s launchd is an improvement, despite it being one of the most painful parts of advanced macos administration.
This seems like a rather hot take. Most of the Mac admins I knew when I worked in the field were quite happy to have a single standard way to solve that class of problems.
Why is inetd an anti-pattern? Isn't separating all the work of opening ports and launching daemons into a separate component (so that it's not mixed into every other component) exactly what separation of concerns is all about?
Inetd approach is just fine. Don't listen to to the worshippers of systemd complexity. It's like listening again the guys who insisted that JEE is the future and you are stupid because you prefer simple solutions to simple problems.
> Prior init's were actually quite easy to replace and alternatives were often used, but these days most software assumes systemd and this situation gets worse every year.
Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?). Idiomatic systemd daemons are, comparatively speaking, very straightforward. Of course, there are a lot of nice-to-have features, but to get a running system, you should largely be able to get away with parsing Wants=, After=, and ExecStart=.
> Not really. Sysvinit scripts are... scripts full of sysvinitisms (double forking and PID files, anyone?).
... well, a bunch of scripts are something that's relatively easy to replace. It's not as though other system components have reliance of these scripts and their sysvinit'isms baked in.
From what I've heard (though not verified) - non-systemd distributions seem to manage to work with upstart, or openrc, instead of sysvinit, without much hassle.
Previous inits were glorified loop { fork() } programs.
There were many scripts, many implementations, but interestingly most were compatible with each other.
The largest issue inits had was that the flexibility they provided gave distro maintainers a lot of choice in how those scripts should operate to be consistent with the rest of the system.
Things like log locations.
The issue to be solved is: how to determine parallelism in boot, how to supervise a process with minimum complexity, and how do we do structured logging.
Interestingly: SMF solved all of these a decade before.
> Previous inits were glorified loop { fork() } programs.
Maybe (I'm not an expert on old init systems), but current inits aren't that.
> The issue to be solved is: how to determine parallelism in boot
I think you're replying to another comment of mine. At any rate, there were and are alternative solutions rising to meet this challenge - which involve nothing like the behemoth which is systemd. Examples: OpenRC, runit, GNU Shepherd (sort of).
> I sincerely believe that systemd solved a problem that nobody was willing to solve
What problem was there which nobody was willing to solve?
Also, "solving" a problem by creating a mechanism that in itself highly problematic in other ways does not necessarily count as a solution; it is a shifting-around of problems.
-----
The systemd "holy war", such as it is (mostly complaints on forums and in blog posts and in personal chats; don't remember any violence in this war) - is due to four reasons:
* The significant problems which systemd introduces.
* The way systemd has been developed and managed as project (including some grievances with individuals).
* The fact that distributions have not only adopted systemd, but made it effectively impossible to opt out of.
* Faults with the process in which systemd was adopted as a required default by all of the main distributions - bypassing wide opposition without addressing its criticism. So, the adoption of systemd exposed technical-governance/power-dynamics problems in the Linux distro world, with systemd serving as the symbol for those.
I like the off-by-one there, and choose to interpret it as intentional.
The fourth item is what I find to be the most interesting. I really hadn't thought about it that way before. I remember the discussion in Fedora mailing lists around systemd and iirc the criticisms were mostly actually answered and handled. But in Debian it really wasn't and it fractured the community a lot.
It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard. Perhaps that's the bigger lesson we should all be learning: to listen closely and respond respectfully.
> I like the off-by-one there, and choose to interpret it as intentional.
Amongst our weaponry are such elements as... I'll come in again.
https://www.goodreads.com/quotes/1495-nobody-expects-the-spa...
> It is perhaps the case that the "anti-systemd" crowd wouldn't be "anti-systemd" if they felt like they had been heard.
I would say it is the other way around. systemd would not have been adopted as the default init system, had the anti-systemd crowed not been ignored.
But that's not even the main point. The grievance is not about how some individuals did not "hear". It is about the _possibility_ of such a decision being taken with that level of technical-community resistance; i.e. the expectation is that decent process would not depend only on the benevolence of those in charge. That's why it's a structural rather than a personal failure IMNSHO.
And again, there's the conflation, or bundling, of the multiple decisions:
1. Offer systemd in the distribution
2. Make systemd the default option for the distribution
3. Necessitate installation & use of systemd with the distribution
I am specifically pretty certain that Devuan would never have been forked if systemd were merely a configurable installation option in Debian.
> I am specifically pretty certain that Devuan would never have been forked if systemd were merely a configurable installation option in Debian.
You can't stop 5 or so random people on the internet from starting the 100th irrelevant Debian-based derivative.
"* The fact that distributions have not only adopted systemd, but made it effectively impossible to opt out of."
The job of a distro is to decide what to ship and integrate that in the best way possible. Would a distro let you choose between glibc and musl for example? If you want either, use a distro that chooses that.
This has more to do with systemd trying to be a "system layer" project for Linux, and influential projects (initially Gnome, then KDE, and now a lot of other things) going along with it.
Honestly, I see no reason a distribution couldn't support multiple service managers. Indeed, Gentoo officially support OpenRC and systemd and do just fine. However, you need to have the resources to do it and the desire. I think there are multiple distributions with one or the other, but Gentoo is likely the only with both.
> However, you need to have the resources to do it and the desire.
If resources were lacking to offer both regular and systemd-based init and service management, than a systemd offering should have been deferred until such time when resources become available, or different projects did some work of their own to reduce the amount of necessary resources. Offering multiple other init/service management options apparently doesn't require many resources.
(Of course, they could have saved a lot of resources by dropping GNOME until the GNOME people started being civil and not hard-depend on systemd... but I realize that's a bit of a controversial suggestion :-P )
> systemd offering should have been deferred until such time when resources become available
Why would you delay adoption of a technical superior solution if there is no interested/lack of resources in continuing maintenance of the inferior solution?
Distributions did not stop adopting Python 3 and invested in continuing Python 2 either.
> Why would you delay adoption of a technical superior solution if there is no interested/lack of resources in continuing maintenance of the inferior solution?
But a technically superior solution was not on the table to become the default, the suggestion was to use systemd and effectively prevent the use of anything else (including potentially technically superior solutions).
> For starters, (x)inetd is an anti-pattern, everything I understand about systems development indicates we should be seperating concerns as much as possible, having one super-server that launches everything under one daemon is directly opposed to this.
It means that this server had to be carefully secured but the benefit is that you have exactly one bit of heavily-audited code listening to the network, logging activity & problems, starting processes, changing users / dropping privileges, setting up namespaces, etc. I’ve seen a lot of code get various combinations of those wrong so I think that’s a far more nuanced problem than in your portrayal.
Those are all things people should know how to do but I’ve seen Java or PHP running as root in production because someone couldn’t figure out how to drop privileges needed only at startup enough times to appreciate the benefits from systemd making it so much easier to do things right.
>"systemd runs services as independent users, it solves that!"
It's in a terrible way, too. Not sure it improved but I recall somebody once, at a company I worked at, trying to introduce per-unix-user systemd services (e.g. `graphite` user for running Graphite), and there were some atrocious steps required like `loginctl --enable-linger` and God knows what.
We moved to running everything systemd as root, it's easier (and you can specify which unixuser the actual systemd unit runs at, which is "close enough").
Loginctl/per-user systemds are for managing interactive sessions. System services with User= is exactly the way you'd go for isolating background daemons.
What if you want to let a non-root user manage the systemd unit for a particular service? Is there a better way than per-user systemd instances?
Check example 3.4 here for a polkit policy allowing an arbitrary user to restart a single unit:
Does this let them modify the unit file, create units, create timers, etc?
User systemd allows that kind of complete self-service, and so lets you do application deployment and management without touching the root account, which is rather nice.
If they can edit a service that runs with more privileges than they have, that's a privilege escalation vulnerability.
Polkit lets a non-root user restart a root/privileged service without letting the non-root user gain privileges.
chgrp, setfacl, sudo.conf ... lots of choices.
"Nobody was willing to solve" is a bit of a stretch when upstart got shipped at pretty much the same time. (Sponsored by the commercial home of a particular Linux distribution -- but so was systemd.)
I used Upstart a lot but it did less and had various unfixed bugs whose failure mode was becoming unmanageable. I give Ubuntu credit for starting it but was reminded of jwz’s CADT rant periodically until we upgraded to a release which used systemd instead.
> my concerns boil down to the fact that the adoption it has seen has lead indirectly (or, directly) into a monoculture;
Were there multiple implementations of sysv-init being used by different distros on Linux before systemd came along?
yes, at least 5.
sysvinit, s6, openrc, runit and Solaris SMF
Were they alternate implementations of sysv-init? Or did they do their own thing, and also happen to run sysv-init scripts for back-compat? Because systemd also runs sysv-init scripts for back-compat.
Edit: Also, were any distros actually shipping those as supported init systems? I was under the impression that most of them were still in the "experimental" stage and not viable replacements (yet).
OpenRC and Solaris SMF definitely did their own thing. SystemV-init and the "run levels" concept was flawed from the very start when it replaced the previous BSD-style "single user" vs "multi user" boot up.
Linux only adopted SystemV style init because it was the norm in Solaris at the time. It is not a linux-ism.
As this article notes - systemd is actually fine as an init system and hardly anyone denies it. It's all of the other stuff (journald, resolved, timers, etc) + tight coupling + environmental assumptions that is the problem.
I'm unaware of any Linux distribution using SMF. If there was, I would have been running it. I didn't really like pfexec (it felt bolted on and not quite there yet), and I really really hated ZFS (and still do), but SMF made Solaris administration a joy.
I don't think any distribution was using runit before systemd, but it was available in Gentoo as a sysvinit replacement and ISTR it was used in the Rails community for supervising Unicorn.
> now pid 0 is listening to the network
Pid 1, surely.
The Famous Article (TFA) is an exemplar of dispassionate substantive, liberal criticism.
The shortcomings of systemd seem likely the overall shortcomings of open source: none of the $PROJECT maintainers experience $PAIN_POINT, so it is simply not a priority for $PROJECT.
Somehow this seems a variation on the tragedy of the commons => https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons
Without some capitalist skin in the game, the $PAIN_POINTS become difficult to prioritize.
Most people (including most Unix greybeards!) really don't understand Unix's 'init' (AKA "the init daemon", "the init process", etc., etc.) -- much less any of its massively-increasing-in-LOC (and complexity!) successor programs...
So we need to start with 'init'.
'init' -- even in its absolute first, simplest incarnation -- is still too complex to understand correctly!
You see, we need to shift perspectives!
We need to shift perspectives from a longtime System Administrator -- to that of a new barebone OS programmer.
What is 'init'?
Is 'init' a program that handles runlevels, starts and stops services, that mounts filesystems, that processes messages, that captures dead processes, that waits for hardware to become available, that logs and maintains informational/database/etc files, that starts audio, that starts X11, that stars the GUI, that acts as a proxy for sockets, or does anything else with the system?
No!
From the point of view of a new barebone OS programmer (as Dennis Richie and Ken Thompson were when they invented Unix and invented 'init') -- 'init' is NONE of these things!
'init' is only THE FIRST PROGRAM, THE FIRST COMPUTER CODE THAT RUNS IN USER SPACE.
And that's it!
That is all that 'init' ever is, or ever was!
(User space, to recap, is the unprotected AKA "unprivileged" AKA "non-supervisor" memory running unprotected (AKA "user-land") code: https://en.wikipedia.org/wiki/User_space_and_kernel_space)
'init' (and every single 'init' successor program, i.e., OpenRC, systemd, etc.) -- are the first program, the first set of computer code OUTSIDE OF KERNEL CODE (which has been running and is currently still running) to be run by the system.
Now, what should that first program do?
See, that's the magic question -- which gives rise to all that is to follow!
In theory you could have an OS where the 'init' program, or its equivalent -- did absolutely nothing! But that wouldn't be very productive!
If the 'init' program isn't itself a shell program (i.e., sh, bash, etc.) -- then (because there's no GUI at this point) the computer will not be able to accept typed command-line commands -- which is the first thing that you want a new OS to do!
So now our 'init' expands in scope (and lines of code)!
Our 'init' could be hardcoded to launch 'sh' or 'bash' (or whatever shell program exists) -- but what if the user wants to change that?
OK, so now we need our first configuration file. Where to put that exactly?
Oh, it's on a filesystem that hasn't been mounted yet?
Well, maybe init should mount that filesystem!
Point is, there's a set of problems (and sub-problems!) -- which give rise to increasing and increasing init's functionality over time!
init, as the first user-space program for an OS to run, on whatever OS it is ran on, in whatever form it is in -- could simply be written to run and 'outsource' all of its functionality to other programs...
But init (as it evolved into its very large LOC complex descendants) -- became a "dumping ground" -- for functionality that was inconvenient to go in other places and/or to be outsourced to other programs.
See, all of the code in all userland Linux utilities -- could in theory be grafted together into one big super program in userspace.
It would have the same functionality as all of the individual Unix/Linux command-line programs put together (and maybe that would be desirable to some people). But from a Software Engineering "separation of concerns" AKA dependency reduction AKA modularity AKA "do one thing and do it right" AKA loose-coupling perspective -- doing that might not be so desirable!
And yet, with the complexity brought about by 'init' descendants -- it seems like we're going down that exact route!
Which leads us full circle (because history always repeats itself!) -- back to the reason why Unix was created -- because of the complexity and problems brought about by the complexity of its predecessor, Multics!
https://en.wikipedia.org/wiki/Multics
Point is -- 'init' in whatever form it takes -- is by no means obligated to do anything -- although if it is to do nothing, then it should at least launch one other program which will do something! If that's the case, then why not put that under user control? But wait, if we're doing that, why not make it launch multiple other programs! OK, now we need a file to tell init where that should be! But what if the filesystem for that file is not mounted?
Anyway, you see how the "rabbit hole" of problems (and increasing LOC complexity) forms!
Related: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
> That is all that 'init' ever is, or ever was!
Except it is not. The "init" process seen by users as PID 1 on the usual Linux distributions is far from the first program started by the kernel. There are usually hundreds of processes started before that PID 1 process gets started.
>There are usually hundreds of processes started before that PID 1 process gets started.
By the time PID 1 is started, the OS has run OS code, or more specifically, prior to PID 1 being ran, the OS has run OS code -- that much is true!
Loosely, you are calling that OS code -- "processes"
We could call that OS code "processes" -- however, there is a difference between this loose definition of what a "process" is, and what a "process" is defined to be by Unix/Linux standards and conventions.
The OS code that the OS runs, the kernel code -- typically does not have a 'PID' (Process ID) associated with any of it.
To understand this, let's remember what Unix/Linux (or any OS) is -- it's software abstraction layer over hardware.
It's goal is to act as a hardware abstraction layer, but also as a moderator, a resource arbiter for other pieces of software, other code -- to run on the same hardware, the same machine, at the same time.
This other code -- is typically the user space (AKA non-privileged) software, the user-space code.
In order to manage that user-space code, AKA user programs, AKA user-installed software and because several (or more) of these programs might be and probably will be running simultaneously, a unique postive integer handle (functionally the equivalent of a key in key-value programming).
That unique positive integer handle assigned to each user-space program (unique ones can also be assigned to threads of programs) -- is typically the PID, or Process IDentifier.
That's WHY they exist.
PID 1 is the Process Identifier for the first user-space program to be run by the operating system, typically 'init' -- or a descendant.
If "There are usually hundreds of processes started before that PID 1 process gets started", which is what you're calling "processes" (kindly disambiguate exactly what you mean when you say "process" please!), then WHY is it that they are not given ascending unique positive integer numbers and WHY does this numbering, should it exist, force PID 1 to be some other higher-numbered PID?
If "hundreds of [user-space] processes [are] started before that PID 1 process gets started", then WHY exactly, is the first ran user-space program PID 1 -- and not some higher numbered PID?
?
???
What Lennart found is that pid 1 is the only process in the whole userland that can ensure a process is terminated without risk of race conditions. Hence, all the process management stuff HAS to be in pid 1 in order to provide consistency guarantees. Hence, systemd.
PID 1 as the first user-space program/process (historically the 'init' program), has always been "the process management process".
In other words, it has historically been the program/process used to manage other programs/processes.
Consider the simplest possible Unix/Linux system with only two user-space processes. For a Kiosk, let's say.
The first user-space process/program is 'init'.
The second (and last!) user-space process/program is the kiosk software itself. Call it KIOSK.EXE (or /usr/bin/kiosk -- since this is Unix/Linux, or what-have-you)
OK, so now here's the problem!
What should happen if KIOSK.EXE or /usr/bin/kiosk -- should fail for some unexpected reason?
Some stack-heap collision, some segmentation fault, some core dump, some "blue screen of death", some abort, some failure...
OK, so now how do we:
a) Monitor that kiosk software to know if its up and running;
b) Bring it back up and running if it has crashed?
Well, if there's only 'init', aka the PID 1 program/process in the system (other than the kiosk software, and the kiosk software is crashed, no longer functional) -- then it must be 'init' (or PID 1's) responsibility to bring 'kiosk' back up!
To get 'kiosk' back up and running!
That's WHY the "process management stuff" functionality -- historically has been placed into 'init' (PID 1) -- simply because there was no other easy/simple/straightforward/obvious/user-friendly place to put it!
But 'init' ultimately is just a user-space program.
It could be programmed to do nothing, it could be programmed to do everything (its code could in theory be fused with the kiosk program, although then it becomes the OS's responsibility to restart that program if it crashes!)
But ultimately 'init' -- is just a program!
It could, if programmed to, do whatever any other user-space program could do.
It could also be programmed not to do anything -- or offer the barest minimal functionality.
What 'init' does or does not do, or is supposed to do or not supposed to do -- is truly at the behest of any programmer who wish to work with the source code of 'init', in whatever form that may take...
Nothing "HAS to" be any particular way for any OS or user program if you have the source code, if you know what it does, and if you know what you are doing...
There are no "HAS to's" -- for those that have source code and understand it.