A live tree of systemd jobs on the boot console
codeberg.orgThis looks cool, I like this... but if I read the screenshot correctly, it took them 12 seconds from initrd to fully loaded. Maybe this would make sense there, but my laptop boots so fast, those 12 seconds are like two seconds. It would take longer to render the screen than it would just spraying its usual text to the VT.
(Honestly, it spends more time in uefi preboot than it does to actually boot the system.)
That's a taken from the demo `sample-vm-slow-services`, which is intentionally slow to show off all the features.
This is an ongoing pet peeve of mine, how the UEFI of a completely solderd laptop board can take longer than the OS to initialize a fully-known set of components.
It's not really UEFI issue, but what modules were loaded, how it was configured/wired up - and sometimes it ends up serializing on slow boot of some components (like DDR5 memory), and weird crappy add-ons from manufacturer.
On many machines with DDR5 you get the lengthy memory training every boot and that comes on top of the already long UEFI initialization time.
Surely that could run in parallel with initializing everything else, with the amount of CPU cache modern CPUs have?
Also the soldered ram couldn't have changed from last boot, so reuse the last training? That's exactly what I mean.
The memory training happens before UEFI is initialized proper, though it provides a minimal framework for handling such parts - it's mainly so that every bit of very early code can properly pass information later on.
As for reusing the training, that's what normally happens, but there are various things that can trigger retraining beyond swapping anything in memory modules, and it's perfectly possible the vendor somehow managed to write an add-on that triggers that on every boot (just like intel managed to wreck NVRAM on every netboot, for example)
The training reuse is already happening on machines with regular user-replaceable DIMM modules. It's why there is such a scary long wait on the first boot of a newly-assembled system, but not subsequent ones.
So even if it can change, starting out with a simple "Are the same modules still installed?" check saves a lot of time.
`systemctl status` gives you a tree already, just not live view
`watch systemctl status`
I usually don't like TUI's and just want outputs into emacs text buffers, but this interface is beautiful, A nice thing to glance at that tells you everything you need to know about your systemd units. Well done.
Like it :)
What i don't get is why booting still takes seconds. When power is on, the hardware is on.
You can initalize most hardware in parallel, reading the amount of operation system into ram is fast.
This is very cool. It should be the new default on all distros.
It may terrify windows and osx refugees, but that's a sacrifice I am happy to make ;)
This also tickles some part of my brain that likes watching spinners, and it gives you an intuitive idea of what’s happening on your system. But for those who find it scary you could just default to a splash.
love it! finally an improvement after 30 years.
but also no reason to not render the tree with proper graphics, TUI is an UI just like GUI is and living dags are particularly well suited to be animated.
This is outstanding user interface design. Whoever came up with this is a real talent.
I would love to understand the process of coming up with the UX design is, what the principles are, and where the understanding of what "good" looks like, comes from.
Love it. Now I need to figure out if I can do a similar thing with OpenRC. Thanks for the inspiration.
Come to think of it I don't think I even have parallel boot enabled on my OpenRC systems since the Linux boot time is so much shorter than whatever the BIOS does before it hands control over (or before it even displays anything on some machines).
I dont have parallel boot enabled as well as it is disabled by default. Looking at this Systemd post triggered this thought to experiment with it one weekend. I have this laptop where I run Gentoo like a control freak. There is nothing in it other than the core system running as openrc services. So, I might play around adding readiness gates to those and see if anything at all can be parallelized.
> ... Linux boot time is so much shorter than whatever the BIOS does before it hands control over
On which system? Is your machine, for whatever reason, re-doing "memory training" everytime it boots? This is one can be really slow.
Who knows, but any desktop and laptop I had access to in the last decade spent more time in POST or whatever than it does in OpenRC. Might be more of a case of the Linux bootup with a minimal set of services I actually need already being pretty fast even without paralellism rather than those BIOS being particularly slow. Windows for example is a lot slower there from my experience - and the difference is even more ridiculous for shutdown.
I hate systemd with a burning passion, but this is the first time I ever thought it might be slightly redeemed.
I also dislike it very much.
edit: Having written that, this tool looks very cool and useful when you need to deal with a systemd distribution.
> I hate systemd with a burning passion
Why?
For me, it is pushing Linux in the opposite direction of the main reason why I like Linux.
I was attracted to Linux because unlike Windows or macOS, I actually have choice with Linux. I can choose my filesystem, my libc, my init system, my bootloader, my system encryption, my window manager, etc.
Sure, it's not perfect: many programs assume that you have glibc, or dbus, or X11, or wayland, etc. Actually many programs assume that you run Ubuntu. And it is frustrating when you would want to use a program and can't because of such an assumption. But again on Linux, there is (or used to be?) a big open source culture. This program is assuming glibc? I can patch it to work with musl. Etc.
Systemd, to me, sounds like it says: "why would I care about interoperability? I am the best, you should use me. And if you want something else, you should make that something else interoperable with me, because I make the rules". It's not like systemd is trying to standardise stuff to allow alternative components to exist, right? It doesn't actively try to prevent alternatives, but that's pretty much it.
And normally I wouldn't care (again, Linux is about freedom of choice for me). But systemd being as successful as it is dramatically reduces my freedom. And systemd it seemingly trying to take over more and more components. And the more adoption it gets, the more power it has. You didn't want systemd to replace that part of your system? Too bad, it will. What are you gonna do anyway, systemd is too big to care about you at this point.
Not the OP, but to provide a short perspective:
We can discuss these. I'm not the one who wants flame wars.- Too many parts. - Doing too many things, sometimes poorly. - Fights with other services, no graceful fallback & logging (i.e. aggressive). - As a result, unnecessarily complex, brittle and fragile. - Harder to secure (at code level) than simpler tools. - Project is too aggressive and want to replace any and every service. - Sometimes break in unexpected ways. - Not faster than Parallel Sys-V.+ Harder to debug/trace than initscripts.
+ "My way or the highway" attitude from the developers
I'm using systemd for ages, never had a single issue at all with it.
But do you run systemd and complain or why do you even hate it?
its linux right, if a linux distribution decides to use it, its not systemds fault to be used and if you use it and don't take the time and effort to have it your way, you shouldn't complain either?
The definition of my and your ages might differ a little. I have used many things which have came before systemd and did a lot of work on them. As a result, I have seen and experienced a lot of fail-modes of both systemd and others which came before.
One of the most important feature of prior tools were transparency. Since they were single-purpose tools written with UNIX philosophy in mind, they were simpler, more robust and has less fail-modes. Systemd is pretty opaque and non-deterministic in that regard. This is not a "old man yells at cloud" stuff. You can't trust to a piece of code you can't trace and debug. Systemd makes it almost impossible with all the intertwined gears and IPC layers it has.
On the "I didn't have any problems" with it side. It might be working on your system, for your use case. It might not show its bugs on a system with a 90-day average uptime. I don't live on that world. I manage way more systems with way more different, niche, interesting hardware and software than that, and this tends to surface edge cases and where things break.
On the "do you run systemd?" part, yes I run it. On many, many machines 7/24/365.
It's easy to say "Why do you even run it, it's Linux, just distro-hop", when you have a single system. I don't. Projects, software has requirements, compatibility targets, etc. Also, considering the hardware we run, it's not that simple either.
Lastly, I run Debian personally since version 3. We use other distributions alongside Debian in our datacenter. I'm very involved and rooted in these systems, and Debian feels like home after using, contributing, and developing derivatives on top of that.
...and how systemd has been forced into systems is another story. I have been there, watched that, so, errm, it's not all voluntary, peaceful and friendly, to put it kindly.
> On many, many machines 7/24/365
Been there too. The thing I disliked most, that it sometimes get stuck in some strange pose, without any reasonable way of remediation. And because it is so annoying to debug, reboot become the only tool do deal with it.
The sheer complexity of systemd created nasty feedback loop: it is too complex to be debugged; people prefer to reboot it; that keeps problems keep accumulating inside.
I hate this 'unix philosophy in mind' 'principle' as an argument.
We are no longer 50 years ago were a linux system or distro was fitting on a floppy disk.
The handful of tools like grep, cp, etc. have a complexity of a banana and everything else successful is so big, that it doesn't fit at all.
Blender, container, ffmpeg, vlc, vim, firefox and co. Huge code bases. And if someone woudl argue "yeah but firefox is doing one thing, its browsing" thats an argument you can bring for systemd too.
Systemd is just not a 'start services in the right order' tool. Its more holistic
You can hate it all you want, but UNIX philosophy isn't related at codebase size at all.
All the examples you give fit into UNIX philosophy rather well. Firefox doesn't do text editing and ffmpeg doesn't render web pages, and vim doesn't transcode videos. They do one thing and do it well. They are as simple as they need to be not more.
Also, if GNU coreutils have a complexity of a banana, why uutils still can't replicate them as-is? Why they had ~140 CVEs in one release despite being written in Rust? If grep is simple, why we have only two performant implementations, grep and ripgrep?
Even FreeBSD guys discussed this: https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
You're simplifying too much, I believe. Try writing a cp clone, why not?
systemd started as a start services in the right order tool, then bloated from there. Being holistic is not the problem, being holistic in the wrong way is.
Now, they decided that DBus is not nice enough and created their own IPC mechanism. What was that? Varlink? They don't stop there even. They say "Hey, why not try Varlink?". So to translate: "We want to kill DBus so, we will force Varlink on you".
Wait till you do have an issue or even just want to disable one of the 'helpful closely coupled' services something works to better fit what ever you doing or for security/reliability reasons (D-Bus is a great example for me).
It's difficult to avoid systemd at this point and it's not because it's the best technical solution for all use cases in the same way that modern windows is a complete shit show but still everywhere.
My question would be what does systemd do that is not possible outside of openrc and launching what ever daemons you want for the tasks you want or combining it with what ever programs you want to do things like sand-boxing or containerization? Not saying openrc is a panacea but systemd is definitely not either and for many systems is just plain annoying, I can consed for people who want a normal desktop (plug and play distro experiences) experience it can provide that.
How much longer should I wait?
As I said, i'm using it for probably a decade by now.
I reckon using cron vs systemd timers is a good example of some of the points above.
Oh my god yes. Systems timers are doubleplus ungood.
Unless you want to run something every thirty seconds.
You can run the command with time, get the real field, subtract from 30, sleep that amount and run the command again in a script, and call that every minute from cron.
If the command is really instant you can do "command & sleep 30; command".
heh.
It has the exact right amount of parts. A modern Operating system just has a lot more to cover than 1970's Unix...
I can remember having to deal with SysVinit and thank god for Poettering and all the other systemd maintainers for bringing the Linux ecosystem to the 21st century
I think the reasonable argument is not old thing is better ( like some people make, but it has its own problems that are usually because of it trying to do everything.
My main issues stem from it's prevalence in server Distro's especially for micro service images.
D-bus is one of the great offenders for me and I don't like having it on even my desktop system but systemd wants it for many things and some things will just break without it. On a small micro service image, it would be nice to be able to strip lots of it down but systemd fights you on this because of the amount of moving parts and interconnected subs-sytems it introduces.
I concur. My main pain point is the size and aggressiveness of the whole "subsystem". I compare it to Sys-V init, because it's the best example I have used until systemd.
I have been finding myself more and more reaching for alpine linux and have liked openrc.
I am a nixos user so have to use systemd on nixos and that makes it very painful, but I can see myself moving away nixos as my host machines os for many reasons but systemd and the painful way nixos uses it is a big one.
I'm using Debian for more than I like to admit, and I'm entwined with the project somewhat, considering what I do as a work, and how much I like Debian as a whole, distro-hopping looks a bit distant, and I'm not very keen on leaving Debian as an ecosystem considering what's possible at the distro level.
Could you elaborate on the reasons for considering moving away from NixOS? :) I like it in principle, but my main issue is that if you filter out the package activity from the repository, the actual core contribution is low. At least in comparison, and if you filter out the ones with a few one off commits, the actual primary contributors are just a handful so to speak. I worry that, since there are so few who truly understands the internals, if something happens to them, things may go more awry than expected ... or something like that. It's been a few years since I last checked the project. There's also that, at the time, pull requests were growing much faster than being closed or merged, not sure if that's still the case? So very curious to hear. :)
> Could you elaborate on the reasons for considering moving away from NixOS?
Sure I also don't want to take away from what nix and even NixOS does do well, it is a great project and has many great ideas.
Also not sure you are aware (I think you probably are but I will state it any way) Nix != NixOS, and to complicate things further Nix it's self has many parts, it's a build system, a language, a package manager, enviornment profile manager (think like what python venv's provide) and kind of what ever you can make it do.
I will say Nix as a tool is a fantastic thing to have around and now with AI the wrangling and learning can become less painful but it does allow you to do a lot of cool things. Building Deterministically Deterministically building environments, integration tests/ setting up environments for such tests, more easily managing multiple possible builds for a project (well easier then wrangling make files or what ever directly and does provide nicer abstractions (not pain free) for building code). If nothing else give nix a try for building your next project and see what you think, very nice for setting up your dev env as well.
Personally I use something called flakes which is a way of managing a project and pinning it's deps. And this is one of the points things will break down and you will understand some of the issues. flakes are supported and used by many but also many reject them and the tooling is just kind of their and breaks some bits of nix. This aspect of different competing community standards is very very common in the nix world and is one of the major pain points.
> I worry that, since there are so few who truly understands the internals.
I have not filter it in that way but, I am not sure the issue lies with the lack of knowledge of the internals but nix seems to complicate things... a lot.
You have the nix daemon and nix store. Until you have good working knowledge of how nix works (not necessarily knowing the code I have only glanced at the internals of daemon and such and the more important aspect is actually the code written in the nix language generally). Don't touch store directly including trying to copy files manually between stores and such the daemon keeps a sql database of the entries and other things about the nix store means you should not directly touch it until you understand this and it's not intuitive or made to be understood easily.
>NixOS? Finally NixOS it's self is great especially at the start and once you get it but the whole idea is declarative configs and well I know have spent lots of time fighting this and while it does have escape hatches they can be obscure and confusing and error prone. An example of this might be something like llama-swap module, you configure your modules in nix and llama-swap will module will configure a systemd unit for you once enable it which sounds great but in truth you probably want to do things outside the config and make ad-hoc changes and you can set it up so all this is possible but you end up finding your self avoiding a lot of the declarative features and having to edit your config then build it and push it every time for things that are small can be very annoying.
The thing is though you can generally just tell it to use a path to your config or give it a config or even not cofigure a systemd unit at all, but many of these things stack up and you find your self fighting it a lot of the time. The default will have things like a firewall unit that pushes some 'useful' firewall rules that block listening and such and works with declarative configs to enable particular ports which is good, but if you don't know that you might spend a bit of time wondering what the hell is going on.
I would say if you still interested look first at nix tooling and nix and if you want to dip into the operating system you will understand a lot more about why things are the way they are and if you are someone who is finding they want to make a lot of quick ad-hoc changes to the system in question nix can get very annoying. The language is not great and difficult to learn, the main issue is that it's not typed and a lot of the features and right way to do things is arcane and occult, AI will help learning all this a lot though and bare in mind it's a lazy functional language.
Alpine for me is looking more attractive to build my host system on and still use nixos for some dev vms and other vms I use and configure Alpine in a well defined way but still lets me quickly hack a bash script into something. Again you can use nix it's self on any distro and even mac and windows (I think).
This is not a good comparison because even Windows 2000's service model is better than SysV.
It doesn't have to replace the cron, resolver, time synchronization, system logs, bootloader, home folder management, all of the disk encryption, TPM, signing, users, and a whole lot to manage services.
I can understand the frustrations with Sys-V init scripts, but on the other hand, I never had a problem with that both as a sysadmin and a distribution developer.
To be honest, loginctl allowed us to pull some hacks, but to be fair, that hacks were to handle bad behavior from a very bad codebase. Any self-respecting development team wouldn't have written anything which needed OS/kernel level reaping to clean from system correctly.
We have been chatting recently, and someone who's more experienced than me said that they have never seen Nixie's cron to die spontaneously in the last 30 years. systemd's timers not only did that, the systemd didn't bother to restart it. Ironic, considering that the biggest feature of systemd is service supervision and unattended recovery.
But isn't that the point of it having many parts? You don't have to use all of them.
I'm not that much of a sysadmin, but as a Linux user and home lab owner, I definitely prefer the integrated systemd solution. At least I can be certain that the interactions between the components are known and tested. If you instead throw together loads of KISS tools in the old UNIX style, all your tools might be individually correct but their interaction is just bad.
The problem is, these parts sometimes are forced upon you, creating rugpulls. You update your system, and tada! something has changed without you even notice, and that might be something you have relied on for a decade or more.
Learning new things is not a problem, but now I have to adapt into that thing and it might not even perform the same, and a fallback might not be possible. Opening a great can of worms you didn't want in the most unpleasant time.
One of the problems with integrated solutions is how much glue code they contain to handle these interactions. KISS tools talk over sockets, via very simple protocols. i.e. they shout each other over mountains using well defined horns and the voices they use are distinct and simple. Less glue code, simpler codebases, more robust tools.
KISS, Everything is a File, Send conservatively but receive generously are three pillars of what makes *NIXen robust systems. If that interconnectedness and meshing of things were robust, we wouldn't have made Windows a laughing stock for its crashes and security problems.
Tell me, how can a WMF or cursor file can infiltrate a whole OS? How can not you print without installing half of the desktop or literally can't have a terminal only Windows Server system? Because everything is connected and they can't be separated anymore.
These "tools thrown together in a bucket in haphazard manner" have kept systems running for literal years between reboots because they talked over these simple sockets. Heck, many good Linux servers die not because of software, but hardware failure.
And I respect Linux because this is still true despite systemd today.
> At least I can be certain that the interactions between the components are known and tested.
Oh good _lord_ you absolutely cannot. It has been my professional experience that Systemd Project is fractally complex, most of that complexity is accidental, and approximately no one on the project is interested in investigating and fixing the intermittent-but-showstopping problems that arise from that accidental fractal complexity. [0]
Poettering and crew really built the project a fantastic escape hatch with their whole "Your usecase is legacy and we're not supporting it." attitude.
[0] The mess of tools in the Systemd Project work fine until they very suddenly do not, and then the best you can do is bodge a workaround because you're very unlikely to ever get a fix for your problem. The folks that staff the project simply don't have the attitude required to work on and maintain a ever-more-sprawling project that aims to be _the_ userspace thing atop which every Linux distribution is built. "That's hard to fix/do. I don't care that you've had a working system for 10+ years, I'm not fixing the thing that's broken for you." is an absolutely unacceptable attitude for folks involved with such a project to have. [1]
[1] If the project was a little take-it-or-leave-it side project, that attitude would be entirely acceptable... encouraged, even! But because of the position in which the project willingly and aggressively put itself into, it's not a little side project.
> It doesn't have to replace the cron, resolver, time synchronization, system logs, bootloader, home folder management, all of the disk encryption, TPM, signing, users, and a whole lot to manage services.
All of those components you mentioned have improved my linux experience on both servers, desktops and mobile clients.
On the other hand, I have seen no improvements and some regressions.
So, experiences differ, apparently.
There is no one thing a modern operating system should be. Requirements differ and for many systems you do in fact not need to cover more than in 1970's Unix.
But what about muh swarms of containers running microservices written with python or JS, using just 350MB per container on my K8S cluster?
Just wanted to take a jab. Feel free to downvote. =]
> Why?
The thing with the whole systemd hate thing is everyone now knows why people might hate systemd (I am not a fan). I think a lot of features around isolation and systemd containers where useful but now probably using your containerization system that best suits your needs.
For me when I get really mad at it is when I want to build small nixos Images and want to minimize the footprint of systemd and run few services (some of that is on nixos as well).
I think it's better suited for desktop systems then server systems and it's quite annoying how prevalent it is. D-bus is one thing that always annoys me and how everything in systemd wants to use it and you can run into issues disabling it (Also don't like D-bus even on a desktop system).
Overall I think and hope we will see a rise not necessarily in openrc but in simpler init 1 systems that try to do less and provide good strong primitives that allow for easily using what ever daemon configured how you want, rather then trying to provide coupled implementations for everything.
Again I know these criticism are something of a meme, but they still are true.
The XZ backdoor. That's why.
Only Rube-Goldberg complexity on the scale of systemd made the XZ backdoor possible. Systemd was made by despicable individuals who went on to work for despicable companies (as a big fuck you to open-source lovers) and then later on to work on even more despicable surveillance project (as a big fuck you to humanity).
systemd has its tentacles meddling in every single part of the Linux ecosystem.
But thankfully there's light at the end of the tunnel: thanks to VMs and containers and the need for speed, we've seen many minimal distros (I'm thinking stuff like Talos for Kubernetes) and those of course don't use systemd.
As soon as I get the time to switch away from my current hypervisor (Proxmox, so Debian, so systemd) to an hypervisor not running under Linux, like maybe bhyve on FreeBSD, I'll have all my VMs and containers and hypervisor totally systemd free (my VMs and containers are already systemd free: I did in anticipation of, at long last, giving systemd the middle finger).
But of course none of this matter much because according to HN we're already at a point in time where we can write: "Plz rewrite an init system".
The real question is: why don't people hate a rube-goldberg init system made by someone who went to work for Microsoft, while using crypto made up by his brother (certainly secure we're sure), and who then went on to create a full-on surveillance company working on locking down hardware?
Regain control of PID 1 people.
If it only takes an animation to make you like systemd that means you don’t understand systemd and linux init systems at all.