Love systemd timers
blog.tjll.netI haven't used systemd timers enough to disagree, but
> Ambiguous $PATH settings make cron script execution difficult to predict.
What makes you say that? You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
> You might feel cool knowing the scheduling grammar by heart
I've used Linux since 1994 and I don't know it by heart. But luckily it's pre-printed in the crontab as comments:
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
The rest of the complaints, sure. Next time I need a cronjob, I'll try it out.
"You just put numbers aligned with the titles."
That is not a fair summarization of their point because that is not the grammar. There's commas, slashes, asterisks, combinations, and then if you want randomization you need to put it in the command itself because cron can't do it. (Some crons can, but it's not a general capability of cron.) Writing a non-trivial cron spec is not easy.
I am familiar with the syntax, so I am biased ("*/3" and "12,14,20" makes sense if you are familiar with Unix tools), but it is still more intuitive to me than the systemd unit file syntax and usage. I know that I just have to edit /etc/cron or throw any executable file into /etc/cron.d/monthly and it will work on my system, but I cannot write a systemd timer file from scratch without looking it, and to do that I first have to find the directory where the other examples are located. /etc/systemd doesn't appear to be it.
This is generally my only real complaint about systemd. I don't care if it is too monolitic, written in C or whatever, I just want a straightforward syntax for straightforward operations. I'd like it if systemd could recognize if a .target file is a shell script and just do "the right thing". Perhaps it would make sense for a timer file to recognize cron syntax as well. Or at least allow for a kind of extensibility so that I can have it supported.
If systemd had a little more respect for existing conventions, I am pretty sure it wouldn't be so controversial. After all, system administrators like it because they use it all the time, but a regular, full-timer user like me, who only deals with it when something is broken or have to use it as a means-to-an-end to set something up, then all friction is annoying and bad UX. (And no, using Nix is not the solution)
If you want to create a new systemd unit file you can run:
or$ sudo systemctl edit --force --full my-scheduled-work@.timer$ systemctl edit --user --force --full my-scheduled-work@.timerWhy the @?
Systemd lets you create templates that take an argument in from the scheduled service. It gets that from the value after the @. So you can write a unit file that schedules a task to run say every 3 days and in that unit file reference `jobs/%i`, then put your task in a file in jobs and say `systemctl start every-3-days@script1.sh` to run `script1.sh` on your schedule without needing to create a new unit file for each script. StepCA has a nice write up on their site about using these templates to schedule cert renewals for any arbitrary service
Oh cool, thank you
Yeah, it would be nice to have a folder like /etc/systemd-jobs/ where I could put them and where there are no files unrelated to job scheduling. There is /etc/systemd/user, but it does get a bit of pollution depending on the system.
Not sure if you're talking about cron or systemd, but cron definitely has that in /etc/cron.d where you can have arbitrary crontabs, or /etc/cron.{hourly|daily|weekly|monthly} where you can just place arbitrary scripts if you don't care exactly when they run, just the frequency.
you can organize them however you want on your system and then use symlinks to make them available.
there's also `systemctl --all list-timers` to view them.
How do you express those things in a systemd timer? E.g. run something 4x per day, */6 in cron.
That's simple but consider "run something 4x per day but randomize a delay by hour so all of the 200 servers doing that task won't run it all at once"
In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
In systemd timers, it's just
and the offset generated will be stable for the job on a given machine (i.e. always same on this machine but different on others) so you will get nice uniform distribution of load.OnCalendar=0/6:00:00 RandomizedOffsetSec=60mIf you add
the job will also be run once if there was one or more scheduled runs when the machine was downPersist=true> In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
Nope. From crontab(5)
That's from my cronie install, but it looks like this has been a feature of some crons for at least a decade. (Notice that the post date of [0] is in 2016.) Given that cronie is based on vixie-cron, and I think I was was using vixie-cron in 2002, I bet it's been a thing for at least twenty years.The RANDOM_DELAY variable allows delaying job startups by random amount of minutes with upper limit specified by the variable. The random scal‐ ing factor is determined during the cron daemon startup so it remains constant for the whole run time of the daemon.still not a thing in vixie cron installed by default in Debian in 2026.
I'll take payout from your lost bet now
https://dyn.manpages.debian.org/experimental/cron/crontab.5....?
`* 1 * * * sleep $(( $(od -N1 -tuC -An /dev/urandom) \% 45 ))m ; <your command here>`
It's a mystery to me why everyone tries to use OnCalendar here, when "n amount of times within a certain timeframe" can be done much more easily with OnActiveSec, in this case that'd be OnActiveSec=6h.
Something like:
You can test it with:OnCalendar=00/6
The format is `DayOfWeek Year-Month-Day Hour:Minute:Second`systemd-analyze calendar --iterations=6 '0/6:00:00'https://www.freedesktop.org/software/systemd/man/latest/syst...
$ systemctl cat public-inbox-watch@.timer # /etc/systemd/system/public-inbox-watch@.timer [Unit] Description=Periodic fetch of public mailing list [Timer] # twice a day OnCalendar=*-*-* 5,17:35 RandomizedDelaySec=1h Persistent=true [Install] WantedBy=multi-user.targetSo the `OnCalendar` stanza is the same as a Cron job; without the helpful comments of the ordering? And that is considered _easier_??
Just use Cron. It does one thing and does it well.
The systemd one supports timezones. You can say America/Denver and the like, or Etc/UTC
1) Your production equipment doesn't have its TZ set to UTC? Enjoy dealing with the intermittent and irregular hassle of DST changes, I guess.
2) From the crontab(5) on my system:
If you have a job you need scheduled in a different timezone, dump a new file in /etc/cron.d, alter its CRON_TZ variable and go to town, as it were.The CRON_TZ variable specifies the time zone specific for the cron ta‐ ble. The user should enter a time according to the specified time zone into the table. The time used for writing into a log file is taken from the local time zone, where the daemon is running.
I, er, what? It's... the same as cron? I'm confused now. It's not exactly the same, I guess?
Sure, I'll pile on here. To do nontrivial scheduling you'd use the entirely-obvious-and-intuitive syntax described at [0]. For example:
Who'd ever want to go back crontab format for nontrivial scheduling? [1]Mon,Fri *-01/2-01,03 *:30:45[0] <https://www.freedesktop.org/software/systemd/man/latest/syst...>
[1] This question is sarcasm. SystemD is often like this... dead simple things look dead simple, but complex things are -if they're possible at all- at least as complex as they are everywhere else.
I found the systemd time spec syntax you referenced to be logical and well thought out.
Cron syntax is simpler for the easy cases because cron tries to do less. It ignores years and seconds entirely, and doesn't try to adhere roughly to ISO8601 ordering and field separators, instead using space universally for field separation and euro-style least-to-most significant field ordering. I like ISO8601, so I get along with systemd's style better, despite it introducing slightly more cognitive load.
The only thing that threw me for a loop and seems like "special magic" was
> "Mon *-05~07/1" means "the last Monday in May."
But good luck doing that in one line in cron.
Some cron-style libraries seem to support L/W/# for last / nearest-weekday / nth of month, but I don't know if any system crons do. (cronie? dcron? I don't think so. fcron? bcron? I don't see it there either.) '#' is syntactic sugar for DOW + 7-day range, while L is covered by the above quoted syntax.
If your cron has that kind of syntax, then for a case like "weekday closest to 1st of month", "W" is more convenient than writing 3 systemd timer rules to cover the three cases (weekday day 1, monday day 2, friday last day of month), but that's a big if. Generally you'd have to write 3 rules in cron anyway.
> I found the systemd time spec syntax you referenced to be logical and well thought out.
I found this amusing when in combination with
> The only thing that threw me for a loop and seems like "special magic" was
but -regardless- a careful reader notes that I never said that the Timer scheduling syntax was illogical or poorly thought out. It's at least as complicated as crontab-style time syntax, which was my entire point.
Related: Not that it's ether part of the core scheduler syntax or necessarily as nice as having it in the core syntax, but my crontab(5) suggests that one can use things like date(1) to get more fine-grained control over the time of execution:
While I expect that you're not one of those people, I know that folks who are accustomed to working with extremely inflexible tools forget (or never learned) that these sorts of things are possible. I'm very aware that people sometimes cut off their own limbs with power tools, but that's not a good reason to ban their use.As noted above, skip values only operate within the time period they´re attached to. For example, specifying "0/35" for the minute field of a crontab entry won´t cause that entry to be executed every 35 minutes; instead, it will be executed twice every hour, at 0 and 35 minutes past. For more fine-grained control you can do something like this: * * * * * if [ $(expr \( $(date +%s) / 60 \) % 58) = 0 ]; then echo this runs every 58 minutes; fi 0 * * * * if [ $(expr \( $(date +%s) / 3600 \) % 23) = 0 ]; then echo this runs every 23 hours on the hour; fi Adjust as needed if your date(1) command does not accept "+%s" as the format string specifier to output the current UNIX timestamp.Obviously you can do additional time & date checking in a shell wrapping the script or binary cron ultimately runs, just as you could do the time & date checking in the script or binary itself. That's far more complex.
Systemd enables cleaner, simpler syntax for common cases. Instead of "59 23 * * *", simply "23:59". Instead of "0 0 * * sun[day]", simply "sun[day]". 1st of the month and don't care exactly what time? Instead of "0 0 01 * *", simply "*-*-01".
Systemd started with the principle that they wanted to accept ISO8601 timestamps, then extended that with lists, increments, ranges. They developed that into a superset of basic cron capabilities, while maintaining similar syntax as best they could for increments and lists; they diverged for ranges and nth-from-last because those conflicted with the - date separator. They repurposed the little-used ~ cron randomization operator in the process. (systemd uses a separate RandomizedDelaySec line for that, which is also arguably superior because it randomizes per trigger, not on initial load of the timer)
The alternative is how AWS does it. They have separate cron() and at() syntax. at() takes timestamps. cron() takes cron+quartz syntax, mangled to remove seconds. They also have rate(value units), because apparently cron syntax is too complicated for most people. I'm sure in theory they did it to support "every 7 minutes" cases. I bet if you surveyed actual rate() AWS schedules, 99% would be cases that evenly divide a larger time unit, and could therefore be implemented with cron().
Cron syntax is more of a mess than I thought. While basic system crons don't support any advanced features, AWS and Cloudflare have adopted LW# capabilities, derived from quartz job scheduler syntax (Cloudflare references quartz explicitly). Quartz has extra fields for seconds and year (year optional, so it must go last, which breaks d m Y sequencing) so its syntax has 6 or 7 fields vs the standard 5. CF uses 5 fields, no seconds or years. AWS uses 6 fields, years but not seconds. Whenever you interact with cron-derived systems, you have to remember not just whether they support quartz-style syntax, but whether they support seconds and/or years. Is that really simpler?
The one feature I found interesting, "W", may not work the way I anticipated according to quartz's documentation. I thought "nearest weekday to the 1st of the month" might be useful because it maps to some billing cycles, but quartz (at least through 2.3) couldn't do that. The 2.3 docs say that 1W, if the 1st falls on a Saturday, triggers on the 3rd because it won't jump backwards over a month boundary. That was my only interesting use case for "W". Systemd does everything else, though it requires more verbosity in some cases, it's simpler in many others. 2.4/2.5 are from the last couple of years, and their docs are restructured and don't mention that, so maybe it was finally fixed? I'm not about to install java crud to test.
I didn't find the *-*~01 syntax that strange (last day of every month). The "fri *-*~07/1" syntax for "last friday of every month" is what I found magical, but it makes sense now that I've thought about it more.
What would you prefer for 3rd friday of the month? Cron (5-field, quartz syntax)
or systemd0 0 ? * fri#3 (special syntax is needed because cron uses logical-or to combine day and weekday fields)
[cf] https://developers.cloudflare.com/workers/configuration/cron...fri *-*-15..21 (works because systemd day and weekday are joined with logical-and)[aws] https://docs.aws.amazon.com/scheduler/latest/UserGuide/sched...
> Systemd enables cleaner, simpler syntax for common cases.
I disagree. It's all a matter what you're accustomed to. On top of that, what's at least as important as clear syntax for simple cases is the establishing of a handful of rules that well serve both the simple and complex cases. Consistency that leads to complicated cases being made easy should not be avoided just to make the simple cases extremely welcoming to the newcomer. Note that I'm not saying that 'systemd-crond' falls into this trap, but I am saying that oh so many tools that aim to be "approachable by newcomers" enthusiastically throw themselves into it.
> Cron syntax is more of a mess than I thought.
Nah. It's just there are a very large number of cron implementations. If The Systemd Project were actually a thing that people could make an independent reimplementation of, we'd see at least as much inconsistency in 'systemd-cron' scheduling syntax as people extended it in their reimplementations to meet their needs.
> What would you prefer for 3rd friday of the month? Cron (5-field, quartz syntax)
I don't understand why the "day of month" column has a "?" instead of a "*", and I don't understand why "fri#3" isn't rendered as "Fri/3", but whatevz. I prefer that to the systemd syntax.
FWIW, I think you can directly translate your systemd scheduler line into this for most crons:
Edit: Oh, no, you cannot. From crontab(5)* * 15-21 * Fri
Hm... Yeah, the handling of the "day of week" column is a terrible wart on the rules.Note: The day of a command’s execution can be specified in the following two fields — ’day of month’, and ’day of week’. If both fields are restricted (i.e., do not contain the "*" character), the command will be run when either field matches the current time. For example, "30 4 1,15 * 5" would cause a command to be run at 4:30 am on the 1st and 15th of each month, plus every Friday.I think you're talking about syntactic simplicity. Systemd requires three field separators (space, :, -) because that's how ISO8601 dates are written. That means not all fields will be space separated, which may be more challenging to visually parse. It's not going to look as clean. Is that what you mean?
Everything else is simpler. You don't have to count fields to figure out whether you're dealing with a day vs a month, or an hour vs minute, no matter how the fields are aligned or how long or complex they are. You can scan left and right and look for space and dash and colon to figure out where you are. The formats for time and date are distinctive and ubiquitous. You don't have to guess what you're looking at when you just see one of them in isolation.
Very little follows from cron's format. You have to learn which fields are which for your particular implementation: does it include seconds? You have to learn the bizarre interaction between day and weekday, to do something non-trivial with them. You have to carefully keep track of which field you're in because you can't determine that from neighboring syntax. The visual clutter, if you want to call it that, of systemd's iso8601-based syntax only applies if you're doing complicated rules. In the vast majority of cases it would simply be an ISO8601 timestamp with various values replaced with *'s, and at most one /.
That first impression you had is what I thought too, before diving into it for my previous replies. But no, cron logic is that day and weekday are ORed. It lets you do clever things like run a script on the 1st and 15th of every month, AND on mondays, all in one line. I don't think I've ever seen such a thing in the wild. I'd go so far as to say it's stupid, because it's likely to be misinterpreted if anyone did use it that way. All the other fields are ANDed. I would say that systemd's AND logic is better... and more consistent, which is your own acceptance criterion.
> It's not going to look as clean. Is that what you mean?
Nope. I don't give a shit how "clean" something looks, [0] I just care how difficult it is to learn and either retain or remind yourself of the syntax when you need to do something complex with it. In that regard, [1] systemd-crond's scheduler syntax is at least as complicated as that of most crons'.
> You don't have to count fields to figure out whether you're dealing with a day vs a month, or an hour vs minute, no matter how the fields are aligned or how long or complex they are. You can scan left and right and look for space and dash and colon to figure out where you are.
So, you have to learn a tool-specific format. Just like you do for crontab. I've read both man pages, and I'm 100% unconvinced that systemd-crond's scheduler syntax is actually simpler. I do agree that it can express specific schedules that one cannot in the syntaxes used by most cron implementations.
> Very little follows from cron's format. You have to learn which fields are which for your particular implementation:
Sure. And I expect that exactly nothing other than systemd-crond uses its weirdo syntax. And -as I mentioned in the comment you're replying to-
> But no, cron logic is that day and weekday are ORed. It lets you do clever things like run a script on the 1st and 15th of every month, AND on mondays, all in one line. I don't think I've ever seen such a thing in the wild. I'd go so far as to say it's stupid...Nah. It's just there are a very large number of cron implementations. If The Systemd Project were actually a thing that people could make an independent reimplementation of, we'd see at least as much inconsistency in 'systemd-cron' scheduling syntax as people extended it in their reimplementations to meet their needs.On this I agree with you. Changing the rules for the Day Of The Week column (but only when it's used in combination with the day of the month column) is catastrophically idiotic. The only time I've ever had need to use the DoW column was to schedule something to run weekly... so I've been blissfully unaware of how incredibly idiotic that behavior is.
[0] If anything, I've noticed a strong inverse correlation between how "clean" a syntax is regarded to be and how easy it is to both do complicated things with it, and come back to those complicated pieces of work six+ months later and be able to understand what the hell you did.
[1] With the notable exception of cron's DoW column.... read on for further discussion.
I take it that your position is: cron works for me, so why migrate to another system that uses its own "weirdo" syntax. Mine is: why not migrate since they're both "weirdo" syntax, and systemd's is at least based on ISO8601 and has integration with other systemd features.
Nobody is all that happy with cron's limitations. That's why there are so many variants, with varying features. When a variant has more than 5 fields, the syntax becomes ambiguous... not to a parser, but to whoever's reading and writing the lines, unless they already know the exact cron variant in use. Google, the one large cloud provider that seems to have stuck with basic cron syntax without quartz-scheduler extras, even invented their own pseudo-natural-language scheduling syntax to fill in for features cron lacks, or for people who don't like cron syntax.
Systemd's is definitely novel, but as I keep repeating, it's ISO8601 + cron. An alternative would be some extension to ISO8601 interval syntax, but that looks clunky and has probably never been used outside of niche applications for data interchange, probably inside xml files; it's used in https://docs.cibseven.org/manual/2.0/reference/bpmn20/events... (syntax at https://en.wikipedia.org/wiki/ISO_8601#Durations ). I think systemd's syntax is better than that.
The direct benefits, even if cron syntax suffices: Calendar timers are integrated with systemd, with all its process-management capabilities[1]. Unifying to systemd timers also avoids duplication of functionality ("is this scheduled process managed by systemd or cron?"). Cron doesn't let you `systemctl list-timers --all`. You'd have to grep through cron logs, possibly (and ironically) using journalctl, or have those logs already in a management dashboard, and that still doesn't let you see the next time the timer would fire. Cron also doesn't let you beta-test cron lines like you can with `systemd-analyze calendar --iterations 10 "Fri *~8..14/1"`
There's even a systemd-cron translator/generator that's in the official debian/ubuntu repos, and in Arch's AUR of course, which avoids the need for system cron but allows you to retain existing crontabs by importing them as systemd timers. They won't be quite as well integrated because it uses defaults for task names and other parameters, but the crontabs get run... without cron.
> I take it that your position is: cron works for me, so why migrate to another system that uses its own "weirdo" syntax.
I'll repeat my opinion, as expressed in my first post:
To do nontrivial scheduling you'd use the entirely-obvious-and-intuitive syntax described at [0]. For example: Mon,Fri *-01/2-01,03 *:30:45 Who'd ever want to go back crontab format for nontrivial scheduling? [1] [0] <https://www.freedesktop.org/software/systemd/man/latest/systemd.time.html#Calendar%20Events> [1] This question is sarcasm. SystemD is often like this... dead simple things look dead simple, but complex things are -if they're possible at all- at least as complex as they are everywhere else.System cron can't implement your example, so I guess what you say is technically true... EINVAL is more obvious and intuitive than <systemd calendar incantation>. Cron just can't do what you want.
You keep saying systemd calendar format is "at least as complicated", but it seems to me the converse is true. Simple cases are simpler to represent in systemd because fields are optional in various circumstances; complex cases, where they can be implemented in system cron at all, look equivalently complex. Removing the weekday constraint, rounding to minutes, and removing the star for year since it's optional:
What's the point of your gripes? Would you really suggest to someone, who has a systemd-based distro, to spin up systemd-cron or cronie and keep writing crontabs? Do you have some specific complaint about systemd calendar syntax that you think could be improved?31 * 01,03 01/2 * 01/2-01,03 *:31
> But good luck doing that in one line in cron.
Two options:
or, if your cron supports „L“:0 0 25-31 5 10 0 L 5 1Won't your first line mean every Monday in May as well as days 25 to 31 of May?
At least busybox's cron implements it that way:
if (line->cl_Mins[ptm->tm_min] && line->cl_Hrs[ptm->tm_hour] && (line->cl_Days[ptm->tm_mday] || line->cl_Dow[ptm->tm_wday]) && line->cl_Mons[ptm->tm_mon] ) {Woah, really? Usually in crontab lines everything has to match. I consider this to be a bug.
But first I‘d check the behavior of other crons ;-)
The first one works in that specific case, but not more generically. For example, "Last Monday in February", or "last Monday of the month" for multiple months unless they're all 30 or all 31 days.
This was fun to cook up and may (or may not!) break if one's locale changes:
I bet one could do something similar to determine if we're at the "last $NAMED_WEEKDAY in the month" by counting ahead a week and seeing if the month name changes.# Run a command on the last day of the month. Only starts checking at midnight towards the end of the month. Assumes GNU date, which is fair if we're discussing a Linux-only cron-alike. 0 0 28-31 * * if [ $(date +\%d) -eq $(date --date="$(date +\%m)/1 + 1 month - 1 day" +\%d) ]; then /usr/local/bin/runCommand.sh; fiIf I were doing this for real, I'd either switch to a more capable cron, or take a serious try at the date math and then wrap it up as a standalone helper. Or I guess I'd look to see if someone already built that helper. ...I guess...
I am curious whether there's a more capable system cron that supports that kind of thing. Quartz job scheduler (java), AWS, and CF obviously wouldn't qualify. I think this is only possible if you're using a heavyweight job scheduler like those. Or... systemd.
It ultimately doesn't matter "for real", because almost nobody without some horrific legacy system to integrate with would need to use anything more than lists, ranges, and increments. There's a reason nobody's added these features to system crons. Clever trigger times for events that don't really need to be triggered at clever times... sure, but lacking that capability wouldn't change anything, the trigger would just be a more boring one.
Events like "last Friday of the month" or "nearest weekday to the 1st of the month" or even "Friday the 13th" are more for business logic, not system crontabs.
> I am curious whether there's a more capable system cron that supports that kind of thing.
I am no cron scholar, nor am I young enough to have the energy required to do an exhaustive survey... so -sadly- I don't know of one. It seems like there'd be one being used internally in at least one business in the world, though, yanno? That just seems like the sort of thing that at least one bored programmer would take a few days to crank out.
> Events like "last Friday of the month" or "nearest weekday to the 1st of the month" or even "Friday the 13th" are more for business logic, not system crontabs.
Oh quite possibly, yeah. But, system crons are quite fine for one-shot things that are re-run on a regular schedule... as well as things that don't have complicated scheduling and/or retry requirements. If you're working on a Big Enterprise Project [0], then you're almost certainly going to have a scheduler inside of you, and -IME- you're very likely to use it to do a lot of that BEP's scheduled tasks. [1]
[0] Or a tiny one wearing the clothes of a BEP
[1] ...if for no other reason than it requires little effort to get the BEP's data and code into its internal scheduler, and can be a huge pain in the ass to get it into an external one.
To be honest it looks rather easy to digest? I can sort of guess the meaning without documentation.
For cron, despite years of looking at it, I can never remember what those numbers mean and I need to Google almost every time.
If you know the syntax, it's still actually rather trivial. Still easier to read than advanced cron magic.
> Still easier to read than advanced cron magic.
Looking at the other examples on that page, I'm gonna say that it's only arguably easier to read for basic stuff... especially if you're familiar with the syntax. The complex stuff is -at best- just as difficult.
Having had to work on an application supposedly supporting cron expressions: the numbers are just the basic parts of the language.
When someone inputs something ridiculous like "5,3/4 4-8,11 1 4,5,6,9-11 */2" you get to enjoy the fun of reverse engineering what they meant (it's never what they actually wrote).
And that's before you get to all the extensions supported in some cron environments (but not all).
I find systemd timers a lot more manageable. Things like having control over whether or not long-running jobs are allowed to overlap and the ability to run tasks between start-finish rather than a fixed time window are major improvements for me. At some point my VPS went down because the backup job ran into some kind of symlink loop and cron just kept spawning more and more backup tasks even though none of them finished.
Having to re-write commands and scripts because CRON had its own special PATH was also a pain point, but the same can be true for some types of systemd timers. But: you can execute those timers manually if you want instead of updating the crontab to trigger in 30 seconds and simply waiting.
Per other comments though, it looks like systemd's syntax when you want to specify something that's not just that one number is at least as complex.
Is your example (which I agree, looks cryptic) any less cryptic in systemd?
I asked jippity, and it said this:
To which I have to go: "what?"[Timer] OnCalendar=*-04,05,06,09,10,11-01 04..08,11:03/4,05:00 OnCalendar=Sun,Tue,Thu,Sat *-04,05,06,09,10,11-* 04..08,11:03/4,05:00> Things like having control over whether or not long-running jobs are allowed to overlap
With cron that's just prefixing the command with `flock -n <lock>`, but sure the "pick somewhere to put the lock" is probably better with systemd.
> Having to re-write commands and scripts because CRON had its own special PATH
Why? Wouldn't you just put that in the crontab? I don't even see this as different. It's in the cron config or the systemd timer config.
The other improvements you mentioned seem good.
This is like a complaint about regex syntax. It's impossible to comprehend a non-trivial regex in a second or two. However, if you know the rules, it's trivial to step through it. What's the point of complaining? There's no representation that anyone could grok on first impression. This is much simpler than regex.
Nobody's prevented from using cron instead of systemd timers. The significant differences in typical relatively simple cases are ordering:
cron: M H d m Y DOW
systemd: DOW Y-m-d H:M:S [each part optional, with *, *, and 00:00:00 defaults]
And then, because - is taken, ranges use .. in systemd. Aside from that, it's mostly the same for typical cases of simple periodic timers. Even x/n and x-y/n for steps work similarly. Syntax for complex cases start to diverge, for jitter or special numerically-irregular DOW or DOM or multiple non-periodic times.
In your example, adding more spaces between the date and time parts would make it more visually digestible. There's also the .. range operator which jippity strangely didn't use for the month field even though it did for the hours field.
Yup, entirely agree. That's why I thought it odd of jeroenhd to bring it up as if it's an argument against cron or cron time specs.
From what I see not better or worse, it's just different.
> 5,3/4 4-8,11 1 4,5,6,9-11 */2
What's so hard about "At 5 minutes past the hour and every 4 minutes, starting at 3 minutes past the hour, at 04:00 AM through 08:59 AM and 11:00 AM, on day 1 of the month, every 2 days of the week, only in April, May, June, and September through November"?
(I used https://crontab.cronhub.io/ to decode it, to be fair)
Complex expressions are one of the things I don't like in cron. On Debian/Ubuntu servers, I just bite the bullet with systemd timers. On my workstation, I have a personal job scheduler that feels easier and more fun to tinker with. The scheduler uses Starlark functions instead. For example:
This was inspired by GNU mcron. In mcron, jobs can calculate the next time they should run using Guile (https://www.gnu.org/software/mcron/manual/mcron.html#Guile-S...):# Run if at least a day has passed since the last run # and it isn't the weekend. def should_run(finished, timestamp, dow, **_): return dow not in [0, 6] and timestamp - finished >= one_day
I found mcron's scheduling counterintuitive and decided I wanted a function that returned a boolean. I can tentatively recommend it.(job '(next-minute-from (next-hour (range 0 24 2)) '(15)) "my-program")
The main nice thing about the environment in systemd is that it is standard and mostly a blank slate, whereas at least for me I was always getting bit by the fact that the environment in Crontab was completely different from say, the environment inherited by supervisord or sysvinit scripts. In systemd the actual unit that gets executed is the same regardless of what triggers it, so there is no gap.
That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
I'd have to concur that I agree this is an advantage of systemd.
> That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
Odd. This script
when scheduled like so#!/bin/bash set > /tmp/set.txt
Produces this file in /tmp/set.txt which has had a handful of values (HOME, UID, etc) lightly redacted prior to posting here -to remove PII or for length- but its keys are entirely untouched:* * * * * $HOME/bin/testCronScript.sh
Seems pretty clean to me. Even when I run this via /etc/crontab, rather than as a user cron job:BASH=/bin/bash BASHOPTS=<redacted because long> BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() BASH_LINENO=([0]="0") BASH_LOADABLES_PATH=/usr/local/lib64/bash:/usr/lib64/bash BASH_SOURCE=([0]="/home/user/bin/testCronScript.sh") BASH_VERSINFO=<redacted bash 5.3.x> BASH_VERSION=<redacted bash 5.3.x> DIRSTACK=() EUID=13370 GROUPS=() HOME=/home/user HOSTNAME=hostname HOSTTYPE=x86_64 IFS=$' \t\n' LANG=en_US.utf8 LOGNAME=user MACHTYPE=x86_64-pc-linux-gnu OPTERR=1 OPTIND=1 OSTYPE=linux-gnu PATH=/usr/bin:/bin:/usr/sbin:/sbin PPID=1337 PS4='+ ' PWD=/home/user SHELL=/bin/sh SHELLOPTS=braceexpand:hashall:interactive-comments SHLVL=1 TERM=dumb UID=13370 USER=user _=/home/user/bin/testCronScript.sh
I get effectively the same results.* * * * * root /home/user/bin/testCronScript.shMaybe your distro's default cron environment was bad, and you never bothered to check and unset the badness? I'd be surprised if they were unable to make the default environment for Timer Units to be bad.
Regardless of exactly how clean the environment is, my favorite part of systemd is the fact that there is only one regardless of how something was triggered. Whether a unit is triggered via a mount unit, timer unit, udev rule, it's the same units at the end, so it's the same environment.
The same problems that could be caused by a polluted environment in cron can be caused in reverse by a polluted environment elsewhere, when you unwittingly copy a command that depends on some environment being set. If you are using systemd as the service manager, this necessarily doesn't happen because it's all units. (Well, you could still copy something from outside of systemd and run into a similar problem, but at least there's essentially only one set of caveats you have to learn for whatever thing you want executed in the background.)
So I guess this isn't so much cron vs systemd timers, but more cron + other init and service supervisors vs systemd init in general.
> Regardless of exactly how clean the environment is, my favorite part of systemd is the fact that there is only one regardless of how something was triggered. Whether a unit is triggered via a mount unit, timer unit, udev rule, it's the same units at the end, so it's the same environment.
>
> The same problems that could be caused by a polluted environment in cron can be caused in reverse by a polluted environment elsewhere, when you unwittingly copy a command that depends on some environment being set.
I'm confused about what you need this for? Are you running some utility command that needs the same environment provided by the daemon's service file? If so, any competent init system lets you extend upstream-provided service files. In OpenRC:
The environment when the service is starting is effectively identical to the one when our custom function is being called:# tail -n 1000 /etc/*/test-service ==> /etc/conf.d/test-service <== extra_commands="${extra_commands} maintenance" OTHER_THING="overriden other-thing" maintenance () { ebegin "doing maintenance. IV='$INIT_VAR' OT='$OTHER_THING'" set > /tmp/maintenance-set.txt eend 0 } ==> /etc/init.d/test-service <== #!/sbin/openrc-run name="test-service daemon" command=/usr/bin/socat command_user=nobody:nobody command_args="UDP-RECVFROM:6666,fork SYSTEM:'/bin/true'" supervisor=supervise-daemon extra_commands="rebuild" INIT_VAR=${INIT_VAR:-"init var"} OTHER_THING=${OTHER_THING:-"stock other-thing"} depend() { use net } start_pre() { set > /tmp/start-set.txt } rebuild () { ebegin "doing rebuild. IV='$INIT_VAR' OT='$OTHER_THING'" eend 0 } # /etc/init.d/test-service start test-service | * Starting test-service daemon ... [ ok ] # /etc/init.d/test-service maintenance test-service | * doing maintenance. IV='init var' OT='overriden other-thing' ... [ ok ] # /etc/init.d/test-service rebuild test-service | * doing rebuild. IV='init var' OT='overriden other-thing' ... [ ok ] # pgrep --list-full socat 133705 /usr/bin/socat UDP-RECVFROM:6666,fork SYSTEM:/bin/true
So, if you need to do maintenance for a service on a schedule in the same environment that is provided for starting that service, you can simply extend the service script and use cron to execute that functionality.# diff -u0 /tmp/*-set.txt --- /tmp/maintenance-set.txt 2026-06-02 23:53:19.703048431 -0700 +++ /tmp/start-set.txt 2026-06-02 23:53:15.265094855 -0700 @@ -9 +9 @@ -BASH_LINENO=([0]="410" [1]="0") +BASH_LINENO=([0]="409" [1]="0") @@ -11 +11 @@ -BASH_SOURCE=([0]="/etc/init.d/../conf.d/test-service" [1]="/usr/libexec/rc/sh/openrc-run.sh") +BASH_SOURCE=([0]="/etc/init.d/test-service" [1]="/usr/libexec/rc/sh/openrc-run.sh") @@ -30 +30 @@ -FUNCNAME=([0]="maintenance" [1]="main") +FUNCNAME=([0]="start_pre" [1]="main") @@ -64 +64 @@ -PPID=133712 +PPID=133702 @@ -69 +69 @@ -RC_CMD=maintenance +RC_CMD=start @@ -73 +73 @@ -RC_OPENRC_PID=133710 +RC_OPENRC_PID=133700 @@ -75 +75 @@ -RC_RUNSCRIPT_PID=133711 +RC_RUNSCRIPT_PID=133701 @@ -93 +93 @@ -_='doing maintenance. IV='\''init var'\'' OT='\''overriden other-thing'\''' +_=']'But. Another thing that confuses me is why you think that SystemD [0] provides anything special here? If you were to create a service file in most any other service manager and start it with cron, you'd get exactly the same environment sanitization as you get for all other services. Given your testimony, I expect that prior to SystemD, you'd have refused to create service files for things like one-off jobs that weren't system services... so why are you okay with it now that you're using SystemD?
[0] I spell it "SystemD" not to mock it -as I understand some do- but to distinguish The Systemd Project from systemd(1). It sucks minor ass that the two share the same name, but what can you do?
I use cron in OpenBSD and it's a deterministic environment and mostly clean[0]. I like that instead of having other subsystems creep in.
> What makes you say that? You can set the PATH right in the crontab.
OK but I don't want to hardcode $PATH in the crontab just so I can test the cronjob. Barring the hardcode, $PATH is one thing when cron runs and another when you try out the command yourself. systemctl start foo.service starts the command inside with the same environment as when the timer fires so you know it'll work the same.
On the flip side, your cron job will run at the time you specify in the crontab. Your systemd timer, on the other hand, may fire at the specified time (and most of the time, it will), but it can also suddenly stop firing once it has fired on a February 29th and then never fire again, due to logic bugs in systemd, or it may or may not fire when you "restart" the timer unit, due to logic bugs in systemd (that's when it only has OnCalendar, so yes, definitely a bug).
> $PATH is one thing when cron runs and another when you try out the command yourself.
Why would that be different with systemd timers? If my ~/.bashrc adds /opt/foo/bin, that's also not part of the systemd timer's PATH, right?
But I guess you're saying the ability to trigger the systemd timer off-schedule is the difference? Yeah, it's annoying with cron to have to temporarily set the trigger two minutes into the future. :-P
Not sure adding that feature justifies a complete rewrite, but certainly a nice addition.
> due to logic bugs in systemd
Yeah my main gripe with systemd and other Lennartware is the extremely low implementation quality, not necessarily the ideas. Though the idea of killing tmux/screen on logout is downright criminal. And the fd passing nonsense[1] for system services is clearly just the idea of a child that found a tool and is misusing it.
[1] which is an awesome and underused feature (https://blog.habets.se/2025/10/The-strange-webserver-hot-pot...), but completely misapplied by systemd.
>But I guess you're saying the ability to trigger the systemd timer off-schedule is the difference? Yeah, it's annoying with cron to have to temporarily set the trigger two minutes into the future. :-P
>Not sure adding that feature justifies a complete rewrite, but certainly a nice addition.
If there is a feature that justifies using a completely different tool it's obviously this one.
Are you making the assumption that it's impossible to add that feature to cron?
You should not see a missing killer feature and go directly to the conclusion of rewriting.
Sure, greenfield is fun. So there's that.
> Why would that be different with systemd timers?
Because you can test if systemctl start foo.service works and if it does, you know it'll work when foo.timer triggers it.
> my main gripe with systemd and other Lennartware is the extremely low implementation quality
It doesn't stack up that poorly against other projects but it's unacceptably low for a core system component.
> Because you can test if systemctl start foo.service works and if it does, you know it'll work when foo.timer triggers it.
It's a better workflow than the one I mentioned for cron, yes, but it's nothing inherent. I would not call the difference one of being "predictable", though.
The article and thread is full of complaints about cron that are either just missing features that could be added (like this one, trigger on demand), exactly the same ("the time spec is too complex"), or just plain "no you can already do that with cron".
It wasn't until this comment (https://news.ycombinator.com/item?id=48385824) that I saw viable reasons why one would need to start from scratch.
Except, of course, that "green field is more fun than improving things".
> I don't want to hardcode $PATH in the crontab just so I can test the cronjob
How is this different? Any way you slice it, you have to configure it somewhere if you're not lucky enough for the default to be acceptable.
> [Lennartware] doesn't stack up that poorly against other projects but it's unacceptably low for a core system component.
Well put.
> How is this different? Any way you slice it, you have to configure it somewhere if you're not lucky enough for the default to be acceptable.
It's a workaround for a design flaw. If you forget it you'll notice when it fails in production. There should rarely be a need to change the default path (just specify the full path if necessary), it should just be the same when you test and when it runs by itself. Trigger on demand would of course solve that.
> exactly the same ("the time spec is too complex")
Of course, the timespec complaint is bullshit. Both have manpages that will tell you immediately if you don't have it memorized.
> It's a workaround for a design flaw.
That's what I'm saying though; how is it a design flaw? It's arguably a flaw in the default configuration. That's certainly fixable without changing the design at all.
> Trigger on demand would of course solve that.
Is it a design flaw that you can't trigger the job as a test? No, that's just a missing feature.
I wouldn't say that the PATH is ambiguous, but cron does have some problems with PATH:
- the default value is missing some values you would expect, like /use/local/bin and /usr/sbin for root.
- on some distributions (for example Arch Linux) the man page doesn't even say what the default path is, or recommend setting it.
- if you need to add something to the path for a single script, you either need to wrap it with a call to env, set it in a wrapper script, or set the path before the entry and reset it afterwards
- you can't use ~ or $HOME in the path, you have to write out the full absolute path. Which is particularly annoying for user crontabs.
Sure, it isn't too hard to work around those, but IMO systemd timers are a better experience, especially since the default uses the same path as all your other services.
> - the default value is missing some values you would expect, like /use/local/bin and /usr/sbin for root.
What do you mean by "you would expect", that doesn't also apply to systemd timers? /opt/foo/bin is not in the path. Would you expect that?
And if this is an objective problem, can we just change the cron default PATH?
> - on some distributions (for example Arch Linux) the man page doesn't even say what the default path is, or recommend setting it.
Send a PR. This doesn't seem like an inherent problem.
> - if you need to add something to the path for a single script, you either need to wrap it with a call to env, set it in a wrapper script, or set the path before the entry and reset it afterwards
Or on the line, right?
The line can get long, but is this really a problem?* * * * * FOO=bar $HOME/bin/foo.sh> - you can't use ~ or $HOME in the path, you have to write out the full absolute path. Which is particularly annoying for user crontabs.
This is incorrect. You can definitely use $HOME in user crontabs.
I'm still not seeing something that warrants a rewrite. (except what you did not mention, which is the ability to run "trigger this now" as a missing feature)
> What do you mean by "you would expect"
I mean it should include things that are usually on the path, like /usr/local/bin. And for the root user it should include sbin.
I don't really expect /opt/foo/bin to be there, but if you want to have it available by default everywhere then you can just add it to systemd once, rather than having to remember to add it to crontab as well.
> Or on the line, right?
Oh you're right. I forgot cron evaluates the command with the shell (which has its own set of issues...)
> This is incorrect. You can definitely use $HOME in user crontabs.
You can in the command itself, because that is evaluated by a shell, but not when setting an environment variable. From the Ubuntu crontab(5) manpage:
> The value string is not parsed for environmental substitutions or replacement of variables or tilde(~) expansion
Maybe it depends on the cron implementation though. The cronie documentation doesn't say either way.
> I mean it should include things that are usually on the path, like /usr/local/bin. And for the root user it should include sbin.
So changing the default PATH for cron requires a green field rewrite? You would expect /usr/local/bin. cron doesn't include it. It seems like a disagreement on default settings. I would agree with you about what the default should be, but I don't understand how this has anything to do with cron either as code or architecture.
This seems a bit like /sbin being missing in the default PATH for the root user's regular shell being a reason to rewrite bash to systemd shell.
> So changing the default PATH for cron requires a green field rewrite?
I never claimed that.
It's just one of many reasons why I think systemd timers are easier to use.
> You can set the PATH right in the crontab.
Yes, but people don't. I've had to debug other's crontabs many times over the last umpteen years.
But how is that different with systemd timers?
Two environments that set the PATH differently won't have the same value set, either way.
Is this about "yes, but cron doesn't let me trigger through its environment, and systemd timers do"?
problem with vars is that they apply to any subsequent entry in the file so you need to take that into consideration; the nice thing about timers is that all settings are self contained and not affected by previous entries. The standard /10 and similar cron expressions also have thundering herd problem when on bunch of servers, tho some variants like in Jenkins use variant H/10 (H standing for hash) where the thing is randomly shifted in time to not hit same minute on same server/job
another benefit is having logs in one place for the job; cron's "send a mail when there is any amount of output text" is just annoying behaviour, but also only place to get the job output unless you redirect it somewhere. Also starting from timer vs just doing systemctl start job.service is the same so easier to debug
other than that the few improvements in how to specify run time have been pretty useful.
For example, setting timer as "persistent" will mean any run "lost" to machine powered off will just be ran next time after boot, so you can have job on your PC that is just "run backup at 2AM" and if you turn it off before that you get the backup done first thing in the morning
There is also both random, and fixed (depending on machine UUID) random delay so avoiding thundering herd problem with backups is also pretty convenient.
There is even option to wake a device for the job if necessary tho the problem of shutdown is left to the user. And picking whether to start counting to next timer from previous one or from the job's end.
What I would like also is to have job summary page ("hey this job was done X times but failed Y times") but that's probably better left to external tooling
> You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
There is* a common trap as the cron PATH is usually just /usr/bin:/bin so anything in /usr/local/bin, or in /sbin won't be there.
> There is* a common trap as the cron PATH is usually just /usr/bin:/bin so anything in /usr/local/bin, or in /sbin won't be there.
There will always be a default. Is systemd timer's default inherently correct for all users any more than cron's is?
I'm just playing devil's advocate here, but why not just change cron, then? Good for the goose is good for the gander?
> But luckily it's pre-printed in the crontab
Way easier.man 5 crontab> I've used Linux since 1994
Same here.
We are now considered old and therefore irrelevant. The new generation uses timers and couldn't care less about cron that has served us just fine for decades.
I use cron and my general attitude towards LP and systemd is very similar to the attitude of LP and systemd to us.
Total n00b here. My first linux install was pretty recently, in late 1996 or early 1997 (sometime that winter).
I just don't get it. Like is the core sentiment "How dare they address obvious system shortcomings"? Is it "I learned once and how dare you think I'm capable of learning again"? Is it "I want others to suffer the way I did to learn job scheduling"?
cron did a job, but had shortcomings. Systemd addresses many of those shortcomings. One day something else will come along and address the shortcomings of systemd, and no one will care about systemd nostolgia. This is how technology is supposed to work: making progress and fixing the shortcomings of the past generation. It's not a religion, we don't have to maintain the weird old ways from the 80's, your soul won't be saved by cron or corrupted by systemd.
Yeah, I certainly have my complaints about systemd but the parent's point is undermined by the fact that cron still works. If you prefer it, carry on I doubt seriously it's going anywhere. I still do sometimes.
I am not agreeing with egorfine. Indeed, why not improve what we can improve?
> cron did a job, but had shortcomings. Systemd addresses many of those shortcomings
Right, but what are those "many" shortcomings? The article lists four, and fully half of them seem to be nonsense, per my comment. (time spec syntax appears to be equally complex in systemd timers, and I have no idea what they mean about PATH, as it seems equal too)
The remaining two are fairly good points, kind of. Sending mail is a black hole until you look there, sure. Believing that emails get meaningfully delivered on a non-email server is very anachronistic. But isn't logging a black hole until you look there too?
So from the article that leaves "Execution history is difficult to follow and interrogate", which I super agree with. I would argue it's not inherent to cron, and one could have written a small tool that allows following and interrogating.
And… surely that goes for logging too? cron does log to syslog, right?
Maybe there's some integration with other stuff that is better, that I'm not aware of?
I'm not disputing that it's better. I'm sure it is. But where's that list? If it's literally only the `list-timers` command, then that's very underwhelming. Is it the randomness to scheduling? I've never needed it (I just spread them out over an hour by fixed start time), but sure that would have value in other cases where coordination is not possible. You could add it to cron, but not in a nice way.
To me it seems like engineers do what engineers like to do: enjoy greenfield implementations. It's open source. Nobody's going to ding your quarterly performance evaluation for going off on a amusement coding session without a requirements doc. I know that I have written a lot of tooling for amusement and to work the way I want it to work. I certainly understand the engineering mind that would rather write something from scratch than understand the previous system.
I don't mind learning new things, but this article seems to fawn over stuff you can do in systemd timers, where… yeah that was always an option (in cron). I don't have faith that the article writer actually knew cron before they trash it in favor of something else.
On a tangent, I do agree with egorfine that Lennartware inherently has a disdain for users and their workloads (e.g. kills user processes inside a screen/tmux arbitrarily), and that audio on linux was set back 5-10 years just from the mere disastrously bad quality of the pulseaudio implementation. It makes sense that he works for Microsoft.
Well theres a few more:
* coalescing jobs with control over the granularity of it. That means you can say "i want this job run on at 14:30:02 exactly" and I want these jobs run at 19:21 or so, 19:22 or so: and 19:23 and set your resoultion to 10m and they'll all run at once. Great on laptops and other scenarios where you want to reduce power draw.
* System wakeup - you can wake a system from sleep various sleep modes (details depend on hw support) and run a job.
* cohesion with the rest of the system. this is a big deal when you stop playing with just your desktop and pet server and have to deal with 10^4 or more servers. having to deal with the wierd quirks of cron vs inittab vs whatever is frustrating and when there are many people working on it, someone is always going to do something quirky and fragile. Yes you have to know the systemd things, but that's it - a timer starts a unit, any unit, without all the bullshit (e.g, oh im stating with cron, these magic invocations are neeeded, oh im starting it with runit and these different invocations are needed, etc)
I literally never experienced any of the problems people complain about for pulseaudio - at the time it was released it was the smoothest audio experience i ever had on linux. I think some people just want to look cool and complain about the new thing.... But also I read manuals and think for 3 or 4 seconds before doing things, so maybe it has something to do with that.
I agree that's a good list. You did a better job than the article.
As for pulse audio: no, it has absolutely nothing to do with that. That's just being condescending and smug. I've never had a lithium battery fire, so that must mean manufacturing defects never cause them?
I get it. Some software had a couple bugs at first, as does all new software. And now some whiny people who have never, don't currently, and will never provide value to the open source ecosystem have spent more years whining and complaining about the bugs than the entire piece of software was in widespread use.
Ok, now that you're just commenting cope and ad hominem I think we're done here.
> How dare they address obvious system shortcomings
I'm all for it.
However the trust in systemd brand and its leader is deeply negative, so anything that comes from systemd folks is an immediate no.
Wierd. I like systemd. It's given me more stability and control over my systems than anything before provided. I like pulseaudio - it made the linux audio experience better than anything that came before it.
I don't live in terror of new things though, so I don't really understand the propaganda.
> pulseaudio - it made the linux audio experience better
This is a take that is so drastically different to what I (and many other people) have experienced that it now makes sense that systemd is to your liking.
Yeah, it's sad that a few dozen very vocal people got upset that they would have to read the manual and maybe get rid of some of the hacky nonsense they cobbled together to get an equivalent experience to what default pulse provided. Those people have spent decades whining about imagined issues and preventing reasonable discourse about actually good software.
> But luckily it's pre-printed in the crontab as comments
That's true, but most people don't know the numbered manual sections, so they get the docs for the cron table command not the cron table config file.
> That's true, but most people don't know the numbered manual sections, so they get the docs for the cron table command not the cron table config file.
No `man man`? ;)
Man Man: the man with the strength of two men.
A man who was bitten by a radioactive man.
Moved from cronie to systemd timers because they are resilient to system startup times. My backup strategy is to create a borg archive entry every day at a fixed time. With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.
Btw this is my repo for the backup automation: https://github.com/gchamon/borg-automated-backups
Cronie has a mechanism for this, called "anacron", which is called hourly by cron (on my system, /etc/cron.hourly/0anacron), and performs all the /etc/cron.{daily,weekly,monthly} tasks, no matter if the earliest possible schedule was missed (and with a configurable random delay). You can modify /etc/anacrontab to create custom schedules.
To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this.
Many cron implementations have a similar mechanism.
EDITED
This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
> fallback to running when the system comes online.
That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then.
> With @hourly I lose this control and multiple machines could potentially trigger backups at the same time
Then don't use @hourly, use staggered times, it's very easy.
> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
There are two options to fix it;
Disable persist so no catching up on missing scripts. Set OnBoot=5m so it gets ran 5 minutes after boot, so your script (say backup) is ran on boot first, then every time on schedule
Enable persist but just add sleep in ExecStartPre - very "cron" way but there is just no in-systemd option to enable "catch up" script to be delayed
Sadly no option to "run catch-up timers with delay" at least yet
> Then don't use @hourly, use staggered times, it's very easy.
Not in cron. In systemd it's just RandomizedOffsetSec=30m and it is "stable" - same host with same job will always have same delay so on multiple hosts it is spread nicely. There is also non-stable version
> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.
I interpreted it more like "I have these 500 different cronjobs all spread out across $unit_of_time. If the system is down for longer than $unit_of_time and then comes back, does all 500 jobs start running instantly (since they missed their previous deadline)?"
Just to be clear, this isn't default systemd timer behaviour, you need to opt in by setting Persistent=true. If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool because at that scale you'd surely need better observability
You could implement this with a gitlab instance in a separate system, like two VMs in proxmox or two physical machines, and a shell executor running in them. Gitlab CI has a nice feature to limit concurrency by using resource groups. Say you have 500 jobs spread through the day and the system stays offline for a while, when it comes online it'll start processing the jobs, but will only run a single one at a time. You get visibility, logs, queue monitoring and an API to query data.
> If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool
Eh sometimes, but you can get pretty far with one of two approaches:
1. Careful use of Requires= and Wants= to group your scripts into chains of jobs, which achieves fixed parallel (though at 100s of jobs, I hope you're generating those unit files with a tool like Puppet or https://github.com/karlicoss/dron or something and not doing this by hand).
2. Even better, just use a lockfile. `ExecStart="flock -F $TMPDIR/mylock <command>"` is pretty hard to beat. Use -F so as not to confuse KillMode and resource accounting and you're golden. Just don't use flock(1) timeouts; let systemd handle that. Heck, if you have that many cron jobs, you should be doing this even if you don't use systemd; otherwise job latency changes can cause reboot-style thundering herds out of the blue.
If you need semaphore behavior and still don't want a real job queue, waitlock (https://github.com/bigattichouse/waitlock) and many other CLIs have you covered.
1. This is spread across 500 files, maintainability goes out the window
2. If this for some reason fails, misconfiguration or unexpected shutdown, you could have a failure that's hard to track or debug
These are fine with a few services chained together, but this requires a shallow depth of dendencies. To have these theoretical hundreds of jobs chained together like this isn't practical or safe.
If you have 100 different jobs that were supposed to run over the past week, but didn't because offline, when you restart, they they all flood the system on start.
100 jobs all running at different times throughout the week is a very different load than them all falling back and running at the same time on system boot.
I don't, it's a single backup service.
Cron also has @reboot. Not exactly the same, but has been sufficient for me so far.
Snooze offers similar resilience to the system being offline: https://github.com/leahneukirchen/snooze
I find it very easy to reason about: a single process maps to a single recurring task.
It can track last execution into a file, yielding durable schedules when the host is offline.
I'm sorry, I tried Googling the word "tolates" but I can't find any definition that makes sense?
> runs the service as soons as the system is available.
cron has the @reboot option which I use for a few scripts and works great.
Typo, I meant tolerates. Fixed it.
Not an option either, because if I reboot two machines and the backup starts in both of them it'll cripple my NAS
How does systemd on the 2 machines avoid that? Are they communicating somehow?
No, just different cron schedules. If I just reboot a machine the job doesn't get triggered, only if I start a machine after the cron schedule should have been triggered. To be fair, if I start two machines in these conditions this will happen too, but such situation is much more manageable than rebooting too machines in a short period of time.
"tolerates".
> With cronie the system needs to be running at the scheduled time, but systemd timer tolerates this and runs the service as soons as the system is available.
Cronie doesn't have a `@reboot` meta-trigger?
I have a Canon printer, I actually can't trust that their print nozzle won't get jammed up after sitting idle for a while. So I had claude setup a systemd script to print a picture of my dog every week, I ensure it has enough CMYK spectrum to stress the printer. Its a nice surprise every monday as I sit on my desk to see a sudden picture pop up from the printer :)
I wish printers could have a mode like this to print random images from an album, or a calendar, rather than wastefully draining ink into a sponge every few days.
If nothing else, maybe it could be some kid's high school science fair project idea.
How about printing a QR code for a randomly generated private key for Satoshi Nakamoto's Bitcoin wallet, then every few days you get a tiny moment of excitement, hope, and then disappointment. It's still wasteful, but it could pay off big time?
this is an amazing youtube video idea if you could get a type writer to do it.
Maybe I'm misremembering, but I'm sure there was something on HN a few weeks ago about an electric typewriter that someone had connected to (I'm guessing) a Raspberry Pi? My search-fu is currently failing to find anything particularly recently, at the moment.
Or if you have a printer/scanner combo, you can turn it into a pen pal!
Dad had an Deskjet 720 or something like that.
It sat unused and powered off for a couple of years after he passed, until I needed a color print.
Didn't do anything but hook it up to power and print. Took about 1/5 of a page until all colors were back in action, after that it printed about 20 pages flawlessly.
Laser printers are your friend. The savings on consumables alone will make it pay for itself.
Epson Ecotank. I’ve been using mine for years and I only had to buy new ink once.
And I printed a lot of photos, notes, documents, etc
Those will still get their nozzles clogged if you don't run them.
This is the part where I get to point out that Brother inkjets do a little dance ~every day that keeps the heads fresh. They do this on their own as for long as they're powered up.
This allows them to work well even if years go by between prints. It's a very thoughtful design element.
(They don't survive sitting for months and months unpowered on a shelf very well, but... you'll have that.)
I never had that happen to me, in all the years I owned it. It also doesn’t randomly wake up and throw my ink away, and sometimes it sits weeks unused. It’s in my home office, so I would know if it randomly wakes up.
The only bad thing is that it can get messy if you select the wrong paper type and the ink will not be absorbed by the paper, making the rollers dirty. That’s annoying but fixable, and preventable by not selecting the wrong paper type.
Laser printers are great for documents, but not very good for photos.
I have an ink jet printer that I like. I don't print very often (average a couple pages per week) but when I do it's a mix of documents and photos. The ink isn't cheap, but the quality seems good and for the amount I print the expense is minor.
Not to mention more water resistant, when printing things like envelopes.
I used to do something similar with an old Samsung ML-2010 back when I was in college the first time around.
I think it was software and not hardware, but for some reason when I had that printer hooked up to my computer and idle for more than a week, it would simply stop printing. I probably could have dug through logs and figured it out, but I instead set up a cron job to print a test page every Monday and Thursday. The test pages would just have something on the top that said something like LOL PRINTER WORKS.
This wasn't actually as wasteful as it sounds; I was taking a boatload of math courses and needed tons of scratch paper in order to do my problems. Since it was scratch paper and would eventually end up in the trash anyway, I would usually prioritize doing my problems on failed prints and/or test prints, and I would usually exhaust those and then use blank paper afterwards.
I was about to recommend a cheap OKI LED color printer (I think C322dn); alas they withdrew from consumer market :/ The colors are super nice and uniform, even if the maximum resolution is only 600 dpi - and the toner won't dry out, which was my brother's crucial purchase criterion; we had HP inkjet clogged more than once.
I love systemd timers! I've slowly moved all of my ansible-deployed cron jobs to timers (now just an ansible copy!). The integration with journalctl, especially in a newer OS like Debian 13 where syslog is gone, is really nice. It's also really nice to be able to start the service manually for debug. Having a cron job that didn't work was an annoying exercise in copy/pasting or writing an extra shell script. Don't even get me started on the black hole of cron job stdout. I can monitor systemd services like I already do and get a notification on failure.
I've noticed more and more open source projects recommending timers as a deployment method and I think that's great!
> more and more open source projects recommending timers
I am perfectly happy with projects recommending timers as long as I can ignore them and use cron.
NixOS comes with systemd, so I've been using it as a first-class part of managing stuff. It's great, especially coming from macOS' launchd.
Which makes it nice to distribute a tool for NixOS so that it can lean into systemd instead of as some bolted-on afterthought.
Makes me wonder what you'd do if you were distributing a lifecycle-heavy tool for Linux users in general since systemd isn't ubiquitous.
I use a systemd timer to run a monthly scrub for my btrfs pool. Kinda cool how you can do increasingly useful things like skip the next scheduled event if the user initiates a scrub, do or don't accumulate tasks if you have a monthly task but the machine was offline for 6 months -- or fold them into a single task, etc.
+1, NixOS makes working with systemd a breeze. Defining units in Nix beats wrangling INI files.
systemd.services.sync-recyclarr = { serviceConfig.Type = "oneshot"; path = [ pkgs.podman ]; script = '' podman exec -it recyclarr recyclarr sync radarr podman exec -it recyclarr recyclarr sync sonarr ''; }; systemd.timers.sync-recyclarr = { timerConfig = { OnCalendar = "daily"; Persistent = true; Unit = "sync-recyclarr.service"; }; partOf = [ "sync-recyclarr.service" ]; requires = [ "podman-recyclarr.service" ]; wantedBy = [ "timers.target" ]; };I don't currently have a personal use-case for container services, but Quadlets are another example of systemd (and podman) beauty. It looks like someone has gone through the trouble of making the OS+home-manager modules: https://github.com/SEIAROTg/quadlet-nix
I never had the opportunity to try out quadlets, but they seem powerful.
I maintain one of the competitors listed in that README (compose2nix), so I am a bit biased haha.
For now, I prefer the ability to interop with Compose.
is this irony?
No. Is that not readable to you lol? I think anyone with even a passing familiarity with systemd would understand what that chunk of Nix is doing.
Compare it to the alternative of using plain systemd (including command(s) required to enable units).
Also, consider what build-time validation you get prior to starting the unit/timer. Hint: zero.
It's significantly uglier and it also skips the helpful headers / sections in the systemd INI files. `[Unit]` and `[Service]` and `[Timer]` represent different layers of execution. Many Nix people got used to the horrible syntax of Nix, I guess? I still find Bitbake significantly more palatable than Nix.
I do appreciate build-time checking but I think this can be solved at systemd side as a separate tool just as effectively.
Ugliness is subjective :)
But my question was around readability: were you able to understand what the snippet I shared is doing?
Re: build-time checks - but systemd hasn’t done it, and I am also unsure where exactly this verification would even take place given systemd’s configuration model. Unless you’re talking about some kind of language server or IDE integration.
Have you been defining them directly in your flake.nix file? I too am on nixos but I keep all my configurations in their native format and symlink them with nix, that way I can take and reuse that config on a non nixos system easily.
The problem I have found is that nixos doesn't seem to pickup and run systemd timers and services placed into the ~/.config/systems/user folder and additionally things like WantedBy=default.target have no effect.
So after I restart all my services manually on reboot I agree, systems timers are cool.
I define all units in Nix because:
a) It is way nicer and you get decent validation at build time
b) A LLM can port units over if the need arises; it’s a very light abstraction around systemd syntax
c) I personally don’t see how I would ever move to another distro :)
b) Or you could take the compiled units from /etc/systemd and copy them wherever
I believe that probably won’t work unfortunately; the generated unit files have a bunch of hardcoded Nix store paths.
Yes, but I believe there is a symlink from /etc to the Nix store, so systemd can find them.
Timers can work with arbitrary units (not just a similarly-named service unit) so they can be surprisingly flexible. I have a timer on my servers that starts a backup.target that fires off a full "restic backup","restic prune", "restic forget" backup cycle each morning with randomized start times and notifications. The actual restic-* units are Podman Quadlets so the whole setup runs agnosticaly of what's on the server, just as long as it has Podman and Systemd installed.
I will admit thought, timers are up there in terms of being the clunkiest systemd unit type to use on a regular basis. I get why they're split up into two files and require different start vs enable syntax's, but man sometimes I just want to create a file that runs a script and be done with it.
I feel like systemd units could need a layer of abstraction above them, so instead of editing the files manually, a tool would do it, some kind of declarative CLI or something. Probably not really a concern in the age of LLMs anymore, but it feels just slightly too tedious every time.
I do similar stuff for backups. What's bitten me sometimes was that I disabled the timers in order to do other stuff on the restic repo (e.g. cleaning locks) but didn't see that the triggered units were still running.
Yeah I run my steps as one-shot commands to try to avoid that, but the timer/service split can be very annoying like that.
Why do you randomize your backup times?
Should have been more clear: I use RandomizedOffsetSec= to add a random offset to a set start time (usually 4am), to prevent overloading the backup server, not truly random start times.
As someone else noted, that's also a cron feature
A feature of _some_ cron systems
True. But commonly used ones.
I haven't always been the biggest fan of systemd in some regards, but I will say that I mostly agree with this sentiment. I've almost completely quit using cron, and now favor systemd timers for scheduled jobs - at the "system" level anyway. I might still embed Quartz for scheduling that's scoped to a particular application or something.
Why? It's one of those fuzzy and somewhat hard to explain things. The systemd approach just maps more cleanly to my mental model of "how things should work" I guess. And maybe some of it is that I did indeed experience plenty of " Ambiguous $PATH settings make cron script execution difficult to predict" in the past, although it's not just that.
I won't sit here and claim that systemd timers are necessarily better than cron in any universal / objective sense. But they've won me over, for what it's worth.
This is a very good intro to systemd timers -- I think you convinced me to finally start using them. Love the "list-timers" thing as well. With cron, it never seemed easy to me to get a picture of all the cron jobs running on a box. I'd need to check crontab for all users, as well as /etc/cron.d/, as well as the daily/hourly/monthly directories.
And in fact I do have a use-case for needing to run something ~5 minutes after the system boots and then every ~12 hours onward from there. It's great that systemd timers has me covered!
Thanks for the kind words! Especially given how ubiquitous systemd is now, skilling up on the toolbox with commands like `systemd-analyze` and `systemctl list-timers` feels super valuable.
systemd is complex on first view, but after using it you didn't want to use anything else. It's handy to manage everything using systemctl
There are still some weird things/choices in its design I don't fully understand.
For example, why do unit definitions have to be actual files on disk? Then, all of these files are reloaded when the daemon reloads, not just changed ones. But, why couldn't there be an API letting me add units programmatically? (There kind of is but it's constrained/inflexible)
Or, why can't I declare multiple units in the same file? It's really designed around the filesystem instead of abstracting at a different level, which is a choice I don't think is smart. It's not like it follows the unix philosophy though.
As for the format used for unit definitions, I wish TOML had been around so they could have had something sensible...
it's so, but overall linux is not following linux philosophy in many other places, and do it far before systemd. Many tools even running under unix not doing so like gcc, x.org, etc.
You can add units programmatically. Google it!
Downvoting makes no sense.
https://www.google.com/search?q=systemd+add+units+programmat...
You can do it either via the CLI (systemd-run), or via the D-Bus API (StartTransientUnit).
PS There's even systemd.generator for a hybrid approach.
SystemD has no dearth of programmable options.
I didnt downvote and I appreciated your reply. Just to let you know :)
I have tried available options including generators.
That and systemd having actually useful man pages.
This is such a modern view. People used to HATE systemd when it first came out, but I always liked it and knew people would eventually come around and its nice to see they finally did!
Some people are stubborn and refuse to see how obviously superior systemd is to the old ways. Me included.
the thing for me is I started using the init system and while it was fine it always felt brittle for some reason. systemd feels solid and robust like it was well thought out. maybe i'm off base and didn't know how to use init effectively but it was my feeling.
that and cron always felt fragile too with a lot of quirks and limitations you had to work around instead of being a robust thing from the start.
Arguably they (we :-) were right at the time. Around Ubuntu 16.04, the journal was Hot Garbage - to keep a production system working (as in, "didn't randomly stop logging, didn't regularly corrupt logs, didn't uncontrollably fill the disk because none of the limit options actually worked") we eventually backported about 30 fixes from newer versions - by 22.04 or so it was "fit for purpose" out of the box, but earlier than that it earned every bit of hatred it got.
People still hate systemd
First it caused lots of issues. And didn't deliver anything significant
But the biggest issue has always been architectural, the way systemd keeps absorbing existing projects, and functionality. That keep adding to the more than 1 million lines of C monolith, that can burden progress in the futre
But as long people can replace any of systemd tool, for a tool they like better, all good
Personally I am now using desktop/server distros without systemd, and there is nothing that I miss, everything works... cuda/llama.cpp/steam/docker...
And commands always have to google them anyway, or find in history...
It does seem to go against the Linux spirit of one tool one purpose.
I hard disagree. Previously I could use what I knew about Linux commands to read logs and administer the systemm. Systemctl knowledge does not generalize.
I have done scheme all my life, which is why I prefer shepherd. Not only is it in a syntax that i can use elsewhere, I get completion in Emacs.
To continue on this: info use systemd on most of my systems. I dont mind. But despite having used it for about 14 years, I still have to read the manual of journalctl.
Systemctl is OK, but I really do not like that respect of the utilities.
Some people see it differently. As in: after using it they don't want to use it. I hate systemd with passion.
But that's because I'm old because obviously systemd-* is the only right way and everyone else who see things differently is a pundit.
I still hate journald logging though
I've been using Linux for over 20 years, systemd for over 10.
Yet there's always something new to learn and actually consider as another useful tool.
I'm using Linux for about 30 years and apparently we were all wrong using cron for decades.
Not wrong, but there’s room for improvement!
I've converted all my crons to systemd timers+services over the past year but cant help but think it's sort of.. less tangible than cron
Like imagine trying to explain systemd timers and services and unit files to a beginner.
> Like imagine trying to explain systemd timers and services and unit files to a beginner.
I think it's... easier? Like "systemd is the place where your system manages all the processes it needs to run. Part of those processes can be run on a schedule, or on a timer, and you define them using this simple text file".
cron is easier for easy stuff ("just run this every 10 minutes") but harder for hard stuff ("run it every 8 hours but with randomized offset so not all machines at once do it, but also if machine was down when it should run, run it immediately").
It is also easier to debug as every job gets its own log rather than trying to write to system mailer nobody had set up with the job errors
I am not the greatest fan of most of systemd's features. I will always prefer it tho since I just view it as a "packaging format". The same way I view docker. It is just that it happens to be the format that a lot of software is using and I have almost no headache integrating services, timers, logging and such of software I install. Without systemd its a mighty pain. Everyone uses the same one thing and that makes me overlook any drawbacks of the model. Only if the entire system was set up by me and mostly ran my software and I was getting paid for it, I might not use systemd.
But one feature of systemd I will absolutely stand by is nspawn. It's just beautiful.
Yeah nspawn has to be one the most underrated (and 100% optional ofc) components.
I believe one of the major distro lines (redhat or debian, I forget which) uses systemd-cron, where cron is just a thin wrapper around systemd. You get more power from writing the unit files directly, but if all you ever need is a simple cron job, you have the old interface still available.
Yep, I use this for a @reboot job and a few regular jobs on my home server. I use user crontabs, so I can get around the "unknown shell/path/etc." by prefixing every job with
or sometimes/some/shell -l myjob.sh. ~/.profile && cd /some/where && ./job >>cron.log 2>&1
Hey everyone, author here. I spotted the hn traffic a little late but I'm happy for any feedback or comments and will try and address the top-level comments as I can.
(Aside: I wrote this article early last month but it caught on only just recently. For better or worse, touching a third rail topic like systemd seems like a sure-fire way to elicit strong and numerous reactions both positive and negative.)
Oh I love them quite a lot! I use them to run all of our backup jobs, easy to set up and have never had an issue.
I've been almost convinced by systemd (and have switched to using it), but God the syntax of those service files is so ugly ...
Never thought I'd see hackers saying INI format looked ugly of all things. It's basic, sure, but that's a good thing for something meant to be easily editable by hand from any editor. Otherwise, it's just key value pairs in named sections, how ugly can it be about that?
key-value pairs where the = cannot be surrounded by spaces, so I have to write
which fills me with sadness[Service] Type=oneshot WorkingDirectory={{ home }}/current/ Environment=RAILS_ENV=production ExecStart=/bin/sh -lc "bin/db-backup --verbose"Whitespace immediately before or after the equals sign is completely ignored by the parser. Its the standard INI format.
Standard INI format™:
https://specifications.freedesktop.org/desktop-entry/latest/...
What? You absolutely can have spaces; most of mine look more like
[Service] Type = oneshot WorkingDirectory = %h/current/ Environment = RAILS_ENV=production ExecStart = /bin/sh -lc "bin/db-backup --verbose"Friend, you have changed my life
Is this one of those cases where at one point you had an error in the file and you figured it was down to spaces? You fixed that issue, it still didn't work but from that point you never thought to question the assumption.
I find myself doing this sort of thing all the time..
Somewhere in my head I had that spaces caused a syntax error, and the UI for systemd is not obvious when you first start using it ... so if it's working then leave well alone. I'll be making all of my .service files (not so many) human-readable in the near future!
That was (cough still is) ddclient for me.
My epiphany a month ago was that I can use
Environment = MULTIPLE=environment VARIABLES="in single line"Just remember: never use Environment= for secrets, since they are visible to all users on the system.
(Use EnvironmentFile= instead)
TOML would look a lot more quiet, but I'm not sure if TOML would be a good fit
unit files barely have any nesting, so the INI-like format is already 90% of the way towards TOML, no?
I suppose some things that can be multiply specified would be arrays, and maybe there'd be a stronger argument for combining say timer & service in the same file (or that it could be optional) given some structure.
Personally though I'm not really a fan of TOML when the data is highly structured - prefer YAML even a lot of the time for some visual indication. (If I were writing a parser, I have zero doubt I'd rather parse TOML!)
There's definitely some weirdness to certain parts of systemd service files, but was a huge improvement over Upstart and the old SysV-style init scripts.
Over all I think Systemd get way to much criticism. You don't have to use all the parts, but if you care to go through the documentation you'll find interesting features such as journald log-shipping and systemd-machined which can manage containers and VMs.
I love how easy it is to create a completely isolated daemons with systemd. In a single .service file one can define a daemon that has a very limited view to the filesystem, can only open specific devices, uses randomized UIDs, and has limited capabilities: https://www.freedesktop.org/software/systemd/man/latest/syst...
It is way simpler and cleaner than Docker/Podman IMHO.
As a passionate systemd hater I would say I do not agree. Cron syntax is worse.
This is why I like NixOS. Defining systemd services in it is very neat.
Could have been worse.
Could have been YAML.
Could have been XML.
XML would have the advantage of having a grammar so we could validate the config files.
It would also make it much simpler to make good GUI editors for the files instead of the Notepad approach most unix config files take.
The systemd dialect of INI is actually pretty well-defined though.
https://www.freedesktop.org/software/systemd/man/latest/syst...
I'd really like a collection of unit tests for parsers. There is a lot of details that can differ between parsers.
E.g. in "Section C" the resulting KeyThree is "value 3▵▵▵▵▵▵▵value 3 continued" where each "▵" symbol is a space.
I think most people would expect a single (or no) space here.
I would guess that most software would strip the comments in SectionC or rearrange the output so that it will result in a diff even when nothing in SectionC changed.
So if you edit the file by hand in the same style as shown in the examples, then most editors would not be able to make a minor edit without making a large diff as many sections would be formatted differently.
Since systemd is successfully parsing its INI files, and barks at you when you put weird shit into them, a grammar for them does exist as well.
XML is that wonderful format that gave us vulnerabilities like death by million laughs, up to a certain moment, you could MitM DTDs, and a whole slew of everything-XML stuff back when XML was like AI is today, none of which I miss today.
Oh, and remember times when programmers would argue whether argument order in XML files should be significant or not?
But XML books with their idealized XML future description did give me the same warm fuzzies as some intricate clockwork mechanism to a Victorian geek.
There are good GUI editors for XML?
XML - I see you’ve used macOS’ LaunchD, the system that inspired Systemd
Yeah, I'm a man of culture like this. However, systemd with its service dependencies runs circles around launchd in pretty much every aspect.
Service dependency resolution and parallel startup is super in systemd. Big fan of what I can do with it.
Could have been better.
Could have been XML Property Lists.
ducks
To be honest, I think either of those would have been better ...
/me cowers in fear
Oh yes, because the well documented clean syntax of sys v init shell scripts was so nice.
If I never recall hacking in ulimit calls in the top of buggy shell scripts for crappy old services that done respect pam_limits it won’t be soon enough.
Relevant, a golfed systemd polyglot file that is simultaneously an executable script: https://domi.work/blog/posts/compose_polyglot/
Yes, I have too much time sometimes... and I agree, I don't like the syntax.
Hard disagree. Compared to an init script, with all its boilerplate, I'd take a systemd unit file.
Even better is systemd socket activation.
I design all my services expecting to receive sockets this way. It makes sandboxing easy as the service itself doesn't need network access to have a listening socket.
It's a shame docker never supported it. I feel like if they had got on board all those years ago there would be broad support across the software ecosystem for it and we wouldn't need half of these complicated iptables rules and proxies and service mesh. It would be a step towards a capability based system.
This is very interesting. I'm not sure what I'd use it for yet, but I imagine it could be useful for triggering ad hoc jobs over the network. Maybe have Home Assistant make a network call to kick off a daily back up when I leave the office at the end of a work day.
A pretty nice use case I have for socket activation is for isolating containers or applications from the host network. The great thing about socket activation is that opened sockets carry over even if the application/container unshares into a different network namespace! It also works great with Podman pods with networking in the pod completely disabled and, as those are host sockets, does fully retain the connection info of peers (so logs are not just uselessly containing the gateway IP, depending on the container network config)
I believe its original motivation was just speeding up boot times by starting fewer services, even if you'd eventually want the service running. This was achieved in the past with xinetd, but systemd made the approach more popular for the masses.
inetd began to fall out of favour in the mid-late 90s as services became more heavyweight and startup times became longer (think of the initial crypto setup needed by sshd vs rsh/telnetd)
CPU speeds have increased & and i/o latency has decreased so much since then that startup times are generally imperceptible, so the pendulum has swung back to favouring socket activation.
The anti-systemd "traditionalists" never seem to acknowledge that history, though!
This is actually something that I like in systemd.
I am dealing with mostly non systemd system: BSD, Alpine, termux On BSD anacron works well, but I do not why I am always running into problems with the cronie anacron implementation. And it is very hard to debug.
I would really like a simple modern cron/anacron alternative.
Cronicle looked cool but it is node.js, a bit heavy and being replace now by their new product called xyOps anyway.
I will use what I am comfortable with and so should others. CronD, SystemD, atD, multiple conditional checks in a shell script, whatever tickles your fancy. There is no wrong answer, just document what you did and add a comment. Comments are permitted in cron. If someone keeps putting complex obfuscated time structures into cron make them decipher their incantation and keep nagging them until they keep it simple, comment their cron entries or until they and their manager resign.
For what it's worth there are usually web apps popping up that can decipher goofy cron time/date incantations. [1] This one has a git repo in the top right, not my repo. Maybe clone it just in case their site goes away some day.
I setup a few systemd timers last year and created https://systemd.guru to play with different options in OnCalendar expressions.
After years of using orchestration tools like airflow and dagster so many lightbulbs have just lit up in my head.
I wish documentation for tools would explain their abstractions concepts in terms of its primitives.
Great post, thanks!
We have used cron perfectly fine for decades and it served us well within its very clear limitations.
But now obviously we were so blind and wrong all this time and the only true solution is of course systemd.
Has it actually served you well? Because it hasn't served me well at all.
I am not the biggest fan of systemd, but today I will always reach for a systemd timer over cron simply due to the sheer amount of bad experiences I've had with cron. Hours upon hours wasted trying to troubleshoot crons that weren't working due to some stupid obscure issue, having to use dirty hacks to monitor for success or retry failed jobs.
A few years ago I was trying to run a very simple bash script with cron and the script just died halfway through for no reason. Nothing in logs, worked fine when run directly, but in cron it just stopped halfway through a loop. Never figured out the cause, just gave up and used a timer instead, which worked fine. Never touched cron again after that.
The ease and convenience of monitoring and troubleshooting alone are worth switching over.
Let me state once again: "within its very clear limitations".
Once you learn that env in cron is not same as in your shell and once you learn to redirect output to loggers - it works just fine.
It would be a lie to say that I never debugged cron and sure it's annoying.
> and the script just died halfway through for no reason
Unrelated to cron. Bad script.
Systemd will reign supreme for a millennium if the answer to every question or complaint about non-systemd tools is "you're holding it wrong".
As a user I'm kinda whatever about the tools because the answer to my complaints about systemd is also "you're holding it wrong."
I don't agree that these are just limitations. The fundamental problem cron tries to solve is very simple: I want to run a program automatically at specific times. There are probably many features of systemd timers that can be considered niche or extraneous in solving this problem, but the ability to easily know when the program last ran and what its exit code and stderr output were is not one of them. I believe that if an alleged solution to this problem doesn't provide at least this, it's not really solving the problem.
> Unrelated to cron. Bad script
Again, worked fine when run manually, worked fine in a systemd timer. Pretty sure I still have it running today and it continues to work fine without ever failing.
So basically it took you decades to learn all the bugs, UX issues and problematic quirks and now you're complaning someone built something better? :)
If the script is bad, surely cron will give you a way to test that? Right? Right??????
well yes but actually no
I'm sympathetic, but "bad script" is an awful assertion.
We are all guilty of making bad scripts, bash is a disgusting degenerate language (and I love it). The way we learn to write good scripts is by writing bad scripts in enough amounts to get bitten by all the warts.
One thing I really love about cron, is that if you set up mail on the server (which: you should btw), then cron actually sends emails if it sees anything in stdout and stderr.
I am a dyed in the wool systemd non-believer, but I really do like the timers.
Thank Lennart you degenerate apostates are finally starting to see the light of His glorious creation. Hallowed be thy systemd-journald.
> But now obviously we were so blind and wrong all this time and the only true solution is of course systemd.
Oh nooooo... I've been so wrong. Seen the direction most Linux distro took I decided to move to VMs (VMs which runs systemd-less systems), OCI containers (where by definition PID 1 is not systemd) and now an hypervisor to run systemd Linux VM but I'm now into... An hypervisor that is precisely not Linux (so no systemd at all).
I sinned. So now maybe it's time to buy Microsoft stocks, praise Windows [ini] config files, and venere the Linux PID 1 god with its tentacles meddling with every part of the Linux system.
Or not.
After reading this article I'm convinced that what we need is systemd timers fronted by a vibe coded crontab -e emulator.
Coz it's looks crazy complicated to set them up.
> But now obviously we were so blind and wrong all this time and the only true solution is of course systemd.
Come on, dude. That's unnecessarily polemic.
cron et al have served us for decades, yes. But that doesn't mean that cron is the solution that needs to accompany us until the heat death of the universe or year 2038, whatever comes first.
I agree, the systemd folks haven't exactly been the best when it comes to PR or when it comes to being even near feature parity with what they tried to replace. But now, they aren't just at feature parity, they surpassed plain old cron.
Maybe it is time to lay cron to rest, at least slowly.
> that doesn't mean that cron is the solution that needs to accompany us until the heat
Yeah I agree.
> systemd folks haven't exactly been the best when it comes to PR
It's deeper than that. Systemd folks are enemies of Linux. First, it's "fuck your opinion, do as we say" attitude which makes me want to throw away everything that comes from that poisonous well. Second, it's the embrace and extinguish strategy employed by the systemd project. And third, systemd author is up to no good: https://news.ycombinator.com/item?id=46784572
> First, it's "fuck your opinion, do as we say" attitude which makes me want to throw away everything that comes from that poisonous well.
On the other hand, it is a consistently heard argument when debating why the year of the Linux adoption on desktop hasn't happened yet is that there are too many standards, too many cooks. And I kinda agree with that, packaging software for multiple distributions is a hot mess, especially if you're shipping daemons. systemd is at least one worry less, it's a stable API that can be used for all purposes.
> And third, systemd author is up to no good: https://news.ycombinator.com/item?id=46784572
Again, this isn't as black-and-white as it seems on a cursory review.
From the perspective of someone who hates anti-rooting measures on phones, anything moving into the direction of trusted computing is bad.
But from the perspective of someone, say, in Russia, Iran, the US or Germany who might be a journalist or political activist? Suddenly, a way for the OS to attest if the hardware hasn't been manipulated in an evil maid scenario and that any successful attempt of exploiting an OS vulnerability at runtime can be reverted by a simple reboot becomes extremely vulnerable.
My personal opinion, we need strong and good attestation capabilities for a multitude of use cases. But we also need good laws that protect user freedoms, similar to "right to repair" laws we need "right to root" laws that ban applications from requiring unrooted phones.
systemd-as-pid1 is good. I have to admit that. I will not go back to syvinit unless a good reason comes up.
However absolutely fuck all the metastasis surrounding systemd and the shit inside, timers included.
> we need strong and good
Yeah this is true.
Problem is, this technology will absolutely be used for limiting your access to the internet from a distribution that was modified. Thus leading to locked down personal computers.
> However absolutely fuck all the metastasis surrounding systemd and the shit inside, timers included.
Meh. systemctl list-timers is orders of magnitude better than trying to wade through /etc/crontab, /etc/cron.{hourly,daily,monthly,weekly,yearly} and whatever else non-Debian distributions come up with, and cron's arcane syntax is a mess on its own on top of that. Especially if you try to run a cronjob as not-root.
> Problem is, this technology will absolutely be used for limiting your access to the internet from a distribution that was modified. Thus leading to locked down personal computers.
That's a hypothetical threat IMHO. DRM and Cloudflare Turnstile [1] are actual threats that are far bigger in practical usage.
I like these systemd things but I always find it annoying how I have to create multiple separate files (like a file for the service and another for the timer, or similar if I need a socket file). In theory this is more flexible but in practice it's vanishingly rare that I need the same service to be accessible to more than one timer. It would be nice if there were some alternative compound format that could combine the timer and service into one.
In decades of trying, I do not believe there was one time that I ever got a cron job to work properly in the first attempt. Systemd timers are a godsend.
There's another big feature: You're not relying on the time zone to which the server was set (like with cron) but can explicitly specify a time zone:
https://www.freedesktop.org/software/systemd/man/latest/syst...
Of course you can do this trivially in cron as well. It is what the CRON_TZ variable is for.
This is a GNU extension so not portable.
> This is a GNU extension so not portable.
1) It's supported by cronie. I bet it's supported by many other crons.
2) "Great" news! The software in the Systemd Project only officially runs on Linux, so "it's not portable" is a really bad counterargument when "alternatives to some Systemd Project feature" is the discussion topic.
hmm, when did that get added? Last time I checked, the only timezone you could specify was UTC (which was one more than cron supported, but still insufficient.)
I wonder what happened with the heading, it was okay before, and then was mutilated since.
I command you to love systemd timers!
(Yeah, they're pretty useful, especially after you get an LLM to write all the boilerplate for you. The boilerplate was the main reason I preferred crontab before.)
Refreshing to see a positive opinion with regards to systemd. Incidentally, my favorite way to spawn jobs, is well, the job spawn command in nushell.
Is there a way yet to force-trigger a timer? There wasn't the last time I used them, which I found to be super annoying for testing them.
It's covered in the article, you can simply start the unit that would be started by the timer.
Oh but it won't appear in the timer-specific logs, I guess...
It may be a disastrous comment to make but I think I like cron better! A tool designed for a particular job etc.... :/
I designed a tool for flying. It's only designed for flying. It is based on the principles of the brick.
Cron. Initial release May 1975; 51 years ago
It is succesfully flying 51 year. And will work next 50 years. Systemd probably will changes syntax in next 2 years. Modern development mindset: if tool is not rewritten last month - it is outdated and we need to reinvent it. Probably using blockchanin and AI.
I don't love cron's time format - it's easy to make mistakes - but one-line, one file configuration is simple in a nice way. I bet we could make a cron that was easier but still simple.
Crontab could be confusing at first glance but 100 lines (literally) of man page have everything including examples. In my exerience they covers 99% of use cases unless you need something REALLY fancy.
And you have cron on every system including BSD and not sure but probably also AIX/Sun/etc. It is universal and everyone knows about it. If your server doing something weird your probably will check crontab, not some obscure systemd subsystem almost no one knows about.
Instead using standart solutions (POSIX by the way) systemd again found NIH problem with cron and added one more tool in init combain.
My point was purpose built doesn't mean its the best tool for the job.
As for the rest of the drivel: so what it was used for a long time. That just means it was pretty good. That doesn't mean it has to stay forever, just that a new contender should do things better.
Systemd timers address real shortcomings of cron.
Your argument boils down to: everyone should be stuck with shortcomings of the early days of computing because you don't like new things.
> My point was purpose built doesn't mean its the best tool for the job.
But you are agree cron is pretty good so it seems analogy with flying brick does not work here.
> Your argument boils down to: everyone should be stuck with shortcomings of the early days of computing because you don't like new things.
I would say: do not fix it if it's not broken. You need a really serious reason to change a good software to one with dubious quality and future. And without portability.
> Prime Time for a Timer Primer
It's pronounced, "primmer."
TIL that's the standard US pronunciation. I thought you must be joking, referencing something else: OP's title works with BrE pronunciation at least.
Wait, really? I'm a native Midwestern/Great Plains American English speaker (I remember reading the Harry Potter books as a kid and wondering why all the -er words were spelled wrong) and I say "PRY-mer." I have never heard anyone say "PRIM-mer" in my life.
Am...am I being punk'd...?
I too have never once heard primmer. Not that all words follow the rules, but it doesn't make sense (vowel-consonant-vowel has long sounding vowel. That page also has the audio file links backwards (regular vs irregular). The file labeled irregular pronunciation sounds like primmer.
I am too. I've heard of the supposedly correct pronunciation, but I can't bring myself to use it. The "PRY-mer" pronunciation is more common in practice.
Does systemd ship with something to upgrade your cron jobs for you? That would be the friendly way. Write your old school cron jobs, and then a script that converts them to do things the systemd way, documenting its steps, i.e. I created this file and this is why. Friendly "I help you do things better" rather than standoffish "your way is obsolete, you need to do it our way". Oh wait. I get it. LLM agents can do exactly that for you can't they. Another way I'm behind the curve.
I have knocked together a systemd service or three based on google copypasta. But generally, for cron jobs, why make it complicated? One line in /etc/crontab and done. I generally call an encapsulation script that sets the right environment variables, uses absolute paths, captures stdout/stderr if required and so on. I just want the simplest possible way to launch that script on a schedule.
I'm fully ready to drink the "just let systemd do all the things" kool-aid, but I would love to see some sort of introductory/tutorial info into some of the things it can do other than services - i.e. containers and timers. I know man pages exist, but it would be nice if there was more scannable intro out there.
btw that xz hack only effect systemd distros
That would assume I like systemd in the first place.
>And yet. You probably shouldn't use literal cron (or its more modern cousins) for scheduled tasks! In 2026 there are more modern options available
What are people on non-systemd distros like Guix System, Void, PCLinuxOS, and so on using for this? Is there still something better to use than cron?
Admittedly I never learned cron, I use a lot of `sleep` and `countdown` for relative delays instead. Just earlier today I set up a 12h countdown followed by opening a URL with xdg-open since I expect a release around then and don't want to forget. I also threw in a little notify-send command in case my browser isn't visible, I should see that pop up. Considered using espeak, but don't wanna scare myself and/or ruin my watching experience if I'm watching a video at that time.
> What are people on non-systemd distros like Guix System, Void, PCLinuxOS, and so on using for this? Is there still something better to use than cron?
Instead of `sleep` you can use `at`. But for scheduling `cron` is still the best.
Package: at
at, batch, atq, atrm - queue, examine, or delete jobs for later execution
> What are people on non-systemd distros like Guix System, Void, PCLinuxOS, and so on using for this?
We have used and still use cron for decades. It does it's job and does it well.
> stdout and stderr output often ends up in a black hole
Ain't that the truth. Literally every crontab I've written for the last 10 years has had this in it:
2>&1 | logger -t cron-WHATEVER
...and that does a pretty good job of capturing anything that the script emits and making it easy to grep for in syslog the following morning.
But I'm still amazed at how many crontabs I run across that don't capture any output at all.
The “standard” is for the output to go to your user’s mail box. You know, that thing you check with the “mail” command and has a user interface shockingly similar to `ed`. You check that all the time, right? Right?
It’s… certainly a product of its time. (I have my system mailer set up to actually send mail to my Gmail account, with authenticated SMTP via API keys, which I did 15 years ago and have no recollection of how I even did it. It still works… somehow. I don’t even use Gmail any more, and I’ll be damned if I have to figure out how to do it with fastmail, and lord knows doing unauthenticated old-school SMTP is just gonna get sent to fastmail’s black hole, so that idea ain’t gonna work either.)
> humble systemd
That the same cannot be said of its maintainer is another matter.
The main person in charge of Linux itself isn't considered the most humble but they make amazing products.
Linus is pretty humble tbh, he just expects that people don't throw shit over the fence.
> In 2026 there are more modern options available
More modern doesn't mean any better or more powerful.
I for one hate this need for going "more modern" without having clear and factual advantages.
Article author lost me when he demonstrated that two files are needed to schedule something.
That is silly.
Same shit, but you need to create 3 files instead of one line in crontab.
systemd is great all around. Don’t listen to the boomers complaining about it because their cheese was loved.
They should listen to other people talking about their knowledge about cheese, of the single one they know, the cheese slices from the supermarket
(typo, i think). cheese was moved?
You will love SystemD [0] timers until they fuck you over in an entirely inscrutable way and the SystemD maintainers don't care to either fix the problem or update the docs to warn of the shortcoming.
One of our customers called in with a production down incident caused by a full disk. We got a copy of the VM and took a look. Investigation revealed that / was full because /var/log was full and that our 'logrotate' timer unit that was scheduled to run once a day had run either exactly never or exactly once... I can't remember which. Further investigation revealed no difference in software load or configuration between this VM and a VM that had a functional logrotate timer unit. Exactly one VM out of hundreds of identical VMs at this site (and many multiples of that at other customer's sites) were affected by this. Advising the customer to clear out /var/log and reboot did not unstick 'logrotate', and none of the diagnostics or fixes we could find anywhere unstuck it. Once "systemd-crond" decided to never schedule this job ever again, it stuck to that decision.
After a lot of searching, we found an open bug report from a year or three prior where someone reported exactly the same symptoms and was scheduling a unit with pretty much the same set of unit configuration flags that we were using. The conversation from the core devs ran through the pattern that one gets used to seeing when one runs into SystemD bugs that are caused by extremely complex unanticipated interactions between parts of the project: "That's not a bug, only an idiot would want that to work.", "Oh, we don't document that that's not supposed to work?", "Wow, okay, yeah, I can see how that maybe should work. That it doesn't sure does seem weird.", "Having said that, I don't know if it's supposed to work, or if it's unsupported. Someone should really either document that or fix it."... and then the behavior is neither fixed nor documented. [1] Absent any actual explanation for the failure, we ended up swizzling the options in our 'logrotate' unit and praying that satisfied whatever gremlin arose from the depths to trouble our customer.
SystemD contains an enormous -and ever-growing- amount of accidental complexity, and has a set of core maintainers who are generally disinterested in either documenting the places where one or more complex systems bind together to cause stop-the-world problems or fixing the systems involved so that they don't bind up. It's a fine project until it's very, very suddenly not, and then you're absolutely SOL. If you're lucky, you can shuffle around what you're doing [2] and hope that avoids the problem. [3]
[0] Some folks use the spelling "SystemD" to mock the project. I use the spelling "SystemD" to distinguish between "the entire systemd project" and systemd(1). I do this because some folks will make a claim like "systemd is very, very small and self-contained. I don't understand why anyone would say otherwise.", but what they are actually saying is that systemd(1) is a fairly small program that doesn't do all that much when run as PID 1. It sucks minor amounts of ass that the project and the program it runs as PID 1 share the same name, but what can you do?
[1] No, I don't have a link to the open bug report. This was more than a year ago, so the bug ID has been long forgotten.
[2] The term of art for this practice is "wave a dead chicken at it".
[3] Plus, like, even disregarding most of the rest of my report... how in the hell do you design a cron that knows a job is scheduled to be run periodically, can tell you how long it has been since it last ran, but never manages to run it? To me, that's unforgivable. It's a "You had one job!"-tier cockup.
>The conversation from the core devs ran through the pattern that one gets used to seeing when one runs into SystemD bugs that are caused by extremely complex unanticipated interactions between parts of the project
>SystemD contains an enormous -and ever-growing- amount of accidental complexity, and has a set of core maintainers who are generally disinterested in either documenting the places where one or more complex systems bind together to cause stop-the-world problems or fixing the systems involved so that they don't bind up.
excellent comment. thx for the long form. im sure it was fueled by excessive frustration.
imagine my surprise to learn that Systemd was causing my long standing frustration with changing my dns settings. and further surprise to learn that server admins have this same issue and many switch away from using systemd-resolved.
> ... imagine my surprise to learn that Systemd was causing my long standing frustration with changing my dns settings. and further surprise to learn that server admins have this same issue and many switch away from using systemd-resolved.
That's introductory course to systemd's shenanigans. People are going to tell you that you're not doing it properly, that there's of course this setting (unless that other setting takes precedence etc.), yada, yada, yada.
If I really have to suffer systemd the first thing I do is manually edit /etc/resolv.conf and then chattr +i /etc/resolv.conf.
And of course remove/purge systemd-resolved.
Not only is it "always the DNS" but then things turn from bad to worse when "it's the DNS, but with systemd".
Removing systemd-resolved is the first step. The second one is moving to an OS or a Linux distro that doesn't have systemd at all.
resolved is fascinating.
Did you know that it will convert the answer to a relative query [0] that has generated an NXDOMAIN into a REFUSED? It doesn't do this for fully-qualified queries, and it doesn't do this for relative queries that return something other than NXDOMAIN.
Why do they do this... even if ALL of the resolvers configured in resolved return NXDOMAIN for the query? «Because we believe that the standards say that resolvers can return REFUSED for any reason at all. This is any reason. Now get lost.». Why don't they do this for fully-qualified queries? «chirping cricket noises»
[0] I think this might also be known as a "zero dots" query. Assume that your DHCP-provided search domain is home.arpa. You can do 'ping pc' and every resolver I remember using will convert that hostname into 'pc.home.arpa' and do a lookup with that name, rather than the one you entered.
Yeah, quite possibly... that one seems familiar! Your Google-fu is quite strong.
FTFY: You will love SystemD until they fuck you over
And you immediately lose the ability to do `crontab -l` on any server to know its scheduled tasks.
Now you get to look around the myriad of places where you can put systemd files, and figure out which ones are base services and which ones are custom, with no general convention to go about it. Nope.
/etc/crontab /etc/cron.d/* /etc/cron.hourly/* /etc/cron.daily/* /etc/cron.weekly/* /etc/cron.monthly/* /var/spool/cron/crontabs/*Bad memories. I particularly enjoyed fighting with third-party programs that installed system cronjobs in the various tabs, and having to remember to go and find them after package upgrades and try to figure out how to robustly identify when their processes were running so my other cronjobs wouldn't overload or clobber state, since the third-party-installed jobs didn't play along with any lockfile-based coordination we used. Wants/WantedBy/Requires are godsends by comparison.
If you had read the article, you would have seen its answer to this.
systemd list-timers
With —-all