> Ambiguous $PATH settings make cron script execution difficult to predict.
What makes you say that? You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
> You might feel cool knowing the scheduling grammar by heart
I've used Linux since 1994 and I don't know it by heart. But luckily it's pre-printed in the crontab as comments:
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
The rest of the complaints, sure. Next time I need a cronjob, I'll try it out.
That is not a fair summarization of their point because that is not the grammar. There's commas, slashes, asterisks, combinations, and then if you want randomization you need to put it in the command itself because cron can't do it. (Some crons can, but it's not a general capability of cron.) Writing a non-trivial cron spec is not easy.
This is generally my only real complaint about systemd. I don't care if it is too monolitic, written in C or whatever, I just want a straightforward syntax for straightforward operations. I'd like it if systemd could recognize if a .target file is a shell script and just do "the right thing". Perhaps it would make sense for a timer file to recognize cron syntax as well. Or at least allow for a kind of extensibility so that I can have it supported.
If systemd had a little more respect for existing conventions, I am pretty sure it wouldn't be so controversial. After all, system administrators like it because they use it all the time, but a regular, full-timer user like me, who only deals with it when something is broken or have to use it as a means-to-an-end to set something up, then all friction is annoying and bad UX. (And no, using Nix is not the solution)
$ sudo systemctl edit --force --full my-scheduled-work@.timer
$ systemctl edit --user --force --full my-scheduled-work@.timer $ systemctl cat public-inbox-watch@.timer
# /etc/systemd/system/public-inbox-watch@.timer
[Unit]
Description=Periodic fetch of public mailing list
[Timer]
# twice a day
OnCalendar=*-*-* 5,17:35
RandomizedDelaySec=1h
Persistent=true
[Install]
WantedBy=multi-user.target OnCalendar=00/6
systemd-analyze calendar --iterations=6 '0/6:00:00'
https://www.freedesktop.org/software/systemd/man/latest/syst...
In cron, you basically have to either use your configuration management to generate those times, or have a random delay script running before the command
In systemd timers, it's just
OnCalendar=0/6:00:00
RandomizedOffsetSec=60m
If you add
Persist=true
Nope. From crontab(5)
The RANDOM_DELAY variable allows delaying job startups by random amount
of minutes with upper limit specified by the variable. The random scal‐
ing factor is determined during the cron daemon startup so it remains
constant for the whole run time of the daemon.
Mon,Fri *-01/2-01,03 *:30:45
[0] <https://www.freedesktop.org/software/systemd/man/latest/syst...>
[1] This question is sarcasm. SystemD is often like this... dead simple things look dead simple, but complex things are -if they're possible at all- at least as complex as they are everywhere else.
Looking at the other examples on that page, I'm gonna say that it's only arguably easier to read for basic stuff... especially if you're familiar with the syntax. The complex stuff is -at best- just as difficult.
When someone inputs something ridiculous like "5,3/4 4-8,11 1 4,5,6,9-11 */2" you get to enjoy the fun of reverse engineering what they meant (it's never what they actually wrote).
And that's before you get to all the extensions supported in some cron environments (but not all).
I find systemd timers a lot more manageable. Things like having control over whether or not long-running jobs are allowed to overlap and the ability to run tasks between start-finish rather than a fixed time window are major improvements for me. At some point my VPS went down because the backup job ran into some kind of symlink loop and cron just kept spawning more and more backup tasks even though none of them finished.
Having to re-write commands and scripts because CRON had its own special PATH was also a pain point, but the same can be true for some types of systemd timers. But: you can execute those timers manually if you want instead of updating the crontab to trigger in 30 seconds and simply waiting.
# Run if at least a day has passed since the last run
# and it isn't the weekend.
def should_run(finished, timestamp, dow, **_):
return dow not in [0, 6] and timestamp - finished >= one_day
(job
'(next-minute-from
(next-hour (range 0 24 2))
'(15))
"my-program")
What's so hard about "At 5 minutes past the hour and every 4 minutes, starting at 3 minutes past the hour, at 04:00 AM through 08:59 AM and 11:00 AM, on day 1 of the month, every 2 days of the week, only in April, May, June, and September through November"?
(I used https://crontab.cronhub.io/ to decode it, to be fair)
That does require you to still know what the default environment is, but it is a mostly completely clean environment, without any influence from any shell.
I'd have to concur that I agree this is an advantage of systemd.
Odd. This script
#!/bin/bash
set > /tmp/set.txt
* * * * * $HOME/bin/testCronScript.sh
BASH=/bin/bash
BASHOPTS=<redacted because long>
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=([0]="0")
BASH_LOADABLES_PATH=/usr/local/lib64/bash:/usr/lib64/bash
BASH_SOURCE=([0]="/home/user/bin/testCronScript.sh")
BASH_VERSINFO=<redacted bash 5.3.x>
BASH_VERSION=<redacted bash 5.3.x>
DIRSTACK=()
EUID=13370
GROUPS=()
HOME=/home/user
HOSTNAME=hostname
HOSTTYPE=x86_64
IFS=$' \t\n'
LANG=en_US.utf8
LOGNAME=user
MACHTYPE=x86_64-pc-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PPID=1337
PS4='+ '
PWD=/home/user
SHELL=/bin/sh
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=13370
USER=user
_=/home/user/bin/testCronScript.sh
* * * * * root /home/user/bin/testCronScript.sh
Maybe your distro's default cron environment was bad, and you never bothered to check and unset the badness? I'd be surprised if they were unable to make the default environment for Timer Units to be bad.
That's true, but most people don't know the numbered manual sections, so they get the docs for the cron table command not the cron table config file.
No `man man`? ;)
another benefit is having logs in one place for the job; cron's "send a mail when there is any amount of output text" is just annoying behaviour, but also only place to get the job output unless you redirect it somewhere. Also starting from timer vs just doing systemctl start job.service is the same so easier to debug
other than that the few improvements in how to specify run time have been pretty useful.
For example, setting timer as "persistent" will mean any run "lost" to machine powered off will just be ran next time after boot, so you can have job on your PC that is just "run backup at 2AM" and if you turn it off before that you get the backup done first thing in the morning
There is also both random, and fixed (depending on machine UUID) random delay so avoiding thundering herd problem with backups is also pretty convenient.
There is even option to wake a device for the job if necessary tho the problem of shutdown is left to the user. And picking whether to start counting to next timer from previous one or from the job's end.
What I would like also is to have job summary page ("hey this job was done X times but failed Y times") but that's probably better left to external tooling
> You can set the PATH right in the crontab. Is that harder to "predict" than it being set in /etc/bashrc, ~/.bashrc, ~/.profile, ~/.bash_profile, /etc/systemd/…, or wherever else?
There is* a common trap as the cron PATH is usually just /usr/bin:/bin so anything in /usr/local/bin, or in /sbin won't be there.
Same here.
We are now considered old and therefore irrelevant. The new generation uses timers and couldn't care less about cron that has served us just fine for decades.
I use cron and my general attitude towards LP and systemd is very similar to the attitude of LP and systemd to us.
If nothing else, maybe it could be some kid's high school science fair project idea.
It sat unused and powered off for a couple of years after he passed, until I needed a color print.
Didn't do anything but hook it up to power and print. Took about 1/5 of a page until all colors were back in action, after that it printed about 20 pages flawlessly.
I have an ink jet printer that I like. I don't print very often (average a couple pages per week) but when I do it's a mix of documents and photos. The ink isn't cheap, but the quality seems good and for the amount I print the expense is minor.
And I printed a lot of photos, notes, documents, etc
This allows them to work well even if years go by between prints. It's a very thoughtful design element.
(They don't survive sitting for months and months unpowered on a shelf very well, but... you'll have that.)
Btw this is my repo for the backup automation: https://github.com/gchamon/borg-automated-backups
To do this at the user level, you can add something like "@hourly anacron -t /path/to/anacrontab -S /path/to/spooldir" to the user's crontab, though I've never tried this.
Many cron implementations have a similar mechanism.
This isn't the same as with systemd timer because timer lets you specify when you want to run your service exactly and will fallback to running when the system comes online. With @hourly I lose this control and multiple machines could potentially trigger backups at the same time, hogging the physical hard drives and the network.
That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.
I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then.
> With @hourly I lose this control and multiple machines could potentially trigger backups at the same time
Then don't use @hourly, use staggered times, it's very easy.
This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.
100 jobs all running at different times throughout the week is a very different load than them all falling back and running at the same time on system boot.
There are two options to fix it;
Disable persist so no catching up on missing scripts. Set OnBoot=5m so it gets ran 5 minutes after boot, so your script (say backup) is ran on boot first, then every time on schedule
Enable persist but just add sleep in ExecStartPre - very "cron" way but there is just no in-systemd option to enable "catch up" script to be delayed
Sadly no option to "run catch-up timers with delay" at least yet
> Then don't use @hourly, use staggered times, it's very easy.
Not in cron. In systemd it's just RandomizedOffsetSec=30m and it is "stable" - same host with same job will always have same delay so on multiple hosts it is spread nicely. There is also non-stable version
Cronie doesn't have a `@reboot` meta-trigger?
> runs the service as soons as the system is available.
cron has the @reboot option which I use for a few scripts and works great.
Not an option either, because if I reboot two machines and the backup starts in both of them it'll cripple my NAS
I've noticed more and more open source projects recommending timers as a deployment method and I think that's great!
I am perfectly happy with projects recommending timers as long as I can ignore them and use cron.
I will admit thought, timers are up there in terms of being the clunkiest systemd unit type to use on a regular basis. I get why they're split up into two files and require different start vs enable syntax's, but man sometimes I just want to create a file that runs a script and be done with it.
Which makes it nice to distribute a tool for NixOS so that it can lean into systemd instead of as some bolted-on afterthought.
Makes me wonder what you'd do if you were distributing a lifecycle-heavy tool for Linux users in general since systemd isn't ubiquitous.
I use a systemd timer to run a monthly scrub for my btrfs pool. Kinda cool how you can do increasingly useful things like skip the next scheduled event if the user initiates a scrub, do or don't accumulate tasks if you have a monthly task but the machine was offline for 6 months -- or fold them into a single task, etc.
systemd.services.sync-recyclarr = {
serviceConfig.Type = "oneshot";
path = [ pkgs.podman ];
script = ''
podman exec -it recyclarr recyclarr sync radarr
podman exec -it recyclarr recyclarr sync sonarr
'';
};
systemd.timers.sync-recyclarr = {
timerConfig = {
OnCalendar = "daily";
Persistent = true;
Unit = "sync-recyclarr.service";
};
partOf = [ "sync-recyclarr.service" ];
requires = [ "podman-recyclarr.service" ];
wantedBy = [ "timers.target" ];
};The problem I have found is that nixos doesn't seem to pickup and run systemd timers and services placed into the ~/.config/systems/user folder and additionally things like WantedBy=default.target have no effect.
So after I restart all my services manually on reboot I agree, systems timers are cool.
a) It is way nicer and you get decent validation at build time
b) A LLM can port units over if the need arises; it’s a very light abstraction around systemd syntax
c) I personally don’t see how I would ever move to another distro :)
/some/shell -l myjob.sh
. ~/.profile && cd /some/where && ./job >>cron.log 2>&1Yet there's always something new to learn and actually consider as another useful tool.
For what it's worth there are usually web apps popping up that can decipher goofy cron time/date incantations. [1] This one has a git repo in the top right, not my repo. Maybe clone it just in case their site goes away some day.
Like imagine trying to explain systemd timers and services and unit files to a beginner.
I think it's... easier? Like "systemd is the place where your system manages all the processes it needs to run. Part of those processes can be run on a schedule, or on a timer, and you define them using this simple text file".
It is also easier to debug as every job gets its own log rather than trying to write to system mailer nobody had set up with the job errors
Why? It's one of those fuzzy and somewhat hard to explain things. The systemd approach just maps more cleanly to my mental model of "how things should work" I guess. And maybe some of it is that I did indeed experience plenty of " Ambiguous $PATH settings make cron script execution difficult to predict" in the past, although it's not just that.
I won't sit here and claim that systemd timers are necessarily better than cron in any universal / objective sense. But they've won me over, for what it's worth.
I have done scheme all my life, which is why I prefer shepherd. Not only is it in a syntax that i can use elsewhere, I get completion in Emacs.
But that's because I'm old because obviously systemd-* is the only right way and everyone else who see things differently is a pundit.
But now obviously we were so blind and wrong all this time and the only true solution is of course systemd.
I am not the biggest fan of systemd, but today I will always reach for a systemd timer over cron simply due to the sheer amount of bad experiences I've had with cron. Hours upon hours wasted trying to troubleshoot crons that weren't working due to some stupid obscure issue, having to use dirty hacks to monitor for success or retry failed jobs.
A few years ago I was trying to run a very simple bash script with cron and the script just died halfway through for no reason. Nothing in logs, worked fine when run directly, but in cron it just stopped halfway through a loop. Never figured out the cause, just gave up and used a timer instead, which worked fine. Never touched cron again after that.
The ease and convenience of monitoring and troubleshooting alone are worth switching over.
Once you learn that env in cron is not same as in your shell and once you learn to redirect output to loggers - it works just fine.
It would be a lie to say that I never debugged cron and sure it's annoying.
> and the script just died halfway through for no reason
Unrelated to cron. Bad script.
We are all guilty of making bad scripts, bash is a disgusting degenerate language (and I love it). The way we learn to write good scripts is by writing bad scripts in enough amounts to get bitten by all the warts.
One thing I really love about cron, is that if you set up mail on the server (which: you should btw), then cron actually sends emails if it sees anything in stdout and stderr.
I am a dyed in the wool systemd non-believer, but I really do like the timers.
> Unrelated to cron. Bad script
Again, worked fine when run manually, worked fine in a systemd timer. Pretty sure I still have it running today and it continues to work fine without ever failing.
Come on, dude. That's unnecessarily polemic.
cron et al have served us for decades, yes. But that doesn't mean that cron is the solution that needs to accompany us until the heat death of the universe or year 2038, whatever comes first.
I agree, the systemd folks haven't exactly been the best when it comes to PR or when it comes to being even near feature parity with what they tried to replace. But now, they aren't just at feature parity, they surpassed plain old cron.
Maybe it is time to lay cron to rest, at least slowly.
And in fact I do have a use-case for needing to run something ~5 minutes after the system boots and then every ~12 hours onward from there. It's great that systemd timers has me covered!
I wish documentation for tools would explain their abstractions concepts in terms of its primitives.
Great post, thanks!
[Service]
Type=oneshot
WorkingDirectory={{ home }}/current/
Environment=RAILS_ENV=production
ExecStart=/bin/sh -lc "bin/db-backup --verbose"
[Service]
Type = oneshot
WorkingDirectory = %h/current/
Environment = RAILS_ENV=production
ExecStart = /bin/sh -lc "bin/db-backup --verbose" Environment = MULTIPLE=environment VARIABLES="in single line"I find myself doing this sort of thing all the time..
Over all I think Systemd get way to much criticism. You don't have to use all the parts, but if you care to go through the documentation you'll find interesting features such as journald log-shipping and systemd-machined which can manage containers and VMs.
If I never recall hacking in ulimit calls in the top of buggy shell scripts for crappy old services that done respect pam_limits it won’t be soon enough.
Could have been YAML.
Could have been XML.
It would also make it much simpler to make good GUI editors for the files instead of the Notepad approach most unix config files take.
https://www.freedesktop.org/software/systemd/man/latest/syst...
XML is that wonderful format that gave us vulnerabilities like death by million laughs, up to a certain moment, you could MitM DTDs, and a whole slew of everything-XML stuff back when XML was like AI is today, none of which I miss today.
Oh, and remember times when programmers would argue whether argument order in XML files should be significant or not?
But XML books with their idealized XML future description did give me the same warm fuzzies as some intricate clockwork mechanism to a Victorian geek.
Could have been XML Property Lists.
ducks
It's pronounced, "primmer."
Now you get to look around the myriad of places where you can put systemd files, and figure out which ones are base services and which ones are custom, with no general convention to go about it. Nope.
With —-all
I have knocked together a systemd service or three based on google copypasta. But generally, for cron jobs, why make it complicated? One line in /etc/crontab and done. I generally call an encapsulation script that sets the right environment variables, uses absolute paths, captures stdout/stderr if required and so on. I just want the simplest possible way to launch that script on a schedule.
Oh but it won't appear in the timer-specific logs, I guess...
https://www.freedesktop.org/software/systemd/man/latest/syst...
1) It's supported by cronie. I bet it's supported by many other crons.
2) "Great" news! The software in the Systemd Project only officially runs on Linux, so "it's not portable" is a really bad counterargument when "alternatives to some Systemd Project feature" is the discussion topic.
Ain't that the truth. Literally every crontab I've written for the last 10 years has had this in it:
2>&1 | logger -t cron-WHATEVER
...and that does a pretty good job of capturing anything that the script emits and making it easy to grep for in syslog the following morning.
But I'm still amazed at how many crontabs I run across that don't capture any output at all.
It's a shame docker never supported it. I feel like if they had got on board all those years ago there would be broad support across the software ecosystem for it and we wouldn't need half of these complicated iptables rules and proxies and service mesh. It would be a step towards a capability based system.
CPU speeds have increased & and i/o latency has decreased so much since then that startup times are generally imperceptible, so the pendulum has swung back to favouring socket activation.
The anti-systemd "traditionalists" never seem to acknowledge that history, though!
I am dealing with mostly non systemd system: BSD, Alpine, termux On BSD anacron works well, but I do not why I am always running into problems with the cronie anacron implementation. And it is very hard to debug.
I would really like a simple modern cron/anacron alternative.
Cronicle looked cool but it is node.js, a bit heavy and being replace now by their new product called xyOps anyway.
One of our customers called in with a production down incident caused by a full disk. We got a copy of the VM and took a look. Investigation revealed that / was full because /var/log was full and that our 'logrotate' timer unit that was scheduled to run once a day had run either exactly never or exactly once... I can't remember which. Further investigation revealed no difference in software load or configuration between this VM and a VM that had a functional logrotate timer unit. Exactly one VM out of hundreds of identical VMs at this site (and many multiples of that at other customer's sites) were affected by this. Advising the customer to clear out /var/log and reboot did not unstick 'logrotate', and none of the diagnostics or fixes we could find anywhere unstuck it. Once "systemd-crond" decided to never schedule this job ever again, it stuck to that decision.
After a lot of searching, we found an open bug report from a year or three prior where someone reported exactly the same symptoms and was scheduling a unit with pretty much the same set of unit configuration flags that we were using. The conversation from the core devs ran through the pattern that one gets used to seeing when one runs into SystemD bugs that are caused by extremely complex unanticipated interactions between parts of the project: "That's not a bug, only an idiot would want that to work.", "Oh, we don't document that that's not supposed to work?", "Wow, okay, yeah, I can see how that maybe should work. That it doesn't sure does seem weird.", "Having said that, I don't know if it's supposed to work, or if it's unsupported. Someone should really either document that or fix it."... and then the behavior is neither fixed nor documented. [1] Absent any actual explanation for the failure, we ended up swizzling the options in our 'logrotate' unit and praying that satisfied whatever gremlin arose from the depths to trouble our customer.
SystemD contains an enormous -and ever-growing- amount of accidental complexity, and has a set of core maintainers who are generally disinterested in either documenting the places where one or more complex systems bind together to cause stop-the-world problems or fixing the systems involved so that they don't bind up. It's a fine project until it's very, very suddenly not, and then you're absolutely SOL. If you're lucky, you can shuffle around what you're doing [2] and hope that avoids the problem. [3]
[0] Some folks use the spelling "SystemD" to mock the project. I use the spelling "SystemD" to distinguish between "the entire systemd project" and systemd(1). I do this because some folks will make a claim like "systemd is very, very small and self-contained. I don't understand why anyone would say otherwise.", but what they are actually saying is that systemd(1) is a fairly small program that doesn't do all that much when run as PID 1. It sucks minor amounts of ass that the project and the program it runs as PID 1 share the same name, but what can you do?
[1] No, I don't have a link to the open bug report. This was more than a year ago, so the bug ID has been long forgotten.
[2] The term of art for this practice is "wave a dead chicken at it".
[3] Plus, like, even disregarding most of the rest of my report... how in the hell do you design a cron that knows a job is scheduled to be run periodically, can tell you how long it has been since it last ran, but never manages to run it? To me, that's unforgivable. It's a "You had one job!"-tier cockup.