Monitoring backups: know before you need them

Setup

Set up in three steps

1. Create a heartbeat

   In your account, create a new one under “Heartbeats”: a name, how often the backup runs (usually daily) and how much leeway it gets, one hour for example. You receive a ping address of its own, such as https://runcheck.de/a8k3m2x9q1.

2. Append the check-in to the crontab line

   Open the crontab with crontab -e and extend your backup line. $? is the script's exit code; this way we also learn whether it ended with an error. -m 10 limits the check-in to ten seconds, --retry 3 tries again after a network hiccup.

   0 3 * * *  /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 3 https://runcheck.de/a8k3m2x9q1/$? > /dev/null

3. Optional: report the start as well

   If you also call /start before the backup, we measure the duration of every run and alert you when the backup gets stuck and the end does not arrive.

   0 3 * * *  curl -fsS -m 10 --retry 3 https://runcheck.de/a8k3m2x9q1/start > /dev/null; /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 3 https://runcheck.de/a8k3m2x9q1/$? > /dev/null

4. Test it once by hand

   Run the same line without the schedule once on the command line. In your account, “Waiting for the first check-in …” turns green, and from now on we keep watch.

   /usr/local/bin/backup.sh; curl -fsS -m 10 --retry 3 https://runcheck.de/a8k3m2x9q1/$?

The ready-made lines for Linux, PHP, Python, Node.js, n8n, Make, Supabase and Vercel are on the page Cron job monitoring. Your own ping address appears in your account once you create the heartbeat, with the same lines ready to copy.

When something fails

What happens when a run fails

If the check-in is overdue, because the server was down, cron did not run or the script hangs, you get a message. If the script reports an exit code other than 0, we alert you right away, with the code in the message. There is one message per incident and an all-clear as soon as the next backup succeeds. The history shows every run with time and result.

FAQ

Common questions about Backup

How do I monitor a backup that consists of several commands?

Keep the first error in a variable and report it at the end. What counts is the code you send, for example with restic:

#!/bin/sh
restic backup /var/www /etc
rc=$?
restic forget --keep-daily 7 --prune || rc=$?
curl -fsS -m 10 --retry 3 https://runcheck.de/a8k3m2x9q1/$rc > /dev/null

rsync exits with code 24. Did the backup fail?

Not necessarily. rsync reports 24 when source files vanished during the transfer, rotated log files for example. For many backups that is harmless. If it should not be reported, set the code to 0 before checking in: rsync -a /source/ /target/; rc=$?; [ $rc -eq 24 ] && rc=0; curl -fsS -m 10 --retry 3 https://runcheck.de/a8k3m2x9q1/$rc

borg reports exit code 1. Is that an error?

For BorgBackup, 1 is a warning: the run reached its end, but the log contains warnings. Since we treat every code other than 0 as a failure, you hear about it too. If you only want real errors (2 and up) reported, set 1 to 0 before checking in: [ $rc -eq 1 ] && rc=0.

How do I notice when a backup gets stuck?

Call https://runcheck.de/a8k3m2x9q1/start at the beginning, as in the optional step above. Then we know a run has started, measure its duration and alert you if the end does not arrive in time.

Can I send the backup log along?

Yes, via POST up to 10 KB, for example the last lines: tail -n 20 /var/log/backup.log | curl -fsS -m 10 --data-binary @- https://runcheck.de/a8k3m2x9q1/$?. Do not put personal data in it.

Do I need a monitoring system of my own?

No. On the server, curl (or wget) is enough, and most Linux systems have it already. Schedule, leeway and alerts are set in your Cronjob.de account. One heartbeat with email alerts is free.