What it watches · 5 of 14
Backups and disks
Are the backups arriving, fresh and the right size? Did the backup job succeed? Is a RAID array missing a disk, or a disk failing SMART? One file says what each host's backups should be.
Cairn OBS doesn't make backups; it checks yours. A small scan on each host
lists backup folders and reads file sizes and times. It never opens a
backup. It also reads the last result of each backup job, /proc/mdstat
for software RAID, and smartctl -H for every disk.
Set it up
- Write
/etc/obs-beacon/backups.yaml, saying what each host's backups should be (below), and point obs-beacond.yaml at it:backups: /etc/obs-beacon/backups.yaml. - Put the same file on every host it names, and start the scan's timer
there:
sudo systemctl enable --now obs-beacon-scan.timer. It scans every 15 minutes, as root, with no network, and reads only its own host's section. (Installing or upgrading the agent package starts the timer by itself when the file is already there.) - In each of those hosts' agent.yaml, set
backup_scan: /var/lib/obs-beacon-scan/backups.json, where the scan writes its result, so reports carry it.
The Backups page appears once the file is set.
backups.yaml
max_age: 26h # the newest copy older than this alerts (default 26h)
shrink: 0.5 # the newest under half the one before warns (default 0.5)
disk_warn: 0.9 # a backup disk fuller than this warns (default 0.9)
hosts:
backup1:
received: # a folder per backup under root, dated copies inside
- {from: web1, root: /srv/backups/web1, expect: [shop-db, web1-system]}
made: # backups this host makes: the newest file over the globs
- {name: billing-pg, paths: ["/srv/backups/db*/billing-pg/*.dump"]}
jobs: [restic-backup.service, pg-dump.service]
desk1: {} # a workstation: only its disks' SMART health
| Key | What it's for |
|---|---|
received |
Backups other hosts send here: from which host, root the folder, expect the backups that must be in it, one subfolder each |
made |
Backups this host makes, by name, found by the newest file over paths |
jobs |
systemd units that run backups. Their last result counts |
What's judged
| What | Alert |
|---|---|
A backup missing, empty, or its newest copy older than max_age |
Serious |
The newest copy under shrink of the one before |
Warning |
A backup disk fuller than disk_warn |
Warning |
| A job's last run didn't succeed | Serious. A job systemd doesn't know warns |
| A software RAID array missing a member | Critical |
| A disk failing SMART's health check | Critical. One smartctl can't read warns |
| No scan, or one over an hour old | Warning |
A host that's gone silent isn't judged on its backups (its silence already alerts); its last scan stays on the page.