docs
cairnobs.org Demo Source
Esc

What it watches · 5 of 14

Backups and disks

Are the backups arriving, fresh and the right size? Did the backup job succeed? Is a RAID array missing a disk, or a disk failing SMART? One file says what each host's backups should be.

1 min read

Cairn OBS doesn't make backups; it checks yours. A small scan on each host lists backup folders and reads file sizes and times. It never opens a backup. It also reads the last result of each backup job, /proc/mdstat for software RAID, and smartctl -H for every disk.

Set it up

  1. Write /etc/obs-beacon/backups.yaml, saying what each host's backups should be (below), and point obs-beacond.yaml at it: backups: /etc/obs-beacon/backups.yaml.
  2. Put the same file on every host it names, and start the scan's timer there: sudo systemctl enable --now obs-beacon-scan.timer. It scans every 15 minutes, as root, with no network, and reads only its own host's section. (Installing or upgrading the agent package starts the timer by itself when the file is already there.)
  3. In each of those hosts' agent.yaml, set backup_scan: /var/lib/obs-beacon-scan/backups.json, where the scan writes its result, so reports carry it.

The Backups page appears once the file is set.

backups.yaml

max_age: 26h      # the newest copy older than this alerts (default 26h)
shrink: 0.5       # the newest under half the one before warns (default 0.5)
disk_warn: 0.9    # a backup disk fuller than this warns (default 0.9)
hosts:
  backup1:
    received:     # a folder per backup under root, dated copies inside
      - {from: web1, root: /srv/backups/web1, expect: [shop-db, web1-system]}
    made:         # backups this host makes: the newest file over the globs
      - {name: billing-pg, paths: ["/srv/backups/db*/billing-pg/*.dump"]}
    jobs: [restic-backup.service, pg-dump.service]
  desk1: {}       # a workstation: only its disks' SMART health
Key What it's for
received Backups other hosts send here: from which host, root the folder, expect the backups that must be in it, one subfolder each
made Backups this host makes, by name, found by the newest file over paths
jobs systemd units that run backups. Their last result counts

What's judged

What Alert
A backup missing, empty, or its newest copy older than max_age Serious
The newest copy under shrink of the one before Warning
A backup disk fuller than disk_warn Warning
A job's last run didn't succeed Serious. A job systemd doesn't know warns
A software RAID array missing a member Critical
A disk failing SMART's health check Critical. One smartctl can't read warns
No scan, or one over an hour old Warning

A host that's gone silent isn't judged on its backups (its silence already alerts); its last scan stays on the page.