What it watches · 1 of 14
Hosts, containers and services
What every agent reports, how a host, a container and a service are judged, and how to tell Cairn OBS what must keep running.
Every 15 seconds each agent sends a report: CPU, memory, disks, network, uptime, the OS, its containers and its services. The server judges each report as it arrives.
How a host is judged
| Condition | Severity |
|---|---|
No report for longer than stale_after (default 60 seconds) |
Critical: "has gone silent" |
| A disk 90% full or more | Critical |
| A disk 80% full or more | Warning |
| Less than 10% of memory available | Warning |
| A service that must be active isn't, or isn't installed | Critical |
| A container that must run isn't, or is unhealthy or restarting | Critical |
| Docker is there but the agent can't read it | Warning |
| On Windows, a critical event in the event log in the last hour | Warning |
A workstation goes no higher than warning on any of these, going silent included (Desk machines).
When a host that went silent is a VM Cairn OBS can see from its hypervisor, the alert says what the hypervisor says about it: stopped, paused or gone.
What must keep running
Out of the box a host with no expectations is judged on Docker's own states alone, and no service is required. To say what matters, open the host under Configure › Hosts and press Adopt what's running. It lists the containers the agent sees and the services running there; pick:
- the containers that must stay running,
- the ones stopped on purpose, with a note,
- the services that must stay active.
That's the host's part of expected.yaml, which you
can keep in a file instead. Container names can use *, so shop-*
covers every replica.
How a container is judged
| Situation | Shows as |
|---|---|
| Must run; running, healthy or with no health check | OK · Running |
| Must run; health check failing | Critical · Unhealthy |
| Must run; restarting | Critical · Restarting |
| Must run; exited or never started | Critical · Not running |
| A "must run" name matches nothing at all | Critical · Missing |
| Stopped on purpose, and stopped | Off · Stopped on purpose |
| Stopped on purpose, but running | Warning · Running, with the note |
| On a host with expectations, running, but matches nothing | Warning · Not expected |
| On a host with no expectations | Judged on Docker's state alone |
A CI job container (ci_jobs) |
Counted, never listed |
Image drift
The same container name running different images (by name and tag) on different hosts warns once it has lasted 15 minutes: a deploy that didn't reach every host. Digests aren't compared, since hosts that build their own images get different digests for the same content.
What you see
- Hosts opens on one square per host, grouped by site and by role (the host name's first word after its site), so the odd one out stands out, with disks filling up, hosts busier than usual and recent restarts above. List has the table: every host with its state, CPU, memory, fullest disk and uptime. A host's page has its history, disks, network, services, containers and, where there are any, its VMs, certificates and event log.
- Containers lists every container on every host, grouped by compose project. For a container in trouble (restarting, unhealthy, or stopped in the last day) the agent sends its last 20 log lines, which operators and admins can read there and on the alert.
- Today, the dashboard, says first whether anything needs you, then lists it worst first, grouped by likely cause (How alerts work), next to what can wait, your services and what changed since you last looked. Services lists every service by who depends on it, with its last day and month.
Who watches the server
When obs-beacond itself is down, it can't tell anyone. So an agent or two on
other hosts can watch it, with watch_server in their
agent.yaml:
watch_server:
public_url: https://obs.example.org/healthz
after: 3m # reports failing this long before it looks closer
channels: # the same format as obs-beacond's alert channels
- id: ntfy
type: ntfy
topic_file: /etc/obs-beacon/ntfy-topic
quiet: # the server's own maintenance, such as a nightly reboot
- name: nightly reboot
daily: "04:00"
timezone: Europe/Berlin
duration: 15m
When its reports have failed for after, the agent fetches the server's
public health page. If that fails too, the server is down: the agent sends
"Cairn OBS is down" through its own channels, once, and "resolved" when a
report gets through again. If the page answers, only this host's path is
broken, and the server will say the host went silent, so the agent stays
quiet. Set it on hosts other than the one running obs-beacond.