docs
cairnobs.org Demo Source
Esc

What it watches · 1 of 14

Hosts, containers and services

What every agent reports, how a host, a container and a service are judged, and how to tell Cairn OBS what must keep running.

3 min read

Every 15 seconds each agent sends a report: CPU, memory, disks, network, uptime, the OS, its containers and its services. The server judges each report as it arrives.

How a host is judged

Condition Severity
No report for longer than stale_after (default 60 seconds) Critical: "has gone silent"
A disk 90% full or more Critical
A disk 80% full or more Warning
Less than 10% of memory available Warning
A service that must be active isn't, or isn't installed Critical
A container that must run isn't, or is unhealthy or restarting Critical
Docker is there but the agent can't read it Warning
On Windows, a critical event in the event log in the last hour Warning

A workstation goes no higher than warning on any of these, going silent included (Desk machines).

When a host that went silent is a VM Cairn OBS can see from its hypervisor, the alert says what the hypervisor says about it: stopped, paused or gone.

What must keep running

Out of the box a host with no expectations is judged on Docker's own states alone, and no service is required. To say what matters, open the host under Configure › Hosts and press Adopt what's running. It lists the containers the agent sees and the services running there; pick:

  • the containers that must stay running,
  • the ones stopped on purpose, with a note,
  • the services that must stay active.

That's the host's part of expected.yaml, which you can keep in a file instead. Container names can use *, so shop-* covers every replica.

How a container is judged

Situation Shows as
Must run; running, healthy or with no health check OK · Running
Must run; health check failing Critical · Unhealthy
Must run; restarting Critical · Restarting
Must run; exited or never started Critical · Not running
A "must run" name matches nothing at all Critical · Missing
Stopped on purpose, and stopped Off · Stopped on purpose
Stopped on purpose, but running Warning · Running, with the note
On a host with expectations, running, but matches nothing Warning · Not expected
On a host with no expectations Judged on Docker's state alone
A CI job container (ci_jobs) Counted, never listed

Image drift

The same container name running different images (by name and tag) on different hosts warns once it has lasted 15 minutes: a deploy that didn't reach every host. Digests aren't compared, since hosts that build their own images get different digests for the same content.

What you see

  • Hosts opens on one square per host, grouped by site and by role (the host name's first word after its site), so the odd one out stands out, with disks filling up, hosts busier than usual and recent restarts above. List has the table: every host with its state, CPU, memory, fullest disk and uptime. A host's page has its history, disks, network, services, containers and, where there are any, its VMs, certificates and event log.
  • Containers lists every container on every host, grouped by compose project. For a container in trouble (restarting, unhealthy, or stopped in the last day) the agent sends its last 20 log lines, which operators and admins can read there and on the alert.
  • Today, the dashboard, says first whether anything needs you, then lists it worst first, grouped by likely cause (How alerts work), next to what can wait, your services and what changed since you last looked. Services lists every service by who depends on it, with its last day and month.

Who watches the server

When obs-beacond itself is down, it can't tell anyone. So an agent or two on other hosts can watch it, with watch_server in their agent.yaml:

watch_server:
  public_url: https://obs.example.org/healthz
  after: 3m                # reports failing this long before it looks closer
  channels:                # the same format as obs-beacond's alert channels
    - id: ntfy
      type: ntfy
      topic_file: /etc/obs-beacon/ntfy-topic
  quiet:                   # the server's own maintenance, such as a nightly reboot
    - name: nightly reboot
      daily: "04:00"
      timezone: Europe/Berlin
      duration: 15m

When its reports have failed for after, the agent fetches the server's public health page. If that fails too, the server is down: the agent sends "Cairn OBS is down" through its own channels, once, and "resolved" when a report gets through again. If the page answers, only this host's path is broken, and the server will say the host went silent, so the agent stays quiet. Set it on hosts other than the one running obs-beacond.