Alerts and reports · 1 of 4
How alerts work
From a condition to an incident to a notice: the two-minute wait, repeats, acknowledging, resolving, and grouping by cause so one failure is one page, not twenty.
Every few seconds the server judges everything it knows and gets a list of conditions: one thing that's wrong, with a severity. Conditions become incidents, and incidents become the notices people get.
Severities
| Severity | Means | ntfy priority |
|---|---|---|
| Warning | Look at it today | Default |
| Serious | Look at it soon: a backup missing, a job failed | High |
| Critical | Something is down now | Urgent |
Each alert channel has a minimum: a pager channel might take only critical, a chat channel everything.
From condition to notice
- A new condition opens an incident, but nobody is told until it has
lasted
alerts.for(2 minutes), so a blip doesn't page anyone. - Once it fires, every channel whose minimum it reaches is told.
- A critical incident nobody has acknowledged is repeated every
alerts.repeat(15 minutes). One that gets worse is sent again at once. - When the condition goes away, the incident resolves, and the channels that heard about it hear that too. One that never fired just goes.
alerts:
for: 2m
repeat: 15m
public_url: https://obs.example.org # links in notices point here
Acknowledging an incident, on the Alerts page, stops its repeats and records who took it. Operators and admins can acknowledge, and write what the cause turned out to be; the cause shows on the Reports page.
Incidents and the notices waiting to go out are kept in the database, so a restart, or another copy taking over, carries on where it left off and sends what was left. The price is that a notice can now and then arrive twice.
One failure, one page
When a database dies, everything that needs it fails too. Cairn OBS groups alerts by what depends on what before they become incidents:
- dependencies you declared (topology.yaml
needs, and Kubernetes' own Services and Ingresses), - dependencies learned from the service map once they're steady,
- a silent host needs its VM, a VM its hypervisor or node, a pod its controller and node.
When a failing thing needs another failing thing, only the deepest failure becomes an incident. Its detail says "Also affected: …" with how many alerts were held back, and it takes the group's worst severity, so a critical symptom still pages. When a learned dependency did the grouping, the detail says so: "Grouped because webmail has talked to postgresql :5432 since Nov 2." If that's wrong, Not a dependency on the incident splits the group, and the edge is never used again.
Two rules keep grouping safe:
- An alert that had already been sent stays on its own. Nothing reads as "resolved" just because its cause was found later.
- Disk, memory and similar host warnings stand alone: they're trouble on a host, not why its services fail.
The dashboard groups the same way, and the service map names the likely cause on a failing service.
Quiet times
- Maintenance windows hold a host's alerts while it's expected to be down (Maintenance windows).
- Workstations never go above warning.
- Workloads in clusters stay warnings until they're adopted (What pages).