docs
cairnobs.org Demo Source
Esc

Alerts and reports · 1 of 4

How alerts work

From a condition to an incident to a notice: the two-minute wait, repeats, acknowledging, resolving, and grouping by cause so one failure is one page, not twenty.

2 min read

Every few seconds the server judges everything it knows and gets a list of conditions: one thing that's wrong, with a severity. Conditions become incidents, and incidents become the notices people get.

Severities

Severity Means ntfy priority
Warning Look at it today Default
Serious Look at it soon: a backup missing, a job failed High
Critical Something is down now Urgent

Each alert channel has a minimum: a pager channel might take only critical, a chat channel everything.

From condition to notice

  1. A new condition opens an incident, but nobody is told until it has lasted alerts.for (2 minutes), so a blip doesn't page anyone.
  2. Once it fires, every channel whose minimum it reaches is told.
  3. A critical incident nobody has acknowledged is repeated every alerts.repeat (15 minutes). One that gets worse is sent again at once.
  4. When the condition goes away, the incident resolves, and the channels that heard about it hear that too. One that never fired just goes.
alerts:
  for: 2m
  repeat: 15m
  public_url: https://obs.example.org    # links in notices point here

Acknowledging an incident, on the Alerts page, stops its repeats and records who took it. Operators and admins can acknowledge, and write what the cause turned out to be; the cause shows on the Reports page.

Incidents and the notices waiting to go out are kept in the database, so a restart, or another copy taking over, carries on where it left off and sends what was left. The price is that a notice can now and then arrive twice.

One failure, one page

When a database dies, everything that needs it fails too. Cairn OBS groups alerts by what depends on what before they become incidents:

  • dependencies you declared (topology.yaml needs, and Kubernetes' own Services and Ingresses),
  • dependencies learned from the service map once they're steady,
  • a silent host needs its VM, a VM its hypervisor or node, a pod its controller and node.

When a failing thing needs another failing thing, only the deepest failure becomes an incident. Its detail says "Also affected: …" with how many alerts were held back, and it takes the group's worst severity, so a critical symptom still pages. When a learned dependency did the grouping, the detail says so: "Grouped because webmail has talked to postgresql :5432 since Nov 2." If that's wrong, Not a dependency on the incident splits the group, and the edge is never used again.

Two rules keep grouping safe:

  • An alert that had already been sent stays on its own. Nothing reads as "resolved" just because its cause was found later.
  • Disk, memory and similar host warnings stand alone: they're trouble on a host, not why its services fail.

The dashboard groups the same way, and the service map names the likely cause on a failing service.

Quiet times

  • Maintenance windows hold a host's alerts while it's expected to be down (Maintenance windows).
  • Workstations never go above warning.
  • Workloads in clusters stay warnings until they're adopted (What pages).