Run it · 3 of 4
Scale and several copies
From one container on SQLite to several copies of obs-beacond on PostgreSQL with no downtime for upgrades or failures. How copies share the work, how one evaluator is kept, and what's been measured.
Cairn OBS is one program at every size:
- Small: one server or one container, one SQLite file, nothing else to run. A restart is a gap of under a second, and nothing is lost in it.
- Large: several copies of obs-beacond sharing one PostgreSQL database, behind a load balancer or a Kubernetes Service. Upgrades, a copy dying and a database failover take nothing down.
What a copy does
Every copy takes agents' reports and serves the web interface. Neither
needs anything of its own: the latest report per host, sessions, settings
and sign-ins in progress are in the database, and copies tell each other
about changes through PostgreSQL's LISTEN/NOTIFY. No copy owns any
hosts, so adding copies needs no resharding.
One copy at a time is the evaluator. It judges alerts and sends notices, records history, sends the digest, and polls Cloudflare, Akamai, Vault, Patroni and the mail round trip, so each is asked once however many copies run. The others stand by. Which copy it is shows in the logs.
- The role is a lease in the database, renewed every 5 seconds. If the evaluator dies, another copy takes over within about 15 seconds, well inside the 2 minutes before anyone is told about a problem.
- Every write the evaluator makes carries the lease's number, and the database refuses an older one, so a copy that was paused and wakes up can't write.
- A notice goes into the database in the same transaction as the incident it's about, and is deleted once sent. A copy that takes over sends what's left. A notice is never lost; at worst, one is sent twice.
- On a normal stop, the evaluator hands the role over at once.
Upgrading with no gap
- Run three or more copies, with
maxUnavailable: 0or its equivalent, so a new copy is ready before an old one stops. - Check
/readyzfrom the load balancer. It passes only once a copy has caught up with the others, and fails as soon as the copy is told to stop. - Set
shutdown_delay(5 to 10 seconds behind a load balancer): a stopping copy keeps serving that long while traffic moves off, then gives requests in flightshutdown_grace(30 seconds) to finish.
During a rollout, old and new copies serve side by side. The schema only ever grows, and each copy stores its web interface's files in the database, so a page from either version finds its scripts on any copy. Agents of the previous release keep reporting.
/healthz answers while the process runs, for liveness probes; /readyz
is for readiness and load balancers. Both are on the web and agent ports.
The Kubernetes manifests set all of this up, with CloudNativePG for the database.
When the database fails over
Writes fail for a few seconds, and reports are answered with a 503. Agents keep them and send them again, so nothing is lost. Web copies serve from what they have and reconnect. The evaluator's lease survives a short gap.
Agents
Each agent keeps up to 240 reports on disk (an hour at the default 15 seconds) when the server can't be reached, and sends them oldest first when it answers. After an outage each waits a random few seconds first, so a thousand agents don't come back in the same second. A restart loses nothing.
A cluster agent sends thousands of pods as batches of changes, not one report per pod.
Measured
On one workstation, two copies on PostgreSQL 17, 1,000 agents every 15 seconds with 10 containers each:
| Run | Reports | Report p95 | Page p95 | Failures |
|---|---|---|---|---|
| Steady, 3 minutes | 12,000 (66.7 a second) | 2.3 ms | 27 ms | 0 |
| Both copies restarted one at a time | 10,000 | 2.3 ms | 26 ms | 0 |
Each copy used about 45 MB of memory and 3% of a CPU.
With 10,000 workloads added (three clusters of 30 nodes, 3,000 pods and 300 VMs each, changing all the time), the same 1,000 agents, and both copies restarted during the run: 32,000 reports made and 32,000 taken, page p95 17.8 ms, and a change in a cluster showing on both copies in 61 ms at p95. Each copy used about 75 MB.
On Kubernetes with three copies and CloudNativePG, probed every 50 ms:
| Event | Failed requests |
|---|---|
| A rolling upgrade, then a rollback | 0 of about 2,000 |
| The evaluator's pod deleted | 0; the role moved in 0.3 s |
| The evaluator killed outright | 7, in the 1.5 s before Kubernetes noticed; the role moved 15 s later |
| A database switchover | 0 |
| The database primary's pod deleted | 0 |
A single container on SQLite, upgraded under load from 200 agents: 0.6
seconds from docker stop to the new version answering, and every report
made was taken.
The targets Cairn OBS is built to: 10,000 workloads and 1,000 agents, pages under 300 ms at p95, and failover under 20 seconds.