What it watches · 14 of 14
PostgreSQL and the data layer
Patroni clusters, HAProxy backends, NATS, etcd and Garage judged from their own status pages: no leader, a replica behind, a backend with no primary, etcd short of space or slow to sync.
The Data layer page watches the pieces a highly available application
stands on, from their own status pages. It appears once datalayer is in
obs-beacond.yaml.
Patroni
obs-beacond asks each Patroni member's REST API every 15 seconds:
datalayer:
patroni:
- http://10.0.0.11:8008
- http://10.0.0.12:8008
- http://10.0.0.13:8008
sync_standby: true # there should always be one
preferred_leader: db1 # optional
# lag_warn_mb: 16
# dcs_stale: 60s
| What | Alert |
|---|---|
| No leader, or no member answers | Critical |
| A member not running or not streaming, or missing | Warning |
A replica more than lag_warn_mb behind |
Warning |
No synchronous standby, with sync_standby: true |
Warning |
Patroni not hearing from etcd for dcs_stale |
Warning |
The leader isn't preferred_leader: a failover happened, and it should be moved back |
Warning |
A new leader goes in the audit log.
Status pages on the hosts
probes are status pages each agent reads on its own host, on loopback or
private addresses only, so nothing is opened to the network for them:
datalayer:
probes:
- id: haproxy
kind: haproxy
url: http://127.0.0.1:8404/stats;csv
exactly_one: [pg-primary] # one server up, never two
at_least_one: [pg-replicas]
from: [lb1, lb2] # default: every agent
- id: nats
kind: nats
url: http://127.0.0.1:8222/varz
peers: 2
- id: etcd
kind: etcd
url: http://127.0.0.1:2381/metrics
# quota_warn_pct: 75
# fsync_warn_ms: 25 # spinning disks need more, such as 500
- id: garage
kind: garage
url: http://127.0.0.1:3903/health
| Kind | Judged |
|---|---|
haproxy |
Every backend: those in exactly_one need exactly one server up (a database primary), those in at_least_one at least one (its replicas), the rest all of them |
nats |
peers other servers connected; warns when slow consumers grow |
etcd |
Its metrics listener (--listen-metrics-urls). Critical with no leader; warns at quota_warn_pct of its space quota, and when the WAL fsync p99 is over fsync_warn_ms |
garage |
"Fully operational" |
A status page that doesn't answer is critical.
These alerts belong to the service map's patroni, etcd, haproxy, nats and garage services, so a database down groups what it takes with it (How alerts work).