docs
cairnobs.org Demo Source
Esc

What it watches · 14 of 14

PostgreSQL and the data layer

Patroni clusters, HAProxy backends, NATS, etcd and Garage judged from their own status pages: no leader, a replica behind, a backend with no primary, etcd short of space or slow to sync.

1 min read

The Data layer page watches the pieces a highly available application stands on, from their own status pages. It appears once datalayer is in obs-beacond.yaml.

Patroni

obs-beacond asks each Patroni member's REST API every 15 seconds:

datalayer:
  patroni:
    - http://10.0.0.11:8008
    - http://10.0.0.12:8008
    - http://10.0.0.13:8008
  sync_standby: true          # there should always be one
  preferred_leader: db1       # optional
  # lag_warn_mb: 16
  # dcs_stale: 60s
What Alert
No leader, or no member answers Critical
A member not running or not streaming, or missing Warning
A replica more than lag_warn_mb behind Warning
No synchronous standby, with sync_standby: true Warning
Patroni not hearing from etcd for dcs_stale Warning
The leader isn't preferred_leader: a failover happened, and it should be moved back Warning

A new leader goes in the audit log.

Status pages on the hosts

probes are status pages each agent reads on its own host, on loopback or private addresses only, so nothing is opened to the network for them:

datalayer:
  probes:
    - id: haproxy
      kind: haproxy
      url: http://127.0.0.1:8404/stats;csv
      exactly_one: [pg-primary]       # one server up, never two
      at_least_one: [pg-replicas]
      from: [lb1, lb2]                # default: every agent
    - id: nats
      kind: nats
      url: http://127.0.0.1:8222/varz
      peers: 2
    - id: etcd
      kind: etcd
      url: http://127.0.0.1:2381/metrics
      # quota_warn_pct: 75
      # fsync_warn_ms: 25             # spinning disks need more, such as 500
    - id: garage
      kind: garage
      url: http://127.0.0.1:3903/health
Kind Judged
haproxy Every backend: those in exactly_one need exactly one server up (a database primary), those in at_least_one at least one (its replicas), the rest all of them
nats peers other servers connected; warns when slow consumers grow
etcd Its metrics listener (--listen-metrics-urls). Critical with no leader; warns at quota_warn_pct of its space quota, and when the WAL fsync p99 is over fsync_warn_ms
garage "Fully operational"

A status page that doesn't answer is critical.

These alerts belong to the service map's patroni, etcd, haproxy, nats and garage services, so a database down groups what it takes with it (How alerts work).