What it watches · 9 of 14
Vault and OpenBao
HashiCorp Vault and OpenBao clusters read with a monitoring token that can't read a secret: seal, HA, Raft, audit devices, replication, and every PKI certificate before it ends.
obs-beacond reads each Vault cluster itself, every minute, through one address: its load balancer or any server. OpenBao has the same API and is read the same way. Each cluster is a source of its own on Clusters, its Raft servers as its nodes, and its PKI certificates join the certificate inventory.
Never a secret. The monitoring token's policy can read the cluster's state and PKI mounts' public certificates, and nothing under any other secrets engine.
Set it up
-
Write the policy:
path "sys/storage/raft/autopilot/state" { capabilities = ["read"] } path "sys/license/status" { capabilities = ["read"] } path "sys/metrics" { capabilities = ["read"] } path "sys/mounts" { capabilities = ["read"] } path "+/issuers" { capabilities = ["list"] } path "+/issuer/+" { capabilities = ["read"] } path "+/certs" { capabilities = ["list"] } path "+/certs/revoked" { capabilities = ["list"] } path "+/cert/+" { capabilities = ["read"] } path "auth/token/renew-self" { capabilities = ["update"] } -
Make a periodic token from it. It renews itself, so it never ends while obs-beacond runs:
vault policy write obs-monitor obs-monitor.hcl vault token create -policy=obs-monitor -period=24h -display-name=obs-beacond -field=token > vault-token -
Put the token in a file only obs-beacond can read (
root:obs-beacon,0640), and add the cluster to obs-beacond.yaml:vault: - url: https://vault.example.org:8200 token_file: /etc/obs-beacon/vault-token # ca_file: /etc/obs-beacon/vault-ca.pem # or fingerprint: "AB:CD:…" # name: vault # default the URL's host name # every: 60s # min_lifetime: 168h # leaves meant to last less are counted, not judged
The file is read again when it changes, so rotating the token needs no restart. With several copies of obs-beacond, every copy needs the file; only the one judging alerts reads Vault.
What's judged
Vault raises no alerts of its own, so these thresholds are Cairn OBS's:
| What | Alert |
|---|---|
| Sealed, or not initialized | Critical |
| HA on and no leader | Critical |
| A Raft server failed or left | Critical |
| A Raft server unhealthy (autopilot) | Warning |
| Raft unhealthy, or three voters or more and no failure tolerance left | Warning |
| The monitoring token refused | Critical |
| The token ends within 21 / 7 days and isn't renewing itself in full | Warning / critical |
| A PKI issuer (a CA) ends within 21 / 7 days, or has ended | Warning / critical |
| A leaf certificate, the newest for its names, ends within 21 / 7 days (or a third / a tenth of a short life), or ended in the last 7 days | Warning / critical |
| A PKI mount's certificates can't be listed or read | Warning |
| An audit device failed to log | Critical, for 15 minutes after the last failure |
sys/metrics or Raft's state can't be read |
Warning |
| Enterprise: a replication secondary not ready, silent or catching up | Warning |
| Enterprise: no secondary streaming, one cut off from its primary, or a corrupt Merkle tree | Critical |
| Enterprise: the license expires within 21 / 7 days, or has expired | Warning / critical |
| The API doesn't answer | Critical; what was seen stays |