docs
cairnobs.org Demo Source
Esc

What it watches · 9 of 14

Vault and OpenBao

HashiCorp Vault and OpenBao clusters read with a monitoring token that can't read a secret: seal, HA, Raft, audit devices, replication, and every PKI certificate before it ends.

2 min read

obs-beacond reads each Vault cluster itself, every minute, through one address: its load balancer or any server. OpenBao has the same API and is read the same way. Each cluster is a source of its own on Clusters, its Raft servers as its nodes, and its PKI certificates join the certificate inventory.

Never a secret. The monitoring token's policy can read the cluster's state and PKI mounts' public certificates, and nothing under any other secrets engine.

Set it up

  1. Write the policy:

    path "sys/storage/raft/autopilot/state" { capabilities = ["read"] }
    path "sys/license/status"               { capabilities = ["read"] }
    path "sys/metrics"                      { capabilities = ["read"] }
    path "sys/mounts"                       { capabilities = ["read"] }
    path "+/issuers"                        { capabilities = ["list"] }
    path "+/issuer/+"                       { capabilities = ["read"] }
    path "+/certs"                          { capabilities = ["list"] }
    path "+/certs/revoked"                  { capabilities = ["list"] }
    path "+/cert/+"                         { capabilities = ["read"] }
    path "auth/token/renew-self"            { capabilities = ["update"] }
    
  2. Make a periodic token from it. It renews itself, so it never ends while obs-beacond runs:

    vault policy write obs-monitor obs-monitor.hcl
    vault token create -policy=obs-monitor -period=24h -display-name=obs-beacond -field=token > vault-token
    
  3. Put the token in a file only obs-beacond can read (root:obs-beacon, 0640), and add the cluster to obs-beacond.yaml:

    vault:
      - url: https://vault.example.org:8200
        token_file: /etc/obs-beacon/vault-token
        # ca_file: /etc/obs-beacon/vault-ca.pem   # or fingerprint: "AB:CD:…"
        # name: vault                             # default the URL's host name
        # every: 60s
        # min_lifetime: 168h                      # leaves meant to last less are counted, not judged
    

The file is read again when it changes, so rotating the token needs no restart. With several copies of obs-beacond, every copy needs the file; only the one judging alerts reads Vault.

What's judged

Vault raises no alerts of its own, so these thresholds are Cairn OBS's:

What Alert
Sealed, or not initialized Critical
HA on and no leader Critical
A Raft server failed or left Critical
A Raft server unhealthy (autopilot) Warning
Raft unhealthy, or three voters or more and no failure tolerance left Warning
The monitoring token refused Critical
The token ends within 21 / 7 days and isn't renewing itself in full Warning / critical
A PKI issuer (a CA) ends within 21 / 7 days, or has ended Warning / critical
A leaf certificate, the newest for its names, ends within 21 / 7 days (or a third / a tenth of a short life), or ended in the last 7 days Warning / critical
A PKI mount's certificates can't be listed or read Warning
An audit device failed to log Critical, for 15 minutes after the last failure
sys/metrics or Raft's state can't be read Warning
Enterprise: a replication secondary not ready, silent or catching up Warning
Enterprise: no secondary streaming, one cut off from its primary, or a corrupt Merkle tree Critical
Enterprise: the license expires within 21 / 7 days, or has expired Warning / critical
The API doesn't answer Critical; what was seen stays