docs
cairnobs.org Demo Source
Esc

What it watches · 4 of 14

Certificates

Every certificate in the fleet, where it's used and when it ends: found on listening ports, in files, in Windows stores, in Kubernetes, Vault, Cloudflare and Akamai. One alert per certificate, never a private key.

3 min read

The Certificates page answers "what ends this month, and where is it used?" It lists every certificate Cairn OBS has found, soonest end first, with its names, issuer, key, and every place it's used: "web1:443, web2:443, web1:/etc/nginx/tls/shop.pem, shop Ingress in prod". The same certificate found in many places is one row, by its SHA-256 fingerprint.

It's read-only, and public certificates only. No private key is ever read, copied or sent.

Where they're found

Where How
Ports hosts listen on Once a day, the agent tries a TLS handshake on each TCP port its host listens on, with STARTTLS on 25, 587 (SMTP), 143 (IMAP), 110 (POP3), 389 (LDAP) and 5432 (PostgreSQL). It finds what nobody listed. Ports 22, 53, 111, 123, 2049 and 5355 are never tried. cert_scan: false turns it off
Files on Linux hosts The root helper reads the files matching cert_files every ten minutes and gives back only the certificates in them. A file holding a key and a certificate gives back the certificate alone. CA bundles are never read
Windows certificate stores The machine's My and WebHosting stores (cert_stores), read-only
Kubernetes and OpenShift cert-manager's Certificate objects (no Secrets read), and a handshake with each Ingress and Route host, as a visitor would
Vault and OpenBao Every issuer and leaf in their PKI engines (Vault and OpenBao)
Checks The certificate each https or TLS check is served
Cloudflare and Akamai Edge certificate packs, and Akamai's CPS enrollments

The default cert_files on Linux:

cert_files:
  - /etc/letsencrypt/live/*/cert.pem
  - /etc/pki/tls/certs/*.crt
  - /etc/nginx/**/*.{pem,crt}
  - /etc/haproxy/**/*.pem
  - /etc/postfix/**/*.pem
  - /etc/dovecot/**/*.pem

** is any number of folders, {pem,crt} either; [] reads none.

What's judged

  • Ending: 21 days is a warning, 7 critical. A short-lived certificate, under 7 days' life, by a third and a tenth of its life. A certificate only in files, served on no port, is judged at 7 days only: a spare file may be nobody's.
  • Renewed but not reloaded: a port still serving a certificate an hour after its host got a file with a newer one for the same names. The classic outage: certbot renewed it, nobody reloaded nginx. The alert names the file and the ports.
  • Served wrong: a name the certificate doesn't cover, a chain sent without its intermediate, or a self-signed certificate on a port that had a CA-signed one. Warnings.
  • Weak: SHA-1 signatures and RSA keys under 2048 bits. Warnings.

There's one alert per certificate, not per place: "shop.example.org ends in 6 days", with where it's used in the detail. A certificate replaced everywhere stops alerting. One that a check, Vault, Cloudflare or a cluster also has alerts there, so nothing alerts twice.

Fixing it from the page

Cairn OBS only looks, with two narrow exceptions, each off until you turn it on, each a button an admin presses, and each in the audit log.

Reload a service

For "renewed but not reloaded", Cairn OBS can ask the host to reload the service. The host's own agent.yaml must allow it, unit by unit, with the files each serves:

cert_reload:
  - unit: nginx.service
    files: [/etc/letsencrypt/live/*/fullchain.pem]
  - unit: haproxy.service
    files: [/etc/haproxy/certs/*.pem]

The root helper reads that list from agent.yaml itself, so the server's word isn't taken. It reloads only when a listed file holds a certificate for the same names that ends later, has started, and matches its key; only with systemctl reload, never a restart, and never a unit that can't reload. The agent then checks that the port serves the new certificate before it reports done.

Without cert_reload, the page says how to allow it.

Ask cert-manager to renew

For a certificate behind a cert-manager Certificate that needs a look, Renew now asks cert-manager to renew it, as cmctl renew does. The cluster agent needs the -cert-renew flag and a small extra role that may patch a Certificate's status and nothing else. Add the cluster with /cluster.yaml?cert_renew=1 instead of /cluster.yaml to get both.

Cairn OBS never issues certificates or holds their keys: certbot, cert-manager and Vault Agent already do that well.