docs
cairnobs.org Demo Source
Esc

Install · 3 of 8

Kubernetes and OpenShift

obs-beacond on OpenShift, OKD or any Kubernetes from the kustomize manifests: one pod on a volume, or three or more copies on a CloudNativePG database.

4 min read

The obs-beacond image runs as whatever user ID the cluster assigns, in group 0, with a read-only root and no capabilities. That fits OpenShift's and OKD's default restricted-v2 security context constraint, and the Kubernetes "restricted" pod security standard, unchanged.

The server runs in the cluster; agents stay on the hosts they watch (Adding Linux hosts). To watch the cluster itself, add a cluster agent (Watching clusters, below).

Get the image

ghcr.io/coffey-labs/cairnobs:<version>. Take the manifests from the release you deploy (its tag in the source, such as v0.1.5): their base/kustomization.yaml names that release's image. If your clusters can't reach GitHub's registry, copy it to one they can, OpenShift's internal one included (The server in a container).

Choose an overlay

The manifests are in deploy/kubernetes in the source, as a kustomize base and overlays:

Overlay For Outside access
openshift/ OpenShift and OKD Two Routes: the web interface with edge TLS, the agent port passed through
plain/ Any other Kubernetes An Ingress for the web interface, a LoadBalancer Service for the agent port
openshift-ha/, plain-ha/ The same, with three or more copies on PostgreSQL As above

One copy

Before applying, set in your copy of the manifests:

  • base/kustomization.yaml: the image's registry path and tag, if you copied it.
  • base/obs-beacond.yaml: alerts.public_url (the web address) and join.agent_url (the agent address as hosts reach it: port 443 behind an OpenShift Route). Everything else in obs-beacond.yaml works here too.
  • openshift/routes.yaml or plain/ingress.yaml: the host names.
  • base/pvc.yaml: the storage class and size.

Then, in a namespace of its own:

oc new-project cairnobs
# The state key seals secrets in the database. Make it once and keep a copy
# somewhere safe: without it the database's secrets can't be read.
oc create secret generic obs-beacon-state-key \
  --from-literal=state.key="$(head -c 32 /dev/urandom | base64)"
oc apply -k deploy/kubernetes/openshift
oc logs deploy/obs-beacond        # the one-time link for the first admin

With kubectl, the same with kubectl create namespace and deploy/kubernetes/plain.

Storage

The volume holds the SQLite database, the agent CA and the certificate cache. SQLite needs a disk of its own, so the claim is ReadWriteOnce block storage with a file system on it:

Storage Usual class
Pure FlashArray through Pure CSI pure-block
Portworx px-csi-db, or a class with repl: "2" or more
OpenShift Data Foundation ocs-storagecluster-ceph-rbd

Never put it on a shared file system (NFS, pure-file, CephFS, ReadWriteMany): SQLite's locking isn't safe there. 10 GiB is plenty for dozens of hosts. The Deployment uses the Recreate strategy, so during an upgrade the old pod lets go of the volume before the new one mounts it.

Several copies

openshift-ha/ and plain-ha/ run three or more copies of obs-beacond on one PostgreSQL database, so a pod, a node or an upgrade takes nothing down:

  • A CloudNativePG Cluster, obs-beacon-db: three PostgreSQL instances that fail over by themselves. Install the operator first (OperatorHub on OpenShift and OKD, or its release manifest elsewhere). It makes the Secrets with the database password and the CA obs-beacond checks the database's certificate against.
  • The Deployment at three replicas, rolling with maxUnavailable: 0, so a new copy is ready before an old one stops. Pods are spread across nodes where possible.
  • A PodDisruptionBudget (one copy down at a time) and a HorizontalPodAutoscaler (3 to 10 copies at 70% CPU).
  • No volume claim: with PostgreSQL nothing on the pod needs keeping.

Set components/postgres/obs-beacond.yaml instead of base/obs-beacond.yaml (it replaces it), and the storage class and size in components/postgres/database.yaml. The same state key Secret serves every copy.

How copies share the work, and how to move a one-copy install onto PostgreSQL: Scale and several copies.

Agents reaching the server

Agents use TLS with client certificates from Cairn OBS's own CA, so their connection must reach the pod untouched:

  • OpenShift and OKD: the obs-beacond-agents Route uses passthrough. Agents connect to https://<its host>:443, which join.agent_url names.
  • Elsewhere: the LoadBalancer Service on port 8461, or an ingress controller with TLS passthrough.

The image doesn't carry the agent packages, so set join.packages_url to a release's download address (Adding Linux hosts).

Watching clusters

Each cluster Cairn OBS watches, this one included, runs one cluster agent from the same image. It reads the cluster with get, list and watch only.

  1. Clusters pull the agent from ghcr.io/coffey-labs/cairnobs, at obs-beacond's own version. If they can't reach it, set join.image in obs-beacond.yaml to your copy (without a tag, obs-beacond adds its version). For the cluster obs-beacond runs in, join.agent_url may be the Service, https://obs-beacond.<namespace>.svc:8461; other clusters need the agent Route or load balancer.

  2. Configure › Clusters › Add a cluster gives one command, good once for an hour. Run it as a cluster admin:

    curl -fsSL -H 'Authorization: Bearer cobsj1_…' https://obs.example.org/cluster.yaml | kubectl apply -f -
    

It makes the cairn-obs-agent namespace (Pod Security restricted), a read-only ClusterRole, and the agent. What it reads and what pages: Kubernetes and OpenShift clusters.

Day to day

  • Upgrade: change the tag and apply again. With one copy the pod is replaced, a short gap; with -ha the copies roll one at a time. Then upgrade each cluster agent with the same address, without the token: curl -fsSL https://obs.example.org/cluster.yaml | kubectl apply -f -.
  • Back up the volume (a CSI snapshot is fine) and, separately, the state key Secret. On PostgreSQL, back up the database (Its database and backups).
  • Moving to another namespace: OpenShift gives each namespace its own user ID range, and files the old pod wrote are readable only by the old ID. Restore the volume with a pod that runs chown -R <new uid> on it first, or give the new namespace's openshift.io/sa.scc.uid-range annotation the old range.