docs
cairnobs.org Demo Source
Esc

What it watches · 6 of 14

Kubernetes and OpenShift clusters

One read-only cluster agent per cluster brings in its nodes, pods, controllers, jobs, volumes and KubeVirt VMs. A node down is one incident, an OpenShift update isn't an outage, and nothing pages until you say so.

4 min read

Nobody installs an agent in a pod. Instead each cluster runs one cluster agent, which watches the Kubernetes API with its own read-only service account and sends what it sees to obs-beacond over the agent port, as a host's agent does. It works the same on Kubernetes, OpenShift and OKD, for the cluster obs-beacond runs in and for any other.

  • Credentials stay in the cluster. obs-beacond holds none.
  • Only outbound connections, so a cluster behind NAT needs no way in.
  • What's sent is trimmed. Pod specs, environment variables and Secret names never leave the cluster.

Adding a cluster

Configure › Clusters › Add a cluster gives one command, good once for an hour. Run it as a cluster admin:

curl -fsSL -H 'Authorization: Bearer cobsj1_…' https://obs.example.org/cluster.yaml | kubectl apply -f -

It makes the cairn-obs-agent namespace (Pod Security restricted), the service account, the read-only ClusterRole, and the agent, from the same image as obs-beacond: ghcr.io/coffey-labs/cairnobs, or your copy if join.image in obs-beacond.yaml names one. The agent joins and keeps its key and certificate in a Secret only it may read. The token travels in a header, so it stays out of proxies' logs.

  • Upgrade the agent: after upgrading obs-beacond, the same address without the token: curl -fsSL https://obs.example.org/cluster.yaml | kubectl apply -f -.
  • Remove a cluster: Remove under Configure › Clusters revokes its certificate, then curl -fsSL https://obs.example.org/cluster.yaml | kubectl delete -f - takes the agent out.

What it reads

With get, list and watch only:

Kind What's taken
Nodes Ready and the pressure conditions, unschedulable, kubelet version, capacity, roles
Pods Phase, node, owner, each container's ready, restarts, waiting reason (CrashLoopBackOff, ImagePullBackOff), last termination (OOMKilled)
Deployments, StatefulSets, DaemonSets Desired, ready, available, updated; a stuck rollout
Jobs and CronJobs Failed, last successful run
Volume claims and volumes Phase (Pending, Lost), capacity, storage class
Warning events The detail for an incident: FailedScheduling, FailedMount, Evicted
Services, Ingresses What's behind them, for the service map and the certificate inventory

And, where the API has them:

Kind What's taken
KubeVirt VMs and VM instances Status, node, guest agent, guest OS, addresses, migrations, and each guest file system's use, so a VM's full disk is caught with no agent in it
CDI DataVolumes Import or clone failures
OpenShift ClusterVersion, ClusterOperators, MachineConfigPools Version, update in progress, operators available or degraded, pools updating
OpenShift Routes Host, backends, TLS; an inline certificate, never its key
cert-manager Certificates Names, issuer, dates, Ready; never the Secret
metrics.k8s.io CPU and memory per node and pod, every 60 seconds

Not read, and not in the ClusterRole: Secrets, ConfigMaps, pods/exec, pods/attach, pods/portforward and nodes/proxy.

What's judged

Condition Severity
The cluster agent sends nothing for 2 minutes, or can't reach its API Critical, one for the whole cluster; its workloads keep their last state
A node is down Critical, one for the node, naming the pods and VMs on it
A node under pressure or not ready yet; one updating for over 30 minutes Warning
A controller short of ready pods, or with pods crashing or failing to start, for 5 minutes Warning; critical when none are ready or all are crashing. Its pods are judged with it, never one by one
A failed job Warning
A volume pending for 5 minutes Warning; lost is critical. Three or more in one storage class are one condition
A VM down or crashing Critical
A VM file system at 90% / 95% Warning / critical
OpenShift and OKD: an operator degraded or not available Critical; a warning while the cluster updates
OpenShift and OKD: an update failing, or a machine config pool degraded Critical

Updates aren't outages. OpenShift and OKD upgrades reboot every node in turn. A node the machine config operator is working on is Updating, not down, even while it reboots, and pods and VMs moving off it are expected to. A maintenance window can name a cluster, which holds everything about it.

openshift-* and kube-* namespaces are judged through the cluster operators, so their pods raise nothing unless you adopt them.

What pages

At fleet size, paging on every namespace would be noise. So only the cluster's own trouble and its nodes page at first. Every other workload stays a warning until it's adopted:

  • In Cairn OBS: Configure › Clusters › the cluster lists what's running. An admin sets a namespace, a controller or a VM to page, warn or off, and can mark a VM stopped on purpose.
  • In the cluster: annotate a namespace, controller, job, volume or VM with obs.cairnobs.org/alert: page (or warn, or off). Teams that own a namespace can decide in their own manifests, with no access to Cairn OBS.

When they disagree, what's set in Cairn OBS for the workload itself wins, then the annotation (the workload's own, its controller's, then its namespace's), then what's set in Cairn OBS for its namespace. An adopted controller or VM that disappears is reported missing.

Adoption is kept in expected.yaml under clusters:, in a file or in the UI.

What you see

  • Clusters: each cluster's platform (Kubernetes, OpenShift or OKD), version, nodes, operators and update state, then its namespaces, problems first.
  • Workloads: one table for every kind from every cluster and hypervisor, filtered by cluster, platform, namespace, kind, node and state, and paged. Filtered to VMs, it's the VMs page. Each row says which cluster or host it's in and what platform that is, and the platform filter counts what's on each: /workloads?platform=OpenShift is everything on OpenShift.
  • A controller, VM or node's own page: its pods, warning events, history and incidents.

When a VM also runs a Cairn OBS agent, the two are linked by its hardware ID and shown as one machine (Hypervisors and VMs).