What it watches · 6 of 14
Kubernetes and OpenShift clusters
One read-only cluster agent per cluster brings in its nodes, pods, controllers, jobs, volumes and KubeVirt VMs. A node down is one incident, an OpenShift update isn't an outage, and nothing pages until you say so.
Nobody installs an agent in a pod. Instead each cluster runs one cluster agent, which watches the Kubernetes API with its own read-only service account and sends what it sees to obs-beacond over the agent port, as a host's agent does. It works the same on Kubernetes, OpenShift and OKD, for the cluster obs-beacond runs in and for any other.
- Credentials stay in the cluster. obs-beacond holds none.
- Only outbound connections, so a cluster behind NAT needs no way in.
- What's sent is trimmed. Pod specs, environment variables and Secret names never leave the cluster.
Adding a cluster
Configure › Clusters › Add a cluster gives one command, good once for an hour. Run it as a cluster admin:
curl -fsSL -H 'Authorization: Bearer cobsj1_…' https://obs.example.org/cluster.yaml | kubectl apply -f -
It makes the cairn-obs-agent namespace (Pod Security restricted), the
service account, the read-only ClusterRole, and the agent, from the same
image as obs-beacond: ghcr.io/coffey-labs/cairnobs, or your copy if
join.image in obs-beacond.yaml names one. The agent joins and keeps its key and certificate in a
Secret only it may read. The token travels in a header, so it stays out of
proxies' logs.
- Upgrade the agent: after upgrading obs-beacond, the same address
without the token:
curl -fsSL https://obs.example.org/cluster.yaml | kubectl apply -f -. - Remove a cluster: Remove under Configure › Clusters revokes its
certificate, then
curl -fsSL https://obs.example.org/cluster.yaml | kubectl delete -f -takes the agent out.
What it reads
With get, list and watch only:
| Kind | What's taken |
|---|---|
| Nodes | Ready and the pressure conditions, unschedulable, kubelet version, capacity, roles |
| Pods | Phase, node, owner, each container's ready, restarts, waiting reason (CrashLoopBackOff, ImagePullBackOff), last termination (OOMKilled) |
| Deployments, StatefulSets, DaemonSets | Desired, ready, available, updated; a stuck rollout |
| Jobs and CronJobs | Failed, last successful run |
| Volume claims and volumes | Phase (Pending, Lost), capacity, storage class |
| Warning events | The detail for an incident: FailedScheduling, FailedMount, Evicted |
| Services, Ingresses | What's behind them, for the service map and the certificate inventory |
And, where the API has them:
| Kind | What's taken |
|---|---|
| KubeVirt VMs and VM instances | Status, node, guest agent, guest OS, addresses, migrations, and each guest file system's use, so a VM's full disk is caught with no agent in it |
| CDI DataVolumes | Import or clone failures |
| OpenShift ClusterVersion, ClusterOperators, MachineConfigPools | Version, update in progress, operators available or degraded, pools updating |
| OpenShift Routes | Host, backends, TLS; an inline certificate, never its key |
| cert-manager Certificates | Names, issuer, dates, Ready; never the Secret |
| metrics.k8s.io | CPU and memory per node and pod, every 60 seconds |
Not read, and not in the ClusterRole: Secrets, ConfigMaps, pods/exec,
pods/attach, pods/portforward and nodes/proxy.
What's judged
| Condition | Severity |
|---|---|
| The cluster agent sends nothing for 2 minutes, or can't reach its API | Critical, one for the whole cluster; its workloads keep their last state |
| A node is down | Critical, one for the node, naming the pods and VMs on it |
| A node under pressure or not ready yet; one updating for over 30 minutes | Warning |
| A controller short of ready pods, or with pods crashing or failing to start, for 5 minutes | Warning; critical when none are ready or all are crashing. Its pods are judged with it, never one by one |
| A failed job | Warning |
| A volume pending for 5 minutes | Warning; lost is critical. Three or more in one storage class are one condition |
| A VM down or crashing | Critical |
| A VM file system at 90% / 95% | Warning / critical |
| OpenShift and OKD: an operator degraded or not available | Critical; a warning while the cluster updates |
| OpenShift and OKD: an update failing, or a machine config pool degraded | Critical |
Updates aren't outages. OpenShift and OKD upgrades reboot every node in turn. A node the machine config operator is working on is Updating, not down, even while it reboots, and pods and VMs moving off it are expected to. A maintenance window can name a cluster, which holds everything about it.
openshift-* and kube-* namespaces are judged through the cluster
operators, so their pods raise nothing unless you adopt them.
What pages
At fleet size, paging on every namespace would be noise. So only the cluster's own trouble and its nodes page at first. Every other workload stays a warning until it's adopted:
- In Cairn OBS: Configure › Clusters › the cluster lists what's running. An admin sets a namespace, a controller or a VM to page, warn or off, and can mark a VM stopped on purpose.
- In the cluster: annotate a namespace, controller, job, volume or VM
with
obs.cairnobs.org/alert: page(orwarn, oroff). Teams that own a namespace can decide in their own manifests, with no access to Cairn OBS.
When they disagree, what's set in Cairn OBS for the workload itself wins, then the annotation (the workload's own, its controller's, then its namespace's), then what's set in Cairn OBS for its namespace. An adopted controller or VM that disappears is reported missing.
Adoption is kept in expected.yaml under
clusters:, in a file or in the UI.
What you see
- Clusters: each cluster's platform (Kubernetes, OpenShift or OKD), version, nodes, operators and update state, then its namespaces, problems first.
- Workloads: one table for every kind from every cluster and
hypervisor, filtered by cluster, platform, namespace, kind, node and
state, and paged. Filtered to VMs, it's the VMs page. Each row says
which cluster or host it's in and what platform that is, and the
platform filter counts what's on each:
/workloads?platform=OpenShiftis everything on OpenShift. - A controller, VM or node's own page: its pods, warning events, history and incidents.
When a VM also runs a Cairn OBS agent, the two are linked by its hardware ID and shown as one machine (Hypervisors and VMs).