docs
cairnobs.org Demo Source
Esc

What it watches · 7 of 14

Hypervisors and VMs

VMs from libvirt/KVM, Hyper-V, Proxmox VE, VMware vSphere and Nutanix, read with read-only accounts. A VM is a VM wherever it runs, and one that also runs an agent shows up once.

3 min read

Every hypervisor's VMs become the same kind of record, so a VM looks and alerts the same whether it runs on KVM, Hyper-V, Proxmox, vSphere, Nutanix or KubeVirt. They're on the Workloads page (filtered to VMs, the VMs page), and each hypervisor shows on Clusters as "read by" the host whose agent reads it.

Each VM still says what it runs on. Clusters, the Workloads rows, a host's VM line and the network map name the platform: VMware vSphere, Nutanix, Proxmox VE, Hyper-V, KVM, or the Kubernetes or OpenShift cluster running KubeVirt. Workloads filters by it, so every VM on vSphere is one click: /workloads?kind=vm&platform=VMware vSphere.

One machine, two views

The agent reports each machine's SMBIOS UUID. The hypervisor knows the same number from outside: libvirt's domain UUID, Hyper-V's BIOS GUID, Proxmox's smbios1, vSphere's BIOS UUID, Nutanix's biosUuid, KubeVirt's firmware UUID. When they match, the pages show one machine with both views: where it runs and its state from the hypervisor, its services and disks from the agent.

When that host goes silent, the alert says what the hypervisor sees ("It's VM db on kvm1, which is stopped", or "running as far as the hypervisor knows: look at the guest, its agent or the network"), and the VM down and its agent silent are one incident, not two. A UUID that two machines claim links nothing, nor does a firmware placeholder: a wrong link could hide a real outage.

On the host: libvirt and Hyper-V

libvirt/KVM needs nothing set up. The root helper reads the host's VMs from libvirt's read-only socket at every report: state, vCPUs, memory, CPU time and autostart. It speaks only the few read-only calls it needs, so it can't change anything.

Hyper-V is read by the Windows agent through WMI, when the agent's account may see the VMs (Hyper-V's VMs).

Both show as "Runs 12 VMs, 10 running" on the host's page. A VM paused by an error, crashed or failed to start is down; a VM shut off normally is no condition. The host's VMs alert as warnings.

From one host: Proxmox, vSphere, Nutanix

These are read through their APIs by the agent on any host that can reach them, with a section in its agent.yaml. The agent only ever asks to read. Each is a source of its own on Clusters.

For each, the API's certificate is checked against the system's CAs, or a ca_file, or pinned with fingerprint (its SHA-256) for a self-signed one. Password and token files hold the secret alone and should be readable by the agent only (obs-beacon:obs-beacon, 0600).

Proxmox VE

On one node of the cluster, with an API token whose user has the built-in PVEAuditor role:

pveum user add obs@pve --comment "Cairn OBS, read only"
pveum user token add obs@pve cairnobs --privsep 1
pveum acl modify / --tokens 'obs@pve!cairnobs' --roles PVEAuditor
proxmox:
  token_file: /etc/obs-beacon/proxmox.token   # obs@pve!cairnobs=<the secret it printed>
  # url: https://127.0.0.1:8006                # the default
  # ca_file: /etc/pve/pve-root-ca.pem          # the default
  # every: 30s
  # name: proxmox                              # kept as <host>:proxmox

Set it on one node only: two would be two copies of one cluster. It reads the cluster's quorum (lost quorum is critical), nodes, and every VM and container, with HA states: an HA error or fence is down, a migration is updating.

VMware vSphere

A vCenter user with the built-in Read-only role, propagated from the top:

vsphere:
  url: https://vcenter.example.org
  username: obs-reader@vsphere.local
  password_file: /etc/obs-beacon/vsphere.password
  # ca_file: /etc/obs-beacon/vmca-root.pem     # the vCenter's VMCA root
  # every: 60s

It reads ESXi hosts (not responding is down, maintenance mode is updating), VMs (inaccessible or orphaned is down, a snapshot over 7 days old degraded), and datastores (85% full degraded, 95% or inaccessible down). A red alarm in vCenter that nobody acknowledged marks what it's on as degraded, saying which.

Nutanix

Prism Central, with the built-in Prism Viewer role over every cluster, through a service account's API key (Prism Central 2024.3 and later) or a user and password:

nutanix:
  url: https://prism-central.example.org        # port 9440 unless given
  api_key_file: /etc/obs-beacon/nutanix.key
  # fingerprint: "AB:CD:…"                      # for its self-signed certificate
  # every: 60s

It reads every cluster's AHV hosts, VMs and storage containers through the v4 API, and Prism's open alerts: a critical one nobody acknowledged marks what it's about as degraded.

Status

Hyper-V and KubeVirt have run against the real thing. The libvirt, Proxmox VE, vSphere and Nutanix readers have been tested against simulators and fakes built from each API's published documentation, not yet against real installs. If you run one of them, tell us how it goes.