docs
cairnobs.org Demo Source
Esc

What it watches · 10 of 14

IBM i and Power servers

IBM i partitions read through IBM's SQL services over Mapepire with a profile that has no special authorities, and the Power servers and every partition on them through the HMC's REST API, read-only.

4 min read

Many shops still run their core business on IBM i, beside the Linux, Windows and OpenShift estate. Cairn OBS reads IBM i partitions without installing an agent on them: a host agent nearby queries IBM's SQL services (the views in QSYS2 that answer what an operator looks at on a green screen) over the network, read-only. Each partition is a source on Clusters, "PROD1 · read by mgmt1", on the same pages as everything else.

The Power servers themselves, and every partition on them (IBM i, AIX, Linux and VIOS), come from the HMC.

What's read from a partition

Every minute, all queries of views:

From Becomes
SYSTEM_STATUS_INFO_BASIC, ENV_SYS_INFO, GROUP_PTF_INFO The partition: CPU, system ASP use, jobs, temporary storage, restricted state, the attention light, "IBM i 7.5 TR6"
ASP_INFO, SYSDISKSTAT Each ASP and its disk units. A failed ASP is down; units missing, failing, rebuilding parity or with mirroring suspended make it degraded. ASP use warns at the ASP's own threshold, IBM i's, not one of ours. An independent ASP varied off is noted, not judged (PowerHA keeps it so on the standby)
SUBSYSTEM_INFO Subsystems. Those you name as expected must be active (below)
MESSAGE_QUEUE_INFO for QSYSOPR Unanswered inquiry messages of the last 30 days, with the job that sent each (a job stuck in MSGW): warning. Severity 90 and up in the last day: critical; 80 to 89: warning
JOB_QUEUE_INFO A queue with jobs ready to run that's held, or that no active subsystem takes from: warning
NETSTAT_INTERFACE_INFO An interface set to start with TCP/IP that isn't active: degraded, or down when its line failed

What IBM i says about itself it judges itself: the ASP threshold, the message severities. Cairn OBS adds only what must be running and what's been stuck too long.

Expected subsystems

Name the subsystems that must be active in expected.yaml under clusters:, by the partition's source name. One that isn't active, or isn't there, is critical:

clusters:
  mgmt1:prod1:
    subsystems: [QINTER, QBATCH, ORDLIB/ORDERS]   # library/name when two libraries have one

Setting up a partition

On the partition, as a security officer:

  1. Install Mapepire, IBM's open-source database server for clients that aren't Java, from IBM's open-source packages (yum install mapepire-server, or ACS's Open Source Package Management), and start it with Service Commander: sc start mapepire. It listens on port 8076 with a certificate it makes itself.

  2. Make a profile with no special authorities and no command line:

    CRTUSRPRF USRPRF(CAIRNOBS) PASSWORD(...) USRCLS(*USER) SPCAUT(*NONE)
              LMTCPB(*YES) INLMNU(*SIGNOFF) TEXT('Cairn OBS, read only')
    

    If your passwords expire, so does this one, and the partition then shows as not answering until it's changed. PWDEXPITV(*NOMAX) is your call.

  3. Hold Mapepire to read-only for that profile, whatever the agent asks, in /QOpenSys/etc/mapepire/iprules.conf (the last matching rule wins), with the agent host's address:

    deny cairnobs@*
    allowbasiconly [email protected]
    

    The agent asks for read-only itself too: Mapepire then has IBM i check each statement and refuses anything that isn't a query.

  4. Optional: *USE on two objects names the partition as the HMC does and gives its Technology Refresh. A shop's own subsystem descriptions may need *USE before they can be expected.

    GRTOBJAUT OBJ(QSYS/QPMLPMGT) OBJTYPE(*SRVPGM) USER(CAIRNOBS) AUT(*USE)
    GRTOBJAUT OBJ(QSYS/WRKPTFGRP) OBJTYPE(*CMD) USER(CAIRNOBS) AUT(*USE)
    

Then, on the agent's host, the password alone in a file readable by the agent only, and in its agent.yaml:

ibmi:
  - address: prod1.example.com      # :port if not 8076
    user: CAIRNOBS
    password_file: /etc/obs-beacon/prod1.password
    fingerprint: "AB:CD:…"          # Mapepire's own certificate, SHA-256

Get the fingerprint with openssl s_client -connect prod1.example.com:8076 </dev/null | openssl x509 -noout -fingerprint -sha256.

The Power servers: the HMC

The HMC's REST API is to Power what vCenter is to VMware. A host agent with an hmc: section reads it every minute, all GETs after logging on:

From the HMC Becomes
Each managed system A server: down when the HMC can't reach it or it's in error; degraded with its attention light on (a serviceable event is open); stopped in standby or powered off
Each partition and VIOS A VM on its server, with what runs in it: down in error (with its reference code), stopped when not activated, updating while it starts, stops or moves (Live Partition Mobility). A VIOS the HMC can't reach over RMC is degraded: its partitions have one path less

An IBM i partition also read through Mapepire is linked to its server here, by the server's serial and its partition number.

Setting up the HMC

In the HMC's web console, as an administrator: Users and Security › Users and Roles › Manage User Profiles and Access. Add a user with the built-in hmcviewer task role (it can look at everything and change nothing), and under User Properties allow remote access via the web, which the REST API needs.

hmc:
  - url: https://hmc1.example.com     # port 12443 is added
    username: cairnobs
    password_file: /etc/obs-beacon/hmc1.password
    fingerprint: "AB:CD:…"

Where two HMCs manage the same servers, read one: both would list the same servers and partitions.

Status

Tested against fakes built from IBM's documented results and from real HMC answers kept in IBM's open-source Python library, not yet against a real partition or HMC. If you run IBM i, tell us how it goes.