Monitor a Cyber Power UPS with node_exporter, Prometheus and Grafana

Whilst wrangling some kettle cords on this very busy desk I was reminded of that unused USB port on the back of my UPS and thought "these are a lot of devices for one 15 amp circuit, I should probably measure my consumption".

I have two Cyber Power CP1500 connected to mains power; one for my three workstations and bench gear, the other powers my three servers. I have exactly one power outlet in my office (ikr!) which I'm hoping to remediate in 2023.

Proof-of-concept

I connected the UPS USB cable to my laptop - unfortunately there is no officially supported PowerPanel® for Arch but a simple yay -yS powerpanel was a quick and easy fix.

I enabled and started the service with

sudo systemctl enable pwrstatd.service
sudo systemctl start pwrstatd.service

Then to make sure it's working

pwrstat -status

It worked, neat!

My next step is to see if there are any pre-built solutions for monitoring power consumption over time. Unfortunately I couldn't find anything in nut and the only other officially supported solution was to connect the UPS to my Apple computer and use the GUI - we can do better.

The pwrstat help text does not contain any means to produce machine-friendly output so it's scripting time. I chose Python for the PoC because 1) it's ubiquitous and 2) it's less ugly than bash - perhaps I'll riig in the near future.

Anyway, here's what I came up with

#!/usr/bin/env python3
import re
import subprocess
import time

# we will configure node_exporter to include this file in it's output
promfile = "/var/lib/node_exporter/ups.prom"
while True :
    # run the pwrstat command, capture output, decode to utf-8
    output = subprocess.check_output(["pwrstat", "-status"]).decode('utf-8')
    # split command output by line into a list, ignore empty lines
    lines = [x for x in output.split("\n") if x]
    # we can re-use this variable, not sure if this is `pythonic` though
    output = ""
    for line in lines :
        # we don't care about lines with colons
        if ":" in line : continue
        # replace all periods (.) with a single colon, lowercase everything
        line = re.sub('\.+', ':', line).lower()
        # split keys and values, strip padding whitespace
        parts = [x.strip() for x in line.split(":")]
        # EAFP : skip non-numeric values
        try :
            key = "ups_metrics_%s" % parts[0].replace(" ", "_")
            value = float(parts[1].split()[0])
            output += "# TYPE %s gauge\n%s %f\n" % (key, key, value)
        # we don't care about text values (for now?)
        except :
            pass

    """ not sure if opening and closing the file every second is a good idea or not, 
        should probably refactor this to limit i/o ¯\_(ツ)_/¯
    """"

    with open(promfile, "w") as fh :
        fh.write(output)
    fh.close()
    time.sleep(1)

Place in /usr/local/bin/ups_metrics, set execution bit and run it (nb: there wont be any terminal output)

chmod +x /usr/local/bin/ups_metrics
sudo /usr/local/bin/ups_metrics

Time to test

curl -sSL http://localhost:9100/metrics | grep "ups_metrics"

# HELP ups_metrics_battery_capacity Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_battery_capacity gauge
ups_metrics_battery_capacity 100
# HELP ups_metrics_load Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_load gauge
ups_metrics_load 261
# HELP ups_metrics_output_voltage Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_output_voltage gauge
ups_metrics_output_voltage 122
# HELP ups_metrics_rating_power Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_rating_power gauge
ups_metrics_rating_power 900
# HELP ups_metrics_rating_voltage Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_rating_voltage gauge
ups_metrics_rating_voltage 120
# HELP ups_metrics_remaining_runtime Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_remaining_runtime gauge
ups_metrics_remaining_runtime 22
# HELP ups_metrics_utility_voltage Metric read from /var/lib/node_exporter/ups.prom
# TYPE ups_metrics_utility_voltage gauge
ups_metrics_utility_voltage 122

Success!

Installing as a system service

I created the file /etc/systemd/system/upsmetrics.service with the following content

[Unit]
Description=upsmetrics
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/ups_metrics
Restart=on-failure
RestartSec=30
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Then, reload systemd and enable the upsmetrics service

sudo systemctl daemon-reload
sudo systemctl enable upsmetrics
sudo systemctl start upsmetrics

I followed these instructions to install node_exporter (if you installed using a package manager your service file and paths may differ) - here's my /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter

[Install]
WantedBy=multi-user.target

The significant line here is

ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter

Which looks for /var/lib/node_exporter/*.prom files - once I verified the service was running and could see my UPS metrics in the /metrics endpoint I created this simple dashboard (I will write about the temperature metric in a future article).

pwrstat-grafana

Hope someone finds this helpful :)

:wq