Build a Linux Service Monitor With Python

In this tutorial we build a minimal monitor on a single Linux host. The same ideas extend to every device in your fleet once an agent reports state centrally.

What We'll Build

1. Read systemd unit state
2. Detect failures or changes
3. Emit a structured event

 1import hashlib
 2import subprocess
 3import time
 4from pathlib import Path
 5
 6CONFIG = Path("/etc/myapp/config.ini")
 7
 8def unit_active(name: str) -> str:
 9    result = subprocess.run(
10        ["systemctl", "is-active", name],
11        capture_output=True,
12        text=True,
13        check=False,
14    )
15    return result.stdout.strip()
16
17def file_hash(path: Path) -> str:
18    return hashlib.sha256(path.read_bytes()).hexdigest()
19
20if __name__ == "__main__":
21    last = file_hash(CONFIG) if CONFIG.exists() else None
22    while True:
23        state = unit_active("nginx")
24        if state != "active":
25            print({"event": "service.degraded", "unit": "nginx", "state": state})
26        current = file_hash(CONFIG) if CONFIG.exists() else None
27        if current != last:
28            print({"event": "config.changed", "path": str(CONFIG)})
29            last = current
30        time.sleep(30)

How to Read the Script

unit_active asks systemd for the current state instead of grepping a status page. active is the only healthy answer you should treat as success. activating, failed, and inactive are different incidents and should stay different events.

file_hash watches one config file. A hash change is not automatically bad — deploys change files on purpose — but a change you did not expect is drift. Keep the path list short. Hashing all of /etc on a small edge box wastes the disk and the CPU you are trying to protect.

The loop sleeps 30 seconds so a lab demo is easy to watch. On a fleet, do not run this as a tight poll from cron on every device without a central collector. You will get duplicate events, clock-skewed timestamps, and no one place to ask "which devices hashed differently since yesterday?"

Run It Locally

1python3 monitor.py
2# in another shell, after you trust the lab box:
3sudo systemctl stop nginx

You should see a service.degraded event with the unit state. Start nginx again and confirm the event stops. Then change the config file and confirm a single config.changed event, not one per loop.

What to Add Before This Leaves the Lab

• Emit JSON lines with timestamp, hostname, and boot_id so events can be joined later.
• Track NRestarts from systemctl show, not only is-active. A unit can be active and still be crash-looping.
• Back off when the host is under memory pressure. A monitor that gets OOM-killed is not a monitor.
• Ship events out when the link is up, and keep a small on-disk queue when it is not.
• Alert on the fleet aggregate. One stopped nginx is a ticket. Fifty in the same hour is a bad rollout.

Why This Matters at Fleet Scale

The same pattern repeats per device with central aggregation. Run this on one device and you have a hack. Run it everywhere with centralized events and you have observability.

How EdgeProtocol Helps

EdgeProtocol's agent extends this pattern across your fleet. The point of the exercise is the event model: service state and config identity, comparable across hosts. EdgeProtocol collects that picture for the fleet so you are not deploying a one-off loop onto every box by hand.

Try EdgeProtocol →