Observability without the Prometheus tax

I have built the full church twice. Prometheus, Grafana, Loki, exporters on everything, a dashboard folder named WIP that became load-bearing. Both times the same arc: two glorious weeks of graphs, then a year of the monitoring stack being the most operationally demanding thing in the lab. Disk-hungry, upgrade-shy, and its failures didn’t page me, because it WAS the pager. When I caught myself debugging Loki at midnight while the thing Loki monitored ran fine, the lesson finally landed.

A homelab’s real observability requirement is two questions. Is anything down? What did it log when it broke? Everything past that is a hobby (a fine hobby) that should be chosen, not defaulted into. Four small tools answer both questions.

Metrics: Beszel

Hub and agents, almost insultingly simple. One hub container on hermes, one agent per box talking over the overlay, one shared key.

# per host:
  beszel-agent:
    image: henrygd/beszel-agent
    network_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - KEY=${BESZEL_AGENT_KEY}
      - LISTEN=10.99.0.20:45876    # overlay only, post 5's pattern

You get per-host CPU, memory, disk, network, temperatures, and (the view I actually open) per-container stats. “bender is at 90% memory” is trivia; “qBittorrent grew 6GB overnight” is a diagnosis. Threshold alerts are built in. Setup time: minutes.

What’s missing versus Prometheus: long retention, PromQL, graphing anything against anything. At five hosts I’ve needed that twice, and both times docker stats plus thinking sufficed.

Logs: Dozzle

Dozzle live-tails every container’s logs on every host (agents again, over the overlay), one browser tab, with search. No shipping, no indexing, no retention policy to operate. It reads Docker’s log stream and shows it to you.

This tool deleted most of my SSH sessions. The debugging loop used to be: ssh, docker ps to remember the name, docker logs --tail 100 -f, wrong container, repeat. Now: click the box, click the container, read. During an incident, the log line you need is seconds away, and seconds matter exactly then.

The honest gap: history ends where the container’s log buffer does. “What happened last Tuesday” is a real aggregator’s job. So far, post-incident curiosity has always fit inside the buffer.

Up/down: Uptime Kuma plus ntfy

Dashboards are pull; pages are push. Uptime Kuma probes whatever you point it at. When something fails it notifies ntfy, a self-hosted push server with a phone app that subscribes to topics. Service drops, pocket buzzes.

The design decision that matters is WHAT to probe. Container-up is the wrong check: Docker restarts crashed containers itself, and post 10’s deunhealth handles unhealthy-but-running. The checks that earn their keep probe the full path: the public URL of each public service, through real DNS, the edge, the overlay, to the app. The route a stranger takes. A container can be perfectly healthy behind a broken route all day, and only a path check notices. I learned this from a failed cert renewal: every container green, every user-facing page a TLS error, for six hours.

Complete it with the meta-check: who watches the watcher? Uptime Kuma pushes a heartbeat; ntfy alerts if the heartbeat goes missing. The monitoring being down should be the first page, not a surprise during the next real incident.

The front door: Homepage

A static dashboard. Every service is a card, grouped by host, with live widgets pulling each app’s own stats over the overlay: queue depth, request counts, disk free, who’s streaming. The YAML lives in the repo; widget API keys ride the sops pipeline.

Day to day it’s a bookmark bar with vitals. Operationally it’s a deploy check: push a change, glance at the card, the widget either answers or it doesn’t. And when the household asks “is the thing broken?”, I answer from one page in five seconds, which is the actual service-level objective of a homelab: the operator sounds like he knows.

The scorecard

Four small tools, each replaceable in an afternoon, none capable of becoming the incident. If the lab outgrows this, Prometheus will still exist. Observability is the easiest layer to retrofit and the worst one to gold-plate on day one. Graph what you’ll act on; everything else is screensaver.

Next: backups. restic to two destinations, keys on paper, and the restore drill that turns a rumor into a guarantee.