Good news, everyone! I rebuilt the entire homelab, and this time I wrote it down.

I’ve been doing this a long time. It started the way it starts for most people in IT: a spare tower under a desk sharing files, then a succession of “temporary” machines that each outlived whatever they were temporary for. Eventually it became a proper homelab, which is to say: a place where I run things the way I think they should be run, with no change-advisory board except my own regret.

This series documents the latest rebuild, in the order you’d actually do it. Not because the hardware died. Because the way it was organized died.

How it got bad

See if any of this sounds familiar.

By last year I had three servers, each running a pile of Docker containers, and each pile had grown differently. One box had everything in a single 1,800-line docker-compose.yml that I was honestly afraid to touch: change one service, recreate them all, hold your breath. Another was managed through Portainer, which meant its real configuration lived in Portainer’s database; the compose files on disk had drifted from reality years earlier and nobody told them. The third had containers started by hand with docker run flags that existed only in shell history, on a host I’d since reinstalled.

Each server ran its own reverse proxy, configured its own way. Two had their own identity provider, because I set up SSO twice and never finished consolidating. Secrets were in .env files, some gitignored, some (I found out during the migration) not. Backups covered whatever had scared me at some point, which is not the same list as what mattered.

None of it was broken, exactly. It all ran. But every change had become archaeology. Which box is that on? Is the file on disk what’s actually running? Where did I put that API key? When a disk started throwing SMART errors, I had to admit the disaster-recovery plan for two of the three machines was “reconstruct from memory.”

The lesson, after a long time of not learning it: infrastructure you can’t rebuild from a file is a liability wearing a hobby’s clothes.

The rewrite

So: tear it down. Not the data, the organization. Every service re-declared from scratch, in one git repository, with rules. The rules are the real product of this series; the specific apps are almost interchangeable.

1. The repo is the only truth. Every service on every host is defined in one repository. Servers receive deploys from it. Nobody edits anything on a server, ever. If I’m SSH’d into a box editing a config file, something has already gone wrong. (Post 9 covers how deploys flow; the discipline is the point.)

2. One identity provider. A single SSO instance guards everything, and protecting a new app costs one line. No more “I’ll add auth later.” Later never comes, and unauthenticated admin panels are how homelabs end up in somebody’s botnet writeup.

3. Exactly one machine faces the internet, and it’s the one I care about least: a cheap VPS with a static IP, running a reverse proxy and nothing stateful. It reaches back to the other sites over an encrypted overlay network. My home IP appears in no DNS record anywhere. If the VPS ever gets popped, the rebuild is ten minutes and the attacker got a proxy and a bad mood.

4. Secrets are encrypted in the repo, value by value, next to the compose files that use them. “Did that .env make it into git” is no longer a question, because the answer is yes, on purpose, safely.

5. Pick boring on purpose. Observability is a dashboard and live logs, not a Prometheus deployment that becomes a second hobby. Backups are restic on a timer, restore-tested. Every component had to pass one test: will this still make sense at 2 a.m. in eighteen months?

The fleet

Three machines across three sites. The fleet is named after Futurama characters, because if you can’t have fun naming servers, why are you even self-hosting:

                        internet
                           |
                  +--------v---------+
                  |  url   (site C)  |   the VPS edge
                  |  static IP       |   Traefik + SSO outpost + Crowdsec
                  +--------+---------+
                           |  ZeroTier overlay (10.99.0.0/24)
             +-------------+--------------+
   +---------v----------+      +----------v---------+
   | hermes  (site A)   |      |      site B        |
   | SSO core, deploys  |      | bender:    *arrs,  |
   | HA, vault, mail    |      |   downloads, reqs  |
   +--------------------+      | nibbler:   storage |
                               | professor: Plex    |
                               +--------------------+
  • hermes, site A (local): the bureaucrat. Hermes Conrad files everything, and so does this box: the SSO core, the deploy orchestrator, Home Assistant, the password vault, the mail server. If it holds state I’d cry about, it lives here.
  • bender, site B (remote): the download box. Library automation, the request portal, VPN’d download clients. Bender downloads things. It’s what he does.
  • nibbler, site B: storage. A little box attached to a frankly alarming amount of disk, exactly like his namesake. Exports the big filesystem the rest of site B works against.
  • professor, site B: Plex, with a repurposed Nvidia gaming GPU doing the transcodes. Good news, everyone, the old graphics card found work.
  • url, site C (VPS): Officer URL, the robot cop. Works the door. Routing, auth enforcement, bouncing scanners. Three containers and no feelings.

The split matters more than the hardware. Control plane, bulk data, and public exposure each get their own blast radius. Site B can drop off the map without taking down site A’s door-lock automations, and the VPS can be rebuilt without touching either site.

Hardware, for the curious: hermes is a VM with 8GB on a small hypervisor, bender is a repurposed desktop with a mid-range Intel CPU (the iGPU matters for Plex, see post 15) and a stack of big disks, and url is whichever provider’s bottom-tier VPS had a clean IP reputation that week. None of this needs serious iron. The organization is the expensive part, and it’s free.

What the series covers

In dependency order, the same order you’d rebuild from bare metal:

  1. This post: the shape and the rules
  2. Domains and DNS: naming, wildcards, scoped API tokens
  3. The repo: layout, the infra/app split, conventions
  4. Secrets: SOPS + age, encrypted env in git
  5. ZeroTier: the three-site overlay
  6. Traefik, inside: per-host proxies, wildcard certs, labels
  7. Traefik, edge: the public front door, Crowdsec
  8. Authentik: one login for everything
  9. Komodo: git push to deploy
  10. The download pipeline: gluetun and kill-switch networking
  11. Observability: without the Prometheus tax
  12. Backups: restic, restore-tested, keys on paper
  13. Split-horizon DNS: same names, shorter path at home
  14. Mail: the boss fight

Plus standalone deep dives where one app earns a full page (Plex and the *arr naming stack are posts 15 and 16).

The mistakes stay in. Every post has a pitfalls section and none of it is hypothetical. Each one cost me an evening at some point, and each comes with the symptom you’d actually see, because “works once you know the trick” is the true shape of this hobby.

Next up: domains and DNS, including the API-token hygiene I wish someone had explained to me much earlier.