Everything in this series is reproducible from the repo except the part you’d actually cry about. Compose files redeploy; the vault’s database and fifteen years of photos do not. I’ve lost data twice. Once to a dying disk in the era before I took backups seriously (tuition), and once, more instructively, WITH backups, when a restore revealed the job had been silently skipping the directory that mattered for eight months. Green checkmarks the whole time. That second story is why half this post is about restoring.

You can only back up what you can see

A post 3 decision pays off here: all state lives in bind-mounted appdata directories, never named volumes. One tree per host:

~/appdata/
+-- vault/
+-- sso-postgres/
+-- homeassistant/
+-- ...

That tree IS the answer to “what needs backing up.” The backup tool points at one path, nothing hides inside Docker’s volume store, and auditing coverage is a one-minute diff of ls ~/appdata against the include list. Do that audit occasionally. It’s how you catch the new service that snuck in since.

Step 1: restic, to two places

restic: encrypted, deduplicated, snapshot-based, one static binary. Runs are incremental; every snapshot restores as a complete point in time. Same source, two repositories:

  • Local: a disk on the host. For the common case: I broke it, I deleted it, the upgrade ate it. Restores at disk speed.
  • Backblaze B2: object storage, about $6/TB-month. For the case backups exist for: the disk, the box, or the building is gone.
restic -r /mnt/backup/restic backup ~/appdata \
    --exclude '**/cache' --exclude '**/Cache' --exclude '**/Logs'
restic -r b2:lab-backups:hermes backup ~/appdata --exclude ...
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

The B2 credential is an application key scoped to its one bucket (post 2’s blast-radius thinking again): a compromised host can write its own backups but can’t read another host’s bucket or touch the account.

Step 2: schedule with systemd, alert on failure

Timers beat cron here for concrete reasons: logs land in journald, OnFailure= gives you a failure hook for free, and systemctl list-timers shows the next run instead of making you simulate cron in your head. The failure hook pushes to ntfy (post 11).

A backup system that can fail silently is a random-number generator with a progress bar. My eight-month story predates the failure hook. Never again.

Step 3: databases get dumped, then backed up

File-copying a RUNNING database snapshots a moving target: usually restorable, occasionally and silently not, the worst distribution of outcomes on offer. The fix is one line per database, just before the file backup:

docker exec vault-db pg_dump -U vault vault | gzip \
    > ~/appdata/vault/db-dump.sql.gz

The dump lands inside appdata and rides along. SQLite (half of every homelab) is more forgiving, but sqlite3 app.db ".backup backup.db" into the same pattern removes the asterisk for free. Mind the order: dump, THEN backup, or tonight’s snapshot carries yesterday’s dump.

Step 4: the keys live on paper

restic repos are password-encrypted; B2 has keys. Now trace the dependency chain on the worst day: the building burns, the backups are safe in B2, and the restic password lives in… the password vault, which is in the backups, which need the password. I drew that circle on a whiteboard during planning and felt appropriately silly, which is the best time and place to feel it.

The backup credentials must live somewhere that survives the lab and doesn’t depend on it. Mine: the password manager for convenience, plus a printed sheet in a drawer at site B. Paper has no firmware and no ransomware exposure. Of all my technology opinions, “print the keys” is the one I’d defend in court.

Step 5: the drill (a backup you’ve never restored is a rumor)

Quarterly, per service that matters:

restic -r b2:lab-backups:hermes restore latest \
    --include /home/lab/appdata/vault --target /tmp/drill
# point a scratch compose file at /tmp/drill, then:
docker compose -f drill.yml up -d
# open it. LOG IN. read your actual data. then tear it down.

Not restic check. That verifies restic’s internals, not your assumptions. The drill catches the system’s bugs: the exclude pattern that ate a real directory, the dump running after the backup, the app that keeps state outside appdata (looking at you, every app with a data directory AND a config directory). My eight-month near-miss would have been a ten-minute drill finding. Your first drill will find something. That’s a prediction from experience, not rhetoric.

The drill also produces the artifact you’ll want most during a real disaster: a restore runbook you have actually executed, written calmly, tested twice. Its margins collect the institutional memory nothing else captures: which UI settings aren’t in config (post 8 listed several), what’s deliberately not backed up, where the paper is.

Milestone: the lab can now lose any disk, any box, or any building and come back with its state.

Next: a pure quality-of-life upgrade. The same public names, the ten-foot path instead of the thousand-mile one, when you’re at home.