[{"content":"Every guide says don\u0026rsquo;t self-host email, and it\u0026rsquo;s worth being precise about where they\u0026rsquo;re right. It\u0026rsquo;s not receiving; receiving is easy. It\u0026rsquo;s not the software; modern servers are excellent. It\u0026rsquo;s IP reputation. The big providers judge your mail substantially by where it comes from, and a residential IP or a cheap VPS comes from nowhere good. One major provider blocklists entire hosting ranges wholesale. I\u0026rsquo;ve read my own bounce messages telling me my IP was rejected for the company it keeps. No amount of correct SPF and DKIM repairs a reputation you were never allowed to build.\nThe guides\u0026rsquo; conclusion (\u0026ldquo;so don\u0026rsquo;t\u0026rdquo;) is where I get off the bus. The right conclusion: design so your own IP\u0026rsquo;s reputation never matters. Accept that constraint up front and self-hosted mail stops being a heroic struggle and becomes one more service with one weird dependency. I waited a long time to own my mailbox. This is the architecture that finally made it boring.\nThe shape inbound: internet -\u0026gt; MX: postfix relay on url :25 -\u0026gt; overlay -\u0026gt; Stalwart (hermes) clients: IMAP/JMAP/submission -\u0026gt; url TCP-passthrough (SNI) -\u0026gt; Stalwart outbound: Stalwart -\u0026gt; relay-provider ladder -\u0026gt; world (url\u0026#39;s own IP: last resort only) Stalwart is the mail server, the modern all-in-one: IMAP, JMAP, SMTP, CalDAV/CardDAV, spam filtering, web admin, one container. It runs on hermes with every port bound to the overlay IP only (post 5\u0026rsquo;s pattern at its best). The machine holding years of mail is unreachable from the internet. Not firewalled. Absent.\nurl plays two small parts. A tiny Postfix relay is the public MX: receives on :25, forwards across the overlay, and holds mail briefly if the tunnel is down (a feature: the internet retries on its schedule, a relay you control retries on yours). And post 7\u0026rsquo;s Traefik TCP passthrough carries IMAPS and submission by SNI. Stalwart terminates its own TLS; the edge never holds mail plaintext or mail keys. Officer URL checks IDs at the door, he doesn\u0026rsquo;t read your mail.\nOutbound: assume your IP is worthless All sending goes through relay providers that have what we can\u0026rsquo;t build: reputation. The SMTP2GO/Brevo/Mailjet/Resend class all run free tiers (hundreds per day or thousands per month; personal volume rounds to zero against any of them). Pick one and you\u0026rsquo;ve shipped. But one provider is one point of failure, so Stalwart\u0026rsquo;s outbound walks a ladder keyed on retry count:\nattempt 1-2 -\u0026gt; provider A attempt 5-6 -\u0026gt; provider C attempt 3-4 -\u0026gt; provider B attempt 7-8 -\u0026gt; provider D attempt 9+ -\u0026gt; url\u0026#39;s own IP (delivers most places; beats queueing forever) A provider outage, a quota blip, an account flag: the message walks down a rung and delivers a few minutes late, no human involved. To be clear about what this is: redundancy, not quota games. Month to month, the fallback rungs barely see a message. It\u0026rsquo;s RAID for SMTP.\nThe detail that makes the ladder work: DKIM alignment on the root domain. Every provider gets domain-verified for example.dev itself, each with its own DKIM key in DNS, so every rung signs d=example.dev. DMARC passes by DKIM alignment no matter who carried the message, and p=reject stays safe. Avoid providers that only verify a subdomain; your identity shouldn\u0026rsquo;t depend on routing luck.\nCheck each rung before trusting it: send a real message through it and read Gmail\u0026rsquo;s \u0026ldquo;show original.\u0026rdquo; You want spf=pass, dkim=pass (d=example.dev), dmarc=pass, per rung.\nThe pitfalls, in the order they hurt The EHLO mail loop. The strangest bounce of the build: \u0026ldquo;mail forwarding loop\u0026rdquo; on a message that touched exactly two of my machines. Cause: the relay on url greets the world as mail.example.dev (it\u0026rsquo;s the MX; its EHLO must match). I had also, per every tutorial ever written, set the internal Stalwart\u0026rsquo;s hostname to mail.example.dev. Postfix connected downstream, heard its own identity echoed back, reasonably concluded it was talking to itself, and bounced everything.\nTwo SMTP hops cannot share a name. The internal server keeps its internal hostname. The world-facing identity belongs to the relay alone. Tutorials miss this because tutorials only have one hop.\nThe stale config cache. Stalwart persists settings to its database immediately, and the running process can keep serving the OLD routing config from memory. I ran two \u0026ldquo;successful\u0026rdquo; test sends through a rule I had already replaced, and only caught it because the provider dashboards disagreed with my expectations. House rule: restart after every routing change, then verify with a real message, checked at the receiving end. \u0026ldquo;It saved\u0026rdquo; and \u0026ldquo;it\u0026rsquo;s live\u0026rdquo; are different claims.\nSilent relay rot. The relay is the inbound MX and the outbound last resort, singular enough that when mail dies in BOTH directions at once, you check it first. Mine broke once from a poisoned anonymous volume: stale generated config baked into a volume that outlived its generator (post 3\u0026rsquo;s no-anonymous-volumes rule, enforced by irony). Triage from anywhere on the overlay: echo \u0026gt; /dev/tcp/10.99.0.30/25. Port dead, suspect found.\nApps are senders too. The vault\u0026rsquo;s 2FA codes and the SSO\u0026rsquo;s password resets matter more than your correspondence. Every app submits to Stalwart over the overlay (port 587, a dedicated account) and rides the same ladder. In this lab the apps all send as the hypnotoad, because when an email demands your attention, ALL GLORY TO THE HYPNOTOAD. One outbound pipeline, one set of logs, one DMARC story, instead of six apps with six half-configured SMTP settings.\nThe DNS appendix Mail is half DNS. The full inventory, so nothing is mystery meat: MX points at the relay. SPF authorizes your senders (mind the 10-lookup limit, which provider include chains eat quickly). DKIM gets one key per ladder rung, all on the root domain. DMARC starts at p=quarantine with reports and graduates to p=reject once a week of reports shows only you. MTA-STS and TLS-RPT if you\u0026rsquo;re feeling thorough. Stalwart can publish its records itself through a DNS API token, scoped per post 2. Same religion, last verse.\nWorth it? The longest chapter of the build, and the only one where the adversary was other people\u0026rsquo;s policy rather than my own mistakes. In exchange: my mail on my domain, my server\u0026rsquo;s logs when something looks odd, calendars and contacts on the same box, and every app\u0026rsquo;s outbound mail in one pipeline I can actually read.\nThat\u0026rsquo;s the lab, documented end to end. Post 3\u0026rsquo;s repo holds all of it; everything else is a service file away. The deep dives (Plex, the *arrs) live alongside as posts 15 and 16, and whatever breaks next gets a post of its own.\nSeries complete. Thanks for reading along.\n","permalink":"https://hi.khuo.ng/posts/2026-06-11-self-hosted-mail/","summary":"The boss fight, documented: inbound through the VPS, a mailstore the internet can\u0026rsquo;t reach, and outbound through a retry-ladder of relay providers, DMARC-aligned on the root domain no matter which rung delivers.","title":"Self-hosted mail: Stalwart behind a smarthost chain"},{"content":"Here\u0026rsquo;s an inefficiency the lab lived with for months: standing at site A, opening vault.example.dev, my request left the building, traveled to the VPS, and came back through the overlay to a server ten feet away. Round trip, city-scale. Useful distance, three meters.\nWorse than the latency is the dependency. That hairpin means local services need the internet working. The day the site A connection wobbled, I couldn\u0026rsquo;t reach things running in the same building, which offends me on a level I can\u0026rsquo;t fully articulate. Post 2 promised this itch would get scratched with smarter DNS rather than riskier records. Here\u0026rsquo;s the smarter DNS.\nThe trick: different answers for different askers Split-horizon DNS: the same name resolves differently depending on where you ask from. Outside, public DNS answers with url, as always. At site A, the LAN\u0026rsquo;s resolver lies, helpfully, and hands back hermes\u0026rsquo;s LAN address.\nMy LAN resolver is AdGuard Home on the firewall (it was already there eating ads). The feature you want is DNS rewrites, one per public name worth short-pathing:\nvault.example.dev -\u0026gt; 192.168.x.10 # hermes\u0026#39;s LAN address auth.example.dev -\u0026gt; 192.168.x.10 home.example.dev -\u0026gt; 192.168.x.10 requests.example.tv -\u0026gt; 192.168.x.10 Devices on site A wifi get the short answer automatically. The same devices on cellular get the public answer. Nobody\u0026rsquo;s bookmarks know the difference, which is the entire point.\nStep 1: the internal proxy holds up its end Hermes\u0026rsquo;s Traefik is about to receive browsers asking for PUBLIC hostnames. Two things must be true first.\nIt needs the public wildcard cert too. It\u0026rsquo;s been minting *.hermes.zt.example.dev (post 6); browsers will now ask it for vault.example.dev. DNS-01 doesn\u0026rsquo;t care that the host is private. Add a second domain to the entrypoint:\nentryPoints: websecure: http: tls: domains: - main: \u0026#34;*.hermes.zt.example.dev\u0026#34; - main: \u0026#34;*.example.dev\u0026#34; # split-horizon names It needs routers for those names, with the SAME auth posture as the edge. One \u0026ldquo;split\u0026rdquo; router per service, mirroring the public route:\nlabels: - traefik.http.routers.vault-split.rule=Host(`vault.${PRIMARY_DOMAIN}`) - traefik.http.routers.vault-split.entrypoints=websecure - traefik.http.routers.vault-split.tls=true # if the edge protects it with SSO, so does this router: - traefik.http.routers.vault-split.middlewares=sso@file That last line is a security rule, not a style note. A name behind forward-auth at the edge but naked on the internal path means being on the wifi is an auth bypass. Guests are on the wifi. IoT devices of dubious provenance are on the wifi. Mirror the middleware list, every time.\nCheck it worked, from a site A device:\ndig +short vault.example.dev # should be 192.168.x.10 on the LAN curl -sI https://vault.example.dev # 200/302 with a valid cert Steady state, the tax is small: a new public service costs one rewrite plus one split-router label on top of what posts 6 and 7 already required.\nWhy not just use internal names at home? The tempting alternative: skip all this and type vault.hermes.zt.example.dev when you\u0026rsquo;re at site A. It fails on sessions and muscle memory. Post 8\u0026rsquo;s cookies are scoped to the registrable domain, so two names for one service means two sessions, OAuth redirect URIs that match only one of them, a password manager that fills on one and shrugs at the other, and a phone app configured against whichever URL you typed at setup. One name everywhere, with DNS choosing the path, keeps every client config true in both worlds. The name is the interface. Don\u0026rsquo;t fork the interface.\nThe quirk the shortcut exposed One app\u0026rsquo;s login broke only at site A, only sometimes, with a TLS error mid-OAuth-dance. I\u0026rsquo;ll spare you the wrong theories I committed to first (the cert, then AdGuard, then \u0026ldquo;the phone is being weird\u0026rdquo;) and give you the real one: during the redirect chain, one hop sent an SNI that didn\u0026rsquo;t match its Host header. The edge had been absorbing this quietly, because its routing for that name happened to match anyway. The internal proxy\u0026rsquo;s stricter Host-based router missed, and the request fell into the 404 void. Only on the internal path. Only at home. Sometimes.\nThe patch, once diagnosed, is a high-priority fallback router that catches the auth path BY PATH when the Host header is unhelpful (priorities explicit, because post 6\u0026rsquo;s footgun 2 never stops being true):\n- traefik.http.routers.sso-fallback.rule=PathPrefix(`/application/o/`) - traefik.http.routers.sso-fallback.priority=2000 - traefik.http.routers.sso-fallback.service=sso-core The transferable lesson outranks the patch: split-horizon doesn\u0026rsquo;t create bugs, it reveals assumptions the single-path setup was silently absorbing. When something works remotely and fails locally, or vice versa, stop debugging the app and start diffing the paths. The app is fine. The paths differ. They always differed; now it matters.\nNext, and last on the main line: the boss fight. Self-hosted email that actually delivers.\n","permalink":"https://hi.khuo.ng/posts/2026-06-07-split-horizon-dns/","summary":"Public hostnames that resolve to the local proxy when you\u0026rsquo;re at site A: same URLs everywhere, no hairpin through the VPS for local traffic, and the OAuth quirk the shortcut drags into the light.","title":"Split-horizon DNS: same names, shorter path"},{"content":"Everything in this series is reproducible from the repo except the part you\u0026rsquo;d actually cry about. Compose files redeploy; the vault\u0026rsquo;s database and fifteen years of photos do not. I\u0026rsquo;ve lost data twice. Once to a dying disk in the era before I took backups seriously (tuition), and once, more instructively, WITH backups, when a restore revealed the job had been silently skipping the directory that mattered for eight months. Green checkmarks the whole time. That second story is why half this post is about restoring.\nYou can only back up what you can see A post 3 decision pays off here: all state lives in bind-mounted appdata directories, never named volumes. One tree per host:\n~/appdata/ +-- vault/ +-- sso-postgres/ +-- homeassistant/ +-- ... That tree IS the answer to \u0026ldquo;what needs backing up.\u0026rdquo; The backup tool points at one path, nothing hides inside Docker\u0026rsquo;s volume store, and auditing coverage is a one-minute diff of ls ~/appdata against the include list. Do that audit occasionally. It\u0026rsquo;s how you catch the new service that snuck in since.\nStep 1: restic, to two places restic: encrypted, deduplicated, snapshot-based, one static binary. Runs are incremental; every snapshot restores as a complete point in time. Same source, two repositories:\nLocal: a disk on the host. For the common case: I broke it, I deleted it, the upgrade ate it. Restores at disk speed. Backblaze B2: object storage, about $6/TB-month. For the case backups exist for: the disk, the box, or the building is gone. restic -r /mnt/backup/restic backup ~/appdata \\ --exclude \u0026#39;**/cache\u0026#39; --exclude \u0026#39;**/Cache\u0026#39; --exclude \u0026#39;**/Logs\u0026#39; restic -r b2:lab-backups:hermes backup ~/appdata --exclude ... restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune The B2 credential is an application key scoped to its one bucket (post 2\u0026rsquo;s blast-radius thinking again): a compromised host can write its own backups but can\u0026rsquo;t read another host\u0026rsquo;s bucket or touch the account.\nStep 2: schedule with systemd, alert on failure Timers beat cron here for concrete reasons: logs land in journald, OnFailure= gives you a failure hook for free, and systemctl list-timers shows the next run instead of making you simulate cron in your head. The failure hook pushes to ntfy (post 11).\nA backup system that can fail silently is a random-number generator with a progress bar. My eight-month story predates the failure hook. Never again.\nStep 3: databases get dumped, then backed up File-copying a RUNNING database snapshots a moving target: usually restorable, occasionally and silently not, the worst distribution of outcomes on offer. The fix is one line per database, just before the file backup:\ndocker exec vault-db pg_dump -U vault vault | gzip \\ \u0026gt; ~/appdata/vault/db-dump.sql.gz The dump lands inside appdata and rides along. SQLite (half of every homelab) is more forgiving, but sqlite3 app.db \u0026quot;.backup backup.db\u0026quot; into the same pattern removes the asterisk for free. Mind the order: dump, THEN backup, or tonight\u0026rsquo;s snapshot carries yesterday\u0026rsquo;s dump.\nStep 4: the keys live on paper restic repos are password-encrypted; B2 has keys. Now trace the dependency chain on the worst day: the building burns, the backups are safe in B2, and the restic password lives in\u0026hellip; the password vault, which is in the backups, which need the password. I drew that circle on a whiteboard during planning and felt appropriately silly, which is the best time and place to feel it.\nThe backup credentials must live somewhere that survives the lab and doesn\u0026rsquo;t depend on it. Mine: the password manager for convenience, plus a printed sheet in a drawer at site B. Paper has no firmware and no ransomware exposure. Of all my technology opinions, \u0026ldquo;print the keys\u0026rdquo; is the one I\u0026rsquo;d defend in court.\nStep 5: the drill (a backup you\u0026rsquo;ve never restored is a rumor) Quarterly, per service that matters:\nrestic -r b2:lab-backups:hermes restore latest \\ --include /home/lab/appdata/vault --target /tmp/drill # point a scratch compose file at /tmp/drill, then: docker compose -f drill.yml up -d # open it. LOG IN. read your actual data. then tear it down. Not restic check. That verifies restic\u0026rsquo;s internals, not your assumptions. The drill catches the system\u0026rsquo;s bugs: the exclude pattern that ate a real directory, the dump running after the backup, the app that keeps state outside appdata (looking at you, every app with a data directory AND a config directory). My eight-month near-miss would have been a ten-minute drill finding. Your first drill will find something. That\u0026rsquo;s a prediction from experience, not rhetoric.\nThe drill also produces the artifact you\u0026rsquo;ll want most during a real disaster: a restore runbook you have actually executed, written calmly, tested twice. Its margins collect the institutional memory nothing else captures: which UI settings aren\u0026rsquo;t in config (post 8 listed several), what\u0026rsquo;s deliberately not backed up, where the paper is.\nMilestone: the lab can now lose any disk, any box, or any building and come back with its state.\nNext: a pure quality-of-life upgrade. The same public names, the ten-foot path instead of the thousand-mile one, when you\u0026rsquo;re at home.\n","permalink":"https://hi.khuo.ng/posts/2026-05-30-backups-restic/","summary":"Nightly restic snapshots of appdata to a local disk AND cloud object storage, databases dumped properly, keys that live on paper, and the quarterly restore drill that finds what the green checkmarks hide.","title":"Backups: restic to local + B2, restore-tested"},{"content":"I have built the full church twice. Prometheus, Grafana, Loki, exporters on everything, a dashboard folder named WIP that became load-bearing. Both times the same arc: two glorious weeks of graphs, then a year of the monitoring stack being the most operationally demanding thing in the lab. Disk-hungry, upgrade-shy, and its failures didn\u0026rsquo;t page me, because it WAS the pager. When I caught myself debugging Loki at midnight while the thing Loki monitored ran fine, the lesson finally landed.\nA homelab\u0026rsquo;s real observability requirement is two questions. Is anything down? What did it log when it broke? Everything past that is a hobby (a fine hobby) that should be chosen, not defaulted into. Four small tools answer both questions.\nMetrics: Beszel Hub and agents, almost insultingly simple. One hub container on hermes, one agent per box talking over the overlay, one shared key.\n# per host: beszel-agent: image: henrygd/beszel-agent network_mode: host volumes: - /var/run/docker.sock:/var/run/docker.sock:ro environment: - KEY=${BESZEL_AGENT_KEY} - LISTEN=10.99.0.20:45876 # overlay only, post 5\u0026#39;s pattern You get per-host CPU, memory, disk, network, temperatures, and (the view I actually open) per-container stats. \u0026ldquo;bender is at 90% memory\u0026rdquo; is trivia; \u0026ldquo;qBittorrent grew 6GB overnight\u0026rdquo; is a diagnosis. Threshold alerts are built in. Setup time: minutes.\nWhat\u0026rsquo;s missing versus Prometheus: long retention, PromQL, graphing anything against anything. At five hosts I\u0026rsquo;ve needed that twice, and both times docker stats plus thinking sufficed.\nLogs: Dozzle Dozzle live-tails every container\u0026rsquo;s logs on every host (agents again, over the overlay), one browser tab, with search. No shipping, no indexing, no retention policy to operate. It reads Docker\u0026rsquo;s log stream and shows it to you.\nThis tool deleted most of my SSH sessions. The debugging loop used to be: ssh, docker ps to remember the name, docker logs --tail 100 -f, wrong container, repeat. Now: click the box, click the container, read. During an incident, the log line you need is seconds away, and seconds matter exactly then.\nThe honest gap: history ends where the container\u0026rsquo;s log buffer does. \u0026ldquo;What happened last Tuesday\u0026rdquo; is a real aggregator\u0026rsquo;s job. So far, post-incident curiosity has always fit inside the buffer.\nUp/down: Uptime Kuma plus ntfy Dashboards are pull; pages are push. Uptime Kuma probes whatever you point it at. When something fails it notifies ntfy, a self-hosted push server with a phone app that subscribes to topics. Service drops, pocket buzzes.\nThe design decision that matters is WHAT to probe. Container-up is the wrong check: Docker restarts crashed containers itself, and post 10\u0026rsquo;s deunhealth handles unhealthy-but-running. The checks that earn their keep probe the full path: the public URL of each public service, through real DNS, the edge, the overlay, to the app. The route a stranger takes. A container can be perfectly healthy behind a broken route all day, and only a path check notices. I learned this from a failed cert renewal: every container green, every user-facing page a TLS error, for six hours.\nComplete it with the meta-check: who watches the watcher? Uptime Kuma pushes a heartbeat; ntfy alerts if the heartbeat goes missing. The monitoring being down should be the first page, not a surprise during the next real incident.\nThe front door: Homepage A static dashboard. Every service is a card, grouped by host, with live widgets pulling each app\u0026rsquo;s own stats over the overlay: queue depth, request counts, disk free, who\u0026rsquo;s streaming. The YAML lives in the repo; widget API keys ride the sops pipeline.\nDay to day it\u0026rsquo;s a bookmark bar with vitals. Operationally it\u0026rsquo;s a deploy check: push a change, glance at the card, the widget either answers or it doesn\u0026rsquo;t. And when the household asks \u0026ldquo;is the thing broken?\u0026rdquo;, I answer from one page in five seconds, which is the actual service-level objective of a homelab: the operator sounds like he knows.\nThe scorecard Question Tool Weight Is the box healthy? Beszel a container + agents What\u0026rsquo;s it logging? Dozzle a container + agents Is it up, from outside? Uptime Kuma + ntfy a container each Where is everything? Homepage YAML you\u0026rsquo;d want anyway Four small tools, each replaceable in an afternoon, none capable of becoming the incident. If the lab outgrows this, Prometheus will still exist. Observability is the easiest layer to retrofit and the worst one to gold-plate on day one. Graph what you\u0026rsquo;ll act on; everything else is screensaver.\nNext: backups. restic to two destinations, keys on paper, and the restore drill that turns a rumor into a guarantee.\n","permalink":"https://hi.khuo.ng/posts/2026-05-23-observability/","summary":"Four small tools instead of a monitoring distro: live metrics, every host\u0026rsquo;s logs in one tab, uptime checks that page the phone, and a dashboard tying it together. What you give up, and why it doesn\u0026rsquo;t hurt at this scale.","title":"Observability without the Prometheus tax"},{"content":"Post 10 built the plumbing. This page is about the brains: Sonarr and Radarr, and the three decisions inside them that determine whether your library is a library or a junk drawer. I\u0026rsquo;ve run these tools for a long time; what follows is the settled residue of getting each piece wrong at least once.\nThe mental model A *arr is a reconciliation loop. You declare what you want (this show, this quality range), it continuously compares that against what you have, and it closes the gap: searches the indexers, hands the pick to a download client, waits, then renames and moves the result into the library. That last step is where most setups quietly rot. The *arr is the single naming authority for your library; everything in this post flows from taking that seriously.\nProwlarr sits beside them as the indexer hub: sources get configured once there and sync to every *arr. Add an indexer in one place, it appears everywhere. The alternative, maintaining indexer lists per app, is a five-minute job that multiplies into an evening.\nStep 1: naming templates, set once, never touched These pair with the Plex scanner expectations from post 15 (year, provider ID, SxxEyy). Sonarr, Settings, Media Management:\nSeries folder: {Series TitleYear} {tvdb-{TvdbId}} Season folder: Season {season:00} Episode (standard): {Series TitleYear} - S{season:00}E{episode:00} - {Episode CleanTitle} [{Custom Formats }{Quality Full}] Radarr, same idea:\nMovie folder: {Movie TitleThe} ({Release Year}) {imdb-{ImdbId}} Movie file: {Movie TitleThe} ({Release Year}) {imdb-{ImdbId}} [{Custom Formats }{Quality Full}] What each piece does: the year disambiguates remakes; the {imdb-}/{tvdb-} token makes Plex\u0026rsquo;s matching deterministic (the scanner reads the ID and stops guessing); CleanTitle strips characters that upset some filesystems and at least one TV client; the quality block in brackets is for humans, handy for seeing at a glance what an upgrade replaced.\nTurn on Rename Episodes, and under Importing leave \u0026ldquo;Use Hardlinks instead of Copy\u0026rdquo; enabled. That setting only delivers if you built the /data single-mount convention from post 15. If your imports are slow copies, fix the mounts, not this checkbox.\nStep 2: quality profiles that terminate The classic failure: enable every quality from DVD to remux, allow upgrades across the whole range, and spend the year re-downloading the same titles in ever-fatter files. A profile should answer two questions: what\u0026rsquo;s acceptable now, and where does upgrading STOP.\nMine, after plenty of oscillation:\nTV (WEB-1080p). WEBDL-1080p and WEBRip-1080p acceptable, upgrade-until WEBDL-1080p. TV gets watched once on TVs and phones; 1080p WEB is the sweet spot of quality, size, and availability. Movies (HD-1080p). Bluray-1080p preferred, WEBDL-1080p acceptable, upgrade-until Bluray-1080p. No remuxes: a 60GB file whose extra bits my screens can\u0026rsquo;t show is just a slower backup. 4K: a deliberate non-goal until every link in the chain (displays, storage, transcode math) says yes. Half-committing to 4K gets you a library that\u0026rsquo;s 10% huge files you transcode down anyway. The principle: upgrades must terminate. Every profile gets an explicit upgrade-until ceiling. Once a title reaches it, the loop goes quiet. A profile without a ceiling is an agreement to churn bandwidth forever.\nStep 3: custom formats, using other people\u0026rsquo;s homework Quality alone can\u0026rsquo;t express \u0026ldquo;prefer proper releases, avoid x265 re-encodes of WEB sources, never touch these groups.\u0026rdquo; Custom formats can: they\u0026rsquo;re matchers that add or subtract score from candidate releases, and the *arr picks the highest score within the allowed qualities.\nThey\u0026rsquo;re also a rabbit hole. The TRaSH guides (trash-guides.info) maintain battle-tested format definitions and scoring for exactly the profiles above: unwanted-release filters, repack handling, codec steering, group lists. Use theirs. Hand-rolling custom formats is for the day their sets don\u0026rsquo;t cover your problem, and that day took years to arrive for me.\nStep 4: recyclarr, so the settings live in git Everything above is configured through web UIs, which means none of it is in the repo, none survives a rebuild, and none has a diff history. recyclarr fixes that: a small container that reads YAML and pushes TRaSH quality definitions, custom formats, scores, and naming config into Sonarr and Radarr on a schedule.\n# config/recyclarr/recyclarr.yml (excerpt) sonarr: main: base_url: http://sonarr:8989 api_key: !env_var SONARR_API_KEY quality_definition: type: series custom_formats: - trash_ids: # repack/proper handling, unwanted filters, ... assign_scores_to: - quality_profile: name: WEB-1080p The YAML lives in the repo, the API keys ride the sops pipeline (post 4), the sync runs nightly. Now the *arr config has the same properties as everything else in the lab: versioned, reviewable, rebuildable. When a TRaSH update changes scoring, it shows up as a diff in recyclarr\u0026rsquo;s logs instead of as mystery behavior.\nThe caveat: recyclarr owns what it manages. Hand-edit a custom format it controls and the next sync reverts you. That\u0026rsquo;s the two-authorities problem from post 15 in a different hat. Decide per-knob whose it is, and don\u0026rsquo;t share custody.\nThe end-to-end trace One request, no humans:\n1. someone adds a show in the request portal 2. portal -\u0026gt; Sonarr: monitored series, WEB-1080p profile 3. Sonarr -\u0026gt; Prowlarr-synced indexers: search 4. candidates scored by profile + custom formats 5. winner -\u0026gt; qBittorrent (inside gluetun, post 10) 6. download lands in /data/torrents/... 7. Sonarr imports: HARDLINK to /data/media/tv/Show (Year) {tvdb-...}/ Season 01/Show - S01E01 - Title [WEBDL-1080p].mkv 8. webhook -\u0026gt; autoscan -\u0026gt; Plex scans one directory 9. it\u0026#39;s on the TV before the requester\u0026#39;s kettle boils Steps 4 and 7 are this page, and they\u0026rsquo;re the two you can\u0026rsquo;t patch by hand later without making everything worse. Good news, everyone: you only have to set them up once.\nNext, back on the main line: knowing when any of this breaks before the household does.\n","permalink":"https://hi.khuo.ng/posts/2026-05-16-arr-naming-quality/","summary":"The deep dive on library automation: exact naming templates that feed Plex cleanly, quality profiles that stop the upgrade churn, custom formats without the madness, and recyclarr so none of it lives only in a web UI.","title":"The *arrs, properly: naming templates, quality profiles, and recyclarr"},{"content":"The media server is the one service in the lab with users: people who didn\u0026rsquo;t sign up for a hobby and just want the thing to play when they press play. That changes the engineering calculus completely, which is why this app gets a full page.\nPlex vs Jellyfin: the honest version I wanted to want Jellyfin. It\u0026rsquo;s open source, it doesn\u0026rsquo;t phone home, nobody can paywall a feature I depend on, and the server itself is genuinely good. I run open source everywhere else it\u0026rsquo;ll go.\nI still run Plex, and the reason is the part people undersell: the client fleet. My users watch on a mix of smart TVs, phones, a games console, and one ancient streaming stick. The Plex client on every one of those platforms is mature, resumes properly, handles subtitle quirks, and looks like a product. Jellyfin\u0026rsquo;s clients have improved a lot, but across that specific zoo the experience is still uneven, and I don\u0026rsquo;t operate the TVs. When playback stutters at someone else\u0026rsquo;s house, the support request comes to me. Plex costs me money; Jellyfin would cost me support calls. I picked the one I can afford.\nThe rest of the ledger, for honesty\u0026rsquo;s sake. In Plex\u0026rsquo;s favor: brokered remote access means people outside the network just work without me exposing anything (more below), watch-state and profiles are effortless, and Plexamp is the best music client I\u0026rsquo;ve used, period. Against it: the features that matter need Plex Pass (hardware transcoding chief among them; lifetime is the only subscription math that works), the company keeps shuffling features between tiers (the 2025 remote-streaming changes were a warning shot), and auth rides their cloud, so a plex.tv outage degrades login to your own local server. That last one genuinely bothers me.\nSo: why not Jellyfin YET. I re-evaluate roughly yearly. The media is named to open standards (below), so nothing locks me in, and the day the client gap closes on my actual devices, migration is an afternoon. That\u0026rsquo;s the posture I\u0026rsquo;d recommend: choose for your users, keep the exit cheap.\nStep 1: the compose file Plex gets its own box: professor, a repurposed desktop whose retired gaming GPU now earns its keep transcoding. The library lives on nibbler, the storage box, exported over NFS. One file, compose/professor/services/plex.yml, per post 3\u0026rsquo;s conventions:\nservices: plex: image: lscr.io/linuxserver/plex:latest container_name: plex restart: unless-stopped network_mode: host # earns it, see below runtime: nvidia # the repurposed GPU, see step 2 environment: - PUID=1000 - PGID=1000 - TZ=America/Toronto - VERSION=docker - PLEX_CLAIM=${PLEX_CLAIM} # first boot only, via sops - NVIDIA_VISIBLE_DEVICES=all - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility volumes: - ${USERDIR}/appdata/plex:/config - /mnt/nibbler/data/media:/data/media # nibbler\u0026#39;s NFS export - /dev/shm:/transcode # transcode in RAM Three choices worth defending:\nnetwork_mode: host. I run almost nothing with host networking; the proxy-network pattern is the whole point of posts 6 and 7. Plex is the exception that earns it: GDM discovery (how TVs find the server), DLNA, and a pile of client behaviors work properly when Plex sees the real LAN. You can run it bridged and forward 32400 plus a handful of UDP ports. I did, for a while. Every odd \u0026ldquo;server not found\u0026rdquo; complaint went away when I stopped fighting it.\nRAM transcode. Pointing the transcoder at /dev/shm keeps constant scratch writes off the disks; transcode chunks live exactly as long as a play session, which is what RAM is for. Doubly true here, where the alternative would be scratch writes over NFS.\nThe claim token ties a fresh server to your account. It expires in minutes, so grab it from plex.tv/claim right before first boot. It goes through sops like every secret, mostly so the compose file stays paste-able.\nStep 2: make hardware transcoding actually engage A retired mid-range Nvidia card chews through several 4K-to-1080p NVENC streams without noticing, IF the pieces line up. The checklist, in the order I got each one wrong:\nPlex Pass. Hardware transcoding is a paid feature. No pass, no NVENC, no matter how correct the rest is. The host can see the card. Driver installed, nvidia-smi prints the GPU. No smi, no transcode, stop here and fix it. Docker can hand it to containers. Install the NVIDIA Container Toolkit and check with a throwaway container: docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi. The compose wiring. runtime: nvidia plus the NVIDIA_VISIBLE_DEVICES / NVIDIA_DRIVER_CAPABILITIES env vars from step 1. The video capability is the one transcoding needs; it\u0026rsquo;s also the one generic toolkit examples leave out. The checkbox. Settings, Transcoder, \u0026ldquo;Use hardware acceleration when available.\u0026rdquo; Yes, I have shipped a \u0026ldquo;broken\u0026rdquo; transcode setup that was this checkbox. Check it worked: force a transcode (set a client to 720p), then look at the dashboard. The stream should say (hw) next to Transcode. No (hw), no hardware. The other tell is nvidia-smi on professor showing a plex process under the enc/dec columns while the CPU stays flat.\nOne footnote that used to matter: consumer GeForce cards historically shipped with a driver-imposed cap on concurrent NVENC sessions, and a well-known community patch lifted it. Recent drivers raised the cap high enough that a homelab won\u0026rsquo;t hit it. If an older setup mysteriously refuses a third or fourth simultaneous transcode, that cap is what you\u0026rsquo;re remembering.\nStep 3: the /data convention (the most important section here) If I could teach one thing about media servers it would be this, and it isn\u0026rsquo;t even a Plex setting. Every container in the pipeline mounts the same storage root at the same internal path:\nnibbler:/data/ # ONE filesystem, exported over NFS +-- torrents/ # download clients (bender) write here +-- usenet/ +-- media/ # the library, Plex (professor) reads here +-- movies/ +-- tv/ # on bender, the SAME mount in qbittorrent, sabnzbd, sonarr, radarr: volumes: - /mnt/nibbler/data:/data # on professor, plex needs only the library half: - /mnt/nibbler/data/media:/data/media Site B splits the roles across boxes (bender downloads, nibbler stores, professor serves), and the convention is what makes that split free: everything mounts nibbler\u0026rsquo;s ONE exported filesystem, so an import is still a rename on the same fs, even though the *arr asking for it and the disks doing it live in different machines.\nWhy it matters: when Sonarr imports a finished download, it moves /data/torrents/... to /data/media/tv/.... If both paths live on one filesystem inside one consistent mount, that move is a hardlink: instant, free, and the original keeps seeding with no duplicate copy. If instead each container mounts its own slice at its own path (/downloads here, /tv there, the way every quick-start does it), the same import becomes a full copy across mount boundaries: slow, doubles disk usage during a backlog, breaks seeding.\nI ran the fragmented version for years and just accepted slow imports as the weather. The fix was one afternoon of remapping volumes. The TRaSH guides hammer this point, and they are right to.\nThe companion rule: path consistency beats path cleverness. Same /data prefix in every container, no exceptions. Sonarr tells qBittorrent \u0026ldquo;the file is at /data/torrents/x,\u0026rdquo; and that sentence must be true in both containers\u0026rsquo; worlds.\nStep 4: naming the scanner loves Plex\u0026rsquo;s scanner is good the way a tired librarian is good: give it ambiguity and someone else\u0026rsquo;s poster ends up on your home videos. The structure that never misfires:\nmedia/movies/The Thing (1982) {imdb-tt0084787}/ The Thing (1982) {imdb-tt0084787} [Bluray-1080p].mkv media/tv/Futurama (1999) {tvdb-73871}/ Season 01/ Futurama (1999) - S01E01 - Space Pilot 3000 [WEBDL-1080p].mkv The load-bearing parts: the year in the folder (disambiguates the forty movies named Crash), a provider ID in braces ({imdb-...}, {tvdb-...}) which removes matching guesswork entirely, and SxxEyy markers. The quality tag in brackets is for humans; Plex ignores it.\nAnd the rule that keeps it correct: you never name anything by hand. The *arrs own renaming. Configure the template once (post 16 has my exact templates) and every import lands right forever. The moment a human \u0026ldquo;fixes\u0026rdquo; filenames in the library, you have two naming authorities, and they will fight.\nStep 5: library settings worth changing Defaults are mostly fine. The ones I change:\nScan my library automatically: on, but know its limit: it rides inotify, and inotify events do not cross an NFS mount. Plex on professor will never hear about a file bender imported onto nibbler. The fix is an autoscan sidecar: the *arrs call its webhook on import and it pokes Plex about exactly the changed directory. With split storage this isn\u0026rsquo;t an optimization, it\u0026rsquo;s the only thing that works. Periodic scan: off once event-driven pickup works. A full scan of a big library is an I/O storm that buys nothing the events didn\u0026rsquo;t. Intro/credit detection, thumbnails: per-library, not global. Real CPU cost and gigabytes of preview data on first run. TV skip-intro is worth it; turning it on everywhere because the checkbox existed is not. Agents: barely matter once IDs are in the folder names, which is the point. Metadata follows the ID. Step 6: remote access without exposing anything Plex\u0026rsquo;s brokered remote access (the server registers with plex.tv, clients connect via NAT traversal) is the rare phone-home feature I keep: zero exposed ports for full remote streaming, which fits \u0026ldquo;one machine faces the internet\u0026rdquo; better than self-hosting the ingress would. url routes nothing for Plex.\nOne subtlety for your own devices on the overlay: Settings, Network, Custom server access URLs, add http://10.99.0.22:32400 (professor\u0026rsquo;s overlay IP). Devices on the overlay then connect direct instead of falling back to Plex\u0026rsquo;s relay at degraded quality from hotel wifi, even though you have a perfectly good tunnel home.\nThe integration map request portal (post 10) -\u0026gt; *arrs (post 16) -\u0026gt; clients in the VPN ^ | namespace (post 10) people ask | rename + hardlink into /data/media v autoscan -\u0026gt; Plex (this post) -\u0026gt; TVs, phones Every arrow is a webhook or a filesystem event. Nothing polls, and after the one-time setup, nothing needs me. The best media server is the one whose admin forgets it\u0026rsquo;s running.\nNext: the other half of the pipeline. Naming templates, quality profiles that stop the churn, and recyclarr keeping it all in git.\n","permalink":"https://hi.khuo.ng/posts/2026-05-13-plex/","summary":"The whole Plex page: the honest Plex-vs-Jellyfin call, hardware transcoding that actually engages, the /data mount convention that makes hardlinks work, naming the scanner loves, and wiring the *arrs in cleanly.","title":"Plex, properly: a full setup, and why not Jellyfin (yet)"},{"content":"Site B runs the media pipeline across three boxes: bender (request portal, *arrs, download clients), nibbler (the storage), and professor (Plex). This post is the back half, the plumbing, and especially the networking pattern, which is the most reusable idea on the box. The brains got their own pages: Plex is post 15, the *arr naming and quality machinery is post 16.\nThe obligatory paragraph, meant sincerely: automation is content-neutral, your sources shouldn\u0026rsquo;t be. Point this at media you have the rights to: your own rips, freely-licensed content, backfills of things you own. The tooling neither knows nor cares. What you feed it is on you.\nThe cast request portal (Overseerr) \u0026lt;- the only public-facing piece | Sonarr / Radarr \u0026lt;- library managers (post 16) | Prowlarr \u0026lt;- indexer hub, synced to the *arrs | +------------------------------+ | gluetun (VPN client) | \u0026lt;- everything in this box exits | +-- qBittorrent | via the tunnel, or not at all | +-- SABnzbd | +------------------------------+ | hardlink into /data/media (post 15\u0026#39;s convention, | on nibbler\u0026#39;s exported filesystem) Plex (on professor) The pattern: a VPN network namespace I\u0026rsquo;ve done VPN-for-downloads every fragile way: VPN on the whole host (everything tunnels, including things that shouldn\u0026rsquo;t), app-level proxy settings (which apps bypass for DNS or trackers, or quietly reset after an update), and the built-in \u0026ldquo;kill switch\u0026rdquo; checkboxes, which are promises. What you want is physics.\nGluetun is physics. It\u0026rsquo;s a container that connects to your VPN provider and lends out its network namespace:\n# compose/bender/services/gluetun.yml (trimmed to the idea) services: gluetun: image: qmcgaw/gluetun container_name: gluetun cap_add: [NET_ADMIN] networks: [proxy] environment: - VPN_SERVICE_PROVIDER=${VPN_PROVIDER} - WIREGUARD_PRIVATE_KEY=${VPN_WG_KEY} # sops, as ever - SERVER_COUNTRIES=${VPN_COUNTRY} volumes: - ${USERDIR}/appdata/gluetun:/gluetun qbittorrent: image: lscr.io/linuxserver/qbittorrent network_mode: service:gluetun # \u0026lt;- the entire trick volumes: - ${USERDIR}/appdata/qbittorrent:/config - /mnt/nibbler/data:/data # nibbler\u0026#39;s export, post 15 sabnzbd: image: lscr.io/linuxserver/sabnzbd network_mode: service:gluetun volumes: - ${USERDIR}/appdata/sabnzbd:/config - /mnt/nibbler/data:/data network_mode: service:gluetun means qBittorrent has no network identity of its own. No interface, no routes, no DNS except gluetun\u0026rsquo;s. Every packet it can physically emit goes through the tunnel, because there is nothing else. When the VPN drops, gluetun\u0026rsquo;s internal firewall closes and the clients go dark instead of leaking onto the ISP connection.\nTest it yourself once, it\u0026rsquo;s worth seeing: kill the tunnel on purpose mid-download and watch the peer list freeze. After years of trusting checkboxes, watching the leak be impossible is the moment this pattern sells itself.\nThe routing consequence everyone hits Namespace-sharers don\u0026rsquo;t exist on the proxy network, so Traefik can\u0026rsquo;t see them, and labels on their own containers do nothing. The labels go on gluetun, which owns the network identity. One router/service pair per app, distinguished by port:\ngluetun: labels: - traefik.enable=true # qBittorrent - traefik.http.routers.qbit.rule=Host(`qbit.${INTERNAL_DOMAIN}`) - traefik.http.routers.qbit.middlewares=sso@file - traefik.http.routers.qbit.service=qbit - traefik.http.services.qbit.loadbalancer.server.port=8080 # SABnzbd - traefik.http.routers.sab.rule=Host(`sab.${INTERNAL_DOMAIN}`) - traefik.http.routers.sab.middlewares=sso@file - traefik.http.routers.sab.service=sab - traefik.http.services.sab.loadbalancer.server.port=8081 This is THE faq of the pattern. The symptom (\u0026ldquo;Traefik 404s but the app is running fine\u0026rdquo;) points everywhere except here, because the app\u0026rsquo;s own compose file looks perfectly normal. Note the explicit .service= on each router: with multiple pairs on one container, Traefik\u0026rsquo;s auto-wiring guesses wrong.\nThe 3 a.m. gotcha: namespace death When gluetun restarts (VPN hiccup, image update, your own redeploy), its network namespace is destroyed and recreated. The clients inside keep running, attached to a namespace that no longer routes anywhere. They don\u0026rsquo;t crash. They sit there looking healthy, downloading nothing, until something restarts them.\nThe fix is an auto-healer: give the clients a healthcheck that proves real connectivity, and run a small watcher (deunhealth) that restarts anything marked unhealthy:\nqbittorrent: labels: - deunhealth.restart.on.unhealthy=true healthcheck: test: [\u0026#34;CMD\u0026#34;, \u0026#34;curl\u0026#34;, \u0026#34;-sf\u0026#34;, \u0026#34;https://api.ipify.org\u0026#34;] interval: 60s retries: 3 That specific check has a bonus property: it fails both when the namespace is dead AND if the container somehow ends up with naked internet. Either way, a restart into gluetun\u0026rsquo;s fresh namespace is the correct response. The VPN blipping overnight becomes a log line you read at breakfast instead of a morning incident.\nExposure: one door Per post 3\u0026rsquo;s pattern: Overseerr gets an internal route AND an edge route. It\u0026rsquo;s the household\u0026rsquo;s front door, with its own accounts and a UI built for \u0026ldquo;request a thing.\u0026rdquo; Everything else (*arrs, clients, Prowlarr) stays internal-only behind the SSO middleware. Nobody outside needs a download client\u0026rsquo;s UI; the request portal is the public API to the whole machine. People ask, the pipeline does, Plex delivers, and my involvement rounds to zero. Bender handles it. It\u0026rsquo;s what he does.\nNext: the deep dives this post keeps pointing at. Plex first.\n","permalink":"https://hi.khuo.ng/posts/2026-05-09-media-stack/","summary":"The plumbing half of bender\u0026rsquo;s media stack: download clients that live inside a VPN network namespace, the label trick namespace-sharing requires, and the restart gotcha that otherwise pages you at 3 a.m.","title":"The download pipeline: gluetun and the VPN-namespace trick"},{"content":"Everything so far deploys by hand. This post adds the missing verb: push. Edit a service file on the laptop, commit, push, and the right host pulls, decrypts its secrets, and recreates exactly the affected stack. This is where the repo stops being documentation that happens to be accurate and becomes the control plane.\nWhy Komodo The candidates I\u0026rsquo;d actually lived with, and why they lost:\nPortainer: the real config migrates into Portainer\u0026rsquo;s database and the compose files on disk become historical fiction. Post 1 told you how that ends. Ansible: push-based and drift-prone between runs, and a pile of playbooks is its own codebase with its own bugs. I wanted less custom orchestration, not differently-shaped. Watchtower: bumps image tags. Can\u0026rsquo;t add a service, change a label, or know what a stack is. Fine at its actual job, which isn\u0026rsquo;t this one. Komodo matches the shape we already have: a core (UI plus state, on hermes, behind post 8\u0026rsquo;s SSO via OIDC) and a periphery agent on each host that executes deploys. Each host\u0026rsquo;s compose.yml registers as a \u0026ldquo;stack\u0026rdquo; pointing at the repo. A webhook on push means: pull, diff, redeploy what changed.\nThe agent is systemd, not a container, on purpose Periphery runs as a native systemd service. That looks like a step backward from containerize-everything until you trace what a deploy needs: the pre-deploy hook decrypts SOPS secrets (post 4), which needs the host\u0026rsquo;s age key. A containerized agent would need that key mounted in, and now the key\u0026rsquo;s blast radius is \u0026ldquo;anything that can reach that container\u0026rdquo; instead of \u0026ldquo;root on the host.\u0026rdquo; We\u0026rsquo;d be un-scoping the thing post 4 carefully scoped.\n# /etc/systemd/system/komodo-periphery.service (the idea) [Service] ExecStart=/usr/local/bin/periphery Environment=PERIPHERY_ROOT_DIRECTORY=/home/lab/komodo Environment=PERIPHERY_BIND_IP=10.99.0.20 # overlay only Restart=always That bind line is post 5\u0026rsquo;s pattern again: the API that can recreate any container on the box does not exist on any public interface.\nSecrets at deploy time The pre-deploy hook on each host:\nsops decrypt secrets/shared.env \u0026gt; ~/secrets/shared.env sops decrypt secrets/bender.env \u0026gt; ~/secrets/bender.env The stack consumes those as extra env files. Plaintext lives outside the checkout, regenerated every deploy, scoped per host.\nThe flag that costs people an hour: mark those env files untracked (track=false in Komodo\u0026rsquo;s stack config). Tracked files get validated BEFORE the pre-deploy hook runs, so a fresh host fails with \u0026ldquo;env file missing\u0026rdquo; because the thing that creates the file hasn\u0026rsquo;t been allowed to run yet. The error message names neither the chicken nor the egg.\nThe automation must not manage itself Post 3 drew the line: the infra stack (proxy, SSO outpost, the deploy agent itself) deploys manually; the app stack deploys automatically. Now you can see the line under load. If Komodo managed its own stack, a bad push would recreate the deployer mid-deploy. Nothing is deploying, the change is half-applied, and the recovery is SSH and hands. The deployer\u0026rsquo;s failure domain has to exclude the deployer, the proxy you reach it through, and the SSO you log into it with. Those deploy via a deliberately boring script, and boring is the feature.\nTreat the webhook like the weather Two operational truths, both paid for:\nWebhooks die silently. A delivery fails upstream, a secret rotates, an endpoint moves, and pushes just stop deploying, with no error anywhere you look. Mine was dead for four days once while I kept \u0026ldquo;deploying\u0026rdquo; at the speed of nothing.\nChange detection has blind spots. A push that only touches secrets shows no compose diff, so no stack redeploys. Your rotated password sits in git while the old one keeps serving traffic.\nOne answer covers both: deterministic scripts as the primary path, the webhook as a fast path. Two scripts live in the repo: a sync-hosts script (pull everywhere, redeploy infra if infra files changed, re-decrypt if secrets changed) and a force-redeploy for app stacks. After any push where the outcome matters, run the script, or at least verify the deploy landed (post 11 makes that a ten-second glance).\nThe honest framing of homelab GitOps: the repo is always RIGHT; whether it\u0026rsquo;s APPLIED is a question you must be able to answer, and force, with one command at 2 a.m.\nThe stale-clone corollary Periphery executes hook scripts from the host\u0026rsquo;s clone of the repo, which updates when something pulls it, not magically on push. Change a deploy script, forget the host hasn\u0026rsquo;t pulled, and the host runs the old script while you stare at the new one. The sync script\u0026rsquo;s first act on every host is git pull for exactly this reason. Symptom to file away: \u0026ldquo;the fix I definitely pushed is definitely not what just ran.\u0026rdquo;\nMilestone: a service edit now goes laptop to running container with one push, and the parts that could brick the pipeline are the parts the pipeline can\u0026rsquo;t touch.\nNext: the download pipeline on bender, and the networking trick that turns compose syntax into a kill-switch.\n","permalink":"https://hi.khuo.ng/posts/2026-04-25-komodo-deploys/","summary":"Komodo Core watches the repo and redeploys app stacks on push. Secrets decrypt in a pre-deploy hook, the agent runs as systemd on purpose, and the webhook gets treated like the weather: usually fine, never trusted.","title":"Komodo: git push to deploy"},{"content":"Count your services, then count their login pages. For years my honest tally was: about thirty services, six with real passwords, ten sharing a password I\u0026rsquo;d rather not discuss, and the rest \u0026ldquo;temporarily\u0026rdquo; unauthenticated on the theory that nobody knows the URL. Security through apathy.\nThe fix isn\u0026rsquo;t discipline (post 4 covered how well discipline scales). The fix is making authentication cheaper than not having it. That\u0026rsquo;s what SSO does here: protecting a new app costs one label, so everything gets protected, and the apathy path stops existing.\nTopology: one brain, many hands The Authentik core runs on hermes: server, worker, its own postgres and redis, the stateful-with-db pattern from post 3. Every host, including hermes itself and including url, runs a lightweight outpost container that enforces auth locally and keeps a websocket open to the core.\nConfession: I originally ran an independent Authentik per host. It seemed robust. It was actually three user databases, three configs drifting apart, three upgrade dances, and no shared session, so \u0026ldquo;single\u0026rdquo; sign-on involved signing in three times. The core/outpost topology is the shape the product wants: state in one place, enforcement everywhere, sessions that mean something.\nThe two integration modes Forward-auth, for apps that can\u0026rsquo;t (or shouldn\u0026rsquo;t be trusted to) do their own auth. The proxy intercepts each request and asks the local outpost who this is; unauthenticated users get bounced to the login flow:\n# defined ONCE, in each proxy\u0026#39;s dynamic config: http: middlewares: sso: forwardAuth: address: http://sso-outpost:9000/outpost.goauthentik.io/auth/traefik trustForwardHeader: true authResponseHeaders: - X-authentik-username - X-authentik-email - X-authentik-groups Per app, the entire cost of protection:\n- traefik.http.routers.app.middlewares=sso@file Those response headers are a quiet bonus: apps that understand proxy-auth headers (more than you\u0026rsquo;d expect) get not just a locked door but a logged-in user with groups.\nOIDC, for apps that speak it: the deploy UI, the request portal, the code editor. They get a real provider in Authentik and run the redirect dance themselves. Proper tokens, proper logout, permissions mapped from Authentik groups. Users see the same account and session either way and can\u0026rsquo;t tell which mode an app uses, which is the point.\nThe cookie mechanics from post 2 pay off here: the forward-auth session rides a cookie on the canonical domain, so one login covers every *.example.dev admin tool. This is why everything sensitive lives under one roof.\nConfig as code, where the product allows Authentik applies YAML blueprints from a watched directory: providers, applications, the app library. Mine bind-mount straight from the repo, so a rebuilt Authentik comes back configured instead of blank:\n# config/sso/blueprints/app-vault.yaml (the shape) entries: - model: authentik_providers_proxy.proxyprovider identifiers: {name: vault} attrs: external_host: https://vault.example.dev mode: forward_single - model: authentik_core.application identifiers: {slug: vault} attrs: name: Vault provider: !KeyOf vault The honest caveat: not everything is blueprint-able. Some settings stay UI-made (session lengths among them, below), and those belong in your rebuild runbook. \u0026ldquo;After rebuild, re-apply the following by hand\u0026rdquo; is an ugly sentence that beats rediscovering each item in production.\nFour pitfalls, ranked by how down production went 1. The outpost dials the core by hostname. Keep it the INTERNAL name. The outpost\u0026rsquo;s AUTHENTIK_HOST is where it phones home AND part of token validation. Point it at a hostname that later answers with a redirect (say, after a domain change) and the redirect strips the Authorization header. The outpost gets 403, every protected app on that host returns 500, and no log says \u0026ldquo;your hostname went stale.\u0026rdquo; Reference the internal domain variable so it moves when the domain moves, mechanically. This one took everything down twice before the rule existed.\n2. Outposts match the core\u0026rsquo;s version, exactly. A version-skewed outpost falls into a websocket reconnect loop (close code 1006, on repeat) and auth goes intermittent by the minute. Core and outposts upgrade as one commit, all hosts, same day.\n3. Set an explicit session lifetime. The default session is browser-scoped: close the browser, log in again, every user, all day. One field fixes it (validity on the login flow\u0026rsquo;s stage; mine is 14 days). It\u0026rsquo;s UI-made, not blueprint-managed, so onto the rebuild runbook it goes. Users notice this one loudest.\n4. Make identities survive a domain move. OIDC apps store the issuer URL and a subject ID. Configure providers with per-provider issuer mode and a hashed user ID as the subject, and the subject survives an issuer-host change. A domain migration then becomes \u0026ldquo;update the issuer URL in each app\u0026rdquo; instead of \u0026ldquo;every OIDC account is an orphan attached to a dead issuer.\u0026rdquo; Costs nothing at setup. Saves a weekend later.\nBreak-glass, before you need it The day SSO fronts everything is the day an SSO outage locks you out of the tools you\u0026rsquo;d use to fix the SSO. Before that day: a second admin account with a recovery path around the SSO, tested while you can still get in, written into the runbook. Cut the spare key before the door locks.\nMilestone: one account, one session, and every service in the lab behind it for one label apiece.\nNext: making git push deploy all of this.\n","permalink":"https://hi.khuo.ng/posts/2026-04-11-authentik-sso/","summary":"A single Authentik core with a lightweight outpost on every host: forward-auth for apps that can\u0026rsquo;t do auth, OIDC for apps that can, config as code where the product allows, and four pitfalls ranked by how down production went.","title":"Authentik: one login for everything"},{"content":"Post 1 made the promise: exactly one machine faces the internet, and it\u0026rsquo;s the one we care about least. This post builds that machine. In this lab it\u0026rsquo;s called url, after Futurama\u0026rsquo;s robot cop. Officer URL works the door, checks IDs, and owns nothing worth stealing.\nurl runs three containers: Traefik, an SSO outpost (post 8), and Crowdsec. No databases, no user data, no credentials beyond the scoped DNS token from post 2. Anyone who\u0026rsquo;s run public servers for long has a short list of boxes that got popped on their watch; the goal here isn\u0026rsquo;t pretending you\u0026rsquo;ll never be on that list again, it\u0026rsquo;s arranging that the internet-facing mistake budget lands on a machine whose complete recovery is \u0026ldquo;rebuild from the repo, ten minutes.\u0026rdquo;\nRoutes live in a file, and the file is the point Internal proxies route by container label because their containers are local. The edge proxies to other machines, so its routes live in the file provider. That file quietly becomes one of the most useful artifacts in the repo: the complete, reviewable inventory of what the internet can reach.\n# traefik/dynamic/public-routes.yml (the shape) http: routers: requests-public: rule: Host(`requests.example.tv`) service: requests-backend middlewares: [secure-headers, crowdsec] tls: {certResolver: dns} vault-public: rule: Host(`vault.example.dev`) service: vault-backend middlewares: [secure-headers, crowdsec] tls: {certResolver: dns} services: requests-backend: loadBalancer: servers: - url: http://10.99.0.20:5055 # bender, over the overlay vault-backend: loadBalancer: servers: - url: http://10.99.0.10:8090 # hermes \u0026ldquo;What\u0026rsquo;s exposed right now?\u0026rdquo; used to be archaeology across three servers. Now it\u0026rsquo;s one file with a git history. Every public service appears here AND keeps its internal-label route on its home box (post 3\u0026rsquo;s rule): the edge route is how the world arrives, the internal route is how you reach it when the WAN is down and you\u0026rsquo;re debugging why.\nAuth at the door Services with real logins of their own (the vault, the request portal) route straight through. Everything else gets the SSO forward-auth middleware at the edge, so strangers are challenged on the VPS before a packet crosses into either site. One middleware reference on the router, same as the internal pattern. It converts \u0026ldquo;I accidentally exposed an admin panel\u0026rdquo; from a disclosure into a login page.\nThe TLS error whose message will not help you Sometimes the edge proxies to a backend Traefik instead of straight to an app, useful when a host should own its own routing for a service. HTTPS to HTTPS. The first time you wire it:\n500 Internal Server Error ... remote error: tls: unrecognized name Here\u0026rsquo;s what\u0026rsquo;s happening. The edge dials the backend by IP (https://10.99.0.10), so the TLS handshake carries no useful SNI. The backend proxy picks certificates BY SNI; given nothing, it refuses the handshake or serves a default cert that doesn\u0026rsquo;t match. The router rule never even runs. This dies during the handshake, which is why staring at routing config gets you nowhere.\nThe fix: tell the dialer what name to present.\nserversTransports: hermes-tls: serverName: vault.hermes.zt.example.dev services: vault-backend: loadBalancer: serversTransport: hermes-tls servers: - url: https://10.99.0.10 And the companion discipline: when a backend route skips auth because \u0026ldquo;only the edge can reach it,\u0026rdquo; make that true with config, not vibes. An ipAllowList middleware on the backend router, pinned to url\u0026rsquo;s overlay IP. Topology is a fact about today; an allow-list is a guarantee about tomorrow.\nCrowdsec: the bouncer Within hours of your DNS records existing, the scanners arrive: /wp-login.php, /.env, /.git/config (rude), exploit probes for appliances you\u0026rsquo;ve never owned. Background radiation. Mostly harmless, entirely tireless.\nCrowdsec tails the edge\u0026rsquo;s access logs, matches them against community-maintained attack scenarios, and issues decisions; a bouncer plugin in Traefik enforces them, refusing decisioned IPs before they reach any router. Setup is the agent, the bouncer, and an API key between them, about half an hour. The community blocklist means known-bad IPs get bounced on their first request to your server, paid for by their behavior on someone else\u0026rsquo;s.\nIt runs on url only. Internal proxies see traffic that already passed the edge or originated inside the overlay; a bouncer there is moving parts with nothing to bounce.\nCheck it\u0026rsquo;s working after a day:\ndocker exec crowdsec cscli decisions list | head If that list isn\u0026rsquo;t filling with strangers probing for WordPress, your DNS records haven\u0026rsquo;t propagated yet.\nNot everything is HTTP The edge also forwards raw TCP. In this lab that\u0026rsquo;s mail, via Traefik\u0026rsquo;s TCP routers with SNI passthrough:\ntcp: routers: imaps: rule: HostSNI(`mail.example.dev`) entryPoints: [imaps] service: imaps-backend tls: {passthrough: true} # the mail server does its own TLS services: imaps-backend: loadBalancer: servers: - address: \u0026#34;10.99.0.10:993\u0026#34; passthrough: true is the important line: the edge reads the SNI to route and forwards the still-encrypted stream. The mail server terminates its own TLS, and url never holds mail plaintext or mail keys. Post 14 leans on this hard.\nMilestone: the internet now reaches exactly one machine, every exposed service is listed in one reviewable file, and the scanners are somebody else\u0026rsquo;s log line.\nNext: one login for everything.\n","permalink":"https://hi.khuo.ng/posts/2026-03-28-traefik-edge/","summary":"A cheap VPS fronts everything: TLS, auth at the door, Crowdsec bouncing the scanners, then proxying over the overlay to services whose disks it can\u0026rsquo;t even see. Plus the TLS error whose message will not help you.","title":"Traefik part 2: the public edge"},{"content":"Somewhere around service fifteen, hand-maintained nginx configs stop being configuration and become a part-time job. I did the nginx years: include files, renewal cron jobs, the ritual nginx -t, the one vhost nobody remembered the purpose of. Traefik\u0026rsquo;s pitch is that the proxy should configure itself from what\u0026rsquo;s actually running, and it delivers: a service declares its routing as labels on its own container, the proxy watches Docker, and https://app.hermes.zt.example.dev exists with a valid certificate the moment the container starts.\nOne proxy per host, defined once Every host runs its own Traefik, but it\u0026rsquo;s the same Traefik: one definition in compose/_common/traefik-internal.yml, included by each host\u0026rsquo;s infra stack (the manually-deployed layer, post 3). Host differences ride the variables from post 2: ${INTERNAL_DOMAIN} is hermes.zt.example.dev on hermes and bender.zt.example.dev on bender. One file to upgrade, three boxes that can\u0026rsquo;t drift.\nThe static config, trimmed to the decisions that matter:\nproviders: docker: endpoint: \u0026#34;tcp://socket-proxy:2375\u0026#34; # not the raw socket, see below exposedByDefault: false # opt-IN routing file: directory: /dynamic # middlewares + shared routes watch: true entryPoints: web: address: \u0026#34;:80\u0026#34; http: redirections: entryPoint: {to: websecure, scheme: https} websecure: address: \u0026#34;:443\u0026#34; http: tls: certResolver: dns domains: - main: \u0026#34;*.hermes.zt.example.dev\u0026#34; certificatesResolvers: dns: acme: email: certs@example.dev storage: /acme/acme.json dnsChallenge: provider: cloudflare # reads the scoped INTERNAL token from env exposedByDefault: false is non-negotiable. Containers get routes only by asking (traefik.enable=true). The default-on alternative publishes every half-finished experiment you docker run at 1 a.m., and you discover which ones when the SSO middleware isn\u0026rsquo;t on them.\nAbout that socket-proxy endpoint: Traefik needs Docker API access to read labels, and the raw socket is root-equivalent on the host. A small filtering proxy that exposes read-only container info (and nothing like /containers/create) turns \u0026ldquo;proxy compromised\u0026rdquo; from a host takeover into an information leak. One extra container, much smaller blast radius.\nWildcard certs with zero open ports Internal hosts can\u0026rsquo;t answer Let\u0026rsquo;s Encrypt\u0026rsquo;s HTTP-01 challenge. There\u0026rsquo;s no path from the internet to them, which is the entire design. DNS-01 proves domain control by writing a TXT record through the provider\u0026rsquo;s API instead, which means real, browser-trusted wildcards on machines with no inbound exposure at all. No self-signed warnings to train your household to click through (a habit they will happily apply to actual attacks someday), no internal CA to distribute to every device.\nThe entrypoint-level domains: block requests *.hermes.zt.example.dev once and serves every service on the host from it. No per-router issuance, no rate-limit roulette when you deploy eight services in a day.\nCheck it worked:\ndocker logs traefik 2\u0026gt;\u0026amp;1 | grep -i \u0026#39;certificate obtained\u0026#39; curl -vI https://anything.hermes.zt.example.dev 2\u0026gt;\u0026amp;1 | grep \u0026#39;issuer\u0026#39; A service declares itself services: app: image: ... networks: [proxy] labels: - traefik.enable=true - traefik.http.routers.app.rule=Host(`app.${INTERNAL_DOMAIN}`) - traefik.http.routers.app.entrypoints=websecure - traefik.http.routers.app.tls=true - traefik.http.routers.app.middlewares=sso@file - traefik.http.services.app.loadbalancer.server.port=8080 Five labels: route, port, auth. That sso@file reference is post 8\u0026rsquo;s payoff; the middleware is defined once in the dynamic directory, and protecting any app forever is that one label. This is also the argument for a proxy per host instead of one central proxy: the labels live with the service, the proxy lives with the containers, and nothing needs to know what\u0026rsquo;s running on another box.\nThree footguns, ranked by hours lost Footgun 1: Traefik runs Go templating on the ENTIRE dynamic file, comments included. The file provider renders {{ }} before parsing YAML, on every byte. I once documented a middleware with a commented-out example containing {{ env \u0026quot;X\u0026quot; }}. The template engine executed my comment, failed, and took down every middleware and route in that file, SSO included. Every protected app on the host returned 500, and the log blamed a \u0026ldquo;template\u0026rdquo; I didn\u0026rsquo;t think of as code.\nRule. Never put {{ }} inside a dynamic-file comment. Write \u0026ldquo;env X\u0026rdquo; in prose. And validate the honest way, render then parse:\npython3 -c \u0026#34; import re, yaml raw = open(\u0026#39;traefik/dynamic/middlewares.yml\u0026#39;).read() yaml.safe_load(re.sub(r\u0026#39;{{[^}]*}}\u0026#39;, \u0026#39;DUMMY\u0026#39;, raw)); print(\u0026#39;ok\u0026#39;)\u0026#34; Footgun 2: default router priority is the rule\u0026rsquo;s character length. Not declaration order. Not specificity. String length. The day two routers can match one request (a shared host with a path carve-out, say), the longer rule text wins, silently, and requests arrive at the wrong backend with no error anywhere. A sprawling Host(a) || Host(b) || Host(c) beats your surgical Host(a) \u0026amp;\u0026amp; PathPrefix(/api) because it has more characters. Set explicit priority= on every overlapping router. I treat a missing priority on overlapping rules as a review-blocking bug.\nFootgun 3: mount the dynamic DIRECTORY, not individual files. A single-file bind mount pins an inode. Editors and git pull replace files (write-temp, rename-over), so the container keeps watching the old inode while you edit the new file. No error, no reload, your changes just don\u0026rsquo;t exist. Mount the directory and the watcher sees replacements. Cheap lesson here, half an hour of confusion otherwise.\nMilestone: every internal service on every host now gets HTTPS, a real certificate, and optional SSO, for the price of five labels.\nNext: the other Traefik, the one the internet actually talks to.\n","permalink":"https://hi.khuo.ng/posts/2026-03-14-traefik-internal/","summary":"One Traefik per host, wildcard certs via DNS-01 with zero open ports, and routing declared as labels on the containers themselves. Plus three footguns, ranked by how many hours each took from me.","title":"Traefik part 1: proxies inside the walls"},{"content":"The lab spans three sites: site A (local, where hermes lives), site B (remote, bender\u0026rsquo;s home), and site C (the VPS). The edge needs to reach internal services, backups cross sites nightly, and I want to administer everything from wherever my laptop happens to be.\nThe traditional answer is site-to-site WireGuard. I\u0026rsquo;ve built that before. It works, and it also means hand-managing keys, endpoints, routing tables, and firewall holes at three locations, two of which sit behind ISP routers I\u0026rsquo;d rather not depend on. The overlay-network answer (ZeroTier here; Tailscale and Nebula occupy the same niche) is: every machine joins a virtual network and gets a stable private IP that works no matter where it sits or how many NATs it\u0026rsquo;s behind. A node joins, you authorize it, it has an IP. The first time the VPS pinged a VM at site A with zero router configuration, it felt like cheating. It is. Cheat.\nWhat the lab does with it Three patterns that earlier posts have been quietly promising:\nThe edge reaches inward. Post 7\u0026rsquo;s url proxies http://10.99.0.10:... and it works, even though the target is a VM behind CGNAT at site A. DNS can tell the truth. Post 2\u0026rsquo;s internal wildcards (*.hermes.zt.example.dev to 10.99.0.10) resolve to addresses that work from any enrolled device, anywhere. Binding to the overlay IS the firewall. Services that should never face the internet bind to the overlay IP and nothing else. The mail server\u0026rsquo;s IMAP port exists at 10.99.0.10:993 and at no other address. Not blocked. Absent. Deploy agents, database ports, and url\u0026rsquo;s own SSH get the same treatment. Step 1: plan the addresses like you mean it It\u0026rsquo;s tempting to let the controller auto-assign. Resist. You will type these IPs into proxy backends, DNS records, and firewall rules for years, so make them carry information:\n10.99.0.10-19 site A (local) hermes, the hypervisor 10.99.0.20-29 site B (remote) bender, nibbler, professor 10.99.0.30-39 site C (VPS) url 10.99.0.40-49 bare metal the machines the VMs live on Months later you\u0026rsquo;ll read 10.99.0.2x in a log and know it\u0026rsquo;s site B without looking anything up. Thirty seconds of ceremony per node, repaid every debugging session after.\nStep 2: enroll curl -s https://install.zerotier.com | sudo bash sudo zerotier-cli join \u0026lt;your-network-id\u0026gt; sudo zerotier-cli listnetworks # wait for OK and your assigned IP Then in the controller: authorize the node, assign its planned IP, and name it immediately. An unnamed node ID six months from now is a security review with extra steps. The matching habit: deauthorize retired hardware the day it retires. The network is exactly as private as its membership list.\nCheck it worked, from any other member:\nping 10.99.0.10 The pitfall: two nodes behind one NAT This one\u0026rsquo;s worth the price of the whole post. I ran ZeroTier on the site A router and on a VM behind it. Both spoke from UDP 9993, the default. From the internet\u0026rsquo;s perspective, one public IP was making two contradictory claims about who owned port 9993. The NAT mapping flapped, and ZeroTier did what it\u0026rsquo;s designed to do under hostile conditions: it degraded instead of failing. One node lost its direct paths and fell back to relaying through external infrastructure.\nNothing went down. Everything from that node just got slow. Ten times the latency, single-digit throughput, intermittently. You will blame the ISP, the switch, and at least one innocent cable before you find it.\nThe diagnostic:\nsudo zerotier-cli peers # the tell: a nearby peer showing 180ms with path type RELAY The fix, one line on one of the colliding nodes:\necho \u0026#39;{\u0026#34;settings\u0026#34;:{\u0026#34;primaryPort\u0026#34;:9994}}\u0026#39; | sudo tee /var/lib/zerotier-one/local.conf sudo systemctl restart zerotier-one Rule of thumb. Every ZeroTier node sharing a NAT needs its own primary port. Router plus a VM, two VMs on one host, doesn\u0026rsquo;t matter. And after any topology change, watch zerotier-cli peers for thirty seconds: DIRECT good, RELAY bad. Relayed paths are a silent tax no dashboard will flag.\nWhere this leaves us Every machine now shares a flat encrypted /24 that follows it across buildings. The architecture\u0026rsquo;s spine is in place: DNS points at overlay IPs (post 2), the proxies ride them (posts 6 and 7), the deploy plane binds to them (post 9), and the public internet sees exactly one machine.\nNext: Traefik on every host. Wildcard certificates with zero open ports, routing as container labels, and three footguns ranked by hours lost.\n","permalink":"https://hi.khuo.ng/posts/2026-02-28-zerotier-overlay/","summary":"One flat 10.99.0.0/24 spanning site A, site B, and the VPS. No port forwards, no router surgery, and the two-nodes-one-NAT collision that silently makes everything slow.","title":"ZeroTier: a private /24 across three sites"},{"content":"During the migration I searched the old setup for anything that looked like a credential. The highlights: an API key in a compose file comment (\u0026ldquo;temporary\u0026rdquo;), a database password in a shell script I\u0026rsquo;d once shared in a gist, and the prize, a .env committed to a repo years ago because the .gitignore entry had a typo. *.env versus .env. Four characters, years of exposure.\nThe lesson is not \u0026ldquo;be more careful.\u0026rdquo; A long time in IT has convinced me that careful doesn\u0026rsquo;t scale. The lesson is that plaintext secrets need to be structurally impossible, which is what this post sets up: secrets that live in the repo, encrypted, where committing them is correct instead of catastrophic.\nWhy SOPS + age SOPS encrypts values, not files. An encrypted env file still reads like an env file:\n$ git diff secrets/hermes.env +VAULT_DB_PASSWORD=ENC[AES256_GCM,data:Tr7nQ...,type:str] Diffs stay reviewable. You can see that a commit added a vault password without seeing the password. Compare git-crypt, where every encrypted diff is binary soup, or an external vault server, where the repo doesn\u0026rsquo;t even know the secret exists.\nage is the keying layer that doesn\u0026rsquo;t fight back: a key is one line in one file. No keyservers, no expiry ceremonies, no GPG agent moods. If you\u0026rsquo;ve spent years in intermittent GPG combat, this part feels like a vacation.\nThe rejected alternatives, briefly: HashiCorp Vault is a high-availability service that must be running before anything else can boot, which points the dependency arrow the wrong way for a three-box lab. And plain .env plus .gitignore: see paragraph one.\nStep 1: keys, one per machine Generate on each machine. Never copy a private key between machines.\n# your workstation age-keygen -o ~/.config/sops/age/keys.txt # each server, during bootstrap (as root) age-keygen -o /etc/sops/age/key.txt Step 2: the recipient matrix .sops.yaml at the repo root says which keys can decrypt which files:\ncreation_rules: - path_regex: secrets/shared\\.env$ # domains, cert email, DNS tokens key_groups: - age: [\u0026lt;workstation\u0026gt;, \u0026lt;hermes\u0026gt;, \u0026lt;bender\u0026gt;, \u0026lt;url\u0026gt;] - path_regex: secrets/hermes\\.env$ # hermes-only credentials key_groups: - age: [\u0026lt;workstation\u0026gt;, \u0026lt;hermes\u0026gt;] - path_regex: secrets/bender\\.env$ key_groups: - age: [\u0026lt;workstation\u0026gt;, \u0026lt;bender\u0026gt;] The scoping principle: a host decrypts only what it runs. When bender is eventually compromised (assume eventually), the attacker reads media credentials. Not the SSO database password. Clone the repo without a listed key and you have variable names and ciphertext.\nStep 3: the key that cannot die Notice the workstation key is in every rule. It reads everything, which means:\nWarning. If your workstation dies and its age key existed nowhere else, every secret in the repo becomes unrecoverable ciphertext. The hosts keep running (they decrypt their own slices), but you can never again edit a secret or add a host.\nSo the minute you generate it, the key goes in your password manager. And watch for the recursion trap: if your password manager is self-hosted, in the lab, backed up by the lab, then the key that unlocks the lab lives inside the thing it unlocks. The key (and the backup credentials, post 12) must exist somewhere that survives the lab and doesn\u0026rsquo;t depend on it. Mine: the password manager, plus a printed copy in a drawer at site B. Paper doesn\u0026rsquo;t get ransomwared.\nEnroll a second machine early, too. Add its key to .sops.yaml, run sops updatekeys, and workstation death becomes an inconvenience.\nThe daily verbs sops secrets/hermes.env # edit in $EDITOR, re-encrypts on save sops set secrets/hermes.env \u0026#39;[\u0026#34;NEW_VAR\u0026#34;]\u0026#39; \u0026#39;\u0026#34;value\u0026#34;\u0026#39; # one value, scriptable sops decrypt --extract \u0026#39;[\u0026#34;VAR\u0026#34;]\u0026#39; secrets/hermes.env # read one value sops updatekeys -y secrets/*.env # after recipient changes Two habits turn tooling into a system.\nHabit 1, the diff check. After any secrets edit, before committing:\ngit diff secrets/ | grep \u0026#39;^\\+\u0026#39; | grep -v \u0026#39;ENC\\[\u0026#39; If anything prints beyond sops metadata, you\u0026rsquo;re about to commit plaintext: you edited without encrypting, or added a var outside sops. This one-liner has caught every near-miss I\u0026rsquo;ve had, including the day muscle memory typed vim instead of sops.\nHabit 2, the example file. Every encrypted env gets a committed plaintext .env.example twin: variable names, a comment per variable saying what it\u0026rsquo;s for and how to generate it, no values. The encrypted file is the data. The example file is the documentation, and after a rebuild it\u0026rsquo;s the file you\u0026rsquo;ll be gladdest to have.\nHow secrets reach containers At deploy time (post 9 has the pipeline), each host decrypts its slice to a path outside the repo checkout, and stacks consume the result as env files:\nsops decrypt secrets/shared.env \u0026gt; ~/secrets/shared.env sops decrypt secrets/hermes.env \u0026gt; ~/secrets/hermes.env # compose interpolates ${VAULT_DB_PASSWORD} normally Plaintext exists only on the host that needs it, regenerated every deploy, never inside the checkout where a stray git add -A could find it. The four-character typo from the top of this post is now a class of incident this system cannot express. That\u0026rsquo;s the only kind of careful that has ever worked for me.\nNext: the network that makes three sites act like one LAN, and the NAT collision worth knowing about before it finds you.\n","permalink":"https://hi.khuo.ng/posts/2026-02-14-secrets-in-git/","summary":"Encrypted .env files committed beside the compose files that use them: per-machine keys, per-file recipient rules, the one backup that cannot live inside the lab, and the git-diff habit that catches every near-miss.","title":"Secrets in git: SOPS + age"},{"content":"Post 1 told you about the 1,800-line compose file I was afraid to touch and the Portainer install whose on-disk config had been fiction for years. This post is the structure that replaced them. The layout is just the reasoning written down as directories, so I\u0026rsquo;ll give you both.\nThe prime directive, restated because everything follows from it: the repo is the only source of truth. Servers receive deploys. Nobody edits a file on a server. The moment you \u0026ldquo;quickly fix\u0026rdquo; something over SSH, your repo is a lie, the next deploy silently reverts the fix, and you get to debug the same outage twice. Ask me how I know.\nThe tree compose/ _common/ # fragments shared by every host traefik-internal.yml # the internal proxy, defined ONCE traefik-public.yml # the edge variant sso-outpost.yml # auth outpost, same everywhere hermes/ compose.yml # app services (deployed by automation) compose-infra.yml # infra services (deployed manually) services/ # one file per service vault.yml homeassistant.yml bender/ # same shape url/ # same shape traefik/dynamic/ # proxy middlewares + shared routes secrets/ # encrypted .env files (post 4) scripts/ # bootstrap, deploy, backup docs/ # runbooks. write them. future-you forgets. Three hosts, identical shape. Once your hands learn it on one box they know it on all of them.\nThe split that matters: infra vs apps Every host runs two compose projects, and this boundary is the most load-bearing decision in the repo:\ncompose-infra.yml: the reverse proxy, the SSO outpost, the deploy agent. Deployed manually, by a dumb script over SSH. compose.yml: everything else. Deployed automatically on git push (post 9). Why not automate everything? Because the deployer cannot manage itself. If the automation redeploys the stack that contains the automation, one bad push kills the deployer mid-deploy, and now you have no deployer, a half-applied change, and a recovery that starts with SSH\u0026rsquo;ing in like an animal. The same logic shields the proxy and the SSO: they\u0026rsquo;re what you need working in order to reach and fix everything else. The blast radius of \u0026ldquo;automation gone wrong\u0026rdquo; must end at the app layer.\nOne service, one file compose.yml contains no services. It\u0026rsquo;s an include manifest:\nname: hermes include: - path: services/vault.yml - path: services/homeassistant.yml - path: services/uptime-kuma.yml Each service lives alone under services/. The payoffs are immediate: adding a service is one file plus one include line, reviewed as one commit (feat(hermes): add vault). Removing one is a deletion. git log services/vault.yml is that service\u0026rsquo;s complete history. And you never again scroll two thousand lines hunting for the right environment: block.\nThe four service patterns Every service file copies the nearest of four shapes. This is what makes service number sixty as easy as service number six: you\u0026rsquo;re not designing, you\u0026rsquo;re instantiating.\nPattern 1, internal-only. The default. Proxy labels, SSO middleware, nothing public:\nservices: app: image: ... restart: unless-stopped networks: [proxy] volumes: - ${USERDIR}/appdata/app:/config labels: - traefik.enable=true - traefik.http.routers.app.rule=Host(`app.${INTERNAL_DOMAIN}`) - traefik.http.routers.app.entrypoints=websecure - traefik.http.routers.app.tls=true - traefik.http.routers.app.middlewares=sso@file - traefik.http.services.app.loadbalancer.server.port=8080 networks: proxy: external: true Pattern 2, public-facing. The same file, plus a route in the edge\u0026rsquo;s inventory (post 7). Both routes, always. A public service with no internal route can\u0026rsquo;t be reached while your WAN is down, and an internal label with a forgotten edge route is how \u0026ldquo;I swear I deployed that\u0026rdquo; happens.\nPattern 3, VPN-shared. network_mode: service:gluetun, labels on the network owner. Post 10 covers it in full.\nPattern 4, stateful-with-db. The app plus its own postgres or redis as siblings in the same file, with healthchecks so startup order is deterministic:\napp-db: image: postgres:16-alpine healthcheck: test: [\u0026#34;CMD-SHELL\u0026#34;, \u0026#34;pg_isready -U app -d app\u0026#34;] interval: 30s app: depends_on: app-db: condition: service_healthy Sibling, not shared: one database container per app that needs one. A communal postgres is a single point of failure with a guest list, and \u0026ldquo;upgrade postgres\u0026rdquo; becomes a negotiation among every tenant. Disk is cheaper than coordination.\nConventions that do the remembering Each of these is boring. Each is also a debugging session you\u0026rsquo;ll never have:\nContainers: lowercase, hyphenated, matching the service name. No host prefixes; the directory already says which box. Volumes: bind mounts into ~/appdata/\u0026lt;service\u0026gt;/ for anything stateful. Named volumes hide data inside Docker\u0026rsquo;s store where your backup job (post 12) can\u0026rsquo;t see it honestly. Anonymous volumes orphan data on recreate. ls ~/appdata should BE your state inventory. Env vars: SCREAMING_SNAKE_CASE, prefixed with the service name. SONARR_API_KEY, not API_KEY. The day two apps want the same unprefixed name in a shared env file, one loses silently. Image tags: stateful services pin a major version (postgres:16-alpine, traefik:v3.3). A surprise database major bump is a corruption story. :latest is fine for the *arrs and friends, whose upgrade culture is genuinely non-breaking. Check your work without deploying Compose validates without a daemon:\ndocker compose -f compose/hermes/compose.yml config --quiet \u0026amp;\u0026amp; echo ok Typos, broken includes, undefined variables: caught at the keyboard instead of as a failed deploy. Run it before every compose commit until your fingers do it for you.\nMilestone reached: you have a repository that can describe every service on every host, and a layout that won\u0026rsquo;t fight you at scale.\nNext: the part everyone gets wrong. Secrets that live IN the repo, encrypted, instead of beside it, leakable.\n","permalink":"https://hi.khuo.ng/posts/2026-02-01-the-gitops-repo/","summary":"A repo layout that scales past 60 services without becoming a junk drawer: the infra/app split, four reusable service-file patterns, and naming conventions that do the remembering for you.","title":"The GitOps repo: compose, one file per service"},{"content":"Before a single container runs, get the naming layer right. Everything in this series hangs off DNS, and the two habits in this post (variables instead of literals, scoped tokens) are the ones you\u0026rsquo;d least enjoy retrofitting. I\u0026rsquo;ve renamed this lab\u0026rsquo;s domains twice. The first rename took a month of whack-a-mole. The second took an afternoon, because of these rules. Let me save you the month.\nWhat you\u0026rsquo;ll need One domain you consider \u0026ldquo;canonical\u0026rdquo; (your main personal domain) Optionally a couple more for public-facing and media services A DNS provider with an API and scoped tokens (I use Cloudflare; anything comparable works) Step 1: give each domain a job Role Example What lives there Canonical example.dev SSO, vault, admin tools, mail Public example.cool things you share with friends Media example.tv the request portal Internal *.\u0026lt;host\u0026gt;.zt.example.dev every service, on every host The canonical choice matters most. Your SSO will live there, and forward-auth session cookies are scoped to a registrable domain: a session on auth.example.dev covers *.example.dev and can never cover example.cool. Keep everything sensitive under the canonical roof and the SSO story (post 8) stays one cookie simple. Scatter admin tools across domains and you\u0026rsquo;ll log in all day, or worse, \u0026ldquo;temporarily\u0026rdquo; skip auth on the strays.\nStep 2: never type a domain into a config file Every compose file and proxy label references variables, never strings:\nlabels: - traefik.http.routers.vault.rule=Host(`vault.${INTERNAL_DOMAIN}`) # never: Host(`vault.hermes.zt.example.dev`) The values live in one secrets file per host (post 4):\nPRIMARY_DOMAIN=example.dev MEDIA_DOMAIN=example.tv INTERNAL_DOMAIN=hermes.zt.example.dev # per host This looks like pedantry right up until you change domains. My afternoon rename was: swap the variable values, redeploy, fix the handful of apps that store their own URL internally. The month-long rename was a find-and-replace across a hundred files on three hosts, with regressions surfacing for weeks. Adopt the variable rule on day one. It costs nothing.\nStep 3: two wildcards, then stop touching DNS Public records. The apex and a wildcard, both pointing at the VPS:\nexample.dev. A 203.0.113.10 ; url, the edge VPS *.example.dev. CNAME example.dev. That wildcard means every future public service is just a new route on the proxy. I have not created a per-service public DNS record in years.\nWarning: the apex points at the VPS. Always. At some point you\u0026rsquo;ll be tempted to point one busy hostname straight at your home connection to skip the hop. Don\u0026rsquo;t. Home IPs change, get CGNAT\u0026rsquo;d, and when your home line drops it takes that name down with it. The performance itch gets scratched properly in post 13, with smarter DNS instead of riskier records.\nInternal records. One wildcard per host, answering with overlay IPs (the overlay network is post 5; for now read it as \u0026ldquo;private IPs that work from anywhere\u0026rdquo;):\n*.hermes.zt.example.dev. A 10.99.0.10 *.bender.zt.example.dev. A 10.99.0.20 *.url.zt.example.dev. A 10.99.0.30 Yes, these are public DNS records with private answers, and that\u0026rsquo;s fine. RFC1918 addresses tell an attacker nothing useful without overlay membership, and in exchange every laptop and phone gets correct answers with no internal DNS server to depend on. We add local DNS much later (post 13) as an optimization. Bootstrapping your lab on DNS that lives inside the lab is a circular dependency you\u0026rsquo;d meet at the worst time.\nCheck it worked:\ndig +short vault.example.dev # the VPS address dig +short app.hermes.zt.example.dev # 10.99.0.10 Step 4: scoped tokens, one per consumer Several things will need DNS API access: the proxies do DNS-01 challenges for wildcard certificates (post 6), and a ddns updater tracks site B\u0026rsquo;s drifting IP. The lazy move is one all-powerful API key pasted everywhere. I did that for years and nothing bad happened, which taught me nothing. Domains are the root of trust for email, certificates, and OAuth callbacks; a leaked global key means someone can re-point your MX and read your password resets.\nMint one token per consumer class:\nToken Held by Can touch INTERNAL proxies on internal hosts DNS edit, ONE low-value zone EDGE url\u0026rsquo;s proxy DNS edit, the public zones DDNS the ddns updater DNS edit, its one zone The reasoning: internal hosts are numerous, and the most likely place for a token to leak into a backup, a log, or a compromised container. So their token is the least useful one to steal. The edge genuinely needs broader cert access, but it\u0026rsquo;s one hardened machine and its token revokes without touching anything else.\nWhen a token does leak (mine went into a paste once, attached to a debug log), revocation is one click and one consumer to re-credential. Not a rotation across everything you own.\nProxied vs DNS-only Everything above is DNS-only (grey cloud, in Cloudflare terms). The edge terminates its own TLS, Crowdsec handles the bouncing (post 7), and half the lab\u0026rsquo;s traffic isn\u0026rsquo;t HTTP anyway: mail and SSH don\u0026rsquo;t survive an HTTP CDN proxy. There\u0026rsquo;s a fair case for proxying public HTTP services (origin hiding, DDoS absorption). My advice: start plain, where dig tells the truth while you debug, and add proxying when you have the problem it solves.\nThat\u0026rsquo;s the naming layer done. Two wildcards, four variables, three tokens, and you shouldn\u0026rsquo;t need this provider\u0026rsquo;s dashboard again for months.\nNext: the repository all of this lives in, and the infra/app split that protects you from your own automation.\n","permalink":"https://hi.khuo.ng/posts/2026-01-18-domains-and-dns/","summary":"Domain-per-purpose, two wildcard records that mean you never touch DNS again, and scoped API tokens so no leaked credential can take the whole thing down.","title":"Domains \u0026 DNS: one canonical domain, many purposes"},{"content":"Good news, everyone! I rebuilt the entire homelab, and this time I wrote it down.\nI\u0026rsquo;ve been doing this a long time. It started the way it starts for most people in IT: a spare tower under a desk sharing files, then a succession of \u0026ldquo;temporary\u0026rdquo; machines that each outlived whatever they were temporary for. Eventually it became a proper homelab, which is to say: a place where I run things the way I think they should be run, with no change-advisory board except my own regret.\nThis series documents the latest rebuild, in the order you\u0026rsquo;d actually do it. Not because the hardware died. Because the way it was organized died.\nHow it got bad See if any of this sounds familiar.\nBy last year I had three servers, each running a pile of Docker containers, and each pile had grown differently. One box had everything in a single 1,800-line docker-compose.yml that I was honestly afraid to touch: change one service, recreate them all, hold your breath. Another was managed through Portainer, which meant its real configuration lived in Portainer\u0026rsquo;s database; the compose files on disk had drifted from reality years earlier and nobody told them. The third had containers started by hand with docker run flags that existed only in shell history, on a host I\u0026rsquo;d since reinstalled.\nEach server ran its own reverse proxy, configured its own way. Two had their own identity provider, because I set up SSO twice and never finished consolidating. Secrets were in .env files, some gitignored, some (I found out during the migration) not. Backups covered whatever had scared me at some point, which is not the same list as what mattered.\nNone of it was broken, exactly. It all ran. But every change had become archaeology. Which box is that on? Is the file on disk what\u0026rsquo;s actually running? Where did I put that API key? When a disk started throwing SMART errors, I had to admit the disaster-recovery plan for two of the three machines was \u0026ldquo;reconstruct from memory.\u0026rdquo;\nThe lesson, after a long time of not learning it: infrastructure you can\u0026rsquo;t rebuild from a file is a liability wearing a hobby\u0026rsquo;s clothes.\nThe rewrite So: tear it down. Not the data, the organization. Every service re-declared from scratch, in one git repository, with rules. The rules are the real product of this series; the specific apps are almost interchangeable.\n1. The repo is the only truth. Every service on every host is defined in one repository. Servers receive deploys from it. Nobody edits anything on a server, ever. If I\u0026rsquo;m SSH\u0026rsquo;d into a box editing a config file, something has already gone wrong. (Post 9 covers how deploys flow; the discipline is the point.)\n2. One identity provider. A single SSO instance guards everything, and protecting a new app costs one line. No more \u0026ldquo;I\u0026rsquo;ll add auth later.\u0026rdquo; Later never comes, and unauthenticated admin panels are how homelabs end up in somebody\u0026rsquo;s botnet writeup.\n3. Exactly one machine faces the internet, and it\u0026rsquo;s the one I care about least: a cheap VPS with a static IP, running a reverse proxy and nothing stateful. It reaches back to the other sites over an encrypted overlay network. My home IP appears in no DNS record anywhere. If the VPS ever gets popped, the rebuild is ten minutes and the attacker got a proxy and a bad mood.\n4. Secrets are encrypted in the repo, value by value, next to the compose files that use them. \u0026ldquo;Did that .env make it into git\u0026rdquo; is no longer a question, because the answer is yes, on purpose, safely.\n5. Pick boring on purpose. Observability is a dashboard and live logs, not a Prometheus deployment that becomes a second hobby. Backups are restic on a timer, restore-tested. Every component had to pass one test: will this still make sense at 2 a.m. in eighteen months?\nThe fleet Three machines across three sites. The fleet is named after Futurama characters, because if you can\u0026rsquo;t have fun naming servers, why are you even self-hosting:\ninternet | +--------v---------+ | url (site C) | the VPS edge | static IP | Traefik + SSO outpost + Crowdsec +--------+---------+ | ZeroTier overlay (10.99.0.0/24) +-------------+--------------+ +---------v----------+ +----------v---------+ | hermes (site A) | | site B | | SSO core, deploys | | bender: *arrs, | | HA, vault, mail | | downloads, reqs | +--------------------+ | nibbler: storage | | professor: Plex | +--------------------+ hermes, site A (local): the bureaucrat. Hermes Conrad files everything, and so does this box: the SSO core, the deploy orchestrator, Home Assistant, the password vault, the mail server. If it holds state I\u0026rsquo;d cry about, it lives here. bender, site B (remote): the download box. Library automation, the request portal, VPN\u0026rsquo;d download clients. Bender downloads things. It\u0026rsquo;s what he does. nibbler, site B: storage. A little box attached to a frankly alarming amount of disk, exactly like his namesake. Exports the big filesystem the rest of site B works against. professor, site B: Plex, with a repurposed Nvidia gaming GPU doing the transcodes. Good news, everyone, the old graphics card found work. url, site C (VPS): Officer URL, the robot cop. Works the door. Routing, auth enforcement, bouncing scanners. Three containers and no feelings. The split matters more than the hardware. Control plane, bulk data, and public exposure each get their own blast radius. Site B can drop off the map without taking down site A\u0026rsquo;s door-lock automations, and the VPS can be rebuilt without touching either site.\nHardware, for the curious: hermes is a VM with 8GB on a small hypervisor, bender is a repurposed desktop with a mid-range Intel CPU (the iGPU matters for Plex, see post 15) and a stack of big disks, and url is whichever provider\u0026rsquo;s bottom-tier VPS had a clean IP reputation that week. None of this needs serious iron. The organization is the expensive part, and it\u0026rsquo;s free.\nWhat the series covers In dependency order, the same order you\u0026rsquo;d rebuild from bare metal:\nThis post: the shape and the rules Domains and DNS: naming, wildcards, scoped API tokens The repo: layout, the infra/app split, conventions Secrets: SOPS + age, encrypted env in git ZeroTier: the three-site overlay Traefik, inside: per-host proxies, wildcard certs, labels Traefik, edge: the public front door, Crowdsec Authentik: one login for everything Komodo: git push to deploy The download pipeline: gluetun and kill-switch networking Observability: without the Prometheus tax Backups: restic, restore-tested, keys on paper Split-horizon DNS: same names, shorter path at home Mail: the boss fight Plus standalone deep dives where one app earns a full page (Plex and the *arr naming stack are posts 15 and 16).\nThe mistakes stay in. Every post has a pitfalls section and none of it is hypothetical. Each one cost me an evening at some point, and each comes with the symptom you\u0026rsquo;d actually see, because \u0026ldquo;works once you know the trick\u0026rdquo; is the true shape of this hobby.\nNext up: domains and DNS, including the API-token hygiene I wish someone had explained to me much earlier.\n","permalink":"https://hi.khuo.ng/posts/2026-01-06-homelab-from-scratch/","summary":"Years of accumulated server cruft, three boxes that each grew their own personality, and the decision to tear it down and rebuild as one git repo. The architecture, the rules, and why.","title":"A homelab from scratch: what we're building and why"}]