Everything so far deploys by hand. This post adds the missing verb: push. Edit a service file on the laptop, commit, push, and the right host pulls, decrypts its secrets, and recreates exactly the affected stack. This is where the repo stops being documentation that happens to be accurate and becomes the control plane.

Why Komodo

The candidates I’d actually lived with, and why they lost:

  • Portainer: the real config migrates into Portainer’s database and the compose files on disk become historical fiction. Post 1 told you how that ends.
  • Ansible: push-based and drift-prone between runs, and a pile of playbooks is its own codebase with its own bugs. I wanted less custom orchestration, not differently-shaped.
  • Watchtower: bumps image tags. Can’t add a service, change a label, or know what a stack is. Fine at its actual job, which isn’t this one.

Komodo matches the shape we already have: a core (UI plus state, on hermes, behind post 8’s SSO via OIDC) and a periphery agent on each host that executes deploys. Each host’s compose.yml registers as a “stack” pointing at the repo. A webhook on push means: pull, diff, redeploy what changed.

The agent is systemd, not a container, on purpose

Periphery runs as a native systemd service. That looks like a step backward from containerize-everything until you trace what a deploy needs: the pre-deploy hook decrypts SOPS secrets (post 4), which needs the host’s age key. A containerized agent would need that key mounted in, and now the key’s blast radius is “anything that can reach that container” instead of “root on the host.” We’d be un-scoping the thing post 4 carefully scoped.

# /etc/systemd/system/komodo-periphery.service (the idea)
[Service]
ExecStart=/usr/local/bin/periphery
Environment=PERIPHERY_ROOT_DIRECTORY=/home/lab/komodo
Environment=PERIPHERY_BIND_IP=10.99.0.20   # overlay only
Restart=always

That bind line is post 5’s pattern again: the API that can recreate any container on the box does not exist on any public interface.

Secrets at deploy time

The pre-deploy hook on each host:

sops decrypt secrets/shared.env > ~/secrets/shared.env
sops decrypt secrets/bender.env > ~/secrets/bender.env

The stack consumes those as extra env files. Plaintext lives outside the checkout, regenerated every deploy, scoped per host.

The flag that costs people an hour: mark those env files untracked (track=false in Komodo’s stack config). Tracked files get validated BEFORE the pre-deploy hook runs, so a fresh host fails with “env file missing” because the thing that creates the file hasn’t been allowed to run yet. The error message names neither the chicken nor the egg.

The automation must not manage itself

Post 3 drew the line: the infra stack (proxy, SSO outpost, the deploy agent itself) deploys manually; the app stack deploys automatically. Now you can see the line under load. If Komodo managed its own stack, a bad push would recreate the deployer mid-deploy. Nothing is deploying, the change is half-applied, and the recovery is SSH and hands. The deployer’s failure domain has to exclude the deployer, the proxy you reach it through, and the SSO you log into it with. Those deploy via a deliberately boring script, and boring is the feature.

Treat the webhook like the weather

Two operational truths, both paid for:

Webhooks die silently. A delivery fails upstream, a secret rotates, an endpoint moves, and pushes just stop deploying, with no error anywhere you look. Mine was dead for four days once while I kept “deploying” at the speed of nothing.

Change detection has blind spots. A push that only touches secrets shows no compose diff, so no stack redeploys. Your rotated password sits in git while the old one keeps serving traffic.

One answer covers both: deterministic scripts as the primary path, the webhook as a fast path. Two scripts live in the repo: a sync-hosts script (pull everywhere, redeploy infra if infra files changed, re-decrypt if secrets changed) and a force-redeploy for app stacks. After any push where the outcome matters, run the script, or at least verify the deploy landed (post 11 makes that a ten-second glance).

The honest framing of homelab GitOps: the repo is always RIGHT; whether it’s APPLIED is a question you must be able to answer, and force, with one command at 2 a.m.

The stale-clone corollary

Periphery executes hook scripts from the host’s clone of the repo, which updates when something pulls it, not magically on push. Change a deploy script, forget the host hasn’t pulled, and the host runs the old script while you stare at the new one. The sync script’s first act on every host is git pull for exactly this reason. Symptom to file away: “the fix I definitely pushed is definitely not what just ran.”

Milestone: a service edit now goes laptop to running container with one push, and the parts that could brick the pipeline are the parts the pipeline can’t touch.

Next: the download pipeline on bender, and the networking trick that turns compose syntax into a kill-switch.