Secrets in git: SOPS + age

During the migration I searched the old setup for anything that looked like a credential. The highlights: an API key in a compose file comment (“temporary”), a database password in a shell script I’d once shared in a gist, and the prize, a .env committed to a repo years ago because the .gitignore entry had a typo. *.env versus .env. Four characters, years of exposure.

The lesson is not “be more careful.” A long time in IT has convinced me that careful doesn’t scale. The lesson is that plaintext secrets need to be structurally impossible, which is what this post sets up: secrets that live in the repo, encrypted, where committing them is correct instead of catastrophic.

Why SOPS + age

SOPS encrypts values, not files. An encrypted env file still reads like an env file:

$ git diff secrets/hermes.env
+VAULT_DB_PASSWORD=ENC[AES256_GCM,data:Tr7nQ...,type:str]

Diffs stay reviewable. You can see that a commit added a vault password without seeing the password. Compare git-crypt, where every encrypted diff is binary soup, or an external vault server, where the repo doesn’t even know the secret exists.

age is the keying layer that doesn’t fight back: a key is one line in one file. No keyservers, no expiry ceremonies, no GPG agent moods. If you’ve spent years in intermittent GPG combat, this part feels like a vacation.

The rejected alternatives, briefly: HashiCorp Vault is a high-availability service that must be running before anything else can boot, which points the dependency arrow the wrong way for a three-box lab. And plain .env plus .gitignore: see paragraph one.

Step 1: keys, one per machine

Generate on each machine. Never copy a private key between machines.

# your workstation
age-keygen -o ~/.config/sops/age/keys.txt
# each server, during bootstrap (as root)
age-keygen -o /etc/sops/age/key.txt

Step 2: the recipient matrix

.sops.yaml at the repo root says which keys can decrypt which files:

creation_rules:
  - path_regex: secrets/shared\.env$       # domains, cert email, DNS tokens
    key_groups:
      - age: [<workstation>, <hermes>, <bender>, <url>]
  - path_regex: secrets/hermes\.env$       # hermes-only credentials
    key_groups:
      - age: [<workstation>, <hermes>]
  - path_regex: secrets/bender\.env$
    key_groups:
      - age: [<workstation>, <bender>]

The scoping principle: a host decrypts only what it runs. When bender is eventually compromised (assume eventually), the attacker reads media credentials. Not the SSO database password. Clone the repo without a listed key and you have variable names and ciphertext.

Step 3: the key that cannot die

Notice the workstation key is in every rule. It reads everything, which means:

Warning. If your workstation dies and its age key existed nowhere else, every secret in the repo becomes unrecoverable ciphertext. The hosts keep running (they decrypt their own slices), but you can never again edit a secret or add a host.

So the minute you generate it, the key goes in your password manager. And watch for the recursion trap: if your password manager is self-hosted, in the lab, backed up by the lab, then the key that unlocks the lab lives inside the thing it unlocks. The key (and the backup credentials, post 12) must exist somewhere that survives the lab and doesn’t depend on it. Mine: the password manager, plus a printed copy in a drawer at site B. Paper doesn’t get ransomwared.

Enroll a second machine early, too. Add its key to .sops.yaml, run sops updatekeys, and workstation death becomes an inconvenience.

The daily verbs

sops secrets/hermes.env                              # edit in $EDITOR, re-encrypts on save
sops set secrets/hermes.env '["NEW_VAR"]' '"value"'  # one value, scriptable
sops decrypt --extract '["VAR"]' secrets/hermes.env  # read one value
sops updatekeys -y secrets/*.env                     # after recipient changes

Two habits turn tooling into a system.

Habit 1, the diff check. After any secrets edit, before committing:

git diff secrets/ | grep '^\+' | grep -v 'ENC\['

If anything prints beyond sops metadata, you’re about to commit plaintext: you edited without encrypting, or added a var outside sops. This one-liner has caught every near-miss I’ve had, including the day muscle memory typed vim instead of sops.

Habit 2, the example file. Every encrypted env gets a committed plaintext .env.example twin: variable names, a comment per variable saying what it’s for and how to generate it, no values. The encrypted file is the data. The example file is the documentation, and after a rebuild it’s the file you’ll be gladdest to have.

How secrets reach containers

At deploy time (post 9 has the pipeline), each host decrypts its slice to a path outside the repo checkout, and stacks consume the result as env files:

sops decrypt secrets/shared.env > ~/secrets/shared.env
sops decrypt secrets/hermes.env > ~/secrets/hermes.env
# compose interpolates ${VAULT_DB_PASSWORD} normally

Plaintext exists only on the host that needs it, regenerated every deploy, never inside the checkout where a stray git add -A could find it. The four-character typo from the top of this post is now a class of incident this system cannot express. That’s the only kind of careful that has ever worked for me.

Next: the network that makes three sites act like one LAN, and the NAT collision worth knowing about before it finds you.