The lab spans three sites: site A (local, where hermes lives), site B (remote, bender’s home), and site C (the VPS). The edge needs to reach internal services, backups cross sites nightly, and I want to administer everything from wherever my laptop happens to be.

The traditional answer is site-to-site WireGuard. I’ve built that before. It works, and it also means hand-managing keys, endpoints, routing tables, and firewall holes at three locations, two of which sit behind ISP routers I’d rather not depend on. The overlay-network answer (ZeroTier here; Tailscale and Nebula occupy the same niche) is: every machine joins a virtual network and gets a stable private IP that works no matter where it sits or how many NATs it’s behind. A node joins, you authorize it, it has an IP. The first time the VPS pinged a VM at site A with zero router configuration, it felt like cheating. It is. Cheat.

What the lab does with it

Three patterns that earlier posts have been quietly promising:

  • The edge reaches inward. Post 7’s url proxies http://10.99.0.10:... and it works, even though the target is a VM behind CGNAT at site A.
  • DNS can tell the truth. Post 2’s internal wildcards (*.hermes.zt.example.dev to 10.99.0.10) resolve to addresses that work from any enrolled device, anywhere.
  • Binding to the overlay IS the firewall. Services that should never face the internet bind to the overlay IP and nothing else. The mail server’s IMAP port exists at 10.99.0.10:993 and at no other address. Not blocked. Absent. Deploy agents, database ports, and url’s own SSH get the same treatment.

Step 1: plan the addresses like you mean it

It’s tempting to let the controller auto-assign. Resist. You will type these IPs into proxy backends, DNS records, and firewall rules for years, so make them carry information:

10.99.0.10-19   site A (local)   hermes, the hypervisor
10.99.0.20-29   site B (remote)  bender, nibbler, professor
10.99.0.30-39   site C (VPS)     url
10.99.0.40-49   bare metal       the machines the VMs live on

Months later you’ll read 10.99.0.2x in a log and know it’s site B without looking anything up. Thirty seconds of ceremony per node, repaid every debugging session after.

Step 2: enroll

curl -s https://install.zerotier.com | sudo bash
sudo zerotier-cli join <your-network-id>
sudo zerotier-cli listnetworks    # wait for OK and your assigned IP

Then in the controller: authorize the node, assign its planned IP, and name it immediately. An unnamed node ID six months from now is a security review with extra steps. The matching habit: deauthorize retired hardware the day it retires. The network is exactly as private as its membership list.

Check it worked, from any other member:

ping 10.99.0.10

The pitfall: two nodes behind one NAT

This one’s worth the price of the whole post. I ran ZeroTier on the site A router and on a VM behind it. Both spoke from UDP 9993, the default. From the internet’s perspective, one public IP was making two contradictory claims about who owned port 9993. The NAT mapping flapped, and ZeroTier did what it’s designed to do under hostile conditions: it degraded instead of failing. One node lost its direct paths and fell back to relaying through external infrastructure.

Nothing went down. Everything from that node just got slow. Ten times the latency, single-digit throughput, intermittently. You will blame the ISP, the switch, and at least one innocent cable before you find it.

The diagnostic:

sudo zerotier-cli peers
# the tell: a nearby peer showing 180ms with path type RELAY

The fix, one line on one of the colliding nodes:

echo '{"settings":{"primaryPort":9994}}' | sudo tee /var/lib/zerotier-one/local.conf
sudo systemctl restart zerotier-one

Rule of thumb. Every ZeroTier node sharing a NAT needs its own primary port. Router plus a VM, two VMs on one host, doesn’t matter. And after any topology change, watch zerotier-cli peers for thirty seconds: DIRECT good, RELAY bad. Relayed paths are a silent tax no dashboard will flag.

Where this leaves us

Every machine now shares a flat encrypted /24 that follows it across buildings. The architecture’s spine is in place: DNS points at overlay IPs (post 2), the proxies ride them (posts 6 and 7), the deploy plane binds to them (post 9), and the public internet sees exactly one machine.

Next: Traefik on every host. Wildcard certificates with zero open ports, routing as container labels, and three footguns ranked by hours lost.