My homelab had CI. Eighteen lint workflows, one in every project, neatly tucked into each project’s own .github/workflows/.

Not one of them ever ran.

Not once. Not a failed run, not a skipped run. A CI runner only looks for workflows in the .github/workflows/ directory at the root of a repository, and mine were nested inside the projects. And for a while there wasn’t even a repository for them to be in.

That’s the most embarrassing thing I found while putting my infrastructure code in order this autumn. It’s also the one that changed how I think about it. Code you’ve never run isn’t infrastructure. It’s a hypothesis.

This is the story of four weeks spent testing those hypotheses.

What I’m actually running

Here’s the estate, as it stands:

  • Three Proxmox VE nodes, clustered, with Ceph across them. Twelve HDD OSDs, four per node, plus SSDs for the things that need to be fast.
  • Two direct-cabled meshes between the nodes, no switch. One carries Ceph, the other carries the cluster’s heartbeat traffic, and Ceph never touches the second one.
  • Two CephFS filesystems, plus NFS and S3 gateways that run on the nodes themselves behind floating IPs. There’s an SMB gateway too, fully built and switched off.
  • One Debian guest that runs the services the whole thing leans on: Gitea, its CI runner, OpenBao for secrets, Portainer and Uptime Kuma.
  • A firewall VM, a Tailscale subnet router and a control node, all guests on the same cluster. Remember those three. They come back.

It’s built with Ansible (cluster, storage, hosts), OpenTofu (VM templates and guests) and Docker Compose (the services). One operator. Me.

A folder pretending to be a platform

Until mid-September, “the infra” was a directory of eighteen separate projects. Fifteen of them were Ansible projects. Nine of those were slices of the same Proxmox cluster, each in its own little world, each with its own README explaining which of the other eight you had to run first.

It got worse. At some point I’d written a script called remove_git_dirs.sh. It found every .git directory under the tree and deleted it. I ran it. So for a while, none of it was under version control at all.

On September 19 I took a baseline snapshot and collapsed the lot into a single repository. Fifteen Ansible projects became four. The nine Proxmox ones became a single project, thirteen playbooks today, and the order they had to run in stopped being prose spread across nine READMEs. It became one site.yml that imports them in order.

Then I went looking for duplicated files. There were 54 of them: licences in two spellings, thirteen .gitignores, nine pre-commit configs, linter configs, CODEOWNERS files, and those eighteen workflow files. They collapsed into 11.

Some of those had never done anything, ever. The workflows I’ve told you about. The nine nested pre-commit configs, because pre-commit only reads the one at the top. And CODEOWNERS. GitHub only honours that file in three places: the root, .github/ or docs/. Mine sat under opentofu/**/.github/. And the path they so carefully protected, environments/production/, was deleted in that same cleanup. Guards, standing outside a building that wasn’t there, in a spot where no one could see them.

Funny in hindsight. At the time it was a slow, sinking feeling, because I’d trusted both.

The network that broke without an error

If the repo was quietly lying, the network was worse. It failed in ways that looked exactly like working.

The MTU reset. The two meshes are built with Proxmox’s SDN fabrics. It turns out every SDN apply regenerates the network config and resets the fabric interfaces to MTU 1500, whatever you’d pinned. If anything on top expects a bigger frame, full-size packets just vanish. No error. No log line. And on the next reboot it fixes itself, which is the cruellest possible behaviour for a bug. The playbook now re-asserts the MTU at the end of every SDN run and fails hard if it can’t.

The jumbo frames. I set the Ceph mesh to MTU 9050 and tested it the way every guide tells you to: ping -M do -s 8972, no fragmentation, clean replies. It passed. Then the mesh saw its first real sustained load, and one node’s Intel X520 cards started hanging and resetting. The cluster lost monitor quorum. An OSD got OOM-killed. The other two nodes ran identical cards at the same MTU without a hiccup, so it might be a marginal cable or SFP that jumbo frames merely exposed. I reverted on September 15 anyway.

What I wrote down afterwards: a ping test isn’t evidence. It passed right up until the moment it mattered. Test under the load you actually expect, or don’t bother. And at 10 GbE on a disk-bound cluster, jumbo frames were worth maybe 0 to 10%. Nowhere near worth what they cost me.

The clever idea that cost me quorum. On September 14 I moved Ceph’s public network onto one of the SDN virtual networks, so guests could reach the monitors directly. Seemed clever. But Proxmox doesn’t write a host address for a virtual network, so every SDN apply stripped the monitors’ own IPs off the bridge. I added four systemd units to put them back. At boot they lost the race. All three monitors stuck in probing. No quorum. The cluster filesystem went read-only.

I moved it back the next day and deleted the feature that made it possible. The rule since then is short: a virtual network is for guests. Anything a daemon binds to gets a real interface.

There was a sting in the tail, too. Changing that variable back did nothing to the live cluster. Ceph only reads its initial config when there isn’t one yet, and the monitors bind from their own map, not from the file. So the playbook now reads the live config first and refuses to run if it doesn’t match what the code says. A config change your running system will never pick up is a trap, so I made the trap loud.

First contact

On September 17, the storage gateways met the real cluster for the first time. They needed nine fixes. Four of them meant that no gateway could ever have served a single client.

  • The keys were in the wrong drawer. On a Proxmox node, Ceph’s config is a symlink into the cluster filesystem, and it tells every client to look for its keyring there. My role wrote the gateway keyrings somewhere else. Permissions were perfect. Nobody could find their key.
  • The gateway switched itself off every time it won. keepalived moves a floating IP between nodes and runs a script on each transition. I’d used notify_master, notify_backup and notify_fault. Those run the script with no extra arguments. Only the generic notify passes the new state. So my script saw an empty state, hit its “unknown, stop to be safe” branch, and stopped the NFS gateway. On every transition. Including the one where it became master. I’d written the safety net so carefully that it caught everything.
  • AppArmor killed NFS-Ganesha on start-up. Podman’s default profile on Debian 13 denied the Unix socket Ganesha needs, so it exited, the health check failed, and the floating IP bounced from node to node every thirty seconds or so. The fix was a profile that keeps the confinement and opens only that one socket class.
  • NFSv4.2 returned I/O errors on every read. The NFS gateway now speaks 4.1 only, until a newer Ganesha ships in the Ceph image.

The Docker side had the same lesson waiting. My hardened Compose template, the one every service was supposed to follow, had never been run before September 18. It showed. The Caddy container in the Tailscale variant had no network at all, so nothing could reach it. Its Caddyfiles used ${VAR}, which Caddy doesn’t expand. One TAG variable pinned three different images. And its shared .env.common was never read, because Compose only picks up a file named exactly .env for interpolation.

None of that needed a real deployment to find. Most of it shows up the moment you run docker compose config, which renders the file the way Compose actually reads it. That’s now in CI, over every Compose file in the repo, on every push.

Secrets, in tiers

Here’s a loop that kept me up one night: the estate needs credentials to build the thing that will later store the credentials.

The way out is tiers, where each one depends only on the ones above it.

TierWhatWhere
0age private keys, OpenBao recovery keysEach machine’s own key file; the backup key only in a password manager that isn’t hosted on this estate
1Ansible secrets, API tokens, state passphrasesSOPS-encrypted files, committed to git
2Runtime secretsOpenBao, on the services guest

Every encrypted file goes to three keys: my workstation, the control node, and a backup key that lives only in the password manager, never on a disk. Lose one machine, lose one recipient, nothing else.

One rule keeps the whole thing from eating its own tail: nothing needed to rebuild the cluster may exist only in OpenBao or my self-hosted Vaultwarden. Both of them run on the thing they’d be helping to recover.

OpenTofu reads its credentials from SOPS as ephemeral values, so they never land in state or in a plan file, and the state itself is encrypted with a key that also lives in SOPS.

And then there’s the lesson I’d rather have learned from someone else. An audit found a root password sitting in plaintext in one of the old files. I fixed the file. But fixing the file isn’t fixing the secret. The tempting move is to drop the value into SOPS and call it done, and that would be exactly wrong. Encryption preserves a leaked secret. It doesn’t fix it. Rotate first, then encrypt the new value. Never the old one.

Trading safety for space, on purpose

One decision I made with my eyes open: erasure coding.

The general-purpose filesystem used to keep three full copies of every file. That’s 33% storage efficiency, and it heals itself if a whole node dies. I moved its bulk data to an erasure-coded pool, 2 data chunks plus 1 parity, one chunk per node. That’s 66.7% efficiency, and usable space went from 500 GiB to 1,200 GiB on roughly the same disks. 2.4 times as much.

It isn’t free, and I won’t pretend it is. With three nodes there’s no fourth place to rebuild onto, so losing a node leaves that pool running with zero redundancy until I fix the hardware. It also needs min_size 2, or losing one node would block every read and write to it. I took that trade for this one filesystem and nowhere else.

Then the small files bit me. EC pads every chunk to 4 KiB, so a 4 KiB file costs 12 KiB on disk. That’s three times its size, the same as the three-copy replication I’d just walked away from. And I had a share that held nothing but kilobyte-sized JSON manifests. First they got their own replicated pool, as an exception. A day later they moved into an S3 bucket on the cluster’s own RGW, and the exception got deleted.

The list I didn’t want to write

Somewhere in all this, I made the docs a promise. The repo’s main README opens with a line saying everything in it was checked against the working tree, and that anything the code doesn’t do yet is called out instead of repeated. Every project document ends with a section called “Points for improvement”.

Keeping that promise meant writing a roadmap that starts with the uncomfortable stuff. Here’s the top of it.

Nothing is backed up. The old weekly backup job wrote to Ceph storage, on the same cluster it was protecting. I disabled it in favour of a Proxmox Backup Server VM. The code for that VM exists. The VM doesn’t. And a backup server living on the same Ceph would only be a fast-restore tier anyway. The real backup is a second one, on separate hardware, that pulls.

Three critical machines exist only as themselves. The firewall, the Tailscale router and the control node were all built by hand, and none of them were in a backup job. The Tailscale router is my only path to the servers’ management controllers, the thing I’d use to rescue the cluster. It runs on the cluster. The control node holds the only copy of the OpenTofu state for my templates and guests. It also runs on the cluster.

Writing those two sentences down felt worse than any outage. An outage is over in an afternoon. A single point of failure you’ve quietly built for yourself just waits for you.

Almost nothing watches anything. Uptime Kuma checks a few services, from a guest on the very cluster it’s checking. Beyond that: no metrics, no alerts, no drift detection. A degraded Ceph placement group or a hand-edit on a node would be found by accident.

The roadmap orders all of it by one rule: close what can’t be undone first, like lost data and leaked credentials. Then what makes failures visible. Then what makes change safe. Then what makes the estate rebuildable. Hardware last.

What’s actually better

It isn’t all confession. Some things genuinely got fixed.

  • Updates are reviewed now. I used to run Watchtower, which updated every container with a floating tag, every hour, with no review and no way back. It’s gone. Every image in the services stack is now pinned to tag@sha256:digest, starting from exactly what was running that day, and the compose check fails on any image without a digest. Pinning turned up one more quiet lie: a hand-kept .env on the control node had been overriding my version tags with latest all along.
  • CI scans images with Trivy and runs gitleaks, tree hygiene and a docs check that fails when a document cites a path that doesn’t exist.
  • main is protected, as code. A role reads the branch rule over Gitea’s API and rewrites it whenever it has drifted from what the repo says.
  • OpenTofu applies only the plan I reviewed. plan writes an encrypted plan file. apply refuses to run without it, applies exactly that file, then deletes it. If the state moved on since I looked, the stale plan is rejected. I’ll be honest: the first real apply through this path hasn’t happened yet. That run is the test.
  • The build order is executable. One root Makefile runs every stage in order and stops at the first failure. Before anything is applied to production, it asks me to type the word production. Not y. The whole word. It’s a small speed bump, and I like it a lot.
  • Gitea is the source of truth. The control node pulls from it instead of me rsyncing files around. The services guest is hardened against the CIS Debian benchmark, and the security doc keeps an unusually blunt list of the controls it doesn’t implement.

And some things are staying out of CI on purpose, forever. The playbook that tears down Ceph, for a start: it’s the only one in the repo that destroys data, and no approval gate is good enough for that. The SDN playbook too, because of that MTU reset. A pipeline runs unverified code faster than I do, and when it fails, it fails halfway through instead of at a prompt.

What I took away

From the first baseline on September 19 to the last commit on October 15, that’s 169 commits. Most of them aren’t features. They’re the repo slowly learning to tell the truth about the estate.

The biggest lesson is the one from the start. The code that scared me wasn’t the code that failed. It was the code that had never run, and that I’d been counting on anyway. The workflows. The CODEOWNERS files. The gateway’s notify script. The Compose template. Every one of them looked finished. None of them had ever been asked to do its job.

So here’s my suggestion. Open your own infra repo tonight and find one thing you’re relying on that has never actually run. A workflow, a restore, a failover, a runbook. Run it once. And if you can’t run it yet, at least write down that it’s never run. That one honest sentence is worth more than a green badge on a check that doesn’t exist.


Sources