Airlock is a deployment control plane for cloud regions that drop off the network, or never had one. Below is a recorded run of the real code: the leader crashes, a region goes dark, a worker dies halfway through a deploy, and a package gets tampered with on its way across an air gap. Every region still ends up on the right version, exactly once.
Three control-plane replicas keep one replicated log of what every region should run. us-east and eu-west pull from it over the network. adc-1 has no network path at all, so it only ever sees signed packages carried in by hand.
This is not a replay. The Raft implementation, the state machine and the region agents from the repository run in this tab, on a simulated network and clock. Kill the leader during a deploy. Cut eu-west off, deploy twice, then reconnect it. Tamper a package. The counters at the bottom are checked after every tick.
Two kinds of numbers, kept apart on purpose. Failover times come from real HTTP servers with fsync'd write-ahead logs and a wall clock. The 10,000-job run uses a simulated network so every fault is reproducible from a seed, and it is about correctness, not speed.
airlock failover -trials 30Each trial starts three replicas as HTTP servers on loopback, commits 20 jobs, kills the leader's server, and times how long until another replica wins an election and until a client write succeeds again.
airlock benchA deploy is a command in a Raft log. It only counts once a majority of replicas have it on disk, so losing any one replica, leader included, loses nothing.
I wrote the Raft core as a pure state machine with no clock or network of its own. That is why the same code runs over HTTP in the binary, on a seeded network in tests, and in this page.
Releases are signed with Ed25519 over the artifact's SHA-256 and size. A region checks the signature and the bytes before it installs, whatever delivered them.
Air-gap packages are signed again by the control plane, scoped to one region, and numbered, so an old package cannot be replayed to roll a region back.
The job id is the idempotency key. A second ack never changes a result. An agent writes each outcome to a local ledger before it reports back, so a crash between installing and reporting never runs the job twice.
After a backlog, an agent jumps to the newest version and marks the rest superseded. It never installs an older generation over a newer one.
Isolated government and defense cloud regions, edge sites with satellite links, factory floors, ships. Anywhere a deploy has to cross a gap that is sometimes or always closed.
Not a new consensus algorithm. Raft, idempotency keys and signed artifacts are standard. The work was putting them together so that each failure mode has a specific mechanism and a test that breaks if the mechanism stops working.
31 Go tests, including a randomized Raft safety check over 60 seeds, run under the race detector, plus a script that runs six real processes and kill -9s the leader.
The log and state grow without bound. A real deployment needs snapshots and truncation.
Replicas talk plain HTTP and the replica set is fixed at startup. Artifact integrity does not depend on the transport, but confidentiality would need mTLS.
The failover numbers are from one machine. Real cross-region latency would stretch every number, and the simulated benchmark says nothing about throughput.
Fine for small artifacts. Large ones belong in a content-addressed blob store, with only digests in the log.
A deposed leader can serve a stale pending list for one election timeout. That is safe here because every delivery is idempotent, but it is not linearizable.
The release key would live in an HSM or cloud KMS, with signing behind an approval step.