Failover architecture · single-identity OpenVPN fleet

Three failures handled.
A fourth, one config line away.

A full-tunnel exit fleet runs one t4g.micro per region behind an Elastic IP. Making it survive a crashed service, a dead box, and silent drift needs no load balancer, no ASG, and no health-check DNS — just the cheapest mechanism that resolves each failure, layered so each one's blast radius is contained by the next.

• ●  ≈ $0/mo incremental
• no hot standby
• recovery: seconds → ~5 min
• shipped: 2026-06

Defense in depth · cheapest layer first

One layer per failure mode

Ordered by blast radius. A failure is caught by the smallest, cheapest layer that can resolve it; the outer layers exist for what the inner ones can't reach.

1. L0 · service

   systemd restart

   Restart=on-failure on openvpn-server@server. Catches a segfault, or a reboot that comes up without the service.

   blast: one process

2. L1 · instance

   EC2 auto-recovery

   A CloudWatch alarm on the host status check that recovers the same box onto fresh hardware — instance ID, ENI, private IP and Elastic IP all come with it.

   blast: one box

3. L2 · region

   multi-remote profile

   One .ovpn lists every region; the client advances to the next exit on its own. Written and tested — emitted once a second region is deployed.

   blast: one region · not deployed

4. L3 · backstop

   SSM healthcheck

   The existing daily read-only probe. Reddens CI when self-healing didn't catch something the fleet can't see about itself.

   blast: whole fleet

L0 + L1 · inside one region

A region that heals itself

The request path top to bottom, with the two recovery mechanisms hanging off the side. Because L1 recovers the same instance, the Elastic IP and DNS name come back with it — every client profile keeps working untouched, and nothing re-associates.

1. Route 53 · A record

   ohio.vpn.beekley.dev — profiles point at a name, not a raw address, so a replaced Elastic IP is a one-record update instead of a config re-issue.

2. Elastic IP

   Retained across host recovery. The same box keeps its address through the whole process.

3. EC2 · t4g.micro

   Ubuntu 24.04 · arm64 · IMDSv2. Recovered by L1 on a StatusCheckFailed_System alarm, in roughly three to five minutes.

4. systemd unit

   Restart=on-failure. L0 resumes a crashed process in seconds, without anything else noticing.

5. openvpn-server@server

   UDP 1194 · AES-256-GCM · full tunnel. Watched by L3, the daily SSM healthcheck that reddens CI.

The only new resource is one CloudWatch alarm per region. A StatusCheckFailed_System breach recovers the instance onto healthy hardware, keeping the instance ID, ENI, private IP and Elastic IP — so nothing re-associates and no client profile changes. That draws a clean division of labour: L0 resumes a crashed process, L1 a dead host. The one residual gap — the OS up but OpenVPN wedged in a way systemd won't restart, which a system-check alarm cannot see — is exactly what L3 exists to redden.

L1 · the recovery nobody has to run

How a dead host heals itself

A box whose underlying hardware fails between weekly rebuilds. Note what is absent: no ASG, no load balancer, no CloudFormation in the loop, and no re-run of user data. Recovery is a stop/start of the same instance, so its disk, config and address all persist.

1. Two failed system status checks trip the CloudWatch alarm.
2. The alarm's recover action stops and starts the same instance on a new host.
3. The EBS root volume persists — instance ID, ENI and Elastic IP are retained, so there is nothing to re-associate.
4. systemd starts openvpn-server@server on boot; the L0 restart drop-in is already on disk.
5. Same DNS name, same address. The client reconnects on its own through resolv-retry infinite.

Roughly three to five minutes end to end, at the same address, with no human in the loop. Because recovery preserves the instance, this is far simpler than an ASG replacement: no Elastic IP re-association, no fresh AMI, no re-run of bootstrap.py. The root volume — carrying the restart drop-in and the server config — comes back intact. The gate is the system status check, and there is deliberately no tunnel-probing load balancer. The CreationPolicy fail-closed signal still governs deploys; it just isn't part of this autonomous self-heal.

L2 · written, not running

The region layer, and why it is dormant

One identity, one profile naming every exit. No DNS trickery and no server-side moving parts — the failover lives in the OpenVPN client's own retry loop. The client tries the first region; if it is unreachable, connect-timeout 10 gives up and dials the next remote in the list. A full cross-region cutover is about ten seconds, with no infrastructure aware it happened.

It works because every server certificate is signed by the one CA and the client authenticates by certificate (remote-cert-tls server), not hostname — so it already trusts every exit. The profile ships as an extra -failover.ovpn beside the per-region files, leaving deliberate region choice intact.

That is the design, and the code for it is written and tested. It is not what runs today. The fleet was consolidated to a single region for cost, and pki.py guards the profile at len(endpoints) > 1 — with one exit there is nowhere to fail over to, so no failover profile is generated at all. The honest status is that L2 is one entry in config.json away, not that it is protecting anything right now.

The whole story on one line each

Failure → response

Why it stays cheap

The money argument is the design argument

The fleet's cost is set by the boxes that have to run anyway. Resilience rides on mechanisms that are free, already paid for, or client-side — which is also why consolidating to one region was a config change rather than a redesign.

Deliberately excluded

An ASG of one

Would also self-heal an OS-level wedge and a terminated box — but it rewrites the whole tested surface and moves the Elastic IP and the readiness signal into boot steps. L0 and the weekly rebuild already cover service and AMI freshness, so auto-recovery closes the one real gap for a fraction of the change.

A UDP Network Load Balancer

Roughly $16/mo per region to front a single instance. It buys nothing auto-recovery doesn't, and adds a component that can itself fail.

Route 53 health-check failover

Route 53 can't cheaply probe UDP 1194; you would bolt on a TCP health port or a CloudWatch alarm check. L2's client-side retry gets region failover for free — when there is more than one region to get it between.

Stack

AWS CDK (TypeScript) · Python (PKI + orchestration) · OpenVPN 2.6 · Ubuntu 24.04 LTS on t4g.micro · EC2 auto-recovery · Route 53 · Secrets Manager · SSM · GitHub Actions with OIDC · Jest + pytest

One CA whose private key never leaves the laptop; nothing secret is generated on the servers. tools/pki.py signs a server certificate per region and pushes each region's materials into that region's Secrets Manager secret, which the instance reads at boot. Failover by layering, not by spending.

Source on GitHub ↗ back to projects