AWS Multi-Region Kubernetes Disaster Recovery

A warm-standby disaster recovery design for a stateful Kubernetes workload: two regional EKS clusters, an Aurora PostgreSQL Global Database, replicated object storage and an explicit, runbook-driven failover.

Problem

A single-region platform turns a regional outage into a full outage. The hard part is not running a second cluster — it is keeping data, secrets and DNS consistent so the standby can actually take traffic without corrupting state or processing work twice.

Solution

Warm standby rather than active/active: the DR region runs the full stack at reduced capacity, the database replicates through Aurora Global Database, and failover is a deliberate, scripted promotion. Active/active was rejected because the workload holds queue state that cannot safely be written from two regions at once.

Failure and recovery path

1. Health checks or an operator declare the primary region unhealthy.
2. Workers in the primary region are scaled to zero (if reachable) to stop new job pickup.
3. Aurora secondary cluster is promoted (managed failover when the primary is healthy, detach-and-promote when it is not).
4. DR workloads are scaled up and pointed at the promoted writer; queue state is rebuilt from the database.
5. Route 53 failover record shifts traffic to the DR ALB; synthetic checks confirm before the incident is closed.

• Amazon EKS
• Aurora PostgreSQL Global Database
• Route 53
• ALB
• ElastiCache / Valkey
• S3 Replication
• Secrets Manager
• CloudWatch
• Terraform

• Three Terraform roots — primary, DR and global — so the DR region can be changed while the primary region is impaired; per-region VPC, EKS with Pod Identity, Valkey and KMS; Aurora Global Database, S3 replication, replicated secrets and Route 53 failover in the global root.
• Database password never stored in Terraform state: generated ephemerally and written through write-only arguments.
• Hardened Helm chart with separate API and worker deployments, probes, resource limits, zone spreading, PDB, HPA and default-deny NetworkPolicies; helm lint and kubeconform -strict pass for both regions' values.
• A failover orchestrator CLI with status, plan and execute: planned mode uses a managed switchover, unplanned mode detaches and promotes; execution needs a confirm token, writes an audit log and resumes from a state file without repeating steps.
• Runbooks for planned and unplanned failover, failback and game days, with durations written as targets to measure.
• 100 unit tests over the orchestrator, guards, adapters, idempotency, the rendered chart and the Terraform layout.

• Same IAM and network baseline in both regions, generated from the same modules; workloads authenticate with EKS Pod Identity.
• Per-region KMS keys for the Aurora clusters and buckets, so replicas are readable in the DR region without depending on the failed one.
• The failover CLI defaults to fake adapters and only touches AWS with an explicit flag and config file.

• Fencing before promotion: workers in the old primary are scaled to zero and the region is marked fenced before the DR database is promoted.
• Split-brain guards refuse a promotion when the other region is already promoted or two writers exist, and refuse failback until the old primary has been rebuilt as a secondary.
• Duplicate processing is handled with idempotency keys, so a job replayed after failover is skipped.
• After the first apply the CLI owns the DNS failover roles; Terraform ignores them, so a routine apply cannot send traffic back to a fenced region.

• Warm standby mainly costs the secondary Aurora cluster, the DR EKS control plane with minimal nodes, NAT and cross-region transfer (figures in the cost doc are illustrative).
• Cache and queue state are treated as rebuildable instead of paying for a global datastore.

No quantified results are published for this project. Figures from client work are confidential, and this portfolio only shows numbers that can be verified or reproduced.

• DNS failover is the easy part; deciding when to fail over and fencing the old primary is where most of the runbook goes.
• Automatic failback is riskier than automatic failover; keep failback manual and gated.
• Known gaps, stated in the README: nothing has been deployed, the fence is soft (database access is not revoked), state and audit files need shared storage outside both regions, and a few AWS API behaviours are unverified.