AWS Backup Cross-Region DR with Restore Testing

Centralized backup with AWS Backup: tag-based plans, copies to a second region under a different KMS key, vault lock against deletion, native restore testing with an automated validator, and an RPO/RTO report computed from job history.

Problem

Backups that are never restored are a hope, not a recovery plan. Common gaps: copies in the same region and under the same key as the source, recovery points an attacker with admin rights can delete, and RPO/RTO targets nobody measures.

Solution

Back up by tag into a primary vault, copy every gold-tier recovery point to a DR-region vault encrypted with its own key, lock both vaults, and let AWS Backup restore testing restore samples on a schedule. A validator checks each restored resource and reports pass or fail, and a report compares measured backup age and restore time against targets.

Failure and recovery path

1. A failed backup or copy job triggers a vault notification and an alarm on the failed-job metrics.
2. A restore test that fails validation is reported to AWS Backup as FAILED and drops the restore-validation metric, which alarms per resource type.
3. The RPO report exits non-zero when any gold resource's last good backup is older than its target, so it can gate a scheduled CI job.
4. In a regional outage, the runbook restores RDS and DynamoDB from the DR vault into the DR region.

• AWS Backup
• Vault Lock
• Restore testing
• AWS Lambda
• EventBridge
• KMS
• SNS
• CloudWatch
• Backup Audit Manager
• Terraform

• Terraform module calling a regional-vault submodule once per region, wired with provider aliases; a validation harness and an example both validate.
• Selection uses tag conditions so it can express 'gold and not excluded'; cold-storage lifecycle is checked against AWS's 90-day minimum.
• Retention rules, copy targets and lock limits live in one policy JSON read by Terraform and checked in CI: every gold rule copies cross-region, retention fits inside the lock range, schedules are valid.
• Stdlib Python validator Lambda with injected adapters: checks DynamoDB item counts, S3 object samples, EBS availability and RDS reachability, and refuses to touch resources without the restore-testing tag.
• 118 unit tests, plus a test that keeps the README's sample output in sync with the scripts.

• Copies are re-encrypted with the DR vault's own key, so compromising one key does not expose both copies.
• Vault lock in governance mode by default; compliance mode is supported but documented as irreversible after its grace period.
• A vault access policy denies deleting recovery points or shortening their lifecycle except for a break-glass role.
• Restore testing runs under its own role, which can only clean up resources carrying the restore-testing tag.

• Restore tests run on a schedule, so the restore path is exercised continuously rather than during an incident.
• Measured restore duration is compared with the RTO target per resource, not assumed from documentation.

• Warm vs cold storage, cross-region transfer for copies, compute for restore tests and continuous backup are the drivers; cold storage only pays off for long retention (figures in the cost doc are illustrative).

No quantified results are published for this project. Figures from client work are confidential, and this portfolio only shows numbers that can be verified or reproduced.

• A backup strategy is only as good as its last tested restore.
• Separate keys and locked vaults matter more against ransomware than extra copies in the same account.
• Known gaps, stated in the README: the RDS row-count check needs a database driver, the break-glass role and a DLQ on EventBridge targets are documented but not built, and nothing has run in a real AWS account.