EKS Secrets with Secrets Manager and the CSI Driver

Applications on EKS read credentials from AWS Secrets Manager as read-only mounted files through the Secrets Store CSI Driver, authenticated with EKS Pod Identity and an IAM role that can read exactly two secrets. Rotated values are picked up without a restart, and CI scans for secrets that slipped into manifests or code.

Problem

Frontends and APIs often get credentials baked into images at build time or injected as environment variables — visible in image layers, process listings, crash dumps and Kubernetes Secrets stored in etcd. Rotating them means rebuilding and redeploying.

Solution

Keep secrets in Secrets Manager and mount them into the pod as files at runtime. The CSI driver's AWS provider fetches them using the pod's own identity, so nothing secret exists in the image, the manifests or etcd. The application watches the files and reloads rotated values in place.

Failure and recovery path

1. IAM denied or a wrong jmesPath key surfaces as a FailedMount event and the pod does not start — no fallback to a hardcoded value.
2. An exposed credential is revoked and rotated first; CloudTrail then shows where it was used, and only then is history cleaned.
3. If rotation polling stops, the app keeps serving the last good snapshot and its health endpoint shows the loaded secret version (never the value).

• Amazon EKS
• Secrets Store CSI Driver
• AWS Secrets Manager
• EKS Pod Identity
• KMS
• Helm
• kubeconform
• Terraform

• Terraform: the EKS add-on bundling the CSI driver and AWS provider, with sync-to-Secret off by default and rotation polling configurable; CMK-encrypted secrets with values set out of band; a Pod Identity association and least-privilege role.
• Helm chart: SecretProviderClass extracting individual JSON fields to files, a hardened Deployment mounting them read-only, PDB, NetworkPolicy (DNS egress only — the app never calls AWS) and a ServiceAccount with token automount off.
• The chart passes helm lint and kubeconform -strict, including the SecretProviderClass validated against its published CRD schema.
• A rotation drill starts the app, swaps secret files atomically the way the driver does, and measures when the app observes the new version — about half a second in local runs.
• Manifest scanner (no secret-looking env values, no literal Secret data, read-only CSI mounts, automount off) and a hardcoded-secret detector with a reasoned allowlist; 74 unit tests.

• The role can call only GetSecretValue and DescribeSecret on two secret ARNs, and decrypt with the key only through Secrets Manager.
• Secrets are files, not environment variables, and are never synced into etcd unless explicitly enabled for a legacy need.
• The app never logs secret values and its health endpoint reports only whether secrets loaded and a version hash.
• IRSA is documented as an alternative in an ADR; the code uses Pod Identity only, enforced by a test.

• Rotation without restarts: the app re-reads all files as one snapshot when they change.
• PodDisruptionBudget and probes keep the frontend available during node rotation.

• Secrets Manager bills per secret per month and per API call; the rotation poll interval multiplied by pod count drives the call volume (figures in the cost doc are illustrative).

No quantified results are published for this project. Figures from client work are confidential, and this portfolio only shows numbers that can be verified or reproduced.

• Moving secrets from env vars to files is a small application change that removes several leak paths at once.
• Scan for the anti-pattern in CI, or it comes back with the next feature.
• Known gaps, stated in the README: nothing has run on a real cluster, the add-on's rotation setting keys need confirming with describe-addon-configuration, and the drill covers only the application half of rotation.