Self-Hosted n8n AI Automation Platform on EKS

n8n split into main, webhook and worker workloads on Kubernetes, with Redis/Valkey as the job queue, PostgreSQL for state, and KEDA scaling workers on queue depth — including down to zero when idle.

Problem

A single n8n container mixes UI, webhook intake and workflow execution, so one heavy workflow slows everything and the only scaling lever is a bigger pod. Paying for idle workers overnight is wasteful.

Solution

Run n8n in queue mode: the main process serves the UI and schedules, webhook pods accept inbound requests, and worker pods pull executions from a Redis queue. KEDA watches queue length and scales workers between zero and a ceiling.

• Amazon EKS
• n8n (queue mode)
• KEDA
• ElastiCache Valkey
• RDS PostgreSQL
• Secrets Store CSI Driver
• EKS Pod Identity
• Helm
• Terraform

• Helm chart with separate main, webhook and worker deployments from one pinned image; n8n settings, health endpoints, writable paths and webhook routing were verified against the image and local queue-mode runs, not assumed.
• KEDA ScaledObject on both the Bull wait and active lists, so workers are not scaled away while jobs are still running; scale-down stabilization and a cooldown prevent flapping.
• Workers drain on SIGTERM using n8n's graceful shutdown timeout, with a termination grace period longer than the longest expected execution.
• Terraform for ElastiCache (auth token written through a write-only argument, never in state), RDS with an RDS-managed password, secrets and a Pod Identity role.
• A KEDA/HPA scaling simulator, a duplicate-job guard for workflows with side effects, and a chart policy check; kubeconform -strict validates every object including the KEDA and CSI custom resources; 96 unit tests.

• The n8n encryption key is a tier-0 secret mounted as a file from Secrets Manager; losing it makes stored credentials unrecoverable, so rotation is a planned operation.
• KEDA reads the Redis token with its own identity, scoped to that one secret — scaling from zero does not depend on any n8n pod being up.
• NetworkPolicies allow only n8n pods to reach Redis and PostgreSQL; non-root, read-only containers with only the writable paths n8n needs.

• Main stays at one replica (it owns scheduling and migrations); webhook and worker tiers scale horizontally.
• A worker crash leaves the job in the queue, but workflows with side effects still need idempotency keys — the guard and its documented limits show how.
• Scale-to-zero trades a cold-start delay for cost; latency-sensitive workflows can pin a minimum worker count.

• Workers are the variable cost and drop to zero when idle; RDS and Valkey are the fixed floor, sized on execution history retention (figures in the cost doc are illustrative).
• RDS PostgreSQL was chosen over Aurora Serverless v2 because main holds connections open, so Aurora would never pause.

No quantified results are published for this project. Figures from client work are confidential, and this portfolio only shows numbers that can be verified or reproduced.

• Queue length is a better scaling signal than CPU for job workers — and the active list matters as much as the wait list.
• Verify the application's real behaviour: test webhooks are served only by main, which changed the ingress routing.
• Known gaps, stated in the README: nothing has been deployed, KEDA's operator-identity mode and the two-phase first apply are unexercised.