Observability and SRE Platform

Prometheus, Grafana and OpenTelemetry with SLIs defined per user journey, multi-window burn-rate alerts instead of threshold noise, and an incident process tied to error budgets.

Problem

CPU and memory alerts page people for things users never notice, while real user-facing failures go unalerted.

Solution

Alert on symptoms: SLIs measured at the edge of each user journey, SLOs agreed with the product owner, and burn-rate alerts that fire when the error budget is being spent too fast.

• Prometheus
• Alertmanager
• Grafana
• OpenTelemetry Collector
• Prometheus Operator CRDs
• promtool
• Helm
• Docker Compose

• SLOs as code: a stdlib generator turns SLO definitions into recording rules for seven windows and multi-window multi-burn-rate alerts (page at 14.4× and 6×, ticket at 3× and 1×), emitted as plain rule files and PrometheusRule resources, plus a Grafana dashboard per service.
• Generated output is deterministic and checked in; CI fails if regenerating changes anything.
• promtool checks every rule file and runs eight alert unit-test scenarios; amtool validates the Alertmanager config and otelcol validates the collector config.
• A local stack (Prometheus, Alertmanager, Grafana and a demo app with fault injection) paged on a real injected error rate about two minutes after the fault began.
• An error-budget report computes budget remaining, burn rate per window and when each alert would have fired from hourly counts; 79 unit tests.

• The OpenTelemetry Collector drops or hashes personal data attributes before anything is stored; hashed IDs are documented as pseudonymous, not anonymous.
• Hardened collector deployment: non-root, read-only filesystem, PodDisruptionBudget.

• Page alerts inhibit the matching ticket alerts, so one incident produces one page, not a storm.
• A Watchdog alert routed to a dead-man's-switch receiver proves the alerting path itself is working.
• Every alert carries a runbook link; runbooks, an error-budget policy and an incident-response process (roles, severities, comms and blameless postmortem templates) are part of the project.

• Label cardinality, retention and log volume drive cost; drop debug logs and high-cardinality labels at the collector rather than paying to store them (figures in the cost doc are illustrative).
• Managed Prometheus and Grafana are compared as alternatives in the cost doc.

No quantified results are published for this project. Figures from client work are confidential, and this portfolio only shows numbers that can be verified or reproduced.

• Fewer, better alerts: alert on budget burn, not on CPU, and every page should need a human action.
• Sloth or Pyrra would generate these rules in many organizations; the stdlib generator here exists to show the math.
• Known gaps, stated in the README: never installed in a cluster, ticket-inhibition was verified by config and tests rather than captured live, and the dashboards were checked structurally only.