Skip to content

AI-Powered Cloud Operations

Log anomaly detection with guardrailed, human-approved remediation

Working portfolio implementationImplemented & validated locally

Basis: Portfolio implementation informed by CV work on AI-based log anomaly detection and AI/LLM-assisted operations; none of the numbers come from that work. External AI calls are optional and off by default.

Overview

A Python pipeline that detects log anomalies deterministically, correlates them with deploys and alerts, optionally drafts an incident summary with an LLM, and proposes — never executes — remediation without human approval.

Problem

On-call engineers spend the first part of every incident assembling context from logs, deploy history and alerts.

Solution

Do the deterministic work first — baselines, anomaly scores, correlation with recent changes. Use an LLM only to summarise what was found, with token budgets, and require a human to approve any proposed action.

Architecture

  1. Inputs

    • Logs

    • Deploy events

    • Alerts

  2. Analyze

    • Anomaly detection

      rolling baseline

    • Correlation

      time window + service

  3. Summarize

    • Template summary

    • LLM summary

      optional, budgeted

  4. Act

    • Proposal

      allow-listed actions

    • Human approval

    • Audit trail

Deterministic analysis first; the LLM step is optional and every action needs approval.

Technology stack

  • Python
  • Statistical anomaly detection
  • Event correlation
  • Optional LLM summaries
  • Audit log

Implementation

  • Rolling-baseline z-score anomaly detection over per-service, per-time-bucket error rate, correlated against deploys and alerts by time window and exact service match.
  • A template summary is always computed; an optional, budgeted LLM summary only narrates findings the deterministic step already produced.
  • A 4-item allow-list of remediation action names (rollback-deploy, restart-pod, scale-up, escalate); anything else is reported, never proposed, and `AuditLog.approve()` refuses a disallowed proposal.
  • Every proposal and every approve/reject decision is written to an append-only audit log with a timestamp and actor.
  • The LLM provider sits behind a `name` + `summarize` interface with a no-op default, so the pipeline runs with zero network calls and zero API keys unless a caller wires in a real provider from their own code.

Security

  • Log content is redacted (emails, IPs, credential-shaped key=value pairs, bare provider tokens) immediately before a prompt is built, not relied on upstream — the guardrail holds even if a log line slips through with something it shouldn't contain.
  • The analysis process holds no credentials capable of executing anything; `propose()` only ever returns a data object, never an action.

Reliability

  • Works fully offline end to end — detection, correlation, summarization and proposal generation have no network dependency.
  • The optional LLM step falls back to the template summary on empty/`None` output rather than raising, so a provider outage degrades prose quality, never the statistics or the proposal.

Cost

  • Per-incident token budget (default 400 output tokens) bounds the optional LLM call; with the default no-op provider, cost is exactly $0 regardless of traffic volume.

Results

No quantified results are published for this project. Figures from client work are confidential, and this portfolio only shows numbers that can be verified or reproduced.

Code & documentation

View on GitHub

Lessons learned

  • Most of the value is in correlation, which does not need a model — knowing a deploy landed four minutes before an error spike is a sorted comparison over three lists, not an LLM call.
  • The allow-list is the safety boundary, not the model: whatever produces the summary, only one of four pre-agreed action names can ever be approved.
  • A no-op default is a feature, not a placeholder — it means the whole pipeline, its tests and its CI run identically whether or not anyone ever plugs in a real model.