One platform. Eight phases. Zero silent recovery.
The OpsKeeper control plane runs incidents through an explicit state machine. Each transition is an append-only ledger event with a defined guard. Every mutating action is observable, replayable, and provably authorized.
Phases, guards, and ledger events
Every incident runs through the same eight phases. Each phase has explicit inputs, outputs, and a guard that decides whether the loop can advance.
Detected
Multi-source alert intake from Prometheus, Loki, Tempo, webhooks, and on-call channels. Static rules for PG/Redis/Host pre-group incoming alerts.
Correlated
Semantic de-duplication across sources using the DIAGNOSIS_SKILL_MAP and an LLM semantic_dedup stage with a circuit breaker.
Investigated
Read-only root-cause analysis across metrics, logs, traces, git, hosts, and topology. Returns an evidence chain with confidence.
Critiqued
On severity ≥ critical, a peer critic audits the RCA. Emits needs_correction without inventing issues — a fast no-op most of the time.
Approved
Human-in-the-loop on a pending proposal. Exact target match on resource, command, and payload hash. No approval → no execution.
Recovered
Narrowly authorized repair execution. Unknown tools and cross-resource targets fail closed. Every action lands in the audit ledger.
Verified
Independent verifier calls recovery.verify only. Four-metric allowlist, three warning tiers, returns a VerifiedDelta for the manager.
Postmortem
Reporter writes an eight-section postmortem from pre-computed ReportFacts. Resource trends, monitoring coverage, changes — never fabricated.
A proposal is a contract, not a wish.
OpsKeeper's approved phase requires a proposal with explicit blast radius, exact-match guards, and audit hooks. The control plane will refuse to dispatch a recovery without all of them.
- resource, command, and payload hash must match the approved incident exactly
- unknown tools and cross-resource targets fail closed
- audit events append on dispatch and seal on completion
phase: approved
proposal:
incident_id: INC-PG-POOL-001
worker: repairer
blast_radius: pg.connection_pool / one-db
guard:
required_approval: human
exact_match:
- resource
- command
- payload_hash
audit:
on_dispatch: ledger.append
on_complete: ledger.sealWhat the closed loop is built on.
The data plane is intentionally boring: a relational store, a vector store, a config registry, and a tracing standard. Replace any of them without rewriting the loop.
PostgreSQL
Incident memory, append-only ledger (loop_event_log / loop_state / loop_contract), and MySQL GET_LOCK advisory locks for orchestrator serialization.
Qdrant
Vector retrieval over historical incidents. Keyword recall + RRF ranking. Retained candidate-decision evidence per query.
OpenTelemetry
W3C traceparent propagation end-to-end across worker → MCP proxy → control plane → web console.
Nacos Config
Skill registry with HTTP 2.x Config API, local fallback, and 30s polling hot-reload. skill_meta.yaml schema.
Drop a new worker, register a skill, watch the loop route to it.
OpsKeeper ships a manager-style dispatcher with safety levels L0–L3. New workers join by registering a Skill and declaring their tool allowlist.