Seven operational roles. Five specialists. One manager.
Every worker in OpsKeeper has a narrow contract: a defined role, a defined tool allowlist, and a defined maximum turn budget. The manager decides who runs when.
Phases map to roles.
Each operational worker is bound to one or more phases of the closed loop. The contract is declared in skill_meta.yaml and enforced at runtime.
Aggregates alerts from Prometheus, Loki, Tempo, webhooks, and on-call channels. Three static rules for PG / Redis / Host plus nine DIAGNOSIS_SKILL_MAP entries and an LLM semantic_dedup stage with circuit breaker.
Read-only causal-chain tracer across metrics, logs, traces, git, hosts, and topology. Traces back to patient zero and returns an evidence chain with confidence.
Post-RCA auditor on severity ≥ critical. Checks the evidence chain, returns a needs_correction flag. Will not invent issues that are not there.
Second pair of eyes on every mutating or destructive action before HITL. Approves only the minimum-necessary blast radius.
Executes mutating actions. Each action must match one approved incident, manifest, resource, command, and payload hash. Otherwise the call fails closed.
Calls recovery.verify only. Four-metric allowlist, three warning tiers. Returns a VerifiedDelta for the manager.
Writes structured period reports from pre-computed ReportFacts. Resource trends, monitoring coverage, changes — never fabricated numbers.
Attach domain experts to incidents.
Specialists live under agents/ and join incidents by routing rule, not by the manager's whim. Use them to enrich an RCA or to draft a postmortem section.
Site reliability patterns: deployment safety, feature flag analysis, dependency blast radius.
Network-layer diagnosis: DNS, LB, BGP, route propagation, packet drops.
CPU scheduling, container limits, kernel cgroup pressure, throttling detection.
Filesystem pressure, IOPS saturation, replica lag, journal replay.
Operational glue: change windows, on-call rotation, communication templates.
A worker is a skill_meta.yaml plus a tool allowlist.
Skills live in Nacos Config with a 30s polling hot-reload, plus a local fallback for air-gapped installs. The control plane reads the same schema every time.
# skills/alerter/skill_meta.yaml
name: alerter
role: intake
phase: detected,correlated
safety_level: L0
max_turns: 12
tool_allowlist:
- read:alerts
- read:topics
- dedup:rules
- dedup:LLM
inputs:
- AlertSet
outputs:
- Incident
- CorrelationID
description: |
Multi-source alert aggregation + semantic dedup.
Circuit breaker on the LLM dedup stage.