Architecture

Control plane

The control plane runs incidents through an explicit state machine — the closed loop. Each transition is guarded and produces an append-only ledger event.

┌──────────────────────────────────────────────────────────────┐
│                    OpsKeeper control plane                    │
│                                                              │
│   ┌─────────────┐  ┌─────────────┐  ┌────────────────────┐    │
│   │ loopbiz     │  │ manager     │  │ safety boundary    │    │
│   │ (state mch) │→ │ (dispatch)  │→ │ (proposal + audit) │    │
│   └─────┬───────┘  └─────┬───────┘  └─────────┬──────────┘    │
│         │                │                    │               │
└─────────┼────────────────┼────────────────────┼───────────────┘
          ▼                ▼                    ▼
       (ledger)        (workers)            (proposals)

The closed loop

Eight phases, in order. The loop refuses to advance without a satisfied guard:

1. detected — alerts ingested from Prometheus, Loki, Tempo, webhooks.
2. correlated — semantic dedup across sources (rules + LLM with circuit breaker).
3. investigated — read-only RCA across metrics, logs, traces, git, hosts, topology.
4. critiqued — on severity ≥ critical, a peer critic audits the RCA.
5. approved — human-in-the-loop on a pending proposal.
6. recovered — narrow mutator runs the approved action.
7. verified — independent verifier returns a VerifiedDelta.
8. postmortem — reporter writes from pre-computed ReportFacts.

Manager dispatch

The manager owns a decision table that maps (phase, severity, role) to a worker. Every worker declares a safety_level from L0 (read-only) to L3 (mutating-with-proposal). The manager refuses to assign a worker whose level is above what the phase allows.

- phase: detected
  severity: [info, warn, error, critical]
  worker: alerter
  safety_level: L0

- phase: investigated
  severity: [info, warn, error, critical]
  worker: investigator
  safety_level: L0

- phase: approved
  severity: [info, warn, error, critical]
  worker: repairer
  safety_level: L3   # requires pending proposal + human approver

Data plane

• PostgreSQL — incident memory, append-only ledger (loop_event_log, loop_state, loop_contract), MySQL GET_LOCK advisory locks for orchestrator serialization.
• Qdrant — vector retrieval over historical incidents. Keyword recall + RRF ranking. Retained candidate-decision evidence per query.
• Nacos Config — skill registry with HTTP 2.x API, local fallback, 30s polling hot-reload.
• OpenTelemetry — W3C traceparent propagation end-to-end.

Plugin surface

Two plugins ship in the repo:

• agentteams-plugin-installer — turns the AgentTeams Dashboard into the OpsKeeper plugin console (5 extension points, HTTP API).
• opskeeper-teamharness — worker-side plugin that exposes 17 MCP tools over stdio MCP. Bearer + HMAC + W3C traceparent auth.

Append-only ledger

Two durable records back the loop. loop_event_log is the append-only event source of truth — a DB trigger rejects UPDATE/DELETE, so the only way to correct an event is to append a correction. Every write carries an idempotency key, making replays exactly-once. On top of that, every mutating-proposal transition (insert / decide / expire / execute / rollback) appends one row to chat_proposal_audit, a SHA256 hash chain: hash_n = SHA256(prev_hash || canonical_json(payload) || proposal_id || action). Any tampering with payload or order invalidates every subsequent hash, and the in-repo verifier walks the chain to report the first break.

Observability

• Prometheus scrape config ships with the repo.
• Loki log streams and Tempo traces are correlated by trace_id.
• Grafana dashboards are provisioned for the closed loop, audit ledger, and skill health.