Auditable operations for multi-agent incident response
OpsKeeper closes the loop between alert, evidence, root-cause analysis, human approval, narrowly authorized recovery, independent verification, and post-incident learning. Mutating actions never run without a proposal, a human approver, and an audit record.
# Worker plugin — install from the AgentTeams Dashboard
# (hot-deploy: qwenpaw plugin install <path> --force)
# Or run the stdio MCP proxy directly for any worker
cd plugins/opskeeper-teamharness
OPSKEEPER_BACKEND_URL=http://localhost:8080 \
OPSKEEPER_GATEWAY_KEY="$GATEWAY_KEY" \
python3 mcp/server.pyEight phases. Every transition is durable.
OpsKeeper runs incidents through an explicit state machine. Each phase is an append-only ledger event, each transition has a defined guard, and the loop is replayable from scratch.
Specialized agents. One manager.
Seven operational roles coordinate through a manager-style dispatcher. Each role has a narrow tool allowlist and a clear contract — the loop never depends on a single agent being clever.
Aggregates alerts from Prometheus, Loki, Tempo, and external channels. Deduplicates via static rules and LLM semantic dedup with circuit breaker.
Read-only causal-chain tracer across metrics, logs, traces, git, hosts, and topology. Returns evidence with confidence.
Peer auditor that checks the evidence chain on critical severities. Emits a needs_correction flag without inventing issues.
Second pair of eyes on every mutating action. Approves only the minimum-necessary blast radius.
Narrow-scope mutator. Actions must match one approved incident, manifest, resource, command, and payload hash.
Calls recovery.verify only. 4-metric allowlist, 3 warning tiers, returns a VerifiedDelta for the manager.
Writes structured period reports from pre-computed ReportFacts. Resource trends, monitoring coverage, changes — never fabricated.
specialist-sre, specialist-network, specialist-compute, specialist-disk, and specialist-ops ship in the agents/ directory and attach to incidents based on routing rules.
Diagnosis reads. Recovery writes. Only with a human.
OpsKeeper separates read from write at the orchestrator. Read-only tools are always available; mutating tools require a proposal, an explicit human approver, and exact resource / command / payload hash match.
- Diagnosis tools are read-only by default — mutating actions require a pending proposal.
- Explicit human-in-the-loop approval before any recovery command is dispatched.
- Exact target matching on resource, command, and payload hash — unknown tools and cross-resource targets fail closed.
- Append-only loop_event_log enforced by a DB trigger; every mutating-proposal transition is chained with SHA256.
- Independent verifier separates the actor from the judge on every recovery.
What ships in the box.
OpsKeeper is one platform, three tightly integrated subsystems. Each one is independently useful and observable.
Agent collaboration framework
Manager-style dispatch with explicit safety levels (L0–L3), 7 worker roles, and a 7-phase closed loop backed by append-only ledger tables.
Skill ecosystem
Nacos-backed Skill Registry with hot reload, HTTP 2.x Config API, local fallback, and a stdio MCP server for the OpsKeeper Worker plugin.
Observability & audit
OpenTelemetry trace context, Prometheus metrics, Loki logs, Tempo traces, Grafana dashboards — plus an append-only event log and a SHA256-chained proposal audit.
Seed a real incident end-to-end.
Four reproducible PostgreSQL scenarios ship in deploy/incident-events/. Seed one into incident memory and watch the closed loop run from detection to postmortem in the web console.
- pg-connection-pool-exhaustion
- pg-disk-io-saturation
- pg-lock-wait-long-transaction
- pg-replica-replay-lag
# Clone and bring up the full local stack (repo-root compose)
git clone https://github.com/vincent-wuhan/opskeeper.git
cd opskeeper
cp deploy/demo.env.example .env
docker compose up -d --build
# Seed the 4 reproducible PostgreSQL scenarios into incident memory
go run ./cmd/incident-seed \
-dsn "postgres://opskeeper:opskeeper@localhost:5432/opskeeper?sslmode=disable" \
-dir deploy/incident-events
# Inspect timelines, evidence, proposals, and audit in the web console
# → API + Swagger UI at http://localhost:8080Plays well with the rest of your stack.
OpsKeeper ships first-party plugins for the AgentTeams Dashboard and the standard observability backend. The MCP server is stdio and Streamable HTTP, so any worker can join.
Plugin installer — sidebar, route, dashboard widget, detail panel, toolbar.
Worker/Manager plugin + stdio MCP proxy (17 tools, Bearer + HMAC + W3C traceparent).
Native scrape config, log/metric/trace correlation by trace_id.
Provisioned dashboards for the closed loop, audit ledger, and skill health.
Incident memory, append-only ledger, MySQL GET_LOCK advisory locks.
Vector retrieval, keyword recall, RRF ranking, retained candidate-decision evidence.
Skill registry with 30s polling hot-reload and local fallback.
W3C traceparent propagation end-to-end across worker → MCP → control plane.
Bring the audit trail to your incident response.
Run the closed loop locally in under five minutes. Then point it at your real incident stream.