Operations manual

Backups

Back up two things: PostgreSQL (the ledger and incident memory are authoritative) and Qdrant snapshots (vector memory). There is no separate export pipeline — the database is the source of truth.

# 1. PostgreSQL logical backup
pg_dump --schema=public --file=opskeeper-$(date +%F).sql "$POSTGRES_DSN"

# 2. Qdrant snapshot
curl -X POST "$QDRANT_URL/snapshots" -H 'content-type: application/json' \
  -d '{"collection_name":"opskeeper_incidents"}'

Key rotation

• Plugin HMAC secrets: stage a new secret in your secret manager, then roll workers. The MCP proxy reads OPSKEEPER_GATEWAY_KEY from the environment at startup.
• JWT signing key: rotate via the API. Old tokens expire on their next refresh.
• Edge secret keys: RotateSecret regenerates an edge's key and replaces the stored hash.

Scaling

The control plane is stateless; scale horizontally behind a TCP load balancer. The orchestrator serializes per-incident via MySQL GET_LOCK, so contention is bounded by the active incident count, not the control plane size.

Monitoring

Grafana dashboards are provisioned automatically. Key signals emitted by the control plane:

• loop_phase_total / loop_phase_duration_seconds — closed-loop throughput and latency per phase.
• opskeeper_tool_invocations_total / opskeeper_tool_duration_seconds — per-tool call counts and latency.
• opskeeper_llm_requests_total / opskeeper_llm_tokens_total — LLM usage per worker.
• opskeeper_http_requests_total / opskeeper_http_request_duration_seconds — API health.

Incident drill

Run a drill at least once per quarter. The repo ships four reproducible PostgreSQL scenarios; seed them with cmd/incident-seed and confirm:

1. The loop reaches postmortem for each scenario.
2. The proposal audit chain verifies (walk chat_proposal_audit hashes).
3. The verifier returns a VerifiedDelta matching the expected metric allowlist.
4. The reporter writes a postmortem in < 60 seconds.

Data retention

loop_event_log rows carry your tenant and timestamp; set retention with a scheduled purge per tenant. Vector memory is retained until your retention policy deletes it. External anchoring of a daily chain root (transparency log) is on the roadmap — until then, database backups are the durable record.