OpsKeeper
Run as a team

Harness guide

The verification harness is how OpsKeeper proves that a change did not regress the closed loop. It runs three scenarios against a real Postgres + Qdrant + control plane stack: alert_storm, rca_loop, and recovery_verify.

Running the harness

harness
# Bring up the harness stack
make harness-up

# Run all scenarios
make harness

# Run a single scenario
make harness SCENARIO=rca_loop

# Tear down
make harness-down

Scenarios

alert_storm

Floods the alerter with 10,000 synthetic alerts over 60 seconds and asserts that:

  • The dedup circuit breaker never trips more than twice.
  • No more than 50 distinct incidents are created.
  • The critic is invoked on the right severity tier.

rca_loop

Replays pg-connection-pool-exhaustion end-to-end and asserts that:

  • The loop reaches verified within 90 seconds.
  • The investigator returns evidence from at least 3 sources.
  • The proposal carries a valid payload hash and blast radius.

recovery_verify

Mutates a synthetic target via an approved proposal and asserts that:

  • The verifier returns a VerifiedDelta matching the allowlist.
  • A proposal with a mutated payload hash is rejected with proposal_hash_mismatch.
  • The audit ledger is HMAC-valid after the run.

Writing a new assertion

Assertions live under tests/harness/assert/. Each is a Go test that receives the live control plane handle and a recorded run.

assertion_test.go
package assert

import (
  "testing"
  "github.com/vincent-wuhan/opskeeper/harness"
)

func TestRecoveryAppliesOnlyApprovedPayloadHash(t *testing.T) {
  run := harness.LoadRun("recovery_verify")
  for _, evt := range run.Audit {
    if evt.Action == "recovery.dispatch" {
      if evt.PayloadHash != run.Proposal.PayloadHash {
        t.Fatalf("dispatch payload hash %s != approved %s",
          evt.PayloadHash, run.Proposal.PayloadHash)
      }
    }
  }
}

CI

The harness runs in CI on every PR. A red harness is a release blocker.