Alert-to-Runbook Responder
A scheduled responder that turns a watchdog alert into a diagnosis and an allowlisted runbook, with the only auto-applied fix chosen because its failure mode was 970 silent skips.
2,658 recorded ticks and 189 incidents in 18 days, every one with a diagnosis attached, and a structural guarantee that the responder can only run commands from a short allowlist.
- Role
- Solo: designed, built and operate
- Timeframe
- Sep 2026 to present
- first recorded tick 19 Sep 2026 (sre_runs table)
- Team
- 1
- Problem
- The watchdog raised alerts; the steps after that were a person hand-running commands, and when that person was a chat session the knowledge died with it.
- Approach
- A cron tick reads the watchdog's state, opens one incident per alert (enforced by a partial unique index), runs a read-only triage, and names a runbook from a registry. The registry validates arguments and builds argv; the responder never constructs a command. Rate limits live in SQL.
- Result
- One fix is auto-applied (re-pinning a drifted scheduler job: idempotent, reversible, and the failure it fixes is silent and indefinite). Everything else parks for approval or reports only. 15 of 15 guardrail tests pass, each negative case paired with a positive twin.
Every figure above is measured or read from the running system. Where a figure has a source line, that is how and when it was read.
Technology Stack
The structural control
The responder names a runbook. It never writes a command. A registry maps each name to an argument validator and an argv builder, and both run, in that order, before anything executes. Anything that deletes, drops, powers off, reboots, rotates credentials or edits network configuration is absent from the registry by design, and the invariant test asserts that nothing in it is both auto-applied and irreversible.
Host access is a short allowlist with its own SSH key. The database host is deliberately excluded: no runbook needs it, so the responder holds no key there. I verified the key is refused rather than assuming it.
Rate limits are rows, not memory. A cooldown of 120 minutes and a cap of three applications a day are answered by querying the incidents table, so a restarted process cannot forget that it already acted.
Why exactly one fix is automatic
A scheduler job in this fleet once failed 970 consecutive times over four days because its model configuration drifted and the job settled into a skipped state. The fix is re-pinning the job, which is idempotent (re-pinning a pinned job is a no-op) and reversible in one command. The cost of applying it wrongly is far below the cost of not applying it, so that one runbook is auto. Restarting a container is gated behind approval until the auto path has a track record.
Against the live drift alert, the dry run proposed the exact command and did nothing. The live run applied it and recorded auto_applied. Then verification failed to parse, and the responder refused to close the incident and marked it failed. That refusal was the point: an exit code of zero never closes an incident on its own. The following run hit the cooldown and declined to re-apply.
Two bugs that only running it found
The database URL in the config used the postgres:// scheme, and a grep for postgresql:// returned nothing, so the responder tried a local socket and failed quietly. And the watchdog's state file is pretty-printed JSON, while the reader parsed only its last line. Both are the shape every failure on this infrastructure takes: a component that reports success while doing nothing.
The runbook that restarts a container had never worked on any host. SSH joins its remote argv with spaces and the login shell re-parses the result, so a cd and a compose restart arrived as two separate commands and compose ran in the home directory. It had passed its validation tests for weeks, because validation checked the arguments and never ran anything. I fixed it by quoting the whole payload once, then proved it with a real restart and watched the probes go green afterwards.
Honest status
This responds to a personal fleet, and 187 of 189 incidents ended in report-only, which is the right outcome for alerts with no safe automated fix. I would not read the fix count as the result. The result is that every tick and every decision sits in a table I can query, so I can answer what it did and why.
Still open: forced-command SSH keys so the server rejects anything off-list rather than only the Python layer, and an alert when the approval queue fills, because a full queue looks identical to an idle one.