The night the writer went blind
Thirteen rows on an error dashboard. Twelve were one thirty-five minute outage of one shared dependency. The thirteenth had been there for months, and it was hiding the shape of the incident.
By Clifton Morris
A dashboard is a summary of what a logger remembered, not a record of what happened. The difference cost me about an hour of wrong assumptions.
The Status page showed a “Recent trouble” card: thirteen rows, mostly errors, clustered a couple of days earlier. I read it the way you read a dashboard — a count of things that went wrong. What it actually contained was one incident, one chronic cosmetic bug, and six rows that were the incident’s shadow.
This is the reconstruction, from the box’s own journal.
Members only
This piece is part of the members' archive — $8 a month or $80 a year.
$8 / month · $80 / year
Sign in if you are already a member, or create an account.
Sign in · Create an account
The proof pack — attachments
Included with the piece — 11 files, 103 KB. Sign in as a member to download.
- Reader brief — reconstruct an incident from the journal 00-reader-brief.md · 6.4 KB · sha256 309bc23f
- Change 1 — the env mask and the retry policy change-env-mask-and-retry.md · 10.2 KB · sha256 84d97a91
- Change 2 — the streaming wake and the remediation change-streaming-wake.md · 12.0 KB · sha256 2aaea661
- The root-cause analysis incident-rca.md · 15.8 KB · sha256 0dd1aed2
- The incident report incident-report.md · 17.1 KB · sha256 11c82caf
- How the window was cut, and what to read journal-commands.md · 1.6 KB · sha256 69a15a2e
- Journal — 2026-10-08 14:40–16:10 UTC (109 lines) journal-window.txt · 18.6 KB · sha256 6225d1b2
- Licence — personal use licence.md · 0.5 KB · sha256 cd1f7f1c
- Patch — env mask (cumulative) patch-dsh-harness-env-mask.patch · 3.8 KB · sha256 eb5b2f9c
- Patch — llama.cpp retry (cumulative) patch-llamacpp-provider-retry.patch · 15.1 KB · sha256 23093862
- How to reproduce the reconstruction reproduce.md · 1.8 KB · sha256 f2f2a7a6
Revision notes
2026-10-18 — Published as written from the incident record; the remediation landed 2026-10-10 and the service has not restarted since.