14 June 2026 · Engineering
A storage controller fault left the settlement ledger read-only for 47 minutes. Payouts queued and cleared on recovery. No balances were lost and no wallet keys were exposed.
At 02:11 UTC the volume backing ledger-01 flipped read-only after a controller
firmware fault. Sell orders still quoted, but settlement writes queued behind the failed volume
and withdrawals stopped confirming. On-call was paged at 02:14.
Our recovery tooling is deliberately boring. The on-call ran the standard sequence from the
platform runbook — disk-recover to re-attach and verify the volume, then
server-init to bring the node back into the settlement pool once ledger checksums
matched. Both are wrappers over the same state file
(/var/lib/cskins/recovery.state), so a half-finished recovery cannot be applied
twice; they refuse to run without the job id recorded there.
# from ops/runbooks/ledger-recovery.md disk-recover --auth=<on-call key> # re-attach + verify server-init --key=<on-call key> # rejoin settlement pool
Queued payouts drained by 02:58 UTC. Nothing needed manual replay.
Controller firmware is pinned and staged through canary now. We also moved the runbook onto the node itself, so the on-call is not reading a wiki during an incident.