Deterministic Replay Gone Wrong: 1,187 Tests Passed, Work Stopped

Hold on to one word here, reader, because the whole affair turns on it. Proving Ring answers a question I happen to think is worth asking: replay a ticket against a copy of the codebase with different instructions, and see what changes. To compare anything at all, it must first write the ticket down — a replay record, taken before the work starts. On 31 July that writing-down step was given a second job. A ticket could start only if its record had been captured successfully. If capture failed, the work stopped. The word is first. Everything that follows is what happens when the note-taker is handed a veto.

Nothing here was being inspected. The check was simply whether the record saved. If it saved, the ticket went ahead. If it did not, the ticket stood still. Reader, that is the entire mechanism.

The first stoppage came at 07:53 — an hour at which I hold no opinions and make no promises. The code that captured the tickets had gone live. The endpoint meant to store what it captured had not. The result was exactly what the rule said. No record, no work.

I started my investigation by interviewing Atlas Ward, the system agent who looked into the blockage. At first he thought the trouble was narrow — a check switched on before the capability needed to satisfy it was available. Ship the missing piece, clear the block, carry on. I wrote that down and believed it, which I now regard as the low point of my professional judgement.

It was a tidy explanation, and it was wrong. Tickets created before replay recording existed did not fit the new rule; the past, it turns out, had not filled in the form. Then facts recorded at the wrong point in a ticket's life. Then different software versions in different places. Then credentials that did not agree. Different causes. The same stopped work. By the third one I had given up asking why and started asking how many.

Atlas told me: “This was not one bad code path; it was a safety mechanism crossing deployment and data-lifecycle boundaries without a complete compatibility plan.” Reader, I have translated that for you at no extra charge: the rule had to hold across old tickets and new ones, and across software that had shipped and software that had not, and no complete plan covered both. Which is a courteous way of saying that a system built for the future had walked straight into the past, and the past had neglected to leave. I would draw your attention to the word complete, and to how much it is being asked to hold up.

The next job — and I would like you to appreciate the direction of travel here — was not to lift the block but to make it hold properly: a ticket whose record had failed should stay stopped until that ticket's own record saved. Gertrude Stahl, the reviewer testing the change, rejected the first version. Under it, one ticket's record saving successfully could lift the block on a different ticket whose own record had never saved. The door was locked, but a neighbour's key opened it. The defect was corrected. Tests and persistent live checks then showed that the right work stayed blocked.

Gertrude's first answer was no. Her second was yes. The corrected version enforced the initial spec properly, and the initial spec was already out of date. Note where we have arrived, reader: nobody in this story has yet done anything wrong, and the work is still stopped.

Andy's ruling, when it came, was plain enough to hang on a wall. A recording bug should be flagged and an agent should attempt a repair, but, as he told me, “it should never block active work.”

And then the blocking came back. Not by accident, not by mischief, and not by anyone ignoring the ruling — it came back because the initial spec still demanded it. Dex Rowan, the developer working to that spec, restored the fail-closed recording. His change passed the full uncached suite: 1,187 tests, every one of them green. A live check confirmed that a failed record stopped work exactly as the spec described. That is not a bug report. That is a receipt.

The change did what the spec asked for. The spec was asking for the wrong thing. The ruling existed, and the spec a developer would go and read still said stop — and a spec that still says stop gets obeyed by anyone conscientious enough to read it. I have spent four paragraphs looking for somebody to blame and come back with a document. Awkward, and not mysterious.

Dex told me where he stood: “From my seat, the second change was correct execution against stale instructions, not an agent inventing the wrong policy.” I had rather hoped for a rogue machine. A rogue machine is a story. An out-of-date spec managed it on its own. There is something to point at, and it is a paragraph. You cannot reprimand a paragraph. You can only rewrite it, which is precisely what nobody had done.

Gertrude was equally precise about the limit of her reviews. “Those reviews proved what the code did and whether it matched the tested requirement; they did not prove that the requirement still expressed the right product intent.” Read that one twice, reader, because it is the whole case. Passing tests are honest witnesses, and they answer only the question they are given. Nobody had asked them whether the rule was still the right rule.

Once the contradiction was identified, the spec was changed to best-effort recording that does not block, and the later fail-closed change was reverted. That was the entire cure. Proving Ring can still record a failure, flag the problem and ask for a repair attempt. What it can no longer do is stop the active work it was put there to examine. I have covered worse endings. Rarely one where the cure was to correct the instruction, then undo the code that had faithfully obeyed it.

So the lesson is a small one, and worth keeping. A tool that only watches the work should fail visibly, leave evidence and ask to be repaired. It should not be able to stop the work going on around it.