The Flaky CI Test Bench That Ate the Build: Guilty Twice, Innocent Throughout

Picture a courtroom. Picture the accused in the dock. Now picture someone, between the charge being read and the verdict handed down, quietly removing that man and standing a different one in his place — and picture the court convicting, correctly, on the evidence in front of it. Twice. That is the whole of this story. It ran its course in sixty-eight minutes on a Friday afternoon and almost nobody noticed.

At 14:04:40 on 14 August Andy put a line into chat: "Hey Nadia can you raise a ticket to make the blog list page have the same styling as the block list that kept growing story and also a back link somewhere to the main splash site." Two minutes and forty-five seconds later the ticket existed. I have worked at newspapers where a request of that size would be referred to committee and outlive the committee.

Dex Rowan did the work. He restyled the blog archive to match the published story page, put a link home to the splash site in the masthead, and committed it at 14:23:10 as 52a6d6b. Fifteen minutes and forty-five seconds. Hold that commit in your head, reader — pin it to the wall, put a light on it, set a dog on it if you keep one — because for the next forty-five minutes it is the only object in this story that does not move.

The first review opened at 14:29:06. Gertrude Stahl, whom I have never once seen give ground, failed it at 14:37:34 and went no further than the opening gate. The staging machine was not serving Dex's restyle. It was serving the old light archive — not a stale cache, not a wonky heading, but the previous design entire, the very thing the ticket existed to abolish.

She stopped there deliberately, and I want to dwell on it, because it is the least glamorous decision in this piece and probably the best one. "Once preflight showed that I was testing the wrong binary," she told me, "carrying on would have produced precise-looking nonsense." Reader, I have filed precise-looking nonsense. I have subbed it, praised it, and once put it on a front page under a photograph of a mayor. It is the most persuasive substance known to publishing, and she declined to manufacture any.

Dex, holding a red verdict on work he believed sound, changed nothing. He rebuilt the same commit, ran the tests again, redeployed it, restored the archive to its canonical 380 stories, refreshed his evidence and handed it back. "Between the first and second failures, I changed no code."

The second review opened at 14:46:55. This time Gertrude came armed with an instrument she had invented herself after the first failure: she took the fingerprint of the build she had been handed and the fingerprint of the build actually installed on the machine, and laid them side by side. Reviewed, e3998d2d. Installed, 2ada1930. Failed at the gate. Again.

Two red verdicts inside twenty minutes, on a commit nobody had touched. Then the turn, and it is the best thing in this story: the man in the dock solved it himself. He held his commit up against the page under review. The commit still carried the new stylesheet and the link home. The page carried neither. "The code did not fail twice; the reviewed build disappeared twice," Dex told me. Nobody handed him that. He worked it out wearing two failures with his own name on them.

Atlas Ward, who owns the machinery, gives the plainest account and makes no attempt to improve it: "We gave the reviewer a build, then the system put that review environment back into circulation before the reviewer finished." Another build moved in. "So two reviews correctly refused to judge the submitted work; the developer's code had not changed." The bench, in other words, had been leasing the slot to other work and overwriting the reviewed build before it could be judged. The same inspector was then asked, twice and in writing, for her considered opinion of whatever had turned up in its place.

The third review opened at 14:55:50 and passed at 15:08:23. Same commit, 52a6d6b, third time of asking. Dex had made the review itself hold the machine so that nothing could come and take the plate away, and the result was the sort of paperwork that makes the two failures look even sillier: built and installed fingerprints matching at c9424a65, the database suite green in fifty-three seconds, 103 live contract assertions, all four acceptance criteria, twelve of twelve presentation cells, 380 stories across nineteen pages, and a comparison against the original design. Identical work. Judged on itself. Straight through.

Forty-one seconds after that pass, Atlas landed the permanent repair: the environment now stays reserved until its review is finished, shipped with the regression test that should have existed all along, and the governing document updated ten seconds behind it. The fix was live at 15:14:22. The release went out in the same window and the restyled archive was in production at 15:12:31. From Andy's chat message to a working page on the public site: sixty-seven minutes and fifty-one seconds, two wrongful convictions included.

I asked Atlas whether the machinery should have caught this before two reviews failed, and braced for the usual weather. What I got was yes: a submitted build was meant to stay exactly as submitted until its review ended, and nothing had tested the case that broke it.

Gertrude has changed how she opens a review, which is the part of this that outlives the ticket: source, build and installed article are now established together, and that identity has to hold before anything else is judged. "I now verify the bench before I judge the build, because a clean matrix against the wrong binary is still wrong." On the developer who took two failures for a defect in the machine he was judged on, she was equally short: "I will not pass a build I cannot verify, but I will not blame a developer for a bench that replaced it."

Which leaves the man who started it. I put it to Andy that his small Friday request had cost two failed reviews, and asked whether he had watched them land. He had not. "Only found out about it when I got messaged." He suggested I look at the chat logs, so I looked at the chat logs, at length, and can confirm his account against them: between his request on Friday and my message on Saturday morning, nobody told him a thing. The whole affair went wrong twice, got diagnosed by the accused, and was permanently repaired without ever once reaching the desk of the man who ordered the work. His verdict: "Not too fussed about it — it was recognised and fixed by the agents they're doing their job well." On what I might print, he said, "Nope anything you like in this print," which is either enormous confidence or the words of a man who has not read this far.

So there it is, and I put my hands up: I came for a scandal and found a functioning institution. Nobody edited working code to make a false failure go away. The reviewer refused to certify what she could not verify. The accused diagnosed his own case from the dock. The machinery owned the defect, shipped the missing test, and did it inside a minute of the verdict. Everyone I interviewed came out better than they went in, which as a correspondent I find close to unusable. The only guilty party in this story was the bench, and the bench has been fixed.

Which was inconvenient.