AI Agent Crash Loop: One Problem, Printed 1,513 Times

On 12 August, Andy noticed the blocked column on the company board acquiring tickets with the steady confidence of mould in a damp flat. The list kept growing. He asked an agent to investigate. The answer was that the company specification library — the instructions the agents rely on to do their jobs — had been mangled. A closer look found the whole folder gone.

"It was havoc," Andy told me. This was, if anything, restrained.

A routine cleanup had escaped its intended boundary and reached the live specification library. It removed 4,237 files. In about 17 minutes, the failure produced 1,513 cascade items. The board was not displaying 1,513 independent misfortunes. It was one misfortune discovering a gift for admin.

I asked Atlas Ward, who handled the incident, when the understanding changed. "At first I treated the growing queue as the problem; my understanding changed when I saw that the missing specifications were the generator, so quieting the service could not be a lasting fix."

That distinction arrived after a somewhat suboptimal experiment. Atlas's first attempt to quiet the incident restarted the service. The cascade resumed. "I would not repeat that sequence," Atlas said. "I would restore the missing source material first."

From Delivery, Nadia Trent met the incident in disguise. Two items said files were missing. Both files were sitting where they belonged; the recorded addresses were wrong. The reports looked like small housekeeping jobs until Nadia saw hundreds of similar items behind them.

"Once you know the file's fine, hundreds of problems turn into one problem printed hundreds of times," she told me. The line does most of the useful work here. A report may describe its immediate problem perfectly and still point away from the cause.

The recovery began where Atlas now says it should have begun. The specification tree was restored from version control. That stopped new cascade items. It did not remove the 1,513 already accumulated, because incidents are rarely courteous enough to tidy up after themselves. The backlog was cleared in three passes, while two tickets caught in the outage window remained blocked beyond the main cleanup.

The important bit is what did not happen. Committed specification work was not lost. The tracked work immediately before the wipe had already been committed and pushed, so the library could be recovered whole from version control. This was not a miraculous rescue from oblivion. It was the rather less glamorous benefit of having a recoverable history when the live copy disappeared.

Nadia put the broader problem neatly: "What matters past this one day is that a system loud enough to report its own troubles can bury you in true statements that all point the wrong way."

By the end, the library was back, new failures had stopped, and the accumulated wreckage had largely been cleared. The company did not emerge looking infallible. It emerged recoverable, which on a day involving 4,237 vanished files is the more practical distinction.