All posts

10 min readPost 3

Twelve review rounds: how to trust a tool that deletes files

On 8 November the three phases of Obituary went through 40 review rounds: finish a phase, have reviewers read it, verify every finding, fix with tests that fail first, then bring in fresh reviewers until a round turns up nothing serious. How the loop runs, what each round was looking for, and when to stop.

On this page
  1. How a round runs
  2. What each round was looking for
  3. When to stop
  4. How we know it is fixed

On 8 November the three phases of Obituary went through 40 review rounds. From one to seven in the morning we ran 12 rounds on the first phase, in the morning 10 on the delete guard, and from the afternoon until early the next day 18 on the MCP server and the Skill. The method was the write, review, fix loop we had settled on the first night, and that day it was pushed as far as it would go.

PhaseContentRoundsFrom (8 Nov)
M1Automatic registration, the graveyard, integration with Claude Code and Codex, the background runner12about 01:00 to 07:00
M2aThe delete guard10about 07:55 to 12:35
M2bThe MCP server and the Skill188 Nov 13:28 to 9 Nov 09:40

In round 1 of the first phase the two reviewers reported 34 findings. Round 2 had 18, and round 3 dropped to 6, plus 4 that we spotted ourselves. From round 4 to round 11, apart from the cluster in round 7, each round had 7 to 9, until round 12 had no CRITICAL and no HIGH. The curve did not fall smoothly to zero. The early rounds found conspicuous mistakes, the middle ones found interactions between earlier fixes, and the later ones found increasingly strange corners. The 18 rounds of the MCP phase took about 20 hours, which is the most direct cost of the method.

Bar chart of findings per review round in the first phase: 34 in round 1, then 7 to 9 per round, a hatched bar for round 7 where a cluster of related problems forced a redesign, and 0 in round 12
Findings per round in the first phase: the count falls from 34 to 7 to 9, the cluster in round 7 led to a redesign, and round 12 had nothing serious left.

How a round runs ​

A round starts with the review. Two or three Codex reviewers read the code, read-only, each from a different angle: deletion safety and identity checks, integration and installation, and the facts in the documentation together with regressions and tests. Sometimes two reviewers independently report the same problem. We call that a shared finding, and it usually gets fixed first. Claude then opens the code for every finding, because reviewers are wrong sometimes and contradict each other: one complains that the guard stops too little, another that the command it suggests selects more than the original. Only findings that hold up go on the fix list. Codex implements the fixes, writing a regression test that fails first and only then changing the code. Claude reads the diff, runs the tests and then the mutation checks to confirm that the tests really catch the change, and fresh reviewers read the fixed version. A phase passes only when a round has no CRITICAL and no HIGH.

What each round was looking for ​

In round 7 of the first phase, the review found that the half-finished state of a burial was being kept on the ledger entry, and that this led to a string of related problems. Rather than patch one more hole, we moved that state out of the ledger altogether, into a small manifest next to the grave (the previous post describes it), and then tested recovery after an interruption at each of 63 points where the process could be cut off.

Round 3 of the delete guard produced the CRITICAL. The command was rm -r repo || /usr/bin/rm -rf backup. In the original, the fallback after || runs only if the first rm fails. Once the guard replaced the first rm with obit rm, Obituary could fail where rm would have succeeded, for instance by refusing a path, and then the fallback, a path-qualified rm -rf backup that is never intercepted, would really run and permanently delete backup. The fix was to tighten the rule: a deletion that is replaced must be the last segment of the command, preceded only by && or ;, and anything with ||, a pipe or a background operator passes through untouched.

Round 5 had the opposite problem, selecting too much. Take the path missing/../keep.txt, where missing does not exist. The real rm fails because that directory is not there, but if an equivalent command first normalised the path to keep.txt and buried that, it would bury a file the original never meant to touch. The fix was to check a path containing .. level by level, to see that each level exists, before normalising it, and in bash to give no equivalent at all and let the command pass. The same round settled a discipline, a table of main scenarios that runs in every round. Round 8 found no more over-selection, and both HIGH findings were protection failing. When the audit log could not be written, a deny that had already been decided was cancelled, and when the pre-scan of wildcards ran over budget or the current directory could not be read, a command that should have been denied was let through. The last thing a guard should do when something goes wrong is switch itself off.

Round 1 of the MCP phase had 11 findings, one of them HIGH. We edited ~/.claude.json directly, but Claude Code rewrites that whole file while it runs, so the two could swallow each other's writes. We changed to the official claude mcp add and remove, and now only read the file to decide whether an entry is ours. The medium findings of that round are representative as well. A resurrection that never ran was reported as a success with one warning, a request cancelled by the client still executed once it got the lock, in recycle mode the description of obit_bury still said the item could be restored, and a broken project .mcp.json meant even the hooks could not be uninstalled. The rounds after that found stranger things: 8.3 short names, alternate data streams, K and the Kelvin sign treated as the same letter under Unicode folding, volume identity, ownership markers on the graveyard root. Each was correct as logic, but triggering it required someone to construct such a name on purpose.

When to stop ​

Review has no end. As long as there are reviewers, they will find something, so the stopping rule has to be decided in advance and graded by consequence, not by whether the reviewers can still find anything. Late in the MCP phase we noticed that more and more of the HIGH findings needed extended-length paths, names constructed inside WSL or particular Unicode folding to trigger, and even when triggered they only left an item in the graveyard, still restorable, instead of permanently deleting the wrong thing. The next day we wrote the rule down as a grading. Only a finding that can delete, move or overwrite the wrong object counts as HIGH and must be fixed. A finding that can only be triggered by extended paths or names constructed in WSL, and that cannot permanently delete the wrong thing, counts as MEDIUM or LOW. Small fixes and test-only changes get a single round. Review cannot prove the absence of errors; it only raises the bar for getting through, and the height of the bar has to be set by consequence.

How we know it is fixed ​

"It is fixed" is the easiest sentence to fool yourself with, and we use three things to keep it honest. Turning a finding into a regression test matters more than fixing it, because a fix covers this one time and the test covers every time after. So before changing the logic of the guard, of locating, settling, committing or recycling, we write a failing test that proves nothing will be deleted by mistake, and after the fix we temporarily revert it and confirm the test really goes red. The reverted version has to compile, since a build failure does not count as verification, a rule that dates back to the small detour on the first night.

The second is the mutation check. With go test -overlay we break one key decision, for example by turning a condition into one that is never true, recompile, and see whether the test responsible for that spot fails. Every entry in the mutation tables says which file and which piece of it is changed, to what, and which test in which package is supposed to catch it. There are two tables now, 516 entries for the core gates and MCP and 138 for the delete guard, 654 in all. On this machine, timing two of them, going from the source edit through the build to the corresponding test catching it took 3.9 and 4.8 seconds, and the terminal output of the second is below. CI splits them by index across 8 machines.

Terminal screenshot: the same test passes on the unmodified code and fails on a mutant with one decision broken, and both commands are timed with time
The same test passes on the original code and fails on the mutant with one decision broken, so the mutant is caught; the second run took 4.8 seconds from edit to failure.

The third is the main-scenario table of the delete guard, which runs on every round and every change so that the most common ways of writing a deletion stay covered.

The loop left two habits behind. Tests are named for the feature and not for the review round, because a test file named after a round only records which round wrote it and not what it guards, and what each round fixed and why goes into the commit message. Documentation describes today's rules and the reasons and not the history: the architecture document had accumulated passages like "round 11 tried this and backed out", which we rewrote into the current rule plus its reason, and the history stays in commit messages. At the end of each phase we also do a pass that changes structure and not behaviour, splitting large files and moving end-to-end tests into their own package, and the rule is that the number and results of the tests are the same before and after.

All posts