One commit, one tree: the ancestry gate

What happened: the missing files

At 23:34 on 5 October we deployed a release of one of our client sites and checked it live. Every changed file matched the release byte for byte, nine of nine; the private work files we probe for were all hidden; five page loads showed no errors (R1). It looked clean.

The next morning our live crawl found the kiosk demo apps returning errors. The release had been built from a git worktree¹, a second working copy of the project, and a worktree holds only the files git tracks. About 4,900 media files (films, recorded voices, the demo apps' media) were deliberately left out of git², so they were never in that copy. For about 9 hours and 20 minutes those parts of the live site returned 404. We rolled back at 08:54 (R2).

Our check had confirmed that what changed was right. It had never asked whether anything that was live had gone missing. The fixed release was re-checked by counting files against the live site: 4,967 live against 4,968 in the new release, with the one missing file a test that had never been deployed and the extra two the intended new files, and every family of media equal, 1,902 films, 791 voices, 794 archival items (R3). It went live at 09:17 with every media probe returning 200 (R4).

What happened: the missing fixes

Three days later the same shape of mistake came back on Museum Mosaic, this time in the commits rather than the files. A search update (new titles, descriptions, structured data and a sitemap) was checked at 09:42 and passed: 468 pages in the sitemap and 468 on the site, every title unique, no fact without a source (R5). It carried approvals from both of our reviewers.

At 13:03, before it was published, the site's owner held it. The update had been built on an older commit, not the one that was live. Git's own test for this, merge-base with --is-ancestor, which exits with status 0 if the first commit is an ancestor of the second and 1 if not³, answered no. Publishing it would have reverted four fixes already live: the home tab bar, the Stories tab link to the magazine, an overlay in the Memory Match game and a retry button on the story list. 488 files differed (R6).

The cause was one sentence. The build note said the update was "from live", and the person who built it had taken the base from a checkout that held a staged candidate, not the published release. Every approval had rested on that sentence; none had asked git (R8). The approvals were withdrawn that afternoon, and the builder, the checker and an approver each wrote up the miss as their own.

How the gate works

At 13:04 we adopted one check for every review and every publish: a short shell script that takes the repository, the live commit and the candidate commit, and runs git's ancestor test (R7). It answers in one of three ways. PASS means the candidate contains what is live. FAIL means it does not, and it prints how many files differ. The third answer is "did not look": the repository or a commit is missing, so no claim can be made either way. That third answer matters as much as the other two: a check that cannot run is reported as such, never as a pass.

We proved the gate before trusting it, by planting cases with known answers: the held update had to FAIL, and did, on 488 files; a tree that did contain live had to PASS; an unknown commit and a folder that is not a repository both had to say "did not look" (R7).

How we optimised the approval line

The same afternoon the gate ran over every other update waiting to publish (R9). Two passed. One could not be tested, because its folder had been built from a file copy with no git history linking it to live; a content check in its place found a live redirect fix that sat outside the folder and would have been lost. One was passed by argument: the reviewer reasoned that live's only extra change was to the stylesheets of two pages, which the new version had already fixed in its own source. That pass was ruled not a pass. The gate as adopted asks whether the candidate contains live, and that live build was a hand-assembled payload with no commit at all, so there was nothing to test against. The owner was asked to commit it first.

Since then our approval lines name the commit the candidate was built on and carry the gate's answer. Between 13:15 on 9 October and 12:44 on 11 October, 55 lines in our studio log carry a gate PASS (R10). Some builders also plant the reverse case on their own work, testing the older commit against the newer and expecting a FAIL, so the gate is seen to fail when it should.

What we do better next

These are plans, not yet measured. First, the gate moves inside every publish script, before the step that checks the live version number, so a release cannot be published around it. Second, every live build gets a commit, including hand-assembled ones, so "did not look" is never the answer for a live site. Third, the file count against live that fixed the first case becomes a standing step beside the commit gate, because the gate sees commits and cannot see files git was told to ignore.

The rule, for anyone approving a release

Before you approve a new version, make sure it was built on top of what is live now, not on an older copy; a version that looks right but was built on an older copy can quietly undo fixes. Ask for the proof, not the sentence: "built from live" is a claim, and a passing ancestry check plus a file count against the live site is the evidence.

Limits

The two cases are from our own studio in one week; they show the failure exists, not how often it happens elsewhere. The 488 is the number of files that differed, not a count of broken pages. The count of 55 is a text search of our log for the gate's PASS wording, so a PASS written in other words is missed and a line quoting another line's PASS may be counted. The ancestry gate does not see files that git ignores; the first case was caught by a file count, not by the gate. The three steps under "What we do better next" are plans and unmeasured.

Appendix: the record, 5 to 11 October 2026

Sources

1. https://git-scm.com/docs/git-worktree
2. https://git-scm.com/docs/gitignore
3. https://git-scm.com/docs/git-merge-base