The Explanation That Lasted a Day
In our last study we published an explanation of our own failure. It survived less than a day, and the thing that killed it was the next study's registration. We want to tell that story carefully, because the explanation dying is the system working.
Where We Left Off
Study AS benchmarked the pipeline that turns an uploaded PDF into a native editable template, and its quality gates failed in an instructive way. A full-bleed dark cover page scored 0.002 because our brand lint deliberately re-brands foreign palettes, and the similarity metric read the re-branding as total mismatch. We published a diagnosis with the results: the fidelity gap is genuine layout deficits conflated with a pervasive re-branding penalty, and we named three fixes.
The product shipped all three that same evening. The verify loop got a palette-blind structure metric so it judges layout separately from appearance. The importer got a "Match source colors" option that derives the document theme from the source PDF's own palette. And the critic became brand-mode aware, so it stops reporting policy as defect.
Then Study AT did what the discipline requires: it turned our diagnosis into a registered prediction and measured it.
The Prediction Was Wrong
If the re-branding penalty is pervasive, giving the pipeline the source palette should recover most of it. We registered that recovery: with "Match source colors" on, at least 12 of 16 pages should beat the Study AS baseline. Reality returned 4 of 16. Against the contemporaneous default-theme arm, the cleaner pairing since build stochasticity makes cross-run comparisons noisy, it was 10 of 16. Either way, the hypothesis our own explanation implied did not hold, and the gate is published as a fail.
The re-branding penalty is real, but it is concentrated, not pervasive. On the fifteen light-background pages, swapping the org theme for the source palette moved appearance scores by hundredths. And the one catastrophic page, the dark cover, stayed at 0.002 with the correct palette in hand.
The diagnosis of that cell is the study's best moment. Palette extraction was flawless: the derived theme carried exactly the right dark green and the right gold. But nothing paints the page surface with them. The rebuilt widgets sit on a white page, because page-background painting turned out not to exist at the assembly stage at all. Not broken, not mis-tuned: absent. Until that ships, "Match source colors" changes accents and tints, not page surfaces, and the product copy should promise exactly that much.
So the corrected story, one study later: the dominant residual gap was layout and density all along. The 0.002 page was never evidence of a pervasive color penalty. It was one missing feature wearing a dramatic costume.
The Rebuilt Loop, Measured on Its New Metric
Study AS's other finding was a verify loop spending 43 percent of wall-clock to buy a median 0.007, with corrections surviving on one page in sixteen. The rebuilt loop, judged by the new structure metric, kept corrections on eleven to twelve pages of sixteen at 2.4 to 3.7 times the uplift, and the decoupling worked exactly as designed: the dark page's layout score climbed from 0.626 to 0.738 while its appearance score sat at 0.007. Layout truth and palette policy no longer contaminate each other.
It still failed its registered bar, a median uplift of at least 0.03 against the measured 0.017 and 0.026. The bar was guessed with no prior structure-uplift data, and the report owns that. It stays a fail on the books, and the bar gets re-registered only after the upstream fixes give the loop pages whose errors it can actually correct.
Five Studies, Five Gate Failures, One Direction
This research program has now failed gates by letter in five studies while every one of those measurements moved the product forward. That is not a coincidence we apologize for. It is what registered gates are for: they hold against wishful thinking, against vendor claims, and, this week, against our own published narrative. Study AS's explanation was plausible, well argued, and wrong in its central adjective, and it took one registered prediction to find out.
The roadmap the two studies leave behind is concrete. First, deterministic page-background painting at assemble time, which closes the catastrophic cell and delivers what "Match source colors" already extracts. Second, layout fidelity at the decompose and build stages, where compositional pages start at structure scores of roughly 0.5 and the real headroom lives. Third, re-register the loop's bar against data instead of a guess.
The registration, paired records, and full report live in the barkup-bench repository, and the Study AT page is landing on the research dashboard beside AS.
