The Page That Scored 0.002
One page in our newest benchmark scored 0.002 out of 1. Not a crash, not a timeout: the pipeline ran perfectly, produced a clean, editable template, and the similarity metric looked at the result and saw almost nothing of the original. So what happened?
What we measured
Replicator, our CMS, ships a pipeline that turns any uploaded PDF into a native editable template. Six stages: a vision model decomposes the page into regions, a matcher decides which existing widgets to reuse and which to build, new widgets are authored under a brand-compliance lint, the pages are assembled, and a verify loop renders the result, scores its similarity to the original, and applies a critic's corrections with keep-best semantics.
Until last week, every number we had for that pipeline was an anecdote: two development runs on one PDF, final score 0.75. Study AS, the first whole-system study in our Barkup Bench series, replaced the anecdote with a registered baseline: twelve foreign-designed PDFs across three difficulty tiers, sixteen scored pages, four gates fixed before the first run, results published as measured.
The machinery gates passed emphatically. All twelve documents made it through all six stages with zero pipeline errors. Every one of the 85 newly built widgets validated through the lint loop, zero fallbacks. The cleanup contract held even after an external process kill mid-run: the partial artifacts were swept, the item re-ran from scratch, nothing leaked.
Both quality gates failed.
The 0.002 page
The worst cell in the study was the cover of a fictional annual report: a full-bleed dark green page. The pipeline rebuilt it as a clean cover in our client org's brand palette, because that is what the brand-compliance lint is for. It deliberately re-brands foreign palettes into the org's theme. The similarity metric then compared a dark green page against an in-brand one and scored the match at 0.002.
So the pipeline did not fail. It succeeded at the wrong contract. The metric measures pixel-faithful reproduction. The product promises something else: recreate this document in our brand. For the development PDF those two contracts were the same thing, because that PDF was already in the org's brand. For every foreign design they are different things. Our 0.75 anecdote was not measuring the pipeline. It was measuring a coincidence.
This is the quiet failure mode of development numbers. The one PDF you test with is never neutral; whatever it happens to share with your system becomes invisible in the score.
The loop that spends quietly
The second failed gate belonged to the verify loop. Across the corpus it consumed 43 percent of total wall-clock, 19.9 of 46 minutes, and bought a median improvement of 0.007. Fifteen of the sixteen pages ended with their final correction reverted by the keep-best guard.
The tempting reading is that the loop is broken. It is not. The guard did exactly its job, round after round: it tried a correction, scored it, found it no better, and rolled it back. The problem is upstream of the guard. The critic's corrections are layout-scoped by design, they move and resize blocks, and on this corpus the residual error is style: palette, backgrounds, typography. On the development PDF, which had gross layout errors, the same loop moved scores up by 0.2 to 0.4, because there the error class matched the correction class.
A quality loop aimed at the wrong error class does not fail loudly. It spends quietly.
What happens next
The study closes with three registered candidates for the follow-up. Derive the document theme from the source PDF's palette instead of forcing the org default, so re-branding becomes a choice rather than a silent rewrite. Give the verify loop a style-correction channel, or teach it to skip correction when the critic's findings are style-class, since two reverted iterations per page is the pipeline's largest cost for its smallest return. And replace the similarity metric with a palette-blind, layout-only score, so the two gaps this study had to pull apart narratively get separated by construction.
The house point, one more time, because this study is the cleanest demonstration we have had: the gates were registered before the run. An anecdote said 0.75. The benchmark said 0.634, and more usefully, it said exactly which part of the shortfall is a design policy doing its job and which part is a real layout deficit. We have shipped benchmark findings into production the same day before. This one hands the roadmap three items with the evidence attached.
The registration, corpus, raw records, and full report live in the barkup-bench repository, and the Study AS page is landing on the research dashboard.
