A Trade You Take
The last study in this series ended on a warning: an agreement A/B can tell you that two judges differ, but it cannot tell you which one is wrong. Study AU had found that a typed-decision model called Jev answers our PDF pipeline's reuse-or-build question for a thousandth of the cost while holding a visibly looser matching philosophy, and the registered gate refused to fund the expensive phase on a failed interchangeability check. The cheap judge was not the same judge. Whether it was a worse one, nobody knew.
Study AV answers that the only way it can be answered: run both policies through full pipelines and score what comes out the other end.
Reframing the Question Changes the Gates
"Is it the same judge" asks about agreement. "Is the difference free" asks about harm, and that is a different registration. AV ran the entire twelve-document corpus through the pipeline twice, identical in everything but the match engine, both arms on one version-pinned commit, and paired the results page by page. The gates do not mention agreement at all: every item completes, no fallback widgets, and the median paired structure difference stays inside a registered no-harm bound of three hundredths.
The Verdict
All gates pass. The permissive policy reused half the regions where the strict one reused a sixth, built 41 percent fewer widgets, ran the match stage thirteen times faster, and took a quarter off total pipeline time, at about a thousandth of the stage's model cost. The measured fidelity price for all of that disagreement: a median fourteen thousandths of a structure point, inside the bound.
Per the decision rule registered before the run, the recommendation ships. The cheap judge becomes the default; the frontier judge stays behind an environment switch as the rollback. A cost optimization is allowed to change the product's judgment now, because the change has a measured price and the price is acceptable.
Where the Impact is Noticeable
The honest texture is that the fourteen thousandths is not spread evenly, and our registered prediction about where it would concentrate was wrong in the instructive direction.
Study AU's anatomy pointed at heading-heavy, composition-dense pages as the risk: that is where Jev stretched its matches. Those pages absorbed the stretching best. The composition tier came out dead even, and the corpus's one genuinely positive divergence was its densest page, an agenda that improved by seven hundredths with nine of ten regions reused. The real casualties were the simple pages: a one-page memo dropped eleven hundredths, the only cell clearly past the bound on its own, and the simple tier as a whole was the weakest.
The lesson in one line: a loose match hurts exactly where a page has few regions to hide it. On a dense page, one stretched widget is a tenth of the layout; on a memo, it is half the page. The routed hybrid we had sketched, cheap judge for most regions and the frontier judge for the hard ones, stays unfunded because a green gate makes it moot. But if a routing threshold ever gets built, that memo cell is the number it would be built around.
Notes
Three decisions behind the result, briefly, because they are where the discipline lives. The study ran on a version-pinned clone of the app rather than the shared dev checkout, which was a moving target mid-refactor; both arms saw byte-identical code. The baseline arm was re-run fresh for about twenty dollars instead of reusing frozen records from the earlier study, citing our sibling research program's experience of a confound propagating silently through record reuse. And a registered stop rule about engine plumbing, whether each arm really ran its assigned judge when the records carry no engine name, was settled by a latency fingerprint: seconds per page on one arm, milliseconds on the other, with no overlap.
Three studies now make the arc. The first found the judges disagree. The second measured what the disagreement costs. The third shipped the decision, with a rollback. That is the shape we want every cost optimization to take on the way to changing a product's opinions.
The registration, both arms' records, the switch handoff, and the full report live in the barkup-bench repository, and the Study AV page is landing on the research dashboard beside its predecessors.
