The Cheaper Judge Has Different Opinions
Replicator's PDF pipeline spends most of its per-page model budget on one recurring judgment call. For every region the vision stage finds, something has to decide: reuse an existing widget from the org's catalog, or build a new one. The shipped judge is a frontier model, and it takes about ten seconds per page. A typed-decision model called Jev answers exactly that shape of question, one choice from a list plus a build sentinel, in a fifth of a second, for about a thousandth of the cost.
Study AU, the third whole-system study in our Barkup Bench series after AS and AT, asked whether we could swap it in. The important part is how the question was asked. Not "is the cheap judge good," which invites a vibe, but "is it the same judge": two registered phases, where the expensive full-pipeline phase only runs if a cheap offline check passes first. We replayed frozen decompose outputs from Study AT, 102 judged regions across 16 pages, through both engines fresh, byte-identical text inputs, identical deterministic guards, and measured decision agreement against a registered bar of 85 percent.
The Gate Failed and Saved Twenty Dollars
Agreement came in at 67.6 percent. Per the registered decision rule, Phase 2 never ran, the frontier judge stays, and the study cost about a dollar and a half of the expensive model's time instead of another twenty of full pipelines.
This is the discipline angle, and it is the story. A registration that declines to fund its own second phase when the precondition fails is the system working. And the engineering is not wasted: the engine switch ships in production dormant, default unchanged, guards identical, as the re-test apparatus for the day the question gets re-asked. An experiment that fails its gate should leave behind the equipment to run the next one cheaply.
The Failure is Not Noise, it is Policy
Here is why the number is more interesting than a miss. Of the 33 disagreements, 31 lean the same way: Jev reuses where the frontier judge builds. Jev's reuse rate is 45 percent against 17. And on the pages where both engines said reuse, they picked the identical widget 15 times out of 15.
That is not a noisy judge. Jev is confident, with a median probability of 0.71 on its extra reuses, and internally consistent. It simply holds a looser matching philosophy. Nearly half of its extra reuses are heading-like regions matched to a generic heading widget, exactly the case where the prompt's rule, present in both engines' inputs, says to judge by purpose rather than shape and not to stretch a loose match. The frontier judge declines those matches. Jev takes them. Same inputs, same rules on paper, different jurisprudence.
A validation detail rode along that makes the comparison trustworthy: replayed against its own recorded decisions from two days earlier, today's frontier judge agreed with itself 99 percent of the time despite catalog drift in between. The replay method is sound, the self-consistency is real, and therefore the disagreement is real too.
What this Study Cannot Tell You
The honest open question is named in the report and deliberately not run: nobody knows yet whether the strict policy is the right one. The frontier judge built 85 new widgets across the baseline study. If reusing a generic heading widget is actually fine, then strictness over-builds at real cost, and the cheap judge's looser philosophy might produce better pipelines, not worse ones. Answering that requires scoring full pipelines downstream, which is a different registration, "which policy is right," than this one, "is the engine interchangeable."
That distinction is worth closing on, because it generalizes well beyond our pipeline. An A/B that measures agreement can tell you that two judges differ, cheaply and decisively. It cannot tell you which one is wrong. Those are different studies, and pretending the first one answers the second is how a cost optimization quietly changes your product's opinions.
The registration, raw decision records, and full report live in the barkup-bench repository, and the Study AU page is landing on the research dashboard beside AS and AT.
