Study AU · Whole-system baselines
The match-stage engine A/B: Jev is not a drop-in judge, it is a looser policy, and the gate declined to fund finding out more
Whole-system baselines
Study Overview
The Question and the Gated Design
Replicator's PDF-pipeline match stage decides reuse-versus-build for every decomposed region: one fable-5.1 call per page over a text-only prompt, wrapped in deterministic guards. Jev (typesafe-ai jev-1.14.0, a typed-decision model at about $0.042 per million input tokens) answers exactly that shape, one choice per region over the candidate list plus a build sentinel, and if quality held, the stage's model cost would fall about a thousandfold and its latency from seconds to sub-second. The registration bought the answer in two gated phases: Phase 1 (about $2) replayed the frozen Study AT brand-arm decompose outputs, 12 items, 16 pages, 102 regions, judging every page with both engines fresh on byte-identical inputs under identical guards; Phase 2, the full-pipeline arm at about $20, would run only past a registered agreement gate of 0.85 on actions and 0.90 on exact widgets.
The Gate, Published as Measured
AU-G1 failed: action agreement 0.676 against the registered 0.85. Per the registered decision rule, Phase 2 did not run, fable-5.1 remains the match default, and the roughly $20 full-pipeline arm was never spent. That is the quiet discipline angle worth naming: the gate's job was to decline to fund a deeper experiment on a failed interchangeability check, and it did exactly that.
One Direction, Same Widget
The texture of the failure is the finding. Of 33 disagreements, 31 run the same direction: Jev reuses where fable builds, with overall reuse rates of 45.1% against 16.7%. When both engines say reuse, they chose the identical widget 15 of 15. And Jev is not noisy: its extra reuses carry a median probability of 0.71. It simply holds a looser matching policy. Nearly half of its extra reuses (15 of 31) are heading-class regions matched to a generic heading widget, exactly where fable's judge-by-purpose rule, present in both engines' inputs, makes fable decline. The two reverse cases sit in Jev's own low-confidence zone (p 0.45 to 0.53). The engines are not an accurate one and an inaccurate one; they are two policies, one strict and one permissive, applied consistently.
The Replay Validates Itself
A protocol check rode along: judged fresh today on the same frozen inputs, fable agreed with its own AT-recorded decisions on 99.0% of regions despite two days of organic catalog drift. The offline replay method is sound, and the shipped judge is highly self-consistent, which is itself a useful property to have measured for a stage that decides what gets built.
The Economics for the Record
Why the question was worth asking: Jev judged pages in 196 milliseconds mean against fable's 9,580, a 49-times speedup, and the entire 16-page sweep cost about $0.003 of Jev input against about $1.50 of fable. If the policies had agreed, the switch would have been free money. They did not, and the price of knowing was about $2.
What This Does Not Settle
The study names its own open question rather than quietly answering it: whether Jev's permissive policy produces BETTER pipelines. Fable built 85 widgets across Study AS, and if generic-heading reuse is actually fine, the strict policy over-builds at real cost on every run. Answering that requires a full-pipeline arm scored on downstream structure and appearance, which is exactly the Phase 2 this study's gate declined to fund on interchangeability grounds; a future registration could re-ask it as which policy is right rather than whether the engines are interchangeable. Until then, the replicator-side engine switch and the Jev judge ship dormant, default fable, guards identical, as the ready-made re-test apparatus.