For a month, barkup-bench has graded every claim the same way: compute the expected document, compare byte for byte, and let unit-tested code decide. Twenty-one studies of exact edits. But the question that started our last two studies, whether an editing agent without memory can still do strategic work, has a half we had been avoiding because our graders could not reach it: requests like "rewrite the third paragraph so it focuses on our central thesis." There is no byte-for-byte answer to more focused. Somebody has to judge.
Study V is our first judge-graded study, and we spent most of its design budget making sure the judge could not quietly become the weakest link in a series that pre-registers everything. Here is how we kept our own rules, and what the judge saw when the grading finally started.
The judge takes the exam first
Our hard rule has always been that graders get unit tests, because a benchmark with an unvalidated grader measures nothing. An LLM judge gets the same treatment. Before it graded a single real rewrite, each judge faced a committed, seeded exam: thirty pairs where one rewrite is on-thesis and the other provably is not, ten pairs where both sides are literally identical (a fair judge must tie), and ten pairs where the longer rewrite is the worse one (a length-biased judge fails). Every comparison runs twice, in both presentation orders, and only order-consistent answers count. The pass bar was pre-registered, and failure meant the study halts before spending.
Both judges, a frontier model and a small one, neither of which wrote any rewrite being judged, went fifty for fifty. Clean sweep, position bias zero out of ten, length bias zero out of ten. Only then did the scored runs begin.
The setup: a page with a planted problem
The corpus is thirty fictional company About pages, assembled by a seeded generator from committed sentence pools: a mission statement carrying the company's thesis, two on-thesis paragraphs, and one paragraph deliberately built from a different company domain's sentences. An alpine safety company whose third paragraph is inexplicably about port logistics. The instruction is always the same: rewrite that paragraph to focus on the central thesis. What varies, in five arms, is where the thesis lives: stated in the instruction (the control), sitting in the mission node with a view that shows only the target, sitting in the mission node with a view that shows both nodes, declared in earlier conversation with the Study T memo, or declared in earlier conversation with nothing.
Every arm's rewrite then went head to head against the control's rewrite from the same model on the same page, judged pairwise.
What the judge saw
The blind arms lost every single comparison, 120 out of 120, and the way they lost matters. They did not invent a thesis. They polished the port logistics paragraph: tightened it, reorganized it, kept it entirely about freight, and never said a word about being unable to know the goal. Fluent, mechanically valid, useless. Our earlier study found silent invention when a model cannot see a value; this is its qualitative sibling, oblivious polishing, and it is just as invisible in an error log.
The memo carried the goal at full strength. With one note line in the application memo, the central thesis is such and such, rewrites tied the explicit-instruction control statistically, and on one model the memo arm actually won the head-to-head, ten wins to two. Study T's finding generalizes completely: the memo carries intent as reliably as it carries codenames.
The surprise failed our gate. In the arm where the mission node sat right in the view, the models clearly read it. Their keyword coverage of the thesis jumped from zero to three quarters, and every rewrite was about the right subject. And they still lost 117 of 120 comparisons. Put the winning and losing rewrites side by side and the difference is visible to the naked eye: the control opens with the thesis, nearly verbatim, and builds from it; the view arm writes competent prose that orbits the topic without ever anchoring it. Reading a goal is not the same as being told one.
What this means for builders
The last study's rule was that focus ids must include every node the request mentions, because views that hide a needed value produce silent fiction. That rule stands, and Study V draws its boundary: it is a rule about values, the data an edit must read. For goals, the intent a rewrite must satisfy, showing the model where the goal lives is measurably second-best. The recipe:
- Views carry values. Put every node the request mentions in the focus ids, so the model can read what it must copy, match, or preserve.
- Memos carry goals. When the user declares an objective, a thesis, a tone, a standing rule about how things should read, record it and restate it outright in the memo or the instruction. One note line buys full parity with the best case.
- Do not make the model infer intent from the document. It will read the goal and still under-deliver on it, and nothing in your logs will look wrong.
The honest caveats are bigger here than usual, and we state them loudly: this is judge-graded, not deterministic. The verdicts come from a pinned judge under a registered protocol, replicated by a second judge (83% agreement, no conclusion flips), triangulated by a keyword proxy that ranks the arms identically, and the whole track is labeled separately in the REPORT so it can never blend into the exact-edit claims. Two models, one task family, thirty pages, and one voided first run disclosed in the protocol notes (a grader bug we caught before a single verdict was scored). If you believe the judge is wrong, every pair it saw is committed in the repo, and re-judging them is one script.
