barkup-bench is our open, pre-registered benchmark series on how LLM agents should edit structured documents. This entry is a self-audit, and it is the purest one we have run, because the mechanism under test was not a hypothesis we shipped and doubted. It was a commitment we believed in so thoroughly that in thirty-five studies we never once thought to vary it.
When a model's patch fails validation, barkup returns the validator's structured issues verbatim and asks for a corrected patch. The issue code, the offending node id, the path, a human-readable message per problem. This has been baked into every arm of every study, it closes guideline 01 of our playbook, and our production guidance states it as a rule: send the issues back verbatim. The belief behind it is intuitive to the point of feeling unfalsifiable. Of course a model corrects better when you tell it exactly what was wrong. Nobody debugging anything has ever asked for a vaguer error message.
But intuitive is not measured, and this particular mechanism had a distinguishing property: it was the only shipped mechanism in the whole architecture that had never been an experimental variable. Study AJ fixed that.
Seeding the failures
There is a reason the correction loop went unmeasured for so long: it rarely triggers. First-pass patch validity in this series runs 84 to 99 percent, so a study that waits for organic failures spends almost all its budget on cells that never reach the loop. The fix is to stop waiting.
We took 45 tasks from the main corpus, derived the known-correct anchored patch for each, and corrupted it with a registered operator from a pre-committed matrix: a dangling node id, a missing required field, a malformed op kind, a placement anchored to a node that can never be a sibling, an unknown attribute key. Each corrupted patch was validated to fail the shipped applier before any scored call. Then the trick that makes it a correction loop rather than a quiz: the corrupted patch was injected into the transcript as the model's own prior turn. From the model's perspective, it made this mistake, and the app is responding.
The response is the entire experiment. One feedback message, then one reply, graded. Three arms, differing only in that message:
The structured arm sends the full issues, verbatim, exactly as every prior study did. The codes arm sends only the issue codes, a diagnostic skeleton with the ids and messages stripped out. The bare arm sends thirteen words: "The anchored patch was invalid," plus the shared retry instruction and nothing else. No codes, no ids, no hint of what went wrong.
The pre-registered gate: the commitment earns its keep if structured feedback significantly beats bare feedback on at least two of three models.
It didn't matter
Structured beat bare on zero of three models.
Opus recovered all 45 seeded failures in every arm. Given nothing but "the anchored patch was invalid," it looked back at the task, the tree, and its own broken patch, and produced the correct edit 45 times out of 45.
Gemini is the sharper result. It scored 42 of 45 in all three arms, and they are the same three cells every time. Zero discordant pairs: not one cell anywhere in the paired comparison where feedback quality changed the outcome. The three misses all sit in the one genuinely hard corruption class, bad placement anchors, and no amount of diagnostic detail rescued them, and no absence of it created new failures.
Sonnet produced the only gradient in the study, 45 with structured issues, 44 with codes, 42 with bare, and it is not statistically significant. Three discordant cells on a 45-cell corpus is a whisper, and we report it as one.
The class breakdown says the same thing from another angle. Pooled across models, four of the five corruption classes were recovered at 115 of 117 or better in every arm, including bare. Only bad anchors were hard, at 15, 14, and 14 of 18, and the arms are within one cell of each other there too. Difficulty lives in the corruption, not in the feedback.
The mechanism is not mysterious. A failed patch plus the original task plus the original tree is an overdetermined system. The model does not parse your diagnostic and surgically repair the broken field; it throws the broken patch away and re-derives the edit from the request, which it can do because the request and the tree are still right there. The error message is not an input to that process. It is a starting gun.
The failure that runs the wrong way
One pre-registered hypothesis predicted bare feedback would be worse in a specific, dangerous way: without diagnostics, models would flail into patches that are valid but wrong, the silent-failure mode this series keeps finding elsewhere. That is the one outcome worse than not recovering.
It ran the other way. Every arm produced exactly three valid-but-wrong cells, the same count with full diagnostics as with none. Bare feedback's extra failures, the handful it has, stayed invalid: visible, loggable, caught by the applier. Thin feedback did not push models toward plausible wrong fixes. It just occasionally failed to fix, out loud, where your own validation layer sees it.
Keep it, for the humans
So the commitment survives, with its rationale corrected, and the correction matters more than the survival.
Return the issues verbatim because it costs nothing and because correction transcripts get read by people. An engineer debugging a failed edit at 2am wants the log to say which node id was dangling, not "the patch was invalid, and then it got fixed somehow." That is a real reason, and it is sufficient. What we can no longer say, because we measured it, is that error-message quality is load-bearing for recovery. If a future constraint ever forces a choice between polishing error messages and any guardrail this series has actually validated, the focused-view contract, the precedence clause, the ask hatch, app-side fan-out decomposition, the error messages lose, on the data.
The caveats are disclosed and real. Seeded single-op corruptions are a constructed proxy; organic failures may be weirder, though they are also rarer than this design assumes. Recovery was single-shot, where the shipped loop allows three rounds, and extra rounds could only narrow a gap that already measures as parity. One grammar, one patch dialect.
The meta-lesson is the one this series keeps paying us for. Study AF found that a restatement ritual we shipped did nothing. Study AD found the worked examples we thought were essential are training wheels the frontier tier does not need. Now Study AJ finds the mechanism we were proudest of, the humane, detailed, verbatim error message, is for us, not the model. Three different corners of the architecture, one repeated shape: the parts of a system that feel most obviously valuable are precisely the parts nobody thinks to test. Give your best-justified commitment a control group. Ours has been riding along, unexamined, in every study since the first, and it turned out to be a comment addressed to the developers, written in the voice of an API.
The full pre-registration, corpus, seeded-failure generator, and raw analysis are in the repo; charts and the arm table are on the dashboard.
