barkup-bench is our open, pre-registered benchmark series on how LLM agents should edit structured documents. Two studies ago it produced the series' most quotable result: the silent guessing that haunted twenty-eight studies was never blindness, it was obedience. Give a model one sentence of permission to ask, and it asks on every provably-unsolvable task, names the exact missing node, and never asks when it has what it needs (the write-up).
We shipped that sentence to production surfaces the same week. And we shipped it with two disclosed fears, because the study that validated it had tested it only on its home turf, where every unsolvable task was provably, unambiguously unsolvable. Fear one: on ordinary vague requests, the hatch might turn an editing agent into an interviewer, taxing every user with questions the model could have answered itself. Fear two: we had graded the asks but never the aftermath. An ask that dead-ends, or that the model fumbles after the user replies, would be worse than no ask at all.
Study AE tried to realize both fears on purpose.
The ladder
The experiment walks the shipped sentence, frozen verbatim, down a five-level ladder of ambiguity. Fifteen tasks per level, every level's defining property validated by unit tests before any model call, two arms (with and without the hatch), three models.
At the bottom, level zero: precise, fully solvable requests, where asking would be pure tax. Level one: the target named indirectly ("the text-atom whose maxLength is 114") but uniquely, resolution the model's job, value stated. Level two: the target named exactly, the value left to judgment ("make the content of n144 punchier"), where a literal-minded hatch might demand a style brief. Level three: a singular request whose descriptor matches exactly two visible nodes with different current values, so no amount of reading resolves which one the user meant. Level four: the value provably absent, the construction from the original study.
The correct behavior is a step function: act on the bottom three levels, ask on the top two.
Separately, the resume loop: forty-five unsolvable tasks per model where, when the model asks, the harness answers as the user would, in one plain sentence, and the conversation continues. No view re-attachment, no ceremony. Just the answer, the way a person types it.
The two fears, refuted at ceiling
The tax never appeared. Across ninety clear-request cells with the hatch present, three models produced zero false asks, with solve rates identical to the no-hatch control. The indirect references got resolved, not questioned. And "make it punchier" got edited on the spot, forty-three times in forty-five, with no interviews about tone (the frontier model asked about wording twice; we disclose it and shrug). Whatever image you had of a permission-slip sentence turning your agent into a bureaucrat, the data declines to support it.
The dead-end fear did worse. Every single ask, once answered, became a correct edit: 135 of 135 across all three models, zero repeated questions, zero cases of the model taking the answer and applying it wrong, zero correction rounds needed. The model asks about the value it cannot see, receives one conversational sentence, and lands the exact patch. If you have been hesitating to wire an ask path because you weren't sure the second turn would work, stop hesitating. The second turn is the reliable part.
The crack, and it is not where anyone looked
Level three is where the study earns its keep. Two visible nodes match the request equally well. The current values differ, so the choice matters. The request is singular, so editing both is wrong, and reading harder does not help, because the ambiguity is in the request, not the document.
The frontier tier does exactly what you would want. Opus asked on fifteen of fifteen, and every one of its questions named both candidate ids: which of these two did you mean? That is a perfect score on a construction it had never seen, from a sentence that does not mention ambiguity at all.
The mid tiers do not. Sonnet and gemini, with the identical sentence in the identical prompt, asked once each in fifteen. The other fourteen times they resolved the ambiguity themselves, and they resolved it differently, which is its own small horror. Sonnet's habit is to edit both matching nodes, a defensible reading of a plural that this request is not. Gemini's habit is to silently pick one, the pure coin-flip, valid and applied and wrong half the time by construction.
The mechanism is almost embarrassingly textual. The shipped sentence grants permission to ask about what is "not visible in the view and not stated in the request." An ambiguous referent is entirely visible. Opus generalized the rule's intent: when I cannot know what the user wants, ask. The mid tiers applied its letter: I can see everything, so I must patch. If that pattern sounds familiar, it should. Our conflict studies found the same shape, with the same models: the strongest tier reads intent where weaker tiers read text. Knowing when a question is the right answer is, on this evidence, a capability, not a prompt feature.
What this changes for builders
If you run a frontier Claude tier, the ask path you shipped just got better than advertised: it covers missing information and ambiguous references, it costs nothing on clear requests, and the answer loop is measured at ceiling. Wire the reply path and trust it.
If you run below the frontier, or your traffic can fall back there, the fence matters: the hatch catches absence, not ambiguity. Requests that could mean two things will get silently resolved, not questioned, and your only defense is making ambiguity impossible upstream: unique references, deterministic enumeration for bulk edits, selection grounding so the app knows which node the user is looking at.
And the obvious fix, adding "or if the request matches more than one node, ask which" to the sentence? It stays unshipped until a registered test passes it. We have been burned before by prompt clauses that looked obviously right: our conflict studies measured a priority rule everyone would have bet on, and it did nothing. Obvious is not measured.
The usual caveats travel with the numbers: three models, one grammar, seeded corpora, fifteen cells per level per model, deterministic grading throughout. Everything is reproducible from the benchmark repo, pre-registration first, and the chart set lives on the research dashboard. Thirty-one studies in, the ask path has graduated from a promising sentence to a mapped territory, with one honest border drawn where capability ends.
