Undo That: The One-Line Echo That Replaces a Transcript
barkup-bench is our open, pre-registered benchmark series on how LLM agents should edit structured documents. Over twenty-three studies it assembled a session recipe that runs without conversation memory: a fresh view each turn, two worked examples, and an app-maintained memo of everything the user declares, which the agent itself writes faithfully. But every request in every one of those studies shared a property so ordinary we barely noticed it: each one made sense on its own.
Real editing chats don't work that way. The single most common thing a user types after an edit lands is a pointer at the edit itself. "Also make that same node bold." "Do the same to the footer." "Actually, undo that." The word that has no meaning in the document, no meaning in the memo, no meaning anywhere except the flow of the conversation, and our whole stateless recipe had never once been asked to resolve it. Study X asked, on purpose, in the way the series always does: build the failure structurally, then race the cheapest fixes.
The setup: nothing to point at
The corpus plants predecessor and follow-up pairs: a normal, self-contained edit ("set the aspect ratio to 16:9 on the image named maple-ember"), immediately followed by one of three anaphora kinds. Amend targets the same node again without naming it. Repeat applies the same unnamed change to a different, named node. Undo reverses the previous change without restating anything, including the value to restore.
One design choice does all the work. On anaphora steps, the model receives only a skeleton view of the document, because a view focused on the right node would leak the answer. So the target, the attribute, and the value must arrive through whatever carries the conversation, and we raced four carriers: the full transcript, a window of just the last two exchanges, a single app-written line, and nothing at all.
Nothing at all resolves nothing at all
The no-carrier arm went zero for 144, across all three models, including the strongest one. And in the pattern this series keeps finding, not one of the 144 was a refusal or a clarifying question. Every single miss was a valid, well-formed, applied patch aimed at a plausible guess. Asked to undo a change it had never seen, each model confidently changed something. This is the strongest silent-failure result in twenty-four studies, and it lands on the request class your users type most.
The echo beats the transcript
Full history is the obvious fix and it works: essentially perfect everywhere. But the interesting arm is the cheapest one. The application always knows what it just applied, so it can append a single line to the next request with no model in the loop and no judgment required:
Previous edit (applied by the app): set "aspectRatio" from "4:3" to "16:9" on the image-atom with id "n42" (named "maple-ember").
That one line tied the full transcript on every model, statistically, at about half the transcript's cost. The from/to shape matters: because the note carries both values, "undo that" becomes fully answerable, and every undo cell under the echo passed on every model. And then the result we did not predict: on claude-opus-4.8, the tier our production system runs, the echo went a perfect 48 of 48 while the full transcript dropped two undo cells to 46. On the strongest model we tested, one honest line beat the entire conversation it summarizes.
The fine print, disclosed as always. "Apply the same change to X" is the strain point for compressed carriers: one model managed only 7 of 12 with the echo while the other two handled it, and the two-exchange window leaked on that same kind on every model. Carrier advice, like every serialization result in this series, is not model-independent; test on the tier you ship. And if a surface routinely sees "same change everywhere" requests, that's fan-out wearing a trench coat, and the measured answer remains app-side decomposition, not anaphora resolution.
The recipe, complete
With Study X, the measured stateless session stack is finished, every clause with a number: a fresh view per turn, two worked examples in the system prompt, the session-notes memo for what the user declares, and the last-edit echo for what the app just did. Two of those four are lines the application writes mechanically. The model remembers nothing, and nothing measured is lost.
The usual caveats travel with the numbers: three models, one grammar, distance-one anaphora (the follow-up always immediately follows its referent), seeded corpora, and one mid-run gateway outage disclosed in the protocol notes (sessions error unrecorded and resume cleanly). Everything is reproducible from the benchmark repo, and the full chart set lives on the research dashboard. If your users point at things in ways our corpus doesn't, that's the next study, and we want to hear about it.
Update, July 2026: the ask path finally got pointed at this construction (Study AG). With the shipped NEED-INFO sentence and no echo, the 144 silent guesses above became 138 questions across three models, each naming the missing antecedent; the discourse gap turns out to be squarely inside the sentence's letter, so the hatch is a real seatbelt behind the echo on every tier. The same study found the fine print: under skeleton views the sentence's visibility clause makes models ask about nodes they can already patch (they quote the echo's node id, then refuse because it is "not visible"), so the hatch's famous zero tax is conditional on views that carry their targets. The echo stays the right UX (zero questions, perfect solves); the hatch behind it now has a mapped interaction instead of an assumed one. Charts on the dashboard.
Addendum (July 2026): The Seatbelt Meets a Toolbox
Study AN finished this story. The ask rule that backs up the echo turned out to have a sharp edge: under compact outline views, models would sometimes refuse a fully specified edit because the target node was not visible, even though the echo had handed them everything an anchored patch needs. The obvious fix was another prompt rule, something like "prefer fetching a view over asking." We pre-registered it and measured.
The rule lost to the toolbox. Simply having a view-fetching tool available dissolved the refusals on every model tier we test, with no prompt change at all: models fetched one precise view, or skipped the fetch and patched directly from the echo. The clause we almost shipped was redundant where it worked and harmful where it did not; below the frontier tier it converted honest questions into silent guesses or tool-call flailing. It stays unshipped. The echo remains the carrier, the ask rule remains the seatbelt, and the lesson joins the pile: when a model seems to need a smarter instruction, check first whether it just needs a tool.
