Hand It Everything It Needs: 23 Pre-Registered Studies on LLM Document Editing
A month ago we started measuring a narrow question with broad consequences: what is the most reliable way to let an LLM agent edit structured documents, the typed trees behind page builders, document templates, and CMS content? The method was borrowed from fields that learned it the hard way: every study pre-registered by commit before the first scored model call, pass gates stated in advance, deterministic unit-tested graders, seeded corpora, and results published as found, including the ones that embarrassed us.
Twenty-three studies, more than 19,000 scored runs, and five models later, the answer is not a clever prompt. It is a division of labor. The model is a brilliant executor with no context of its own; the application owns the context. Hand the model everything the request assumes, and almost nothing else. Here is that recipe with a number on every clause, followed by the part we think matters more: what we got wrong along the way, and published anyway. (If you are here to build rather than to read, the checklist version of this essay, with code examples ready to copy, is The Builder's Playbook.)
The architecture, one measured clause at a time
1. Give every node a permanent id. The foundation everything else stands on. Across the entire series, zero failures were ever caused by a model mangling a stable id.
2. Edit by id-anchored patches, not whole-document rewrites. At small sizes every id-stable interface ties. Above roughly 300 nodes, anchored patches are the only interface both model tiers hold: 87 to 100% where whole-document rewrite falls to 0 to 80% on the cheap tier, at about $0.26 and 4 seconds per solved 1000-node edit versus $0.88 and 10 minutes. Positional patches (RFC 6902) decay toward 10%. Never make the model reproduce what it is not changing.
3. Show a focused view, and make it cover every node the request mentions. A ~1.5k-token view of the relevant region matches full-document accuracy at 1 to 4% of the input, and view size scales with tree depth, not document size. But the focus list is a correctness contract: an edit that must read a second node, given a view showing only the target, does not fail. It silently invents a plausible value, 90 times out of 90 in our test, with zero refusals. Include every mentioned node and the score returns to 90 of 90, at 25 times less input than the whole document.
4. When the user doesn't name a node, give the model one search call. A skeleton view plus a deterministic keyword-search tool grounds plain-language requests at oracle-level accuracy on the frontier tier, at a median of exactly one call and a tenth of the input. Embeddings measured no better than keyword overlap for structural references. Navigation tools were a trap: oracle accuracy on the frontier model at higher cost than showing everything, collapse on the cheap one.
5. Replace session memory with two worked examples. What conversation history actually contributes to an editing session is teaching, not memory: two canned examples of the patch dialect's tricky operations, about 900 tokens in the system prompt, restore stateless sessions to full-history accuracy. Measured through 36-edit sessions with no decay, edit 36 taught as well as edit 1, at flat ~2.1k input per step while keep-history grows linearly to 24k.
6. For everything the user declares, keep a memo, and let the agent write it. Requests that depend on earlier conversation ("rename it to the codename we settled on") fail stateless editors 100% of the time by construction. A three-line app-held memo of declared facts and rules recovers every one at 2% extra cost, and it carries qualitative goals at full parity with restating them, where merely showing the model the document node the goal lives in loses 117 of 120 judged comparisons. Views carry values; memos carry goals, and no prompt ritual converts one into the other: forced to restate a document-read goal in its own words before rewriting, every model complied perfectly and still lost every decisive judged comparison, zero wins in ninety (Study AF), so where a goal comes from matters more than whether the model says it aloud. And the agent itself writes that memo faithfully: delegated extraction tied our perfect-oracle baseline on all three models tested, retractions handled, zero noise, with no laziness even when a history window made the memo redundant, and at exact parity whether declarations are announced formally or phrased the way people actually talk, with zero false notes from 432 conversational chatter baits. The memo is also measured at scale (Study AH): flawless to its twenty-note cap, no burial at any position, lossless full-replace, with one sharp edge: at the cap, a new declaration silently costs a note, and in thirty of thirty measured cells the note it cost was a goal, the one thing only the memo carries. Cap your memo with a deterministic eviction policy, never a silent truncation. That policy has since been measured at the injury site (Study AK): with a goal-preserving eviction in the update path, every over-cap update became a designed eviction and goal survival at the cap edge went from zero of ten to ten of ten on the frontier tier, with the remaining exposure being notes a model prunes before it ever sends the list.
7. Never ask one prompt for N edits. "Change every X inside Y" broke every strategy we had, including perfect retrieval: models deliver roughly half of N and stop, confident. Prompt-side fixes don't rescue it. The application enumerates the targets with a deterministic query and issues one small edit per node: 90 of 90 tasks, 674 of 674 subtasks, at a third of the cost. Give deterministic work to deterministic code.
Every clause above ships as code in @kevinpeckham/barkup or as a documented pattern with a reference implementation, each gated on its study before release.
What we published against ourselves
The reason to trust the list above is the list below.
The original headline was wrong, and the correction became the most useful finding of the series. We first reported whole-document rewrite beating granular tools by wide margins. The real cause was a silent SDK default that hid the model's own tool calls from multi-turn history (vercel/ai#16840). Under corrected history the interfaces tie, and the footgun itself, which quietly collapses small-model tool reliability to single digits while frontier models mask it, is worth more to a builder than the original claim was.
We refuted our own hypotheses on the record, repeatedly. Statelessness failed its first gate. Positional annotations rescued nothing. Embeddings added nothing over keyword overlap. Prompt interventions did not fix fan-out. A worked example did not teach exhaustiveness. The judge-graded study failed its headline gate in the most useful way possible, by splitting: the memo carries goals, the view does not.
And when a question could not be graded deterministically, we said so out loud instead of pretending. Qualitative rewrites run on a separately labeled track where the LLM judge had to pass its own pre-registered exam, fifty calibration pairs with planted answers, identity probes, and length traps, before grading a single real comparison. Both judges went fifty for fifty. Judge-graded numbers are never pooled with the deterministic claims.
The recurring villain, if the series has one, is silent failure. A model that cannot know something does not tell you. It invents a value that validates, polishes a paragraph it cannot aim, or would leave a safety net unfilled where nothing tests it. None of this appears in error logs. Nearly every component of the architecture exists to hand the model the context that makes silence impossible, and nearly every study worked by constructing the moment where the missing context would otherwise fail invisibly. A late coda recast the villain entirely (Study AC): offered a one-sentence escape hatch, every model asked on every provably-unsolvable task, 270 of 270 against 0 of 270 without it, never asked when it had what it needed, and named the exact missing node each time. The silence was never blindness; it was a protocol that only accepted patches. The write-up is The Model Always Knew What It Couldn't See. The coda then got its own coda (Study AE): the hatch calibrates, with zero false asks on ninety clear requests and a perfect 135 of 135 solves after the user answers, but ambiguity detection turned out to be a capability, not a prompt feature: shown a request matching two visible nodes, the frontier tier asks which one while mid tiers silently resolve it themselves. The hatch covers absence on every tier, and dangling references ('undo that') on every tier too, with one measured condition: its zero tax presumes views that carry their targets, so the hatch and the view contract are one system, not two features (Study AG). Ambiguity it covers only where capability supplies the judgment.
What's still open
Honest boundaries, and one already closed since this post first went up: follow-up requests that point at the previous edit were measured the same day (Study X: without a carrier they fail 0 for 144, every failure a silent guess; a one-line app-appended last-edit echo restores full-history parity at half the cost and beat history outright on the production tier; the write-up is Undo That). Still open: whether production systems can shrink their history windows to the memo recipe's cost is a live question with real dollars attached; and everything above was measured on generated corpora, trees up to about 1,000 nodes, and two to five models per study.
A second closure, added after publication: the standing context block, the brand pack of company facts, client records, and styleguide rules that most production editors ship with every request, is now measured (Study Z). It simply works at production sizes: exact facts copied past same-schema distractor clients, unstated styleguide rules applied at any position, 216 of 216 per arm across three models, zero cross-client contamination in 324 cells, and slicing the pack per request buys nothing while forfeiting prompt caching. The one hazard found is spec conflict: when a standing rule written with "always" collides with what the user asked for, models pick a reading rather than fail. Our first read of that split, that stronger models pick the literal reading, was refuted a day later by its own pre-registered confirmation study (Study AA, the series' second major self-correction): strictness is real per model but is not a capability gradient, and it shifts with context composition. What the confirmation study validated instead: softening rule wording from "always" to "we generally prefer" is the cheap fix that works, a priority meta-rule is not, and restating a rule in the memo steers interpretation so strongly it can override a user's explicit countermand. That last hazard closed within the day, measured: a one-sentence precedence clause inside the memo block restored every trampled countermand at zero cost to the memo's benefit (Study AB), where the same sentence buried in a styleguide had moved nothing. Placement, not phrasing, is what makes precedence language work. The write-up, kept under its original refuted title with the retraction as its foreword, is Stronger LLMs Follow Conflicting Instructions More Literally, Not Less.
A third closure, and the one this architecture was quietly missing: nearly everything above was measured on sonnet and gemini, while the surfaces downstream of this research run claude-opus-4.8, and the series' own findings (the Study Q inversion, the Study AA composition sensitivity) forbid assuming tier transfer. Study AD re-ran the entire core stack on the shipped tier, corpora and prompts verbatim, gates anchored to the weakest previously passing tier. Every gate passed, mostly at the top of the prior bands: the patch dialect's 194 of 200 is the best ever measured on the main corpus, the focused views swept 90 of 90, search grounding landed exactly on the oracle bound, and every session policy was perfect. One result is new rather than confirmatory: the worked examples that rescue sonnet's stateless sessions are not load-bearing on the frontier tier, whose bare stateless sessions also scored 240 of 240. The training wheels are for the smaller bikes, and at 900 tokens they stay bolted on as insurance. Fan-out alone stayed open, at a higher floor and with a third mitigation profile, so app-side decomposition remains the rule at every tier. The charts are on the dashboard.
Everything is reproducible from the benchmark repo: every brief committed before its first scored run, every corpus seeded, every grader unit-tested, every correction kept. The research dashboard has every chart. The original capstone, written at seven studies, remains as the historical record, visible accretions and all. If you reproduce, extend, or refute any of this, we genuinely want the issue.
