This was the series capstone at seven studies, kept intact as the historical record, appended updates and all. The synthesis at twenty-three studies, reorganized around the full measured architecture, is Hand It Everything It Needs.
Over the past week we ran seven pre-registered studies, more than 13,000 scored model runs, against one question: what is the most reliable way to let an LLM agent edit structured data? Typed trees, specifically: page layouts, document templates, anything with a schema, nesting, and rules. We tested four models from three vendors, trees from 5 to 1,000 nodes, and editing sessions up to twelve consecutive edits.
The whole thing compresses to one sentence: give every node a stable id, and never make the model reproduce anything it is not changing.
Here is how we got there, including the part where we had to correct ourselves in public.
The question we started with
The original essay argued that an HTML dialect plus whole-tree rewrite was the right interface for agent editing: readable, familiar, one coherent artifact validated in one shot. We benchmarked it against an equal-strictness JSON twin, granular mutation tools, and patch formats, with every prompt and seed committed before the first scored call.
The format claim died immediately, and cleanly. Modern models write strict HTML and strict JSON essentially perfectly (first-pass validity was at least 99.3% in every arm of every model). Format never decided anything. That was our first null result, and far from the last.
The bug that was better than a finding
The early headline said whole-tree rewrite beat granular tools by 33 points on multi-turn tasks. It was dramatic, it confirmed our thesis, and it was wrong. Our harness had built conversation histories with the AI SDK's response.messages, which silently omits the model's own tool calls. In every multi-turn tools conversation, the model could not see what it had just done, including the insert that returned the id it was asked to edit next.
With correct history, the gap vanished. Rewrite vs tools became 91.9% vs 93.9% (a slight tools edge). The "small models are bad at tools" result vanished with it: gemini-flash went from 3.8% to 71.3% on multi-turn edits, haiku from 28.7% to 98.8%, purely from history construction. The frontier models had masked the bug because they barely need history to re-derive state. We re-ran everything affected, published the correction across every post, and filed the issue upstream (vercel/ai#16840).
Two lessons survived that are worth more than the original claim: audit your history construction, and evaluate on a cheap model, because it will expose plumbing failures your frontier evals hide.
Positions lie, ids don't
One early result did survive everything: RFC 6902 JSON Patch, the standard that addresses nodes by positional paths like /children/3/children/7, collapsed as trees grew. 70% success at 150 nodes, roughly 10% at 1,000. Index arithmetic is the wrong job for a language model.
So we pre-registered a patch dialect where every operation names its target by stable id and placements anchor to sibling ids, no indexes anywhere. It recovered the entire collapse, matched whole-tree rewrite on accuracy, and was the cheapest condition measured. Then the size extension made it decisive: above roughly 300 nodes, whole-tree rewrite becomes a frontier-only technique (gemini-flash solved 0 of 15 tasks at 1,000 nodes; sonnet needed a streaming transport and ten minutes per rewrite), while anchored patches held 87 to 100% for both tiers at every size. Per solved 1,000-node edit: about $0.26 and four seconds, vs $0.88 and ten minutes.
The model doesn't need to see the tree
Every study so far had put the full tree in the prompt. Two follow-up studies asked whether it must. We replaced the full serialization with a focused view: the path from the root to the nodes the edit concerns, their child lists in order, and everything else collapsed to labeled placeholders or omitted with a count.
Accuracy did not move. Statistically indistinguishable from full input for both models at every size, and sonnet on the most aggressive view went 45 of 45. Input tokens fell 96 to 98% at 1,000 nodes, and the view barely grows with tree size at all (about 1,300 tokens at 300 nodes, about 1,500 at 1,000): it scales with tree depth, not node count. Document size stops mattering to your token bill. Rendering the view as HTML vs JSON changed nothing on accuracy, one more format null, though the HTML views were 9 to 24% terser.
Sessions drift, and the fix is free
The last study was the production shape: twelve consecutive edits against one evolving tree in a single conversation. Shown the tree once and then asked to keep patching it, sonnet decayed to 83.8% success by the final third of the session, and the failures were exactly what you would guess: placements computed against a mental child list that three earlier edits had reshuffled. Only 8 of 20 sessions ended with a byte-perfect tree.
Attaching a fresh minimal view to every single turn erased the drift (239 of 240 steps, 19 of 20 sessions intact) and cost four times less, because the once-shown tree rides along in conversation history every turn anyway. Best and cheapest turned out to be the same policy. Whole-tree rewrite as a session protocol failed structurally: the cheap model silently corrupted state (every one of its 44 failures was a valid tree with wrong content), and one frontier session deterministically ran out of context window at step 11, because twelve accumulated rewrites of a 350-node tree simply do not fit.
The checklist for app builders
- Give every node a stable id and guarantee nothing ever renumbers it. Every winning result in seven studies rests on this one property.
- Have the model reply with id-anchored operations, applied and validated by your code. Never positional indexes. Never reproduction of unchanged content.
- Send focused context, not the whole document. A view of the relevant region with honest placeholders matches full-input accuracy at 2 to 4% of the cost.
- In multi-turn editing, refresh the model's picture every turn. It is cheaper than not doing it.
- Validate at the boundary and return structured errors verbatim. The correction loop recovered most first-pass failures everywhere.
- Audit your conversation-history plumbing, and eval on a cheap model. Our worst failure came from the SDK, not the model, and frontier models hid it.
- Whole-tree rewrite is fine below about 200 nodes and for from-scratch builds. Do not scale it, and do not chain it.
Where this sits, and why bother
None of this is wholly novel in isolation, and it would be strange if it were. Aider's edit-format benchmarks showed well before us that the format an LLM uses to express edits measurably changes success rates. The Berkeley Function-Calling Leaderboard documents that granular tool-call reliability varies sharply by model. "Lost in the Middle" established that models use long inputs unevenly, and Microsoft's "LLMs Get Lost in Multi-Turn Conversation" documented multi-turn degradation across model families. If you have been building agents for a while, several results above will read as confirmation rather than revelation.
We think the series earns its place anyway, for three reasons. First, specificity: the general findings do not tell you which editing interface survives those forces on typed trees, where the size crossover actually lands, or which session policy eliminates drift and what it costs. We measured those numbers, with mechanisms attached. Second, the correction: our most thesis-confirming result was manufactured by a one-line SDK misuse, and we would not have caught it without pre-registration and cheap-model coverage. Treat that as a case study in how easily an agent benchmark ends up measuring its own plumbing instead of the model. Third, reproducibility: seeds, prompts, corpora, and unit-tested graders are all committed, and a few hundred dollars of API spend re-runs the entire series from scratch.
Which is also an invitation. If you extend this (other grammars, more model tiers, real retrieval instead of our oracle bound, longer sessions), the repo is set up for exactly that: pre-register a brief, run it, publish whatever you find. We would genuinely like to see the view results broken.
What shipped, and what we still don't know
The validated pieces live in @kevinpeckham/barkup 0.3: the anchored-patch applier and the focused-view renderer, each shipped with conformance vectors generated by the benchmark itself.
Honest limits: the view studies are an oracle bound (our instructions name their target ids, so finding the right node from a vague request is untested); the later studies ran on two models rather than four; and everything ran on one grammar family. Those are the next questions, not footnotes we are hiding.
Update (July 2026): Two of those open questions closed within days, both against our predictions, which is pre-registration doing its job. Study L tested grounding: with id-free, human-style instructions and the full tree in context, accuracy costs 7 to 9 points versus id-anchored instructions; model-driven navigation of a skeleton view matches the oracle bound on the frontier tier but costs more tokens than showing the whole tree and collapses on the cheap tier; naive lexical retrieval lands near 60%. "Know your ids or invest in real retrieval" stands as the boundary. Study M tested memory: stateless sessions (fresh view every turn, no conversation history) confirmed the constant-cost economics but dropped end-state integrity from 19 of 20 sessions to 13 of 20, with the failures concentrated in late-session placements. Keep the history and the per-turn view; they cover different failure modes. Both studies are in the repo with full pre-registrations.
Update, July 2026: two follow-up studies (N and O) moved one of those boundaries. Grounding no longer requires reading the whole tree: give the model a skeleton view plus one deterministic keyword search tool and it matches its id-oracle accuracy at a median of one search call, on both model tiers, at about a tenth of full-tree cost. Off-the-shelf embeddings, tested as the fashionable alternative, added nothing over keyword matching. Sessions, however, still refuse to forget: annotating views with exact node positions did not rescue stateless editing, so keep the history. Details in Then We Found the Cheap Part.
One more update from the same week: Studies P and Q. P found what session history was actually doing all along: teaching, not remembering. Two canned worked examples in the system prompt (about 900 tokens, from a tree the model never edits) recover stateless sessions to full-history accuracy on both models, at half the cost and with no context ceiling. Q found the first task shape that breaks every strategy at once: fan-out edits ("change every X inside Y") drop even oracle-retrieval accuracy to roughly 45 to 69%, and the fix is to decompose in your application and issue single-target edits. Details in Your Agent Doesn't Need a Memory.
Everything is reproducible from the benchmark repo: pre-registrations committed before every scored run, seeded corpora, raw analyses, the correction, and the audit trail. The series in order: the original argument, the benchmark, the footgun, anchored patches, the crossover, focused views, session drift, and the barkup 0.3 release.
And two more, which close the context question: Study T asked what happens when a request depends on something said earlier in the conversation ("rename it to the codename we settled on"). The stateless recipe failed all 160 such steps by construction while scoring perfectly on everything else, and the fix is not a transcript but a memo: an app-maintained note block of declared facts restored every one at 2% extra cost. Study U asked the mirror question about the document: an edit that must read a second node ("set A's caption to match B's title") against a view showing only the target failed all 90 cells, and every failure was a valid patch with a silently invented value. Putting both mentioned nodes in the view fixed all 90 at 25 times less input than the whole tree. Together they complete the map: the model needs no memory and no full document, but your application must hand it a memo of what was said and a view of every node the request mentions. Full story in The Two Things Your Agent Can't See.
And a judge-graded coda: Study V asked whether those two fixes carry qualitative goals ("make this paragraph focus on our thesis") and split the answer: the memo carries intent at full parity with an explicit instruction, while a model that must read its goal from the document writes measurably less focused prose, losing 117 of 120 judged comparisons. The slogan the series ends its context arc on: views carry values, memos carry goals. Method and verdicts in Views Carry Values, Memos Carry Goals.
And the last asterisk, closed: every memo in the studies above was written by the test harness. Study W audited the shipped alternative, the agent writing its own memo through a tool, on sessions long enough to outrun the history window, with a mid-session retraction, on three models including the first data from the tier our production system runs. Both pre-registered gates passed: recall 36 of 36 per model, zero noise, no recording laziness, and the post-truncation safety net held. The whole session arc, from "keep everything" to "a memo the agent writes itself," is now measured at every layer: Who Writes the Memo?.
