barkup-bench is our open research project measuring how large language models read and edit structured document trees. Every study is pre-registered: the hypotheses, task corpora, prompts, and analysis plan are committed to a public repository before a single model is called, and the results are published as found, corrections included.
The series so far: nineteen studies, more than 13,000 scored model runs, four models, trees from 5 to 1,000 nodes, and editing sessions up to 36 edits long. The findings compress to one sentence: give every node a stable id, and never make the model reproduce anything it is not changing. Everything the benchmark validated ships in the open-source barkup library, linked below with the full article series.
barkup-bench · pre-registered · corrected 2026-07-06 · studies F–S
Main study: 200 seeded tasks × 6 conditions × 4 models × 2 regimes, temperature 0 (July 2026), plus follow-up studies G (tool-history footgun), H (300–1000-node size extension), I/J (focused views: partial-context patches, JSON and HTML renderings), K (12-edit sessions under four serialization policies), L/N (grounding without ids, then the retrieval ladder), M/O/P (session memory, positional views, synthetic history), Q/R (fan-out edits and their fixes), and S (36-edit long sessions). All numbers below are protocol-v2 corrected — the original interface-reliability gaps were an artifact of conversation histories that hid the model's own tool calls (vercel/ai#16840). Charts show parity prompts, models pooled unless noted; Wilson 95% intervals. Conditions: A HTML + whole-tree rewrite, B JSON + rewrite, C JSON + mutation tools, D HTML + tools, E JSON Patch, F id-anchored patch (pre-registered addendum, BRIEF-F.md).
H2 · task success by tree size
Whole-tree rewrite (A, B) leads granular tools (C, D) at every size tested. The gap peaks on small/medium trees and narrows — but never reverses — at ~150 nodes. JSON Patch (E) collapses to 69.6% on large trees — and the pre-registered addendum condition F (id-anchored patches) recovers it exactly to rewrite level at the lowest token cost.
H4 · multi-turn reference tasks, by model
Success on “insert a node, then edit it by the id from your own output” — protocol v2. With the model's own tool history visible, every interface is near-parity (A 88.1% vs C 91.3%, n.s.); gemini's residual tools deficit is phase-1 editing accuracy, not follow-up dropout. Zero id-reference failures in any arm, ever. The dramatic v1 gaps live in the footgun chart below.
H3 · mean tokens per solved task
Tools arms (C, D) resend a growing conversation every call, so they burn 4–5× more tokens on small/medium trees. At ~150 nodes the whole-artifact arms grow too — and the HTML dialect (A, 15.6k) undercuts JSON rewrite (B, 23.0k) by ~30%: the one place the format measurably matters.
Overall · success per model × condition (parity, 200 tasks each)
Per-cell task success. Down the sonnet-4.5 and gpt-5.4 rows the interface barely matters; down the haiku-4.5 and gemini-3.5-flash rows the tools arms give up 10–17 points against rewrite.
Study G · the tool-history footgun
The original +33pp multi-turn gap was manufactured by conversation histories that omitted the model's own tool calls (the AI SDK v5→v7 response.messages trap). Same tasks, same prompts — only history construction differs. Under corrected history, all 2,160 Study G cells pass.
Study H · 300–1000 nodes
Whole-tree rewrite becomes frontier-only at scale (gemini-flash: 0/15 at ~1000 nodes; sonnet needs a streaming transport and ten minutes per rewrite). Anchored patches hold 87–100% for both tiers at every size — $0.26 and 4 seconds per solved 1000-node edit. Positional patches decay to ~10%. Solid lines: sonnet-4.5; dashed: gemini-3.5-flash.
Studies I & J · focused views
Show only the path to the referenced nodes plus their child lists (everything else collapsed or omitted with a count), and anchored-patch accuracy is statistically unchanged for both models at every size — sonnet on the minimal view went 45/45 — while the view's size grows with tree depth, not node count. Rendering the view in barkup's HTML dialect instead of JSON changes nothing on accuracy (p = 1.0 in all four paired comparisons) and is 9–24% terser; it shipped as barkup/view in 0.3. Caveat: this is the oracle bound — instructions name their target ids. Solid: sonnet-4.5; dashed: gemini-3.5-flash.
Study K · 12-edit sessions
Twelve sequential patches against one evolving tree. Shown the tree once, sonnet decays to 83.8% by the last third (stale ordinal placements); with a fresh ~1.5k-token minimal view every turn it stays at 100% (239/240 steps) for a quarter of the tokens — the once-shown tree rides along in history every turn anyway. Whole-tree rewrite as a session protocol fails structurally: gemini silently corrupts state (all 44 failures valid-but-wrong; 2/10 sessions end intact) and one sonnet session deterministically exhausted the 200k context window at step 11. Solid: sonnet-4.5; dashed: gemini-3.5-flash.
Study L · grounding without ids
Same edits, instructions rewritten without ids ("the image-atom named 'maple-ember'"), every description verified to match exactly one node. With the full tree in context, models ground at 84–87% (vs 91–96% with ids). Letting the model navigate a skeleton with an expand tool matches the oracle on sonnet — at a median of 54 expansions, costing MORE than showing everything — and collapses on gemini, mostly by exhausting its budget without an answer. Naive lexical retrieval is the floor it was designed to be. The pre-registered gate failed: no partial-context mechanism was both accurate and cheaper — until Study N (next chart) swapped the expand tool for content search.
Study N · the retrieval ladder
Same id-free tasks as Study L; the expand tool replaced by a single content-search tool (a few words in, the 5 best keyword matches out, shown in place). The frontier model matches its id-oracle bound exactly — same 43/45, same two failed tasks — at a median of one search call and ~90% less input; gemini jumps from 23/45 (navigating) to 39/45 (16–0 paired, p < 0.001), its full-tree score at 3% of the cost. Two more rungs: swapping the keyword matcher for text-embedding-3-small changed nothing (target coverage 23/45 vs 24/45 — embeddings can't resolve "the 3rd block inside atlas"), and two-stage grounding (gemini reads the tree, names the ids, sonnet patches a focused view) holds 41/45 with the frontier model's median input at 1,484 tokens (−97.4%). The Study L gate, re-tested, passes twice.
Study M · stateless sessions
If a fresh view arrives every turn, does the model need conversation history at all? No — statelessness confirmed the constant-cost economics (~1.3k input tokens at step 1 and step 12 alike) but failed the accuracy gate: sonnet lost 7 steps to 0 against full history (p = 0.016) and end-state integrity dropped from 19/20 sessions to 13–14/20, with failures concentrated in late-session placement edits. A 2-exchange window sits in between, leaning inadequate. Guidance at the time: keep the history AND the per-turn view. (Study P, two charts down, found what history was actually providing — and how to replace it.) Solid: sonnet-4.5; dashed: gemini-3.5-flash.
Study O · positional views
Study M's stateless failures were all placement edits, so Study O annotated every view child with its true 1-based position (plus a prompt line mapping ordinals to anchors) and re-ran the 2×2 of history × positions. The rescue failed: stateless accuracy moved 3–1 / 1–0 vs plain stateless (n.s.), and sessions ending byte-perfect stayed at 15/20 vs full history's 19/20 — models misplaced nodes that had the correct position printed on them. Whatever history contributes to placement, it is not arithmetic. Positions with history were descriptively the best cells in the whole series (gemini: 20/20 end-states, 60/60 late placements) but not significantly better than history alone: harmless, possibly mildly helpful, not a substitute for memory. (Study P, next chart, found the actual substitute.)
Study P · synthetic history
If history isn't state (the view carries that) and isn't positional help (Study O), maybe it's worked precedent — and precedent can be faked. Two unit-tested examples (an ordinal insert and a move, on a tree the model never edits, ~900 tokens) restore stateless sessions to full-history accuracy on both models: worst paired difference vs K-view anywhere is 2–0 (n.s.), end-states back to 18–20/20, and gemini's examples arm goes 60/60 on the late placements that defined the M failure class — with 20/20 byte-perfect endings. The system-prompt framing ties the fake-turns framing, so the value is content, not conversation. Flat ~2.1k input/step forever, half of keep-history's cost at 12 edits, no context ceiling. The session guidance is revised: fresh view per turn + two worked examples in the system prompt.
Study Q · fan-out edits
One instruction, 2–32 targets, on the same large trees. Even with retrieval solved (every target visible in the view), models finish only part of the set: 62–69% overall, ~45% at 7+ targets, failures 100% partial coverage. The 0.4 search recipe fails its gate (median 6 calls instead of 1; a third of runs over 100k input tokens; −24 pp vs full tree on gemini). And the series' first model inversion: sonnet is significantly better with a view (p = 0.022), gemini significantly better with the whole tree (p = 0.008) — no context strategy is model-independent here. Study R then tested the fixes: prompt-side interventions (a worked fan-out example, a coverage checklist) do NOT rescue it — but decomposition, measured directly, is perfect: the app enumerates targets and issues one single-target edit per node, 90/90 tasks on both models (674/674 subtasks, zero failures) at ~⅓ the input cost of a whole-tree prompt. The flat line at the top is the recipe. Solid: sonnet-4.5; dashed: gemini-3.5-flash.
Study S · long sessions
Every prior session study ran 12 edits. Study S ran the two surviving recipes through 36 (10 new sessions, seed committed) and both held: full-history S-view went 719/720 steps across both models, stateless-plus-worked-examples S-system went 713/720 with no late-third decay (98–99% at steps 25–36 — step 36 is taught as well as step 1), McNemar parity everywhere, zero context ceilings, zero blocked steps. What diverges is cost: keep-history's median per-step input grows linearly to ~24k tokens by step 36 while the stateless recipe stays flat at ~2.1k, which compounds to 449k vs 81k input per session — 5.4–5.6×, against a pre-registered prediction of ≥3×. The gate passed on both models; the worked-examples recipe is the measured default for long sessions. All seven S-system step failures were placement-class (insert/move), the class Studies M/O/P mapped. Solid: sonnet-4.5; dashed: gemini-3.5-flash.
Open-source software generated in response to the research, all MIT licensed.
The library the benchmark shaped. A grammar codec for authoring typed trees as HTML, plus everything the studies validated: id-anchored patches, focused views, content search, and deterministic selection. Every feature was gated on a pre-registered study before it shipped.
The harness itself. Pre-registered briefs, task corpora, committed seeds, the full REPORT.md, and every correction, published as found.
The sibling package. A Markdown round-trip codec born from the same discipline of testable guarantees: one round trip canonicalizes any document, a second is byte-identical.
Every post in the barkup-bench series, in order. Start at the top for the whole story, or jump to the capstone, Stable IDs Are All You Need.