Series note: Fourth post in the barkup-bench series, after the benchmark, the shipped feature, and the footgun that humbled us.
When we pre-registered barkup-bench, one hypothesis felt safest: somewhere as trees grow, whole-tree rewrite (reply with the entire edited artifact) would stop being worth it, and finer-grained editing would overtake it. We called it the crossover, and we expected to find it between granular tool calls and rewrite.
The main study said: no crossover, up to ~190 nodes. Then the protocol correction said: even the gaps you did see were an artifact of your own conversation plumbing. By that point I'd have forgiven the crossover for not existing at all.
It exists. We found it by extending the benchmark to trees three to seven times larger: 300, 600, and 1000 nodes, 45 fresh pre-registered editing tasks, two models spanning the price spectrum. It just isn't between the interfaces anyone was arguing about.
What happens at 1000 nodes
One edit request. A tree of about a thousand typed nodes. Three ways to reply: rewrite the whole artifact (with barkup's HTML dialect), send an RFC 6902 JSON Patch, or send an id-anchored patch. That last one is the dialect we shipped in barkup 0.2 after the earlier rounds of this benchmark: every operation names its target by stable id, and placements anchor to sibling ids, with no positional indexes anywhere.
| Success | ~300 nodes | ~600 | ~1000 |
|---|---|---|---|
| Whole-tree rewrite, claude-sonnet-4.5 | 15/15 | 14/15 | 12/15 |
| Whole-tree rewrite, gemini-3.5-flash | 9/15 | 5/15 | 0/15 |
| Anchored patch, sonnet | 15/15 | 15/15 | 13/15 |
| Anchored patch, gemini | 14/15 | 14/15 | 13/15 |
| RFC 6902, either model | ~53% | ~20% | ~10% |
Task success by tree size, interface, and model, barkup-bench Study H: 45 pre-registered tasks across three sizes, protocol v2, n = 15 per cell.
Three separate stories are hiding in that table.
Whole-tree rewrite becomes a frontier-only technique. Sonnet can genuinely do it. Twelve of fifteen thousand-node rewrites came back byte-faithful except the requested edit, which frankly exceeded my expectations. The cheap model can't do it at all, and its failure isn't loud: it produces plausible trees with quiet damage. If your agent rewrites large artifacts and your evals run on a frontier model, you're measuring a capability your production tier may not have. That's a cousin of the lesson from the footgun post.
The pipe gives out before the model does. A thousand-node rewrite is ~50,000 output tokens: ten minutes of generation. Unstreamed, every single one of sonnet's large rewrites died at the gateway mid-response. We had to switch the harness to streaming just to let the model finish a sentence ten minutes long. And when a rewrite failed and entered the retry loop, one task burned half a million tokens. There's a mechanical ceiling here that has nothing to do with intelligence.
Anchored patches don't care. Both models, every size, 87 to 100%. The patch for a thousand-node edit is a few dozen output tokens, so it completes in seconds, can't hit output ceilings, and failing cheaply is its worst case. Per solved thousand-node task: rewrite on sonnet cost $0.88 and 597 seconds; the anchored patch cost $0.26 and four seconds. On gemini it cost about four cents and two.
The revised map
So the practical guidance, three studies and one correction later:
- Up to ~200 nodes: rewrite and anchored patches are statistically interchangeable on accuracy; rewrite is operationally simplest. (And with correct conversation history, granular tools are fine here too. That gap was our bug, not the models'.)
- From ~300 nodes up: anchored patches are the only interface that's simultaneously reliable across model tiers, fast, cheap, and free of transport ceilings.
- Positional patch formats: not above ~150 nodes, at any tier. Index arithmetic is the wrong job for a language model, and it gets worse with every level of nesting.
The through-line of the whole series is stable identity. Positional patches fail because positions are fragile. Rewrite at scale fails, when it fails, by drifting somewhere in fifty thousand tokens of faithful reproduction. Anchored patches work because every operation names a thing that cannot move out from under it: an id the codec guarantees byte-for-byte. That guarantee was barkup's founding obsession before any of this data existed. It's nice when the benchmark eventually explains why you cared.
Everything is reproducible from the repo: pre-registration (BRIEF-H), corpus, seeds, raw analysis, and the streaming-transport protocol note, all at barkup-bench.
Update (July 2026): One more result since publishing: anchored patches' one remaining cost at scale was resending the whole tree as input on every request. A pre-registered follow-up (Study I in the benchmark repo) replaced the full tree with a focused view centered on the nodes the edit references and found accuracy unchanged, with input tokens down 96 to 98% at about 1000 nodes. The input side is now effectively size-invariant too. A second follow-up (Study J) confirmed the views cost even less rendered in barkup's native HTML dialect, at identical accuracy. Both are written up in The Model Doesn't Need to See Your Tree.
Update (July 2026): A session study (Study K) then extended the crossover to conversations: across 140 twelve-edit sessions, whole-tree rewrite also failed as a session protocol at every tier, with twelve accumulated rewrites deterministically exhausting a 200k context window at the frontier and silently corrupted end states below it (2 of 10 sessions intact), while a fresh focused view per turn was both the most accurate and the cheapest policy tested. Details: Your Agent's Session Is Drifting.
Update (July 2026): A third model, and the one that matters most to us: Study AD re-ran the anchored-patch dialect on claude-opus-4.8, the tier our own product surfaces ship. It scored 194 of 200 on the main corpus, the best number ever measured for the dialect, went 14 of 15 against full thousand-node trees, and the focused-view variants were perfect at every size. The practical rule above survives its third model unchanged. Charts on the dashboard.
