Barkup Bench

H2 · task success by tree size

At benchmark sizes (5–150 nodes), no interface dominates

Whole-tree rewrite (A, B) leads granular tools (C, D) at every size tested. The gap peaks on small/medium trees and narrows; but never reverses; at ~150 nodes. JSON Patch (E) collapses to 69.6% on large trees; and the pre-registered addendum condition F (id-anchored patches) recovers it exactly to rewrite level at the lowest token cost.

H4 · multi-turn reference tasks, by model

Corrected: multi-turn reference edits are near-parity

Success on “insert a node, then edit it by the id from your own output”; protocol v2. With the model's own tool history visible, every interface is near-parity (A 88.1% vs C 91.3%, n.s.); gemini's residual tools deficit is phase-1 editing accuracy, not follow-up dropout. Zero id-reference failures in any arm, ever. The dramatic v1 gaps live in the footgun chart below.

H3 · mean tokens per solved task

Rewrite is cheap where trees are small; HTML is the terser artifact where they're large

Tools arms (C, D) resend a growing conversation every call, so they burn 4–5× more tokens on small/medium trees. At ~150 nodes the whole-artifact arms grow too; and the HTML dialect (A, 15.6k) undercuts JSON rewrite (B, 23.0k) by ~30%: the one place the format measurably matters.

Overall · success per model × condition (parity, 200 tasks each)

Frontier models don't care; smaller models do

Per-cell task success. Down the sonnet-4.5 and gpt-5.4 rows the interface barely matters; down the haiku-4.5 and gemini-3.5-flash rows the tools arms give up 10–17 points against rewrite.

The follow-up series · studies G–AO

Every Study, One Page Each

The main study above set the baseline; everything since has been a pre-registered follow-up, each with its own charts, gates, and verdict. Grouped by theme, or subscribe to the Atom feed to hear about new studies as they publish:

Interface reliability

• Study G · One hidden SDK default, two very different benchmarks The original +33pp multi-turn gap was manufactured by conversation histories that omitted the model's own tool calls (the AI SDK v5→v7 response.messages trap).
• Study H · The crossover, found: above ~300 nodes, only anchored patches hold Whole-tree rewrite becomes frontier-only at scale (gemini-flash: 0/15 at ~1000 nodes; sonnet needs a streaming transport and ten minutes per rewrite).

Views & grounding

• Study I/J · The model doesn't need to see the tree: a ~1.5k-token view matches the full 85k-token input Show only the path to the referenced nodes plus their child lists (everything else collapsed or omitted with a count), and anchored-patch accuracy is statistically unchanged for both models at every size; sonnet on the .
• Study L · Finding the node is the expensive part: grounding costs 7–9 points, and navigation is a frontier-only trap Same edits, instructions rewritten without ids ("the image-atom named 'maple-ember'"), every description verified to match exactly one node.
• Study N · Stop walking, start searching: one find_nodes call grounds the edit on both model tiers Same id-free tasks as Study L; the expand tool replaced by a single content-search tool (a few words in, the 5 best keyword matches out, shown in place).
• Study Q · "Change every X inside Y" breaks everything; including the oracle One instruction, 2–32 targets, on the same large trees.
• Study U · The view that can't see the answer doesn't fail; it makes something up Focused views assume the instruction carries every value an edit needs.
• Study AN · Ask or act: tool availability dissolves the visibility-clause tax With a view tool simply available, the ~70% NEED-INFO tax on fully specified edits vanished on every tier; the tested fetch-before-ask prompt clause was redundant on the frontier and unsafe below it, so it stays unshipped.

Sessions & memory

• Study K · Sessions drift unless you re-show the tree; and showing a fresh view every turn is also the cheapest policy Twelve sequential patches against one evolving tree.
• Study M · The view carries the state, but history still earns its keep If a fresh view arrives every turn, does the model need conversation history at all? No; statelessness confirmed the constant-cost economics (~1.3k input tokens at step 1 and step 12 alike) but failed the accuracy gate:.
• Study O · Printing the position on every node does not rescue stateless sessions Study M's stateless failures were all placement edits, so Study O annotated every view child with its true 1-based position (plus a prompt line mapping ordinals to anchors) and re-ran the 2×2 of history × positions.
• Study P · History was a teacher all along: two canned examples replace the whole conversation If history isn't state (the view carries that) and isn't positional help (Study O), maybe it's worked precedent; and precedent can be faked.
• Study S · 36 edits later, nothing broke; except the cost curve Every prior session study ran 12 edits.
• Study T · The one thing statelessness can't do; and the three note lines that fix it Every prior session study used self-contained requests, so Study T built the missing class: callback steps whose required fact lives only in earlier conversation; a declared codename a later rename must use, and a stand.
• Study W · The agent writes the memo faithfully; even when history makes it redundant Studies T and V validated the memo with oracle extraction: the harness knew what to record.
• Study X · "Undo that" needs an echo, not a transcript The last structurally unmeasured request class: follow-ups that point at the previous edit; "also set that same node's…", "apply the same change to X", "actually, undo that." Anaphora steps ran against skeleton views (a.
• Study Y · The memo survives how people actually talk Every declaration in Studies T, V, and W was announced formulaically ("For later reference: the codename is X").
• Study AH · Perfect to the cap. At the cap, the goals die first. Every study that shipped the session-notes memo measured it at three to six notes; the shipped implementation caps it at twenty, with a normalize step that silently drops the excess.
• Study AK · The fix works where it can, and the frontier tier invented consolidation Study AH found that a twenty-first declaration at the memo's full cap always kills a note, and the victim is always a goal.
• Study AL · The gate fails on a moved baseline; and unproven does not ship Study AK left one pathway open: the app-side eviction cannot restore a note the model pruned before sending, and the mid tiers pruned goals client-side in 4 to 6 cells of ten.
• Study AM · The notice learned to ask for everything back Twice in two studies; both times unregistered; opus answered an eviction notice by re-sending the memo compressed into fewer sentences carrying all twenty-one needles: told a note had died, the frontier tier invented l.
• Study AQ · The powered fence: vindicated where it no longer matters, harmful where it does AL's fence sentence, powered 4× and aimed at the swap candidates: it closes sonnet's prune pathway completely (16/40 to 0/40, p=.00002), does nothing on gemini, is unnecessary on fable-5 (which spontaneously consolidates losslessly), and is actively harmful on kimi-k3 (prunes 5/40 to 18/40) — one sentence, four dispositions, and the fence line closes for the successor tiers.
• Study AR · Advertised headroom: one line of state outdoes the rule, and a binding number is poison A capacity state line ('an update may include up to 24 notes') harms nothing anywhere, erases cap-edge pruning completely on sonnet (18/40 to 0/40) and gemini (15/40 to 0/40, the first intervention ever to move it), perfects fable-5, and is inert on kimi-k3, whose losses turn out to be intrinsic lossy consolidation; the truecap arm shows a binding number is catastrophic.

Qualitative (Track 2)

• Study V · Views carry values, memos carry goals The series' first judge-graded study, run under a pre-registered pairwise protocol: both judges (gpt-5.4 primary, haiku-4.5 sensitivity; neither an editor) first had to pass a 50-pair calibration gate with planted verdi.
• Study AF · Saying the goal out loud doesn't make it yours One clause in the shipped session-notes rule had been inferred rather than measured: restate a goal from the memo in your own words before a goal-directed rewrite.

Standing context

• Study Z · The brand pack works; the rule you forgot is the hazard Production doc editors ship a standing context block with every request; company facts, client records, a styleguide; and nobody had measured whether models actually use it.
• Study AA · We tested our own headline and lost: strictness is not a capability gradient Study Z’s spec-conflict split was an accident with one conflict shape, so Study AA measured it on purpose: three registered conflict kinds (rule vs instruction, an explicit user countermand of a rule, rule vs rule), four.

Asking & calibration

• Study AC · The model always knew what it couldn’t see Twenty-eight studies of silent failure (90/90 invented values in U, 144/144 silent guesses in X, 120/120 oblivious polishes in V, zero clarifying questions anywhere) never once OFFERED the model a way out.
• Study AE · Only the frontier knows when a question is the right answer Study AC left two fears on the record: that the escape hatch would tax clear requests with unnecessary questions, and that an ask might be a dead end.
• Study AI · The obvious fix, measured: it helps exactly where we don't ship Study AE left an obvious fix on the table: the shipped ask sentence covers what the model cannot see, so ambiguous references (visible, but two of them) slipped through on the mid tiers.
• Study AG · The discourse gap closes on every tier; the visibility clause bites One border of the ask-path map was never tested: Study X's discourse construction, where "undo that" against a carrier-less editor failed 0 of 144 with every failure a valid silent guess.

Confirmations & machinery

• Study AD · The core stack transfers to the tier we actually ship Studies F through U validated the core editing stack on sonnet-4.5 and gemini-3.5-flash, but the downstream template-editing surface runs claude-opus-4.8, a tier whose benchmark data began at Study W.
• Study AJ · The error message didn't matter Returning barkup's structured validation issues verbatim has been a design commitment in every arm of this series, the closing instruction of playbook guideline 01, and standing guidance in the production codebase, and i.
• Study OW · The open-weight backfill: the cheapest model passed everything; two frontiers failed the same gate gpt-oss-120b clears all 13 shipped-guardrail regression gates at 33 cents; kimi-k3 and fable-5 both go 12/13, red on the same memo-cap-edge slice via cap-obedient goal pruning; kimi-k3 posts the best adjacent-class tag numbers measured; the guardrail stack proves tier-robust down to the cheapest open-weight model.

Agent-surface steering

• Study AO · Canonical-tag steering: the catalog read is the whole mechanism Unguided models never match a tag catalog's exact casing (0/36 clean everywhere); with a tags_list tool they consult it unprompted and land canonical on the first attempt; warnings alone cannot recover, and hard gating is unnecessary.
• Study AP · The off-catalog fork: one sentence settles the foreign branch; the registration loses the adjacent one The frontier tier refuses to tag off-catalog topics unaided (12/12 empty, replicating AO); one sentence of tool-description guidance closes the sub-frontier gap on every tier; a guarded tag_create tool stays fully disciplined; the adjacent-class gate failed on solo-authored acceptable sets, and the AP′ re-score under an independently authored key passes.