barkup-bench · practical applications
Ten guidelines for building document-editing apps with LLM agents, distilled from twenty-nine pre-registered studies and more than 24,000 scored model runs. Each one is an action you can take this week, with the measurement that earned it a place on the list and a link to its chart on the research dashboard.
The whole architecture follows from one finding: the model is a brilliant executor with no context of its own, and when context is missing it does not fail loudly. It fills the gap with something plausible and moves on. So the division of labor is fixed. The application owns the context; the model owns the edit. Hand it everything the request assumes, and almost nothing else.
01 · The foundation · main matrix + Study H
A user asks your page builder to change one headline in a 1,000-node layout. Whole-document rewrite makes the model reproduce the 999 nodes it is not touching: about $0.88 and ten minutes per solved edit at that size, and cheap-tier models collapse outright. Positional patches (RFC 6902) decay toward 10% because indexes shift under the model's feet. Id-anchored patches were the only interface both model tiers held above roughly 300 nodes, at about $0.26 and four seconds per solved 1,000-node edit.
Assign every node a stable id at creation and never recycle it. Have the model reply with a small array of operations that address nodes by id, with placements anchored to sibling ids (before/after) instead of positions. Apply atomically, validate against your schema, and send any issues back verbatim for one correction round. Never make the model reproduce what it is not changing.
import { applyAnchoredPatch } from "@kevinpeckham/barkup/patch";
// The model replies with operations addressing nodes by id:
// set-attribute, remove-attribute, set-name, remove,
// insert (with a fresh id), move.
const result = applyAnchoredPatch(grammar, storedTree, JSON.parse(reply));
if (!result.ok) return retryWithFeedback(result.issues); // verbatim
persist(result.node); // the input tree is never mutated Measured Anchored patches held 87 to 100% where whole-document rewrite fell to 0 to 80% on the cheap tier (chart). Across the whole series, zero failures were ever caused by a model mangling a stable id. Write-up: We Found the Crossover.
02 · Context · Studies I, J, U
"Make the hero headline match the promo card title" writes one node but reads another. Show the model a view containing only the write target and it will not error, refuse, or ask. It invents a plausible title, formats it perfectly, and applies it: 90 times out of 90 in our test, with zero refusals. Nothing in your logs will ever flag it.
Render a focused view: the relevant region expanded, everything else collapsed to id-bearing placeholders. A view of about 1.5k tokens matches full-document accuracy at 1 to 4% of the input, and its size scales with tree depth, not document size. But treat the focus list as a correctness contract: every node the request mentions, reads, or compares against must be in it, not just the node being written.
import { renderView, VIEW_PROMPT_RULES } from "@kevinpeckham/barkup/view";
// The request reads the card title and writes the hero headline.
// BOTH ids go in the focus list; the rest of the document
// collapses to id-bearing placeholders.
const view = renderView(grammar, storedTree, {
focus: [heroHeadlineId, promoCardTitleId],
}); Measured Target-only views: 0 of 90, every failure a silent invention. Views covering both nodes: 90 of 90 at 25 times less input than the full document (chart). Write-up: The Two Things Your Agent Can't See.
03 · Grounding · Studies L, N
"Fix the typo in the pricing section" names no node. Give the model a skeleton view of the document plus a single deterministic keyword-search tool and it grounds plain-language requests at oracle-level accuracy, at a median of exactly one call and a tenth of the input of showing everything.
Two things not to build: off-the-shelf embeddings measured no better than keyword overlap for structural references, and navigation tools (open node, list children) were a trap: oracle accuracy on the frontier model at higher cost than showing the whole document, and collapse on the cheap one.
import { findNodes, renderSearch } from "@kevinpeckham/barkup/view";
// One tool. Deterministic keyword scoring, no embeddings.
function searchTool(query: string): string {
return renderSearch(grammar, storedTree, findNodes(storedTree, query));
} Measured Skeleton plus one search call: oracle-level grounding on the frontier tier, median exactly one call (chart). Write-up: Then We Found the Cheap Part.
04 · Sessions · Studies M, P, S
What conversation history actually contributes to an editing session is teaching, not memory. Two canned worked examples of your patch dialect's tricky operations, about 900 tokens in the system prompt, restore stateless sessions to full-history accuracy. Measured through 36-edit sessions with no decay: edit 36 solved as well as edit 1, at a flat ~2.1k input tokens per step while keep-everything history grows linearly to 24k.
Pick the examples once, for the operations models actually fumble (an insert with a sibling anchor, a move). They are part of your system prompt, not per-session state, which also makes them cacheable under guideline 08.
// System prompt skeleton for a stateless editing turn: // 1. dialect rules (what operations exist, how anchors work) // 2. worked example: insert with a sibling anchor // 3. worked example: move // -- cache breakpoint -- // 4. fresh focused view of the current tree state // 5. the current request (plus memo and echo, guidelines 05-07)
Measured Stateless with two examples ties full history through 36-edit sessions at 5 to 6 times less input (chart). Write-ups: Two Examples Replace a Memory and The Thirty-Sixth Edit.
05 · Declared state · Studies T, V, W, Y
"Rename it to the codename we settled on" fails a stateless editor 100% of the time by construction: the answer lives in a conversation the model no longer sees. A short app-held memo of declared facts, standing rules, and goals, appended to every request, recovered every such callback at 2% extra cost.
The memo also carries qualitative goals at full parity with restating them in the request, where merely showing the model the document node the goal lives in lost 117 of 120 judged comparisons. Views carry values; memos carry goals. And you can delegate the writing: agent-extracted memos tied a perfect-oracle baseline on all three models tested, handled retractions, survived casual human phrasing at exact parity, and produced zero false notes from 432 conversational chatter baits.
## Session notes (app-maintained memo) Declared facts, standing rules, and goals from this session -- authoritative even when the conversation that declared them is no longer visible. Apply standing rules to every edit they cover without being reminded, and anchor goal-directed rewrites on the goals below. PRECEDENCE: a direct, explicit instruction in the current request overrides any note here for that request -- the memo carries standing intent, not vetoes (a one-off override is not a retraction; keep the note unless the user retracts it). Facts: - The product's launch codename is "Nightjar". Rules: - Write product names in title case. Goals: - The pricing page should read as reassuring, not salesy.
The benchmark-validated block, verbatim including the precedence sentence (guideline 06).
Measured Memo restores history-parity at 1.02 times stateless cost (chart); goals via memo win 117 of 120 over goals via view (chart); agent-written memos tie the oracle (chart). Write-ups: Views Carry Values, Memos Carry Goals and Who Writes the Memo?.
06 · The memo's fence · Studies AA, AB
The memo steers so strongly it has a failure mode: with a standing rule on file ("always include the trademark symbol") and a user asking for a one-off exception ("written plain, no trademark symbol"), the strongest model enforced the memo against the user 12 times out of 12.
One sentence inside the memo block header fixed every case: a direct, explicit instruction in the current request overrides any note here for that request. Placement is the whole trick. The same sentence class buried in a styleguide moved nothing; at the point of injury it restored 0 of 12 to 12 of 12 with zero cost to the memo's legitimate steering. The block in guideline 05 already includes it.
Measured Countermands honored: opus 0 of 12 without the clause, 12 of 12 with it (p = .0005); steering preserved 12 of 12 everywhere (chart).
07 · The previous turn · Study X
"Undo that." "Actually make it shorter." A stateless editor fails these 0 for 144, and every failure is a silent guess at what "that" was. The fix is one app-generated line describing the last applied edit, appended to the request. It restored full-history parity at half the cost, and beat keeping history outright on the production tier.
Your app already knows what it just applied; the echo is a string template, not a model call.
Previous edit (applied by the app): set "content" from "Q3 Draft Overview" to "Q3 Final Overview" on the text-atom "headline" (id n214).
Measured Echo ties full history overall and beat it on opus, 48 of 48 versus 46 of 48, at half the input (chart). Write-up: Undo That.
08 · Standing context · Studies Z, AA
Most production editors ship a standing block with every request: company facts, client records, a styleguide. It simply works at production sizes. Models copied exact facts past three same-schema distractor clients and applied unstated styleguide rules at any position in the pack, 216 of 216 per arm, with zero cross-client contamination in 324 cells. Do not bother slicing the pack per request: it bought nothing and forfeits prompt caching.
Layout matters for cost: put everything static (dialect rules, worked examples, the pack) in one block under a single cache breakpoint, and everything per-request (view, memo, echo, request) after it. The shipped layout read 64 to 68% of input from cache, cutting effective input cost by a quarter to nearly half.
One hazard: when a rule written with "always" collides with what the user asked for, models do not break the spec, they pick a reading, and which reading is not predictable across models. Softening the wording to "we generally prefer" resolved the conflict cleanly where a priority meta-rule in the styleguide did not. Audit your pack for absolutes.
const system = [
{
role: "system",
content: dialectRules + workedExamples + brandPack, // static
providerOptions: {
anthropic: { cacheControl: { type: "ephemeral" } },
},
},
{ role: "system", content: view + memo + echo }, // per-request tail
]; Measured Facts and rules 216 of 216 per arm, zero contamination in 324 cells; cached layout cut effective input cost 25 to 43% (chart); soft phrasing resolved the conflicts a meta-rule could not (chart).
09 · Bulk edits · Studies Q, R
"Change every button in the section" broke every strategy we tested, including perfect retrieval: models deliver roughly half of N targets and stop, confident. No prompt intervention rescued it, not exhaustiveness instructions, not a worked example, not asking it to count.
The fix is structural. Enumerate the targets with a deterministic query, then issue one small single-target edit per node. Deterministic work goes to deterministic code; the model only ever sees tasks shaped like the ones it solves at 100%.
import { selectNodes } from "@kevinpeckham/barkup/view";
// The app enumerates; the model edits one target at a time.
const targets = selectNodes(storedTree, {
type: "button-atom",
within: sectionId,
});
for (const id of targets) {
await applySingleTargetEdit(id, request); // guideline 02 per call
} Measured Decomposition: 90 of 90 tasks and 674 of 674 subtasks, at a third of the single-prompt cost (chart). Write-up: Your Code Finds the Targets Now.
10 · The seatbelt · Study AC
Even with everything above in place, some request will eventually need a value the model cannot see. By default it will not tell you: across five studies of constructed missing-context tasks, models produced hundreds of valid, plausible, silently wrong edits and zero clarifying questions. Not because they could not see the gap. Given permission, every model named the exact missing node every time.
The permission costs one sentence. With it, every model asked on every provably-unsolvable task (270 of 270, against 0 of 270 without), and never once asked when it already had what it needed. An ask_user tool performed identically, so the sentence is the whole fix. Two obligations come with it: wire the reply to an actual UI path (a question that dead-ends is worse than a correction loop), and keep guidelines 02, 05, and 07 anyway. The hatch is a seatbelt; making the question unnecessary is still strictly better than making it possible.
If the request requires a value or a node that is not visible in the view and not stated in the request, do NOT guess: reply with a single line "NEED-INFO: <what is missing and where you would need to read it>" instead of a patch.
The registered sentence, verbatim.
Measured Asks on unsolvable tasks: 270 of 270 with the sentence, 0 of 270 without; false asks on solvable twins: zero in 270 (chart). Write-up: The Model Always Knew What It Couldn't See.