https://www.lightningjar.com/research/barkup-bench/atom.xml Lightning Jar — Barkup Bench research 2026-07-23T12:00:00.000Z Lightning Jar https://www.lightningjar.com/research/barkup-bench/ao Study AO: Canonical-tag steering: the catalog read is the whole mechanism 2026-07-23T12:00:00.000Z Lightning Jar Unguided models never match a tag catalog's exact casing (0/36 clean everywhere); with a tags_list tool they consult it unprompted and land canonical on the first attempt; warnings alone cannot recover, and hard gating is unnecessary. https://www.lightningjar.com/research/barkup-bench/an Study AN: Ask or act: tool availability dissolves the visibility-clause tax 2026-07-22T12:00:00.000Z Lightning Jar With a view tool simply available, the ~70% NEED-INFO tax on fully specified edits vanished on every tier; the tested fetch-before-ask prompt clause was redundant on the frontier and unsafe below it, so it stays unshipped. https://www.lightningjar.com/research/barkup-bench/al Study AL: The gate fails on a moved baseline; and unproven does not ship 2026-07-18T12:00:00.000Z Lightning Jar Study AK left one pathway open: the app-side eviction cannot restore a note the model pruned before sending, and the mid tiers pruned goals client-side in 4 to 6 cells of ten. https://www.lightningjar.com/research/barkup-bench/am Study AM: The notice learned to ask for everything back 2026-07-18T12:00:00.000Z Lightning Jar Twice in two studies; both times unregistered; opus answered an eviction notice by re-sending the memo compressed into fewer sentences carrying all twenty-one needles: told a note had died, the frontier tier invented l. https://www.lightningjar.com/research/barkup-bench/aj Study AJ: The error message didn't matter 2026-07-17T12:00:00.000Z Lightning Jar Returning barkup's structured validation issues verbatim has been a design commitment in every arm of this series, the closing instruction of playbook guideline 01, and standing guidance in the production codebase, and i. https://www.lightningjar.com/research/barkup-bench/ak Study AK: The fix works where it can, and the frontier tier invented consolidation 2026-07-17T12:00:00.000Z Lightning Jar Study AH found that a twenty-first declaration at the memo's full cap always kills a note, and the victim is always a goal. https://www.lightningjar.com/research/barkup-bench/ah Study AH: Perfect to the cap. At the cap, the goals die first. 2026-07-16T12:00:00.000Z Lightning Jar Every study that shipped the session-notes memo measured it at three to six notes; the shipped implementation caps it at twenty, with a normalize step that silently drops the excess. https://www.lightningjar.com/research/barkup-bench/ai Study AI: The obvious fix, measured: it helps exactly where we don't ship 2026-07-16T12:00:00.000Z Lightning Jar Study AE left an obvious fix on the table: the shipped ask sentence covers what the model cannot see, so ambiguous references (visible, but two of them) slipped through on the mid tiers. https://www.lightningjar.com/research/barkup-bench/ag Study AG: The discourse gap closes on every tier; the visibility clause bites 2026-07-16T12:00:00.000Z Lightning Jar One border of the ask-path map was never tested: Study X's discourse construction, where "undo that" against a carrier-less editor failed 0 of 144 with every failure a valid silent guess. https://www.lightningjar.com/research/barkup-bench/ad Study AD: The core stack transfers to the tier we actually ship 2026-07-15T12:00:00.000Z Lightning Jar Studies F through U validated the core editing stack on sonnet-4.5 and gemini-3.5-flash, but the downstream template-editing surface runs claude-opus-4.8, a tier whose benchmark data began at Study W. https://www.lightningjar.com/research/barkup-bench/ae Study AE: Only the frontier knows when a question is the right answer 2026-07-15T12:00:00.000Z Lightning Jar Study AC left two fears on the record: that the escape hatch would tax clear requests with unnecessary questions, and that an ask might be a dead end. https://www.lightningjar.com/research/barkup-bench/af Study AF: Saying the goal out loud doesn't make it yours 2026-07-15T12:00:00.000Z Lightning Jar One clause in the shipped session-notes rule had been inferred rather than measured: restate a goal from the memo in your own words before a goal-directed rewrite. https://www.lightningjar.com/research/barkup-bench/aa Study AA: We tested our own headline and lost: strictness is not a capability gradient 2026-07-13T12:00:00.000Z Lightning Jar Study Z’s spec-conflict split was an accident with one conflict shape, so Study AA measured it on purpose: three registered conflict kinds (rule vs instruction, an explicit user countermand of a rule, rule vs rule), four. https://www.lightningjar.com/research/barkup-bench/ac Study AC: The model always knew what it couldn’t see 2026-07-13T12:00:00.000Z Lightning Jar Twenty-eight studies of silent failure (90/90 invented values in U, 144/144 silent guesses in X, 120/120 oblivious polishes in V, zero clarifying questions anywhere) never once OFFERED the model a way out. https://www.lightningjar.com/research/barkup-bench/x Study X: "Undo that" needs an echo, not a transcript 2026-07-12T12:00:00.000Z Lightning Jar The last structurally unmeasured request class: follow-ups that point at the previous edit; "also set that same node's…", "apply the same change to X", "actually, undo that." Anaphora steps ran against skeleton views (a. https://www.lightningjar.com/research/barkup-bench/y Study Y: The memo survives how people actually talk 2026-07-12T12:00:00.000Z Lightning Jar Every declaration in Studies T, V, and W was announced formulaically ("For later reference: the codename is X"). https://www.lightningjar.com/research/barkup-bench/z Study Z: The brand pack works; the rule you forgot is the hazard 2026-07-12T12:00:00.000Z Lightning Jar Production doc editors ship a standing context block with every request; company facts, client records, a styleguide; and nobody had measured whether models actually use it. https://www.lightningjar.com/research/barkup-bench/v Study V: Views carry values, memos carry goals 2026-07-11T12:00:00.000Z Lightning Jar The series' first judge-graded study, run under a pre-registered pairwise protocol: both judges (gpt-5.4 primary, haiku-4.5 sensitivity; neither an editor) first had to pass a 50-pair calibration gate with planted verdi. https://www.lightningjar.com/research/barkup-bench/w Study W: The agent writes the memo faithfully; even when history makes it redundant 2026-07-11T12:00:00.000Z Lightning Jar Studies T and V validated the memo with oracle extraction: the harness knew what to record. https://www.lightningjar.com/research/barkup-bench/s Study S: 36 edits later, nothing broke; except the cost curve 2026-07-10T12:00:00.000Z Lightning Jar Every prior session study ran 12 edits. https://www.lightningjar.com/research/barkup-bench/t Study T: The one thing statelessness can't do; and the three note lines that fix it 2026-07-10T12:00:00.000Z Lightning Jar Every prior session study used self-contained requests, so Study T built the missing class: callback steps whose required fact lives only in earlier conversation; a declared codename a later rename must use, and a stand. https://www.lightningjar.com/research/barkup-bench/u Study U: The view that can't see the answer doesn't fail; it makes something up 2026-07-10T12:00:00.000Z Lightning Jar Focused views assume the instruction carries every value an edit needs. https://www.lightningjar.com/research/barkup-bench/n Study N: Stop walking, start searching: one find_nodes call grounds the edit on both model tiers 2026-07-09T12:00:00.000Z Lightning Jar Same id-free tasks as Study L; the expand tool replaced by a single content-search tool (a few words in, the 5 best keyword matches out, shown in place). https://www.lightningjar.com/research/barkup-bench/o Study O: Printing the position on every node does not rescue stateless sessions 2026-07-09T12:00:00.000Z Lightning Jar Study M's stateless failures were all placement edits, so Study O annotated every view child with its true 1-based position (plus a prompt line mapping ordinals to anchors) and re-ran the 2×2 of history × positions. https://www.lightningjar.com/research/barkup-bench/p Study P: History was a teacher all along: two canned examples replace the whole conversation 2026-07-09T12:00:00.000Z Lightning Jar If history isn't state (the view carries that) and isn't positional help (Study O), maybe it's worked precedent; and precedent can be faked. https://www.lightningjar.com/research/barkup-bench/q Study Q: "Change every X inside Y" breaks everything; including the oracle 2026-07-09T12:00:00.000Z Lightning Jar One instruction, 2–32 targets, on the same large trees. https://www.lightningjar.com/research/barkup-bench/k Study K: Sessions drift unless you re-show the tree; and showing a fresh view every turn is also the cheapest policy 2026-07-08T12:00:00.000Z Lightning Jar Twelve sequential patches against one evolving tree. https://www.lightningjar.com/research/barkup-bench/l Study L: Finding the node is the expensive part: grounding costs 7–9 points, and navigation is a frontier-only trap 2026-07-08T12:00:00.000Z Lightning Jar Same edits, instructions rewritten without ids ("the image-atom named 'maple-ember'"), every description verified to match exactly one node. https://www.lightningjar.com/research/barkup-bench/m Study M: The view carries the state, but history still earns its keep 2026-07-08T12:00:00.000Z Lightning Jar If a fresh view arrives every turn, does the model need conversation history at all? No; statelessness confirmed the constant-cost economics (~1.3k input tokens at step 1 and step 12 alike) but failed the accuracy gate:. https://www.lightningjar.com/research/barkup-bench/h Study H: The crossover, found: above ~300 nodes, only anchored patches hold 2026-07-07T12:00:00.000Z Lightning Jar Whole-tree rewrite becomes frontier-only at scale (gemini-flash: 0/15 at ~1000 nodes; sonnet needs a streaming transport and ten minutes per rewrite). https://www.lightningjar.com/research/barkup-bench/ij Study I/J: The model doesn't need to see the tree: a ~1.5k-token view matches the full 85k-token input 2026-07-07T12:00:00.000Z Lightning Jar Show only the path to the referenced nodes plus their child lists (everything else collapsed or omitted with a count), and anchored-patch accuracy is statistically unchanged for both models at every size; sonnet on the . https://www.lightningjar.com/research/barkup-bench/g Study G: One hidden SDK default, two very different benchmarks 2026-07-06T12:00:00.000Z Lightning Jar The original +33pp multi-turn gap was manufactured by conversation histories that omitted the model's own tool calls (the AI SDK v5→v7 response.messages trap).