Study 1 · The chart-draft pipeline
The hint-following study: models that can read the notes don't copy the hint
The chart-draft pipeline
Study Overview
The Question
Stem Shovel, Lightning Jar's music collaboration app, drafts a chord chart for a band by showing a model what Spotify's Basic Pitch transcriber heard in each bar of their recording, plus a free template matcher's per-bar chord guess in brackets. The prompt calls the guess "a hint, not an answer," but nobody knew whether the model reads the noisy notes or copies the hint. If it mostly copies, the frontier call is laundering a free heuristic's output; if it mostly reads, wrong hints should barely hurt. Study 1 made the question answerable by construction: 16 hand-authored songs (440 bars, known chords), synthesized to audio, transcribed by the production pipeline with its exact model files and thresholds, and drafted under four arms that vary only the hint: none, the production matcher's, the truth, and a systematically wrong transform (root up a perfect fourth, quality kept).
Ground Truth by Construction
Every prompt text is the production app's verbatim, pinned by test; grading is mechanical (same root pitch class, same canonical quality, enharmonics equal) and committed before any scored run. The free template matcher's own baseline, known at registration, is 90.5% (398 of 440 bars), with a weakness it earns honestly: on the study's power-chord song its bare-fifth penalty names root-plus-fifth bars as major or sus chords and scores 3 of 28. Five models ran the full 64-cell grid: the production draft model claude-fable-5-1, the pilot gemini-3.5-flash, gpt-oss-120b, haiku-4.5, and kimi-k2.6.
The Gradient
The registered headline number is the flip rate: of the bars a model answered correctly with no hint at all, how many did it surrender to a wrong hint? The answer is a capability gradient, not a universal. The two strongest models never flipped a single bar: zero copied roots in 438 chances each, roughly 850 combined. gpt-oss-120b flipped 22.6%, and bimodally: in two songs it surrendered to the wrong hints wholesale (0 of 32 and 0 of 28) while reading the rest clean. haiku-4.5 flipped 49.5% and kimi-k2.6 flipped 69.1%. The registered prediction got the shape right and the universality wrong: it expected wrong hints to hurt every model, and they did not hurt the strong ones at all.
Hints Carry Notation, Not Pitch
The pilot model showed why wrong hints can even help a strong reader. On the power-chord song with no hint, gemini wrote the right roots in the wrong spelling (plain majors instead of fifth chords) and scored 4 of 28 under strict grading. Shown wrong-rooted hints that carried the "5" suffix convention, it wrote the right roots in the right spelling: 28 of 28. The hint acted as a style guide, never as a source of pitches. fable-5-1 went one better on honesty: shown wrong hints, it flagged the disagreement with the matcher in its notes field in 11 of 16 songs. kimi flagged zero.
The Production Design Holds, and Nothing Cheaper Does
The shipped configuration (notes plus the matcher's hint) was the best arm for every model, and for the production model it hit 99.8% bar accuracy, 9.3 points over the free matcher alone, at about $0.10 a draft. The same configuration rescued the cheap models without saving them: haiku rose 38 points to 87.5% and kimi 47 points to 75.5%, both still below the bare matcher's own 90.5%. Reading noisy transcription with no help at all, only the frontier tier beat the free heuristic (gemini 94.1%, fable 98.5% on answered cells). The model choice behind Stem Shovel's chart draft now rests on a measured number instead of a vibe.
The Letters, Honestly
The registered gate asked for a significant correct-hint lift in the pilot model and in the pooled matrix. The pooled matrix passed decisively (37.7 points, p < 0.00005); the pilot clause failed by letter at p = 1.0, because the pilot model was at ceiling everywhere but one song and a paired test with one informative pair cannot reach significance at any effect size. The gate fails as written and both readings are published. One protocol amendment was committed before any affected scored cell: haiku rejects the production thinking settings outright, and kimi with default reasoning spent its entire output budget thinking in 62 of 64 cells, burning $8.70 to produce no JSON, which independently reproduces the reasoning-off lesson already hard-coded in Stem Shovel's own text-to-beat path. The archived failure is published as evidence. One more artifact is on record: a single prompt that deterministically returns an empty completion from fable (zero output tokens, reproduced), scored 0 of 28 under the registered no-retry rule.
What This Licenses
For Stem Shovel: keep the hint line and keep the frontier draft model; the hint is free accuracy for fable and poison only in the hands of models the product does not use. Two specific fixes went back to the app's FINDINGS file: the matcher's bare-fifth penalty mis-names power-chord bars the draft model then has to rescue, and a one-retry-on-empty guard would close the deterministic empty-completion edge. For everyone else: hint-following under pressure is a measurable, strongly tiered property, and a scaffold that is safe decoration for a frontier model can quietly become the answer sheet for a cheap one.
Production Response, and Amendment 2
A benchmark about a product is most useful when the product can answer, and Stem Shovel answered the same day. Version 0.142.0 shipped both fixes the study sent back, citing it: the matcher's flat bare-fifth penalty became a missing-tone penalty (a bar with no third no longer reads as a major chord), and the chart draft gained one retry on an empty completion. The bench re-ported the new matcher, registered Amendment 2, and re-ran the shipped-configuration arm with the hints it now produces: 80 cells, zero errors. The hint itself rose from 90.5% to 96.4% of bars, and every model that leans on it rose with it: kimi 75.5% to 94.8%, near its correct-hint ceiling; haiku 87.5% to 92.5%; gpt-oss 85.9% to 92.0%; gemini 94.3% to 98.2%. fable moved within noise (99.8% to 99.3%, and zero wrong hints propagated under either matcher). The flip rate is a wrong-hint-arm measurement and is unchanged by construction. The numbers elsewhere on this page stand as run, against v0.141. The sharpened conclusion is less flattering for the cheap tier, not more: with the better matcher, haiku, kimi, and gpt-oss all score below the bare matcher's own 96.4% in the shipped configuration, so the hint is doing their work for them; only fable and gemini add value over the hint they are given. Full tables are in the repository's REPORT.md under "Production response and Amendment 2".