The Models That Can Read Music Don't Cheat
Stem Shovel, our music collaboration app, will draft a chord chart from a band's recording. The pipeline is humble: Spotify's Basic Pitch transcribes the stems into a noisy pile of notes, a small template matcher makes a free guess at each bar's chord, and a frontier model reads both and writes the chart. The prompt hands the model the matcher's guess with a warning attached: "a hint, not an answer."
We wrote that prompt, and we could not have told you which one the model was actually using. If it mostly copies the hint, an expensive frontier call is laundering the output of a few dozen lines of free code. If it mostly reads the notes, a wrong hint should barely matter. Nobody knew, so we measured it.
This is the start of our third pre-registered research series, stem-bench, which points the discipline we built for Barkup Bench and AEO Bench at the AI features inside Stem Shovel.
How the study works
We hand-wrote sixteen songs as data: 440 bars, one chord per bar, keys around the circle, the chord qualities a band's chart actually uses. A deterministic synthesizer renders each song to audio, and the production Basic Pitch pipeline transcribes it, with the same model files and the same thresholds the app ships. The transcription comes out realistically messy, which is the point. Because we wrote the songs, we know the right chord for every bar anyway.
Then we ablate the hint. Four arms, identical except for the bracketed guess beside each bar: no hint at all, the production matcher's real guesses, the correct answer, and a systematically wrong answer (every root moved up a fourth, a musically plausible neighbor that the notes in the bar contradict). Five models, one call per song per arm, mechanical grading, everything registered by commit before the first scored run.
The number we registered as the headline is the flip rate: of the bars a model names correctly when there is no hint, the share it hands over to a planted wrong hint when one appears.
Models that read music ignore wrong answers
Hint-following turned out to be a capability gradient, and a steep one.
Claude Fable and Gemini Flash copied a wrong hint zero times. Not rarely: zero, in 876 chances between them. The planted wrong chords, each sharing a key and a plausible function with the truth, moved nothing. gpt-oss-120b flipped 23 percent of the bars it could read on its own, Haiku 4.5 flipped 50 percent, and Kimi K2.6 flipped 69 percent. The weaker the model's own reading, the more the hint became the answer.
The strong models did take one thing from the hints: notation. Our power-chord song is all bare fifths, and with no hint the models spell those bars as plain majors, right root, wrong suffix. Given hints written as power chords, even hints with wrong roots, they kept the suffix convention and took the pitches from the notes. The hint works like a style guide, not an answer key.
Fable did one more thing we did not ask for: shown wrong hints, it flagged the disagreement in its notes field in eleven of sixteen songs. Kimi, which copied the most, flagged it in zero.
What this buys the product
The shipped design held up. Notes plus the matcher's hint, read by Fable, scored 99.8 percent of 440 bars, nine points better than the free matcher alone, for about ten cents a draft.
The cheap substitution we half-expected to recommend does not exist. Haiku and Kimi, run in exactly the shipped configuration, scored below the bare template matcher. Swapping the frontier model out to save money would make the chart worse than the free code it was supposed to improve on.
What went wrong, because something always does
Our registered gate failed on a technicality worth recording. Its pilot clause demanded statistical significance from the pilot model, which turned out to be at ceiling: fifteen of sixteen songs perfect with no hint, one informative pair, p stuck at 1.0 no matter the effect size. The pooled matrix clause passed at p < 0.00005. Lesson registered: a gate clause aimed at a model at ceiling measures the fixture, not the mechanism.
Two models also needed corrected call shapes, committed as an amendment before any of their scored cells. Haiku rejects the adaptive thinking setting our production chart draft uses. And Kimi, called with its default reasoning on, spent its entire output budget thinking and returned no answer in 62 of 64 cells, which cost $8.70 and independently reproduced a lesson we had already learned in production: for this family of tasks, Kimi needs reasoning off.
Every study in this series grades with committed mechanical code, no judged scoring, which matters here for a disclosure we will keep repeating: the analyst model behind this research is Claude Fable 5, and the matrix measures its sibling. The graders do not care.
The full registration, fixture, and results live in the stem-bench repository, and the dashboard at /research/stem-bench will carry each study as it lands.
Postscript (2026-10-21)
Stem Shovel adopted the result the same day. Version 0.142.0 replaced the matcher's flat power-chord penalty with a missing-tone penalty and added a retry on empty completions, both changes citing the study. We re-ported the new matcher into the bench, registered an amendment, and re-ran the shipped configuration against it: 80 cells, about two dollars, zero errors.
The design held up, and the free half of it got better. The matcher's own accuracy rose from 90.5 to 96.4 percent, and every cheaper model rose with it: Kimi gained 19.3 points to 94.8, Haiku 5.0 to 92.5, Gemini 3.9 to 98.2, gpt-oss 6.1 to 92.0. Fable was unchanged within noise and still propagated zero wrong hints. The flip rate above is measured in the planted-wrong-hint arm, so by construction it does not move.
What sharpened is the substitution argument. With the better matcher, the three cheaper models now score below the bare matcher itself in the shipped configuration: they subtract value from the hint they are handed. Only Fable and Gemini add anything over the free code. The full numbers live in the repository's REPORT.md, under Production response.
