Thirty-Eight Percent
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. Twelve studies into the series, the models were retired from hearing. One question from the literature stayed open: the reconciler, the single published shape in which a language model improves an audio pipeline, by reading its symbolic outputs and fixing their disagreements, never touching the audio. The team asked whether it still had value. The bench's answer was a registered conditional: first measure the thing it would improve, then run it only if there is headroom. Study 12's alignment had just made that measurement possible for the first time, a bar grid in recording seconds plus bar-aligned chord truth on a real song.
There was headroom.
The Number the Title is Hiding from Nobody
Production's chord path scores 38.2 percent on a real recording.
This is the pipeline that scored 95.8 percent at home on the synthetic bench, the proud winner over a neural chord model in the home-field rematch. On the real song it reads 29 of 76 bars, and that is with a registered allowance that forgives power-chord voicings doing real work; strict grading says 20. The second, exploratory song reads worse. We predicted a home-field gap of 15 to 30 points; it is nearly 60.
The culprit is not the matcher and not any model. It is the note transcription meeting real synth pads, guitar voicings and sustained textures. The very first week of this program left a fixture note warning that rich harmonics defeat the transcription; Study 13 measured it in production conditions instead of suspecting it. The honest implication deserves plain words: every synthetic benchmark number in this series was a ceiling, not an estimate, and the chord-draft feature deserves a user-interface hedge today, regardless of what else happens.
The Redemption, With Our Own Trap Defused
Arm two handed the model production already runs for chart drafts one text-only pass: the key, the detector's chords, and the transcribed notes per bar. Symbols in, symbols out.
It lifted 29 of 76 to 41 of 76. Twelve fixes, zero breaks, across 27 changed bars. The gate was registered specifically against the trap Study 9 caught, where a plausible consistency signal pointed the wrong way on the only error available: a reconciliation pass had to fix at least twice what it broke. With zero breaks, it passed at infinity, for both models tested. The cost is about a cent a song.
That is the first time in thirteen studies that a language model has improved a hearing result. It did it by never hearing anything, which means the twelve-study law was applied, not broken: models do reading jobs, and the reconciler is a reading job that happens to live inside an audio pipeline.
The Coda
Three disclosures ride along. Our own harness truncated all four first pilot calls at its registered 4000-token output cap while the models narrated their way bar by bar toward the JSON, the same self-inflicted budget wound as Study 8, now twice; the cap was amended to 16,000 and the truncated run stays in the history. Gemini then passed the gate cleanly on the gated song and burned the entire 16,000-token budget narrating the other without ever reaching the JSON, which is that family's budget failure mode from Studies 2 and 8, not the over-editing we predicted for it. And the ceiling gets named rather than celebrated: 54 percent reconciled is better, not good. If the chord feature matters, the next dollar goes to a stronger transcription on real timbres, not to more model.
The ship shape went over with the findings: reconcile inside the chart-draft job the app already runs, keep the detector's chord wherever the model's answer parses to nothing, cap output generously, and hedge the chord hints in the UI until the transcription improves. The ledger: about sixty cents, including the truncated pilot; the program's total model spend since Study 9 is about seventy cents across five studies.
The registration, amendment, both arms and the full report live in the stem-bench repository; the Study 13 page is landing on the research dashboard. The series neighbors: The Last Job and The Home-Field Rematch. The app is Stem Shovel.
