The Last Job
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. Twelve studies ago, the app asked language models to do most of its hearing. Study by study, each of those jobs went to something better. Tempo, key and meter went back to the classical detector that was there all along, vindicated twice. Chords went to transcription plus a template matcher. Structure on synthetic arrangements went to a which-stems-are-sounding check, and structure on real recordings to a 1.5 megabyte neural segmenter. The counting ambiguities went to a data-gated ladder. After Study 8, exactly one hearing job remained with the model, because it had measured perfect there and nothing else could do it: placing section boundaries in seconds against the actual recording.
Study 12 took the last job with fifty-year-old math. Render the chart as dumb sustained chord pads at the declared tempo. Dynamic-time-warp the render against the recording. Read every bar's timestamp off the warping path, and with the bars, every section mark and every chord. No model, no weights, no spend.
The Numbers
On the synthetic corpus: 95.0 percent of 560 bar starts landed within a quarter second, the errors living where physics says they should (reverb tails, dense mixes). All eight contrast section boundaries landed within half a second, and that is the headline, because half-second precision is the metric every listener failed: Study 8's models scored zero of eight there, Study 10's neural segmenter zero of eight, and the literature's best reported model manages 0.423 on real music. Seven of eight tempo-change boundaries landed too, boundaries that are structurally invisible to stem activity and that blind models found half the time. Dynamic time warping does not care that the tempo moved. That is the method.
And the probe we almost did not need: aligning a recording against the wrong song's chart costs 8.6 to 16.3 times more than the right one, on every registered pair. "Is this chart this recording?", a question production currently asks the model to confirm by ear, is a number now.
The Gate that Failed, One Boundary Short
The registered real-recording gate failed, and it deserves full weight, because its anatomy is the study's finding.
On the band's real recording, the six mid-song boundaries landed between 0.02 and 0.11 seconds of the declared truth. Nothing in twelve studies has come within an order of magnitude of that on real audio. The three misses were the edges: the intro boundary off by 3.6 seconds where the alignment's open start absorbed the real intro, and the outro off by 9.1 where the chart's eight-bar outro reading met a three-second performance tail. Six of nine inside half a second, against a registered seven. The neural segmenter edges it on that one song by being merely mediocre everywhere instead of perfect in the middle and lost at the ends.
A second real song, carried with a deliberately uncertain transcription, behaved exactly as predicted: clean placements until the uncertain bridge bars, then visibly adrift, then re-anchored by the outro. And a third public chart could not be transcribed at all, which is a result, not an excuse: it marks the real-world limit of chart-driven anything.
The Constraint Moved
Put those three songs together and the moral is plain. Alignment is sub-tenth-second wherever the performance follows the chart, and it is lost exactly where the chart stops describing the performance. The binding constraint on this feature is no longer hearing. It is chart quality, which is a data problem, not an AI problem, and data problems have mechanical fixes: ship the alignment behind a path-cost gate (the probe's order-of-magnitude floor separates right charts from wrong ones), flag the first and last boundaries as the ones carrying chart-fidelity risk, and show those hedged, with the user as the verifier, the same pattern the app already uses for ambiguous tempo. If the edge flag is accepted as interface, the model's remaining role in this feature is nothing.
The bench registered its own gate and reports its failure by letter, as it has for every study in this series. The ledger line is worth a sentence of its own: this study cost zero dollars, and everything since Study 9 has cost about eight cents in total.
And the arc closes on a sentence we did not expect to write when this program started. The models' one genuine hearing ability, the thing Study 8 went looking for and honestly found, texture contrast, was overtaken twice: by the stems, which name the form for free, and now by the chart, which places it to the hundredth of a second. Every hearing task the app ever asked of a model now has a code-or-data answer that beats or matches it. The models keep the jobs the series kept finding they were actually good at, which were reading jobs all along.
The registration, the aligner, both arms and the full report live in the stem-bench repository; the Study 12 page is landing on the research dashboard. The series neighbors: The Home-Field Rematch and They Hear Texture, Not Form. The app is Stem Shovel.
