The Home-Field Rematch
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. It hears with classical code. Tempo is an autocorrelation of onset envelopes, key is a Krumhansl template correlation, chords are a hand-rolled matcher. The app's own agent, surveying the music-information-retrieval literature, called that stack obsolete: published neural models beat classical baselines by twenty points, and it drafted a study naming four real candidates, small, permissively licensed, runnable on a CPU. Beat tracking, section segmentation, chord recognition, plus two literature key profiles the papers describe as superior.
Study 10 of stem-bench is the third study in this program designed across the agent boundary, and the bench's first move was adversarial in the right direction: verify the challengers exist. Weights downloaded and hash-pinned against published tables, runners built from vendored reference code, a new hard set designed from the review's own cited failure cases (swing, octave-boundary tempi, reverb, dense mixes), and the whole study ran for zero dollars. No language model touched it anywhere.
Act One: All Five Gates Fail, Each In Production's Favor
On the bench's synthetic corpus, the classical stack won every registered comparison. The nuances are where the ink belongs.
The beat tracker's loss is the program's fifth registration lesson. At the registered tolerance of plus or minus 1 bpm, production was exact on 29 of 33 items and the neural tracker on 18. But the neural model derives tempo from a median beat interval on a 20 millisecond frame grid, so it answers 142.9 for a true 140: musically right, numerically outside the letter. At the model-grade tolerance our other studies use, the two are dead tied at 29 apiece, with different misses. An absolute tolerance set at DSP precision measures the candidate's output quantization, not its hearing. And the letter cannot take away what the tracker genuinely has: a per-beat grid at 0.892 F-measure that production simply does not possess, and honest near-silence on the two pulseless pads where our classical detector structurally must answer something.
The segmenter's loss needs no such nuance. LinkSeg invented 54 boundaries across the 22 songs that have no structure at all, about two and a half per song, worse than the chart-anchored language models of Study 8. The loop-invention disease is not cured by graph attention. On synthetic material, the free which-stems-are-sounding detector from Study 9 beat the published neural segmenter outright.
If the story ended there, it would be a comfortable one: naive classical code, vindicated. It does not end there.
Act Two: the Rematch on Real Recordings
A team member pointed out that the MMKK catalog (songs composed, performed and written by a member of the team) is public, so a registered amendment ran the same instruments against the band's real songs and the band's own declarations, which turned out to include declared section start times: data entry of bench-grade quality. The domain warning every benchmark carries in fine print cut both ways at once.
The segmenter flipped from structure-inventor to structure-finder. LinkSeg found 29 of 33 declared boundaries on the real songs, including one song at eight of nine with zero false boundaries. And Study 9's stem-activity detector, perfect on synthetic arrangements, collapsed on real dynamics: 25 of 33 with 19 false boundaries. We therefore retract Study 9's suggestion to ship it, which the real recordings exposed as a synthetic-corpus artifact. This is the program correcting itself again, the same shape as the PDF correction a study earlier in the series, and we would rather keep doing that in public than pretend the first answer was final.
Production's tempo detector kept its known failure classes, now on real music: it read exactly half of one song's declared 224 and exactly two thirds of another's declared 145. The octave and 3:2 confusions are not synthetic curiosities; they are what autocorrelation does to real arrangements, and the neural tracker degrades more gracefully there without being accurate enough to replace anything.
And the key question closed hard. The literature profiles that looked interchangeable on synthetic diatonic pop, one point apart across 33 items, collapsed on real mixes: Krumhansl called 4 of 6 real songs correctly, the two celebrated replacements 1 of 6 each, failing into parallel-minor and subdominant flips. The twenty-point literature gap did not just fail to appear; it reversed.
The Moral, Stated Plainly
A benchmark's corpus is a thumb on the scale. Our synthetic arm made the classical stack look unbeatable and the neural segmenter look broken; the real arm broke one of our own shipped suggestions and rehabilitated the segmenter in the same hour. Neither arm alone would have told the truth. The only honest fix we know is to run both domains and report the flip, which is what this study does.
The ship guidance that survives both acts: keep the tempo detector exactly as it is, keep Krumhansl without a second thought, and adopt LinkSeg, a 1.5 megabyte CC-BY model, only as the section fallback for songs without charts, behind the chart whenever one exists. Five gates, two corpora, zero dollars, and the app changes by one small fallback.
The registration, the hash-pinned weights, both arms and the full report live in the stem-bench repository; the Study 10 page is landing on the research dashboard. The series neighbors: The Stems Name the Form, the Sum Keeps the Time and They Hear Texture, Not Form. The app is Stem Shovel.
