The Stems Name the Form, the Sum Keeps the Time
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. It is built around stems: users upload a song as separate tracks, and the app plays them in sync. But every analysis feature in the app listens to the sum. The tempo detector mixes all the stems' onset envelopes into one autocorrelation, and the AI features only ever receive the mixed-down audio. The one signal the app uniquely has, the separated tracks, went unused everywhere.
The stem-shovel agent drafted the study asking whether that is a waste, and the bench finished and registered it, the second study in this program designed across that agent boundary. Study 9 of stem-bench built six songs that each carry a stem engineered to sabotage the sum: a rubato lead that floats over the grid, an off-grid pad, a noise and room-tone bed, a comp part playing double time. Four analysis instruments ran over 21 items at zero model cost, with the stems shuffled so nothing knew which track was the drums.
The Sabotage Failed
The summed envelope, production's supposedly naive design, was exact on 18 of 21 items, including all six sabotage items. The pad, the noise bed, the rubato lead, both double-time comps: none of them moved it.
The registered alternative, picking the single sharpest stem and trusting its tempo, scored 11 of 21, with seven regressions against the sum and zero rescues. Five of the seven regressions are octave errors, and the mechanism is almost funny: a clean bass line autocorrelates beautifully, at half time. Sharpness rewards exactly the stems that keep a different clock.
So the gate failed decisively, and the failure is the finding, the second time in this program that losing a registered bet has been the most valuable outcome. Summing the stems is not a compromise production settled for. The cross-stem reinforcement at the true period is the tempo signal, and isolating any one voice destroys it. For honesty's sake: the sum's three misses are a shared systematic, compound meters read at their 3 to 2 dotted-pulse confusion, and no stem-aware variant fixed those either. That one belongs to the chart, not to more signal processing.
But the Stems Do Hold One Thing the Mix Hides
The arrangement. A detector that asks nothing more than "which stems are sounding in each bar" and calls a boundary where the active set changes found 8 of 8 section boundaries with zero false boundaries on the uniform loops, identically on the truth grid and on the production-portable detected grid.
Compare Study 8: the models found those boundaries blind too, at 7 and 8 of 8, but invented three to six sections per uniform loop, and even the chart-anchored production model still invented about one. The activity detector cannot invent structure on a loop, by construction: the active set never changes. Its blindness is equally structural, since all six harmony-only and tempo-only boundaries are invisible to it; those belong to the chart. One honest caveat from the sabotage class: a noise track with slow swells toggles its own activity bit, so a shipped version wants the atonal-stem guard the study's other outputs already provide.
Free, browser-side, no model call. The division of labor the series has been converging on gains a third column: the chart states the names, the model places the seconds, and the stems, it turns out, name the form.
The Confidence Trap
The study's quietest result may save the most future pain. A per-stem key agreement score, how many stems individually agree with the sum's key, looked like a natural confidence signal. On this corpus, production's key detection was right 20 of 21 times, and the agreement score gave its maximum value, a perfect 1.00, to the one wrong answer in the whole study. Production's existing confidence measure correctly ranked that same answer third-lowest of 21. A confidence signal that points the wrong way on the only error available is worse than no signal, and it would have shipped looking plausible.
The small model-side pilot closed the remaining door: feeding the model an isolated harmonic stem instead of the mix helped nowhere and hurt twice, with the tempo answers still the familiar genre prior either way. Stem isolation does not make a listener out of a reader.
The scorecard stays honest: the bench's own prediction about the misleading stems was refuted in both halves, since the sum survived the double-time comps and the sharpest-stem rule regressed rather than rescued. The ship guidance went over as three sentences: ship nothing for tempo and key, the current detector won; consider the activity detector for sections, guarded against sustained noise; and keep the model for what text and signal cannot do.
The registration, instruments, and full report live in the stem-bench repository; the Study 9 page is landing on the research dashboard. The series neighbors: They Hear Texture, Not Form and The Fact Line Beats the Instruction. The app is Stem Shovel.
Update (2026-10-30)
The next study withdrew one piece of this article's ship guidance. Study 10 re-ran the stem-activity detector on real recordings, where it collapsed on real dynamics: 25 of 33 section boundaries with 19 false boundaries. Its perfect score here was an artifact of our synthetic arrangements, so we retract the suggestion to consider it for section detection.
