They Hear Texture, Not Form
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. Four studies into measuring Stem Shovel's listening features, our verdict had hardened into a refrain: the models read, they do not hear. Study 4 caught the song-check transcribing whatever text rode beside the audio. Study 7 re-measured the rebuilt feature and confirmed the pattern in a new costume. So for the last unmeasured AI feature in the app, section detection, we went looking for the loophole in our own verdict.
The loophole was always visible in the data. Study 4's blind arms failed every number while still noticing gross structural events in prose: "the tempo doubles here," "the texture thins." Section boundaries are gross structural events. So Study 8 of stem-bench built mixes whose sections alternate sparse texture with a full band, boundaries a human hears instantly, true by construction, and asked the fairest version of the question: is anything in there actually listening? The registration committed in advance to publishing a yes: blind success would be the program's first evidence of useful hearing.
The First Yes
Blind, with no chart and no hints, the production model found seven of eight texture boundaries, and the newer Gemini found eight of eight. The band dropping out and the band coming back are audible to these models. After four studies of "it was reading the whole time," that is the program's first genuine listening result, and we are publishing it with the same prominence we gave the failures.
The Edges of the Yes
It is texture they hear, not form. Boundaries marked only by a harmony change or a tempo change landed half the time blind. And mixes that are one uniform sixteen-bar loop, with no section change anywhere, came back subdivided into invented verses and choruses, three to six false boundaries per item, confident names attached. The model hears the band change; it does not hear the song. One generational note cuts the other way from Study 4's equality result: the newer Gemini is markedly cleaner on noise, about one invented boundary per item against the production model's three.
Losing a Gate by Half a Boundary
With the chart's rehearsal marks beside the audio, detection is perfect: one hundred percent of boundaries found, every placement within four seconds. The study gate still failed, and the margin is almost comic: we registered a noise bound of half a false boundary per item, and the production model landed at 0.53. Eight false boundaries across fifteen items, where the bound allowed seven and a half.
The source of those eight is the study's sharpest nuance. Even chart-anchored, the uniform loops got subdivided. A chart that marks one [A] section does not stop the model from inventing a chorus. Study 7 established that explicit chart text beats wrong settings; Study 8's corollary is that the chart's silence does not beat the model's form prior. A model that has heard a million songs expects a verse and a chorus, and absence of evidence does not outvote expectation. Names complete the pattern: eight of eight correct from the chart's marks, three or fewer without. Names come from text, as everything always has.
Parse what Text States, Ask What Text Cannot Answer
The recommendation completes a rule this series has been converging on for weeks. The chart's rehearsal marks state the section count and the section names; take both mechanically, no model involved. What the chart cannot state, and what measured perfect, is where each boundary lands in the actual recording, in seconds. Ask the model only that. Applied as a rule in the app's gate layer, it zeroes the loop subdivision and passes the failed gate by construction, the same shape as Study 7's substitute-by-value fix.
One amendment, disclosed in full because the symmetry is too good to hide: our own harness's 2000-token output cap truncated the rambling production model mid-JSON in sixteen cells. That is the exact budget-conflates-with-failure mistake we caught in another model's harness in Study 2, now caught in ours. Cap raised, cells re-run before publication, and two production-model cells that still rambled past the raised cap are counted as misses, not excuses.
Eight studies now, and every shipped AI feature in the app has been measured. One law confirmed four ways: models do what the text affords. And now its first honest exception, with the edges mapped: they also hear the band change. Build on the law; allow the exception.
The registration, amendment, and full report live in the stem-bench repository; the Study 8 page is landing on the research dashboard. The predecessors: The Second Opinion Wasn't Listening and The Fact Line Beats the Instruction. The app is Stem Shovel.
