The Second Opinion Wasn't Listening
Stem Shovel's song-check sends a model the actual audio of a song and asks it to confirm or correct what the app's signal-processing detector found: tempo, key, time signature. A second opinion from a model that can listen. Study 4 of stem-bench asked the question the feature's name assumes: is anyone listening?
The setup
We synthesized fourteen mixes where every fact is true by construction: three meters, tempo shifts and key modulations at known seconds, real drum parts, and two pulseless ambient pads whose only correct tempo answer is "none." Production's listening call was ported verbatim. The ablation is what text rides beside the audio: nothing, the detector's correct numbers, systematically wrong numbers, or the song's chart, generated by the same committed code Study 3 measured.
The Reveal
Blind, with nothing but the audio, the production model got tempo right 2 times out of 12 and key once, at confidence 1.0, while narrating genres our synthesized comp does not contain ("lo-fi hip-hop beat"). Given the detector's correct numbers, it agreed. Given wrong numbers, it echoed them, 10 to 12 out of 12 per field. Given the chart, it was perfect: every field, every mix, and all six tempo and key changes found and placed within four seconds.
Same audio in every arm. Only the text changed, and the answers tracked the text. The second opinion reads; it does not hear.
Putting our Own Fixture on Trial
The registered stop rule said: if the model cannot hear the beat, suspect the fixture. It fired twice, so before claiming anything about models we put the fixture itself on trial. We decoded the finished MP3s and autocorrelated their onset envelopes, a committed mechanical instrument, and the registered pulse was physically present in 12 of 12 metered mixes. Then we let a second, newer model listen to one file as a disclosed diagnostic. It said 180 bpm about a verified 120. Confidence 1.0.
Two generations of listener, one mechanically verified pulse, the same deafness. That is the honest answer to "how do you know it is not your synthetic audio": we measured the audio. The finding is about the models, and we published the instrument alongside the claim.
What They Do Hear, and the Gate's Twist
The models are not deaf to everything. They track harmonic texture loosely, confusing relative keys the way a person squinting at a chord might, and they notice gross structural events in prose: "the tempo doubles here," never with a number. What is absent is exactly what the feature needs: absolute tempo, meter, and key.
The honesty gate failed by letter, in the opposite direction from our prediction. We expected fabrication under pressure: show the model a pulseless pad plus a false "120 bpm" suggestion and watch it defer. It resisted, four times out of four. Then, shown the same pad with no suggestion at all, it invented "60" once, unprompted. It fabricates from priors, not from pressure. One more entry for the confidence field's record: 1.0, on the invention.
The newer Gemini generation pushes back on wrong tempo and meter where the production model swallows everything, a real improvement; both accept a wrong key 10 of 11 times. And the swap space here is narrow: this surface is Gemini-only, since the OpenAI audio model is not on the gateway and Anthropic models take no audio at all, which also means that for once the matrix contains no sibling of the analyst.
The Product Read is Genuinely Good News
The chart arm is the feature. With the lead sheet beside the mix, the song-check nails everything, including the one thing the app actually wants from it: where does the modulation land, in seconds. That is a question neither the chart alone nor the detector alone can answer, and the model with both in hand answers it perfectly across both generations.
The ship guidance went over the same hour. Never let the second opinion override the detector on tempo or meter. Lean the feature on the chart-anchoring path, where it is excellent. And ignore the confidence field, which reads 1.0 on fabrications and 1.0 on perfection alike.
Four studies in, the series has a shape. The chart drafter followed the text hints, not the transcribed notes, until the notes were all it had. The beat generator lived or died by a call-shape field, not by intelligence. The score reader was bottlenecked by rasterization plumbing, not by reading. And the listener turns out to be a text instrument wearing headphones. Models do what the text affords far more than what the modality promises, and benchmarking the modality is how you find out which half of your feature is real.
The registration, both amendments, the pulse-verification instrument, and the full report live in the stem-bench repository; the Study 4 page is landing on the research dashboard. The series so far: The Models That Can Read Music Don't Cheat, The Model That Ignored the Off Switch, and Rasterize Your PDFs. The app is Stem Shovel.
