Study 4 · Listening
The listening layer: the second opinion reads, it does not hear
Listening
Study Overview
The Question
Stem Shovel's song-check (askAiAboutMix) has a model listen to a song's mix and confirm or correct the DSP detector's tempo, key and meter, the second opinion for the cases the detector is unsure about. It had never been measured against known audio. Study 4 synthesized 14 MP3 mixes whose facts are true by construction: 12 metered items across 4/4, 3/4 and 6/8 at 60 to 140 bpm, including tempo shifts and modulations at known seconds, plus two pulseless pads whose registered correct tempo is null. Production's call was ported verbatim and pinned, and what rides beside the audio was ablated the way Study 1 ablated the matcher's hints: no candidates, the truth as candidates, registered wrong candidates (the octave trap, a rotated meter, the key up a fourth, and a false 120 bpm asserted over the pulseless pads), and the chart arm, where the matching lead-sheet summary is generated by the same committed code as Study 3's ground truth.
The Stop Rule on Trial
The registered fixture-health stop rule fired on the first pilot (blind tempo 1/12), so the synthesis gained an unmistakable drum layer and the run was archived and discarded. It fired again on the drummed fixture (2/12), and this time the rule's own premise went on trial: a committed onset-autocorrelation instrument decoded every metered MP3 and found the beat, or a small integer multiple of it, within 4 percent, 12 of 12, and a disclosed one-cell second-listener diagnostic reproduced the failure on a verified pulse (180 bpm, wrong key, confidence 1.0, on a drummed 120 bpm item). The premise, that the synthesis does not carry tempo, is therefore false, and the rule was retired as discharged rather than re-thresholded: its action was taken twice, its premise was then tested directly and refuted, and the firing is reclassified as the finding it actually detected. Amendment discipline turned a broken-looking fixture into the study's sharpest instrument.
The Second Opinion Reads; It Does Not Hear
Blind, both gemini generations pin tempo to a 120 to 180 genre prior at confidence 1.0, name wrong keys (1 to 2 of 12), and find zero of the six changes, on mixes with explicit kick, snare and hats whose pulse is mechanically verified. Given text, they transcribe it: correct candidates in, correct answers out (production 11/12 tempo, 12/12 key); wrong candidates in, wrong answers out (production echoes the wrong value on 10 to 12 of 12 per field); the chart in, perfection out. The second opinion is a text channel wearing headphones. What the models genuinely hear is harmonic texture, loosely (relative-key confusions, the occasional real chord motion named), and gross structural events (a tempo double noticed in prose but never quantified); absolute tempo, meter and key extraction are not in the current audio stack, consistently across two model generations.
The Chart Arm Is the Product
With the lead sheet beside the mix, the song-check nails everything, both models: 12 of 12 on tempo, meter and key, and all six tempo and key changes found AND placed within 4 seconds of their true position. That placement is the thing the product actually wants from this feature (where, in seconds, does the modulation land), and it is exactly what the blind listener cannot do at all. The feature's measured value is chart anchoring: given Study 3's reliable score reading and this study's placement result, the two features compose into a pipeline that locates chart events in the recording.
The Gate, and the Swap Space
S4-H1 applied Study 3's every-model lesson and scoped the honesty gate to the production model alone, and it still failed by letter, in the unprompted direction: shown a pulseless pad blind, production invented a tempo of 60 at full confidence, while resisting the false 120 bpm authority on all four pressure cells. It invents on its own, not under social pressure, which is the exact inverse of the registered prediction. The successor tier ran the probe clean and pushes back on wrong tempo and meter (deference 4/11 and 3/11), though it still swallows the wrong key 10 of 11. The registered availability probe answered the swap question structurally: gpt-4o-audio-preview is not on the gateway and no Anthropic model takes audio, so the swap space for this surface is gemini-only.
Honest Accounting
The registered scorecard ran mostly against the predictions: 1 wrong in the instructive direction (the invented tempo came unprompted, not under false authority); 2 half-right (meter weak blind, candidates only partially fix it); 3 wrong (deference was highest for key, not meter, and production deferred on everything); 4 right (the chart lifts change detection to 6 of 6, placed); 5 wrong (one octave error, not two, because the blind misses were prior-pinning, not octave confusion). Total spend about $1.20 of the $13 in fences, with both discarded pilots archived with their anatomy.
What This Licenses
For Stem Shovel, briefed back to the app the same day: never let the second opinion override the DSP detector on tempo or meter; gate or shrink the no-chart path, since a blind listen adds fabrication risk for no measured value; ignore the confidence field, which reads 1.0 on fabricated answers; and treat the song-check as a chart-anchoring tool, where it is measured perfect. For everyone else: an audio-capable model that answers confidently about a mix is most likely reading the text you sent with it, and the cheapest way to know is a fixture whose pulse you can prove.