The Fact Line Beats the Instruction
Four days ago we published The Second Opinion Wasn't Listening: Stem Shovel's song-check, asked to listen to a mix and verify the app's tempo, key and meter detection, was not listening. It read whatever text rode beside the audio, echoing wrong candidates ten to twelve times out of twelve. The app rebuilt the feature in response. The model is now framed as a chart reader with the recording playing. The settings arrive as bar-following facts rather than candidates to confirm. The prompt says, in so many words, do not estimate a tempo by ear. An application gate keeps tempo and meter only where the chart actually states them. And the confidence field, which read 1.0 on fabrications and perfection alike, was deleted outright.
Study 7 of stem-bench ported the rebuilt call verbatim and re-ran the ablations. This is the closing bracket: a production fix provoked by a measurement, verified by the next measurement, which is the loop this whole program exists to run.
The Redesign Works
The headline gate passed decisively. Zero wrong settings survived to application on the production model, against the old design's ten to twelve out of twelve. And the improvement is not just the gate catching things: handed "the song's settings say tempo 240" while the chart's own mark reads 120, the model follows the chart, twelve out of twelve, in the raw answer, before any gate touches it. Where there is a chart statement to transcribe, the chart-transcription framing transcribes it, and the deference failure that named the last article is closed.
About the Gate That Failed
The second gate asked about charts with no tempo mark at all: bare chord charts, where the registered correct answer is null and the instruction says so explicitly. The model never answered null. It also never estimated by ear.
All forty-seven invented tempos, across both models and every cell, are the settings value verbatim. Feed the settings 120, it answers 120. Feed them a lie of 60, it answers 60. The instruction "use them to follow the bars" sits directly beside the fact line "tempo: 120", the two contradict, and the fact line wins every time. Meter behaves identically.
That is the fourth time this program has measured the same law in a new costume. In Study 1 the hint beat the transcribed notes. In Study 2 the off switch lost to the provider's own behavior. In Study 4 the candidates beat the audio. Now a fact line beats the instruction standing right next to it. Models do what the text affords, and an instruction is just more text, outvoted by a fact.
Why Nobody Gets Hurt
The reason this gate failure costs the product nothing is the application gate: wherever the chart states neither tempo nor meter, the code nulls both fields, twenty-four out of twenty-four correct refusals. The study gate fails by letter while the shipped feature contains every consequence. That is defense in depth doing exactly what prompts cannot, and it is worth saying plainly: the rebuilt prompt did not stop the model from inventing answers; the code did.
One leak marks the guard's design limit. A sibling model echoed a wrong setting once under a chart that did state a tempo, and the gate, which filters by presence (the chart says a tempo, so keep the model's), passed the wrong value through. The production model never did this, so the production gate passes. But the leak and the fix point the same direction.
Stop Asking the Model
Here is the recommendation that falls out of the whole study. The chart summary's tempo marks and time signatures are machine-readable text that our own code produced. Nothing about them requires a model. Parse them. Substitute by value instead of filtering by presence, and the one observed leak closes by construction, along with every unobserved one.
Reserve the model for the two things this study measured as actually working: reading keys from chord progressions, which was perfect on bare chord charts, twelve out of twelve on both models, and placing changes in seconds against the audio, the thing neither the chart nor the detector can do alone. Ask models only the questions text cannot answer. Four studies in, that is less a recommendation than the series' refrain.
The registration, corpus, and full report live in the stem-bench repository; the Study 7 page is landing on the research dashboard. The predecessor is The Second Opinion Wasn't Listening. The app is Stem Shovel.
