Seventy-Eight Percent
Thirty-Eight Percent ended with a promissory note: production's chord path reads a real recording at 38.2 percent, the culprit is the note transcription meeting real timbres, and if the chord feature matters, the next dollar goes to a stronger transcription, not to more model. Study 17 spent that dollar. It came to thirty cents, and almost all of it went to proving a negative.
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. The study registered three zero-model successors to the transcription stage, all built from things production already holds: transcribing each stem alone instead of their sum (the app stores stems separately, and the current path feeds the transcriber the polyphonic mixture it fails on), pooling raw chroma into production's own template matcher (nothing says its pitch-class profile must come from transcribed notes), and ChordMini, the published neural chord model that lost our synthetic gate back in the home-field rematch and had never been scored on real audio with bar-aligned chord truth, because that truth did not exist before Study 12. A conditional fourth arm would stack the Study 13 reconciler on whichever successor won.
The Rematch
ChordMini on the mix reads the real recording at 59 of 76 bars. That is 77.6 percent, against the current path's 38.2, and, more consequentially, past 44 of 76, which is what the current path achieves WITH the language-model reconciler under production's shipped configuration. A zero-model front end now beats code plus model, which flips the ship order: the front-end swap outranks the reconciler.
The home-field rule is now three for three. The neural segmenter that invented structure on our synthetic loops found real form; the same-piece constant that synthetic numbers set at 3 needed to be 1.6 on real recordings; and now the neural chord reader that lost the synthetic gate wins the real one by 39 points. Every one of those reversals was invisible until real audio was scored, and each cost nothing but discipline to find.
Roots, Not Dressings
The anatomy matters as much as the number, because the gated chart, from the band Badverbs, is 74 power-chord and 50 sus4 slots across its 76 bars. This band writes roots and colors, not sevenths. ChordMini's entire gated credit arrived through the registered power-chord allowance, with zero strict matches: its vocabulary is root-level triads, and it will never spell F#5 or Esus4. A disclosed root-only read makes the real ranking visible: the current path holds 53 of 76 roots, per-stem transcription 65, chroma 68, ChordMini 70. Every successor out-reads the shipped path at the root level; the registered scores differ in the dressing.
The dressings are where the other two arms died, each instructively. The chroma arm graded 0 of 76 while holding 68 of 76 roots: its pooled mass dresses nearly every right root in a hallucinated seventh, the exact over-read class its own pre-registration self-test surfaced, and a fix would need a tuned threshold the arm was registered not to have. Per-stem transcription split by stem inventory, and the mechanism is legible: the gated song has no vocal stem and improved, the exploratory song has one and got worse, because a transcribed vocal melody pollutes the merged harmony. One song each way is a hypothesis, not a finding, but it ships with a rule attached: if per-stem transcription goes anywhere, vocal stems stay out of the chord evidence.
The Fixer Breaks
Then the conditional arm fired, and the reconciler, which had fixed 27 bars across two studies without breaking one, broke sixteen.
The mechanism is clean. The reconciler reads the transcribed notes beside the detected chords and corrects their disagreements. ChordMini produces no notes, so the registered design paired its chords with the current path's note lines, and those notes are exactly the weak evidence the whole study exists to replace. ChordMini's chords are better than the notes support, so a pass that corrects the chords toward the notes destroys information: 59 of 76 fell to 47, four fixes against sixteen breaks, and the fix-to-break gate we registered against the agreement trap failed for the first time, which is to say it finally earned its keep. The reconciler's value is conditional on its evidence channel being the one that produced the draft. On the old path, it remains measured and real. On the new one, it is a wrecking ball with a confident tone.
The Coda
The scorecard kept us honest: two predictions right, one half, three wrong. Our favorite going in was the chroma arm, which scored zero. Our reconciler prediction said two breaks at most; the measurement said sixteen. Publishing the scorecard beside the result is the point of registering predictions at all.
The ship shape went over with the findings: ChordMini on the mix as the chord front end for real recordings, about 30 MB of pinned weights already vendored since Study 10, four to six seconds a song on CPU, zero API spend; a draft layer that renders root-plus-power spellings, since for this catalog that covers most of the page; the UI hedge from Study 13 can soften as the number warrants; and no reconciler on the new front end, ever, unless a study measures it there first. The ledger: about thirty cents, all of it in the two calls that proved the reconciler harmful.
The registration, the instruments, both results files and the full report live in the stem-bench repository; the Study 17 page is on the research dashboard. The series neighbors: Thirty-Eight Percent and The Home-Field Rematch. The app is Stem Shovel, and the recordings are Badverbs' own, scored against the band's declarations.
