The Detector, Not the Drafter
The last chapter ended with the simplest sentence this pipeline had earned: a stereo file goes straight in as its mono downmix, and Stem Shovel drafts a chord chart from it at the quality the stems themselves would give. Every measurement behind that sentence, though, came from one band's four songs. The app's next question was the obvious and the dangerous one: can the same pipeline face recordings it has never seen, by other artists, from other eras, at other capture quality? The answer decides whether a public signed-out upload tool ships, the kind where anyone drops in a file and gets a chart. So stem-bench, our open, pre-registered benchmark series, assembled the widest corpus it has ever graded: twelve rights-clear recordings spanning a century. Three acoustic-era jazz 78s from 1917 to 1925, now public domain under the 100-year rule, carrying scholarly beat-aligned chord annotations. A gospel quartet. A 1921 waltz. Two songs by Josh Woodward, an artist who publishes his music under Creative Commons and his chord progressions beside it. A ragtime piano showpiece, a live audience tape, a fiddle medley in 6/8. And the band's own songs as controls.
Truth Without Trust
The hard part of grading wild recordings is not running the pipeline. It is building truth that owes nothing to the system under test. For the band's songs we have the band's charts. For a 1923 King Oliver side we have no band to ask. The bench built three instruments to place published chord sequences on a recording's timeline automatically, using the same chroma features everyone uses, then did the thing this series exists to do: validated them against songs where per-bar truth was already known. All three failed, scoring 33 to 74 percent on recordings whose answers we hold, because repetitive vamps let a misplaced sequence score as well as the right one. They were rejected before registration and stay in the repository as evidence. What truth rests on instead is publications: the JAAH annotations fitted to our transfers through a gate that compares each fit against the same fit shifted half a beat, mechanical per-bar chord extraction from sheet-music MIDIs that match the published settings, the artist's own posted progressions, and the band's charts. One recording, the ragtime showpiece, got no accuracy grade at all, because the only path to truth was the bench transcribing the sheet by eye, and a grader cannot audit its own author. It was downgraded to observation rather than graded on trust. No bench-authored harmony exists anywhere in this study.
Both Gates Fail, and the Break Is Upstream
The registered gates asked two things: no drafted chart may be confidently wrong, and at least six of the eight accuracy-graded recordings must be good. Five of eight were confidently wrong. Zero were good. The public tool does not ship. That is the headline, but it is not the finding. The finding is where the failure lives.
A public upload carries no declared tempo, meter or key, so the pipeline leans on its production detector for all three, and that detector was tuned on modern full-band material. On the acoustic transfers and the sparse vocal recordings it missed, and missed structurally. The gospel quartet, four voices and no instruments, came back as 78.6 beats per minute in C minor; the truth is 97.1 in A flat major, and the drafted chart scored zero of sixteen bars. The waltz, a piece in 3/4, came back as 4/4, and scored 8 percent. Dippermouth Blues got a grid at exactly half the true tempo, 57 bars where the annotation holds 114. In every one of these the grid inherited the wrong prior, the draft inherited the wrong grid, and the chart was confidently wrong before the model read a single transcribed note. The model never had a chance to be right, and never had a chance to be the problem.
The proof sits in the same table. On the two modern recordings the detector was right, once to the artist's own declared tempo to the decimal, and the drafts landed at 86 and 89 percent on their chord sequences, just under the registered bar for good. The control reproduced its known ceiling at 67 of 68 roots. And one capability sailed through the wreckage untouched: section boundaries. On Livery Stable Blues the draft placed eleven of eleven section marks while its harmony timeline was wrong, which says structure sensing and harmony ride on different signals, and only one of them broke.
Just Tell It the Tempo
The app read that result the same day and asked the follow-up any product person would ask: the planned tool was never going to detect unattended anyway. A visitor would tap the tempo and pick the meter from a dropdown, the way musicians do. So what happens to the five confidently wrong recordings when a person supplies exactly what the detector got wrong? Study 34b ran precisely that experiment, and the answer reframed the whole program: all five stayed confidently wrong.
First the bench had to know what a truthful visitor would tap, which is harder than it sounds for a hymn with no instruments and a waltz nobody alive has danced to. The instrument it built uses the publication against the recording: a published repeat structure fixes how many beats a section lasts, chroma self similarity finds how many seconds that repeat actually takes on the record, and the tempo falls out of the division, with no production code anywhere in the loop. On the hymn the verse lag agrees with itself at one, two and three multiples within 0.2 percent, landing at 97.1 beats per minute. Our own first report had written roughly 105 for that recording, an unlabeled estimate that the committed instrument now supersedes, and the correction is published in place: the number above is the corrected one, and no class or gate anywhere moves with it.
With true tempo and meter declared, the five failures split into two clean families, and neither is the detector. The two 1917 jazz 78s drift off any fixed grid: the sides were cut unevenly enough that no single bar length exists, and the placed grids settle 7 and 9 percent away from the declared truth they were handed. The other three recordings do the opposite: their grids land to the bar, Dippermouth at 114 of 114, the hymn and the waltz exactly on their published forms, and the drafts are still wrong, at 66 percent, 17 to 31 percent, 6 to 8 percent. The reason is starvation. The note transcriber, tuned like everything else on modern material, pulls 1.3 to 2.2 notes per bar out of acoustic-era audio where a modern song gives it ten or more. The model drafted confidently on grids that were right and evidence that was nearly empty, and nothing in the drafts looks unsure: unknown chord shares near zero, bar counts preserved, every surface signal of a healthy chart.
Two Walls, Both Visible Before the Model
Declared priors were the necessary fix, and they are not a sufficient one. That sentence could have ended the study as pure bad news, except for where the two walls stand: both are measurable before the model runs, from numbers the pipeline already has. A grid that settles more than a few percent away from the tempo the visitor declared has diagnosed its own drift. A transcription feeding a chart fewer than three notes a bar has diagnosed its own starvation. Across all fifteen recordings those two checks flag every confidently wrong chart and not one good one. The detector even carries a usable confession in its own output: its tempo confidence separates its wrong detections from its right ones on this corpus, while its key confidence confesses nothing, happily reporting a wrong key at the maximum confidence it can express.
The app shipped both checks as hold-for-review rules the same day, and the bench's registered verdict on them is the one that matters: necessary, not sufficient. They were read off these fifteen recordings, which makes them hypotheses, not findings, until they survive material they were not tuned on. That out-of-sample test is the next chapter, and it did not go the way anyone predicted.
The Ledger
This series publishes what it cannot know alongside what it can. The study's only refusal came, of all places, on the cleanest input: a direct DAW export with page declarations, where the model returned no parseable answer. Why, we cannot say, because this run stored parsed results rather than raw model text; the runner now keeps the raw draft on every future run, and the question waits there. The app had also just shipped a hold-for-review rule on generated charts, which holds a draft whose bar count changed, and asked the bench to read that rule against these results. The reading is uncomfortable: all five confidently wrong charts would have sailed past it, because the draft faithfully preserved the grid's bar count and the error lived upstream of both numbers the rule compares. The rule is not wrong, it is blind to the failure mode that actually dominated.
Study 34b carried its own disclosures. One registration clause was reconciled before any data existed: both sheet-music recordings demonstrably perform more of the piece than the publication prints, the hymn singing its sixteen-bar form nearly five times through, the waltz adding one repeated strain, so the bar-count grading was re-registered against committed form evidence rather than against a publication the performers did not follow. The tap-tempo cross-check failed outright on the hymn, which is genuinely hard to tap, and disagreed on the waltz, both failures disclosed beside the instrument that replaced them. Study 34 cost 71 cents against a five dollar fence; 34b cost 41 cents against two dollars.
The production answer is a direction, not a patch: fix the front of the pipeline, not the back, and then keep going, because the front has two walls, not one. The drafter is nearly good wherever the grid and the transcription are healthy, and both health checks exist before a single token is spent. The registrations, the rejected instruments, the truth validation and the full reports live in the stem-bench repository; the entries are Study 34 and Study 34b on the research dashboard. The controls are Badverbs' own recordings, the modern wild material is Josh Woodward's, shared under Creative Commons, and the 78s belong to everyone now, which is how a 1917 jazz band ended up stress-testing a 2026 pipeline twice, and winning twice.
