The Band Was Right
On Friday our benchmark told us the band cannot keep time. Two pre-registered studies agreed: the recordings wander off their declared tempos, a simulated perfect tap cannot rescue a chartless chart draft, and the missing piece is true-tempo estimation. The verdict had registered gates, instruments with unit tests, and a paper trail. It also had the band's songwriter in the loop, and he replied with one line: the tempo is precisely metronomically constant. He was right, the benchmark was wrong, and the day that followed is the strongest argument for pre-registration this program has produced.
stem-bench is our open, pre-registered benchmark series measuring the AI features in Stem Shovel, our music collaboration app. The recordings under test are Badverbs' own, scored against the band's declared charts. That matters here, because the correction did not come from a reviewer or a failed replication. It came from the musician who played to the click.
One Hour to Verify
The check cost nothing and involved none of the machinery under suspicion. A comb sweep slides a grid of candidate tempos across the song's raw onset envelope, the note attacks themselves, and scores how much energy lands on the teeth. The sweep returned razor-sharp peaks at 131.97 to 131.99, 120.00, 144.96 to 144.98 and 223.82 to 223.97 bpm, against declared tempos of 132, 120, 145 and 224. All four recordings are click-exact.
It got worse in an instructive way. The second study's own tempo instrument had already found those exact values on every song, to 0.02 percent or better. The bench had graded that perfection as 24 percent off, because it scored the instrument against its own broken ruler.
The Broken Ruler
The ruler was our chart-to-audio alignment, a chroma comparison that had served as bar-level truth since Study 12. The correction forced us to separate two things it does. Its per-bar chord bins stay musically aligned with the chart, which is why eleven studies of chord grades survive untouched. Its bar geometry is unusable: on one song its bar durations span 0.91 to 2.43 seconds against a true 1.818, its errors reach 22 seconds, and its "first bar" sits four bars late on two songs, which had silently turned the simulated perfect tap into a four-bars-late anchor. The whole collapse the studies reported was whole-bar index shifts manufactured by the truth file. The lesson entered the program's ledger in six words: take the alignment's bins, never its geometry.
By the end of the day the tempo verdicts were withdrawn, with dated corrections on the report, on both dashboard entries and in the app's findings channel, and the same data had been re-scored into the opposite story. The corrected guidance to the app inverted the registration's premise: use the declared bpm, it is exact, and anchor bar one near zero.
The repaired ruler paid immediately. Study 25 re-ran the chartless draft on the grid the recordings actually have, and the song that had collapsed to 36 of 76 bars drafted 72 of 76, exact parity with a draft that gets the chart handed to it. A second song came in one bar above its baseline at 152 of 153. Study 26 used the same truth to repair the alignment's broken ends, taking Letter Lands' first and last bars from 7.3 and 15.3 seconds of error to 0.05 and 0.02.
The Second Catch
One claim survived the first correction. Eat All the Clocks' chart seemed to need 172 seconds of audio against a 147.5 second recording, so we published an arrangement mismatch: about 13 chart bars the recording never plays. The songwriter pushed back again, with arithmetic. The chart parser counts 71 bars in that chart, and they match the song precisely.
He was right again. The live chart has 71 bars. Our fixture, copied from the song page three weeks earlier, carried 86: each chorus written as 12 bars where the chart's repeat line expands to 8, and a four-bar outro where the chart holds one bar and a fade. Fifteen phantom bars, and they were the entire mismatch. On the corrected chart the reader's hints match 65 of 71 bars, uniformly from the first bar to the last, the alignment's interior error drops from 7.8 seconds to under half a second, and the end-repair recipe fixes this song like the others. There was never anything wrong with the arrangement. There was something wrong with our copy of it.
The second catch has a twist worth savoring. Study 26 had just shipped a refusal check: when the chart in hand disagrees with the recording too much to trust, the recipe declines to produce a bar grid rather than showing a wrong one. The first real disagreement that detector ever caught was our own mis-copied chart. We built a wrong-chart detector, and it opened its account on its maker.
The Coda
Three things we are keeping from this day.
The musician was the sharpest instrument in the loop. Both corrections began as one-line replies from the person who knows what the click was doing, and both were verified mechanically within the hour. A domain expert reading your results is not a courtesy reviewer; on the right question, they are a better anomaly detector than anything in the pipeline.
Mechanical graders do not protect you from a broken ruler. They make the wrongness precise, reproducible and well-documented. Every number in the withdrawn verdicts was computed correctly by committed, tested code. The code measured against the wrong truth, flawlessly.
Pre-registration did not keep us from being wrong. It made being wrong cheap to repair. Every withdrawn claim had a committed brief, a committed instrument and a dated results file, so the corrections were surgical rather than archaeological, and they are published as corrections on the pages that carried the errors, with the void runs preserved. The gates stand as registered measurements of defective constructions, and the defects are named.
The registrations, the instruments, the void runs and the full report live in the stem-bench repository; the corrected entries are Studies 23b through 26 on the research dashboard. The app is Stem Shovel, and the recordings are Badverbs' own, scored against the band's declarations, which were, both times it mattered, exactly right.
