Stem Bench
Stem Bench is our open research project measuring the AI features of Stem Shovel, Lightning Jar's music collaboration app. Every study is pre-registered: the hypotheses, fixtures, prompts, and graders are committed to a public repository before a single scored run, and the results are published as found, corrections included.
The Findings in One Sentence
Language models are excellent readers and unreliable listeners: given the notes, a chart, or a page they are near-perfect, given audio alone they repeat whatever text rides beside it, so every hearing job went to code or data and the models kept the reading jobs. Every study and article in the series is linked below.
Stem Bench · pre-registered · corrected 2026-11-13 · studies 1 to 52
The models read; they do not hear. Every hearing job Stem Shovel gave a model went to code or data, and the one place a model now improves the audio pipeline is a reading job.
Reading Works
Shown the notes, the chart or the page, the frontier tier is close to perfect. The shipped chart-draft configuration scores 99.8 percent per bar, and the production model reads engraved notation at 98.8 percent from a clean image and 99.3 percent from a clean PDF. It also ignores wrong hints: zero of 438 copied. Cheaper models do not hold that line, and Study 2 found that a call shape that works for one model can silently break its sibling.
Listening Does Not
Given audio alone, the song check found the right tempo 2 times in 12, at full confidence, and echoed whatever numbers rode beside the audio. With the chart beside the mix it was perfect, because the value was the chart, not the hearing. Study 8 recorded the program's first genuine listening result, and it was narrow: models hear texture change, not form.
Code and Data Took the Hearing Jobs
The summed tempo detector survived stems engineered to break it, the classical stack held its ground against published neural models, and a tempo ladder retired the octave error class by representation. Study 12 took the last hearing job with no model at all: aligning a rendered chart against the recording placed mid-song section boundaries within about a tenth of a second on a real song.
Real Audio Resets the Ceiling
Study 13 measured the chord path on a real recording: 38.2 percent, against 95.8 on the synthetic bench, so every synthetic number in this series is a ceiling, not an estimate. A text-only reconciler pass then fixed twelve bars and broke none, the first time in the series a model improved a hearing result, and it did so by reading the pipeline's outputs rather than listening. Study 17 then ran the rematch: the neural chord reader that lost the synthetic gate reads the same recording at 77.6 percent, past the reconciled ceiling, and the reconciler broke sixteen bars the moment the draft no longer came from the notes it reads. Study 18 validated two matcher fixes the app found in the bench's own fixture: the shipped floor moved from 38.2 to 53.9 percent, worth as much as the reconciler pass, with the ceiling unchanged. Study 19 then gated the app's TypeScript port of the winning chord reader at perfect parity, zero disagreeing frames in 7,102, so the real-audio numbers now describe the shipped feature itself. Studies 20 and 20b are the program's first full refutations, published as found: the note channel cannot dress the model's chords in the chart's colors, and the one rule that survived split 81 percent against zero on held-out songs, its precision a function of arrangement density rather than a number a card could print. Study 21 then graded the shipped reader on those same held-out songs: its roots generalize at 90 percent across the whole charted catalog, and the one gate it fails is the grade's own vocabulary. Studies 22 and 23 then ran the recording-to-chart question end to end: the bar grid cannot yet be derived (or even certified, the program's truth being coarser than the question), and the production draft composition fails every gate on the chart the app would parse while its JSON channel quietly holds the best chord numbers the program has measured, so the named fix is deterministic rendering, not better ears. Study 23b then removed the training wheels and collapsed the draft on every song, and its same-day correction is the arc's real lesson: the recordings are metronome-exact at their declared tempos, the collapse came from the bench's own alignment ruler (whose 'first bar' sat four bars late on two songs, so the simulated tap was never a tap), one chart seemed 13 bars longer than its recording plays (a second correction traced that to the bench's own chart fixture), and with the true grid the shipped reader's inputs are already near chart-grade on the arrangement-matched songs, and Study 25 then measured the draft end to end on that grid: exact parity on letter-lands, above baseline on peaceful-dreams, with sub-bar anchor phase left as the named open problem once the second correction resolved the apparent mismatch into a bench fixture error.
Methods & Provenance
Sixty-seven pre-registered studies, October and November 2026: 34,863 scored cells across seven gateway models at temperature 0. Each study is registered by commit before its first scored run and published as found; the full catalog lives in the study index below, one page per study. Most ground truth is built by construction (hand-authored progressions, MusicXML scores, synthesized mixes), and the later studies add real recordings scored against the band's own declarations. Production prompts and call shapes are ported verbatim, and grading is mechanical. Three neural MIR baselines and the zero-spend DSP instruments are not counted in the model census. Every amendment and correction, including the dated correction to Study 3, is disclosed and committed.
The study series
Every Study, One Page Each
Every study is pre-registered before its first scored run and gets its own page with charts, tables, and a verdict. Grouped by theme:
The chart-draft pipeline
- Study 1 · The hint-following study: models that can read the notes don't copy the hint Stem Shovel's chart draft shows a model each bar's transcribed notes plus a free template matcher's chord guess. Across 320 cells on 440 seeded bars, the frontier models copied a systematically wrong hint zero times between them, while kimi-k2.6 surrendered 69% of the bars it could read correctly on its own; the shipped configuration (fable-5-1 plus the hint) scored 99.8%, 9.3 points over the free matcher alone, at about $0.10 a draft.
- Study 23 · Recording to chart: every gate fails on the chart the app would parse, while the same answers' JSON carries the best chord numbers the program has measured Kevin's research question, run as a composition: the transcription, the shipped reader's per-bar hints, and the production draft prompt, grid-given per Study 22, one model call per song on all four charted songs. Every registered gate sat on the chart channel, the draft's markdown read through the app's own barmark adapter, and every gate failed: one chart unparseable, two missing 18 and 56 bars. The anatomy inverts the texture of that verdict. The same answers' bar-exact JSON channel grades as the program's best chord numbers ever: peaceful-dreams 151 of 153 roots, above the reader's own 149, with eleven of its 62 dialect bars exactly right, D7sus4 written strictly eleven times where the reader scores zero by vocabulary, and G7 on the G5(7) bars, the seventh heard and the spelling absent from every example. The model did the musical job and lost it in serialization: cell-level repeats the notation does not define, invited by the production prompt's own compact example, and under-written section repeats. The verdict names the fix: build the chart deterministically from the JSON the model already gets right, and stop trusting its markdown as the document of record.
- Study 23b · The chartless-song grid collapsed the draft, and the same-day correction relocates the cause: the bench's ruler, not the band's tempo Study 23b swapped the end-to-end pilot's chart-anchored grid for the grids a chartless song would have, and both arms collapsed the draft, as far as 151 of 153 roots down to 64. The entry's first publication read that as a tempo problem: the declared bpm is not the recording's tempo, a perfect tap cannot save it, and the missing leg is true-tempo estimation. Kevin corrected the premise the same day and a DTW-free comb sweep verified him within the hour: every recording is metronome-exact at its declared bpm (131.97, 120.00, 144.97 and 223.9 against declared 132, 120, 145, 224). What actually broke was the bench's own alignment ruler, whose bar geometry drifts up to 22 seconds and whose 'first bar' sits four bars late on two songs, so the simulated tap was never a tap: it anchored the grid at the ruler's error, and index-paired grading then punished every uniform arm by whole-bar shifts while the baseline stayed shift-aligned by construction. One apparent mismatch also surfaced, eat-all-the-clocks' chart seeming about 13 bars longer than its recording plays, and a second same-day correction relocated that too: the surplus was the bench's own chart fixture, not the band's chart, which matches the recording precisely. The gates stand as registered measurements of defective constructions; the tempo verdict is withdrawn, and the corrected grid guidance is the opposite of the original: use the declared bpm, anchor bar one near t equals zero, and when a bar map and a recording disagree, audit the chart fixture before blaming the arrangement.
- Study 24 · True-tempo estimation found the tempo was never lost: the combs recover the exact declared bpm, and the same-day correction rebuilt the grid story from the audio up Registered as the fix Study 23b named, the joint phase-and-period comb search ran over the shipped reader's grid-free segment boundaries, and its first reading split the catalog: steady songs reached baseline parity chartless, with peaceful-dreams writing 18 of its 62 dialect bars exactly, above grid-given's 11 and the program's best color number, while letter-lands and eat-all-the-clocks stayed collapsed and the in-run least-squares fit to the aligned truth itself hit almost nothing, which the bench read as tempo wander. Kevin corrected the premise the same day, a DTW-free comb sweep verified him within the hour, and the study's own data re-scored into the opposite story: the recordings are metronome-exact at their declared tempos, the onset joint comb had found that exact tempo on all four songs to 0.02 percent or better, and the bench had graded that perfection as 24 percent off against its own broken ruler. What actually failed was the alignment truth's bar geometry, including two four-bar start errors that had turned 23b's simulated tap into a four-bars-late anchor, and one apparent arrangement mismatch on eat-all-the-clocks that a second same-day correction traced to the bench's own chart fixture, 15 phantom bars; the band's live 71-bar chart matches the recording precisely. The corrected guidance inverts the registration's premise: use the declared bpm, anchor bar one near t equals zero, and audit the chart fixture before blaming the arrangement.
- Study 25 · On the true grid the chartless draft reaches the chart: parity on letter-lands, above baseline on peaceful-dreams, and the two gate failures name the real remaining problems The correction's follow-up put the end-to-end draft on the grid the recordings actually have: the declared period, now known exact, anchored at chart bar one by a committed best-shift procedure disclosed as an oracle construction. The result inverts two studies of collapse: letter-lands drafts 72 of 76 roots, exact parity with the chart-given baseline, and peaceful-dreams drafts 152 of 153, one bar above it. Both registered gates still failed, and both failures are the useful kind. mop-heart missed on sub-bar anchor phase alone: the oracle anchor carried a 115 millisecond residue from the alignment start, about 43 percent of a beat at 224 bpm, while Study 24's onset-comb phase on the same song had drafted 136 of 140, above baseline, so the comb is the better anchor and fast tempos need beat-level precision. eat-all-the-clocks failed its playable-prefix gate, 49 of 71 against 55, first read as the arrangement mismatch extending into the played span; a second same-day correction (Study 25b) relocated the mismatch to the bench's own 86-bar chart fixture, since the band's live chart has 71 bars and matches the recording bar-for-bar, and on the corrected chart the draft reaches 56 of 69 roots, the same 81 percent as its old baseline, so eat's remaining gap is song difficulty, not grid. A same-day amendment excluded a non-physical anchor candidate after the shift search chose bar one 7.6 seconds before the audio on a one-root margin; the void run is preserved and disclosed. The synthesis across Studies 24 and 25 is the measured recipe for the feature: declared bpm as the period, onset-comb phase with bar one near t equals zero, with the better arm per song reaching 72 of 76, 152 of 153 and 136 of 140 against baselines of 72, 151 and 131. The one problem no grid was thought able to close dissolved under the second correction: the mismatch was the fixture's, and auditing fixtures against live charts joins the recipe.
- Study 28 · The capstone chartless draft fails its gate and names the missing instrument: the comb's phase is a beat detector, not a downbeat detector Every chartless number so far was a synthesis of best arm per song across Studies 24 and 25, so this study measured the ONE grid recipe the auto-chart feature would actually ship: period from the onset comb at the declared tempo, anchor from the comb's own lattice registered near the first audible sound, length from the audible span, then one draft call per song graded against the corrected charts. The gate failed honestly on two songs and the failure is the finding. The raw comb phase picks the strongest beat of the bar, not beat one: letter-lands and peaceful-dreams anchored about 2.1 beats late, on the backbeat lattice, drafting 81.3 and 89.5 percent against references of 94.7 and 98.7. mop-heart anchored one beat late and still drafted 135 of 140, above its reference, because long same-chord spans tolerate a beat; eat-all-the-clocks anchored on the downbeat and drafted at its corrected-chart baseline [Correction 4, 2026-11-08: the downbeat claim was the ruler's own; eat's registered anchor was one beat late, so the comb phase missed all four downbeats; see the correction below]. The other gate passed cleanly: the derived bar counts came out 75, 153, 140 and 70 against charts of 76, 153, 140 and 71, every song within one bar from the audible span and the exact period alone, so the length leg of the feature is solid. The comb period stays exact everywhere. What the auto-chart grid lacks is one instrument, downbeat disambiguation, and the app's own backbeat brief (answered the same morning as Study 29) had already proposed its shape: score the grid against itself shifted by whole beats. Named as the next remedy, not yet registered.
- Study 30 · Downbeat disambiguation fails both gates: chroma novelty is not a downbeat detector, and bar one turns out to be a bit the user should supply Study 28 named the chartless grid's one missing instrument, downbeat disambiguation, and the obvious candidates were cheap, so this study registered two and measured them: every whole-beat shift of the Study 28 anchor scored by mean detrended energy at all bar lines, with chroma novelty as the registered primary (harmony moves on downbeats, a signal the backbeat does not carry) and kick-band onsets as the secondary (the mix low-passed at 150 Hz, the exact complement of Study 29's construction). Both gates failed, published as found. Chroma novelty went zero for four, and the committed scores show why: on letter-lands the BACKBEAT outscores the true downbeat 0.0585 to 0.0468, on mop-heart the two leading candidates tie to four decimals, on peaceful-dreams the winning margin is four percent, and on eat-all-the-clocks, the one song whose anchor was already right, novelty moved it off [Correction 4, 2026-11-08: that anchor was one beat late, and novelty's choice was in fact the correct class; see the correction below]. The same chroma smear that broke the DTW's bar geometry makes frame-to-frame novelty mush at beat granularity. The kick band went two of four and also derailed eat. The re-drafts failed by consequence: letter repeated Study 28's 81.3 percent and peaceful got worse, 61.4, because novelty had moved it to a different wrong beat. The cross-study tally is the real product: three cheap chartless scorers are now measured, raw comb phase, chroma novelty and kick band, and none earns the right to overrule a correct base. Everything else in the grid is machine-exact, period to 0.02 percent, length within one bar, lattice right, so the remaining uncertainty is one bit per song, which beat is bar one, and the named remedy is a product gesture: a single bar-one tap from the user, zero DSP risk. The alignment feature's v5 is unaffected; its DTW tether already supplies that bit.
- Study 31 · The witness vote finds every downbeat, exposes the bench's fourth ruler error on the way, and carries the chartless draft to chart parity The app answered Study 30 with a brief of its own: the three dead scorers all read energy at beat granularity and were blind to where a song begins, ends and moves its chords, so measure the witnesses a musician uses, vote among them with pre-registered weights, and keep the humility rule. The bench registered seven witnesses voting by beat class on the Study 28 lattice. The first run failed every gate, and the failure pattern was the finding: time zero, harmonic rhythm (margin 35) and all twenty section boundaries agreed with the truth exactly on letter, peaceful and mop, and agreed WITH EACH OTHER on eat, one beat before the registered truth. They were right. Correction 4: eat's anchor is minus 0.085 seconds, not plus 0.415; the old value was Study 25's hint-fit made against the phantom-bars chart before Correction 2, with an instrument measured flat across a two-beat range, and the chart-informed DTW's dissent had been sitting in Studies 27 and 29's committed numbers all along as the constant 0.474-second offset politely misread as eat's quarter-bar edge. The non-drum stems sound from the file's first sample, so the bar line is at time zero, exactly the app's DAW-export argument, which holds on the whole corpus. On the corrected ruler, Study 31b passed the hard gate: the vote finds the true class on all four songs, five to one, six to one, three to one and five to one, none undecided, and HARMONIC RHYTHM ALONE IS FOUR OF FOUR, the single witness the app ships first. One more defect surfaced and was fixed in a registered amendment: a beat class is modulo one bar, and placing the voted anchor a whole bar wrong had collapsed the re-drafts (it also retro-explains Study 30's peaceful number). With Study 31c's placement rule, bar one contains the first sound, the chartless drafts reached letter 71 of 75, which is 94.7 percent, EXACTLY the chart-given reference, and peaceful 152 of 153, which is 99.3 percent, ABOVE its 98.7 reference: the first chartless drafts at reference parity. The bar-one tap retreats to a fallback behind an honest undecided. The humility screen went vacuous once the ruler was fixed, so the app's fifth and sixth songs are the registered condition for shipping the vote. A coda, same day: the app's agent reviewed the committed results and caught an instrument defect the bench had missed, a 90-second cap on the onset envelope that had left the ends witnesses unmeasured rather than measured, and read time zero's four distances as the envelope's 0.07-second lead. Study 31d lifted the cap and re-scored: the song's end is honestly dead as a witness (ritardando abstentions on all four full songs), the lead-corrected time zero is FOUR OF FOUR with distances of 0.02, 0.03, 0.008 and 0.000 beat, and both vote variants still find every downbeat.
- Study 32 · The adversarial corpus gives the humility gate teeth, the ship vote survives every honest fooler, and the one wrong answer names the rule the app needed Study 31 ended on a condition: the humility screen was vacuous once the ruler was corrected, so the vote ships behind two more songs. The band had no more songs ready, and the app's agent wrote the substitute brief itself: build songs designed to fool the witnesses, and make the gates bite. The bench adopted it whole with six registered reconciliations and ran 37 items, synthetic correct bases at three tempos and three meters, five hostile starts, four harmonic foolers, a backbeat mix, and the four real recordings re-cut by pads and trims against the Correction 4 truths shifted. Run one failed its humility gate by a fixture defect worth publishing: the bench's synthetic kick was a pure 65 hertz sine, nearly invisible to the spectral flux onset envelope, so every CORRECT base sat on beat 2, the backbeat all over again, this time inside the bench's own synthesizer. The real songs supplied correct bases anyway, and the gate's teeth bit something real there: the crash witness voted to move a correct base three times out of three and is now formally dead. Study 32b gave the kick a six millisecond beater click and re-ran everything: every base correct as constructed, and the ship vote, lead-corrected time zero at weight two, ratio-ruled chord changes at weight two, sections at one, never decided a wrong class on any synthetic adversary. The pickup, the half-bar silence, the two-second silence and the four-click count-in all resolved correctly, the count-in self-defeating in 4/4 exactly as predicted. The backbeat case got tested in both directions across the two runs: a wrong base moved to the truth five to nothing, a correct base kept five to nothing. The two-chord-bar fooler produced run one's confident wrong vote and resolved to an honest undecided with the fixture fixed, held not by the registered ratio rule but by the humility margin itself. On the real re-cuts: fifteen right, four undecided, one wrong, and the one wrong names the do-not-place rule, a start that is not beat-aligned silences time zero and the chord witness at once and leaves a one-point section vote holding a wrong base; when both fire, keep the declared grid or ask. The app's eighth-note fix for the end witness was implemented and measured: still abstains on all twelve real-song reads, now as a measurement rather than an artifact, and on synthetics it faithfully votes the last onset's class, which is a beat-four snare, so the end witness's seat is declined on the evidence. Re-drafts on the one-beat pads followed the voted bar one to reference parity, 72 of 76 and 152 of 153. The registered stand-in for songs five and six resolves: the humility gate passes for every shipping witness, no false moves on synthetics, one named real case.
- Study 33 · The stereo file needs no stems: the mono downmix drafts above the stems sum, and every clever split loses Study 32 shipped the chartless grid, so a fresh stereo upload already gets its bars for free; what nobody had measured is the draft, whose reference numbers came from transcribing the non-drum stem sum, something a stereo file does not have. Kevin asked the question directly: how well does the auto-draft work on a stereo file, and is it worth treating the file as two stems and running the same process? A zero-dollar scoping measurement framed it: the four corpus mixes correlate left to right at 0.83 to 0.90, the side channel sits 10 to 13 dB under the mid and weakest below 200 Hz, kick and bass dead center, so each channel is a near copy of the whole band. The registered design put five transcription inputs through one identical pipeline per song: the stems sum as the ceiling, the plain mono downmix, left and right as two pseudo-stems, mid and side as the steelman, and the harmonic half of a median-filtering HPSS as the DSP comparator, graded by root score against the chart with the grid, vote, placement and hints computed once and shared. The reproduction gate passed to the bar on both songs, the stems arm landing exactly on the frozen references, 72 of 76 and 151 of 153. The headline nobody registered: the plain mono mix drafted ABOVE the stems sum on both songs, 74 of 76 and 152 of 153; Basic Pitch transcribes drums weakly, the mix yields fewer notes than the stems sum, and the draft model reads through the bleed. Kevin's question closed in the registered direction twice: the L/R split went one bar under mono on letter and tied it on peaceful, at double the transcription cost both times. Mid/side, the pilot's best arm by one bar, crashed six bars under mono on the matrix song, the side channel's 2,270 extra cymbal and reverb notes reading as harmony in enough bars to poison the draft; the pilot edge was sampling noise with the sign flipped. HPSS failed its gate on the pilot song, two bars under mono, then tied mono on the matrix song, inconsistent and never better, and the zero-dollar panned-synthetic sweep names the trap: on clean synthetic sustains the median filter is perfect, 99.6 to 99.7 percent chord-tone purity at every pan width, and on a real song it reads strummed attacks and vibrato as percussive and takes harmony with them. The production answer is the simplest one on the table: a chartless stereo upload goes straight in as its mono downmix, same grid, same hints, same draft, stems-ceiling quality on both songs measured, no channel splitting, no mid/side, no source separation. Two disclosures published with the results: the matrix runner as first committed would have re-run the pilot song and was corrected to the registered scope before any paid call, and the matrix fence was exceeded, $0.68 spent against the registered $0.60, because the two-buffer arms carry nearly twice the notes and peaceful's drafts grew to about 38k tokens each.
- Study 34 · The chartless draft meets the wild: both breadth gates fail, and the break is upstream of the model Twelve rights-clear recordings, from 1917 acoustic 78s to a live phone-era audience tape, went through the shipped chartless pipeline against publication-grade truth. Both registered gates failed, with zero of eight accuracy-graded slots good, and every confidently wrong chart traced to the production tempo, meter and key detector misreading acoustic-era or sparse-vocal material before the grid ran. On modern full-band recordings the priors held and the drafts landed at 86 to 89 percent sequence roots. The public signed-out upload tool does not ship on this evidence.
- Study 34b · Declared priors are not enough: the five wrong charts stay wrong with the true tempo and meter supplied Study 34 blamed the detector, so the app asked the obvious follow-up: re-run the five confidently wrong recordings with the true tempo and meter declared, the shape its planned tool takes. All five stayed confidently wrong. Two failure modes emerged, both visible before the model runs: unevenly cut 78s drift off any fixed grid at the comb stage, and where the grid lands to the bar the transcription is starved, 1.3 to 2.2 notes per bar against 10 or more on modern material. The detector's tempo confidence separates its own failures exploratorily; its key confidence does not.
- Study 35 · On modern material the draft mostly lands, compound meter is a units bug, and the confidence refusal keeps only mistakes Ten modern CC BY recordings whose artist pages state tempo, key, meter and chords, run in declared mode with the app's new holds registered in advance. The texture flips: every 4/4 grid lands within 1.4 percent, five drafts grade good (three with perfect root sequences, one at a declared 240 bpm) and three more sit at 83 to 86 percent. The two failures are both 6/8 songs, and the cause is a units mismatch, not music: the artist states dotted-quarter tempo, the product reads beats of the bar per minute, and the declared bar comes out three times too long while the drift hold compares the grid to the same misread value. The registered 0.84 tempo-confidence refusal inverts out of sample, refusing both right detections and passing only wrong ones. An exploratory tempo-following beat tracker recovers the unevenly cut 78 that every fixed comb failed, 89.6 percent roots against 17 to 31.
Call shapes and reasoning
- Study 2 · Reasoning off, measured: the claim holds 51 points beyond its wording, and the call shape is not portable Stem Shovel's Text-to-Beat hard-codes reasoning off for its model on an unmeasured comment in the code. Study 2 ported the production call verbatim and measured both halves: for the production model the claim is upheld 51 points beyond its own wording (95.0% vs 44.0%, the program's first by-letter gate pass), while the finding that outranks the gate is that reasoning off is not a portable call shape: gemini accepts the field, disobeys it, and returns zero valid beats in 25 tries under production's token cap, so the documented env-var model swap would silently kill the feature.
Reading the page
- Study 3 · Can a model read sheet music? The frontier tier can, the probe had to be calibrated twice, and the PDF arm was corrected after first publication Stem Shovel's chart-read feature measured as a registered round trip over 11 Verovio-engraved scores with every fact known by construction: the production model reads a clean page at 98.8% exact chord recall as an image and, after a dated correction, 99.3% as a PDF. As first published this page reported a 20-point PDF ingestion gap; Study 5's anatomy then caught the bench's own PDF generation silently dropping every chord accidental, the affected cells were re-run on honest pages, and the finding is replaced: the hazard is PDF text generation at authoring time, detectable but not recoverable. Gates unchanged; haiku remains the sharpest tier cliff measured; the honesty gate still fails by letter through one fabricated pilot-model line.
Listening
- Study 4 · The listening layer: the second opinion reads, it does not hear Stem Shovel's song-check has a model listen to a mix and confirm or correct the DSP detector's tempo, key and meter. Against 14 synthesized mixes with truth by construction, both gemini generations are near-blind without text (production: tempo 2/12, key 1/12, zero changes found, confidence 1.0 on fabricated answers), echo whatever candidates ride beside the audio, and become perfect with the lead sheet in view (12/12 every field, all six changes placed within 4 seconds). The fixture-health stop rule fired twice and was put on trial itself: a committed onset-autocorrelation instrument proved the pulse present in all 12 MP3s, reclassifying the blindness as a model-class finding.
- Study 7 · The redesigned song-check, re-measured: deference is closed, the instruction loses to the text, and the gate catches everything that matters Stem Shovel rebuilt the song-check in response to Study 4, and Study 7 measured the replacement with the v0.192 call ported verbatim over the same truth-by-construction mixes. The verdict splits cleanly: the deference failure is closed decisively (zero wrong settings survived to application, against the retired design's 10 to 12 of 12 echo), while the no-ear-estimates instruction loses to the settings text on chord-only charts, every invented tempo the settings value verbatim, 47 of 47 cells across both models, with the application gate containing all of it. The program's recurring law, text beats instruction, makes its fourth appearance.
- Study 8 · Section detection: they hear texture, not form, and the gate is lost by half a false boundary Section detection was the last unmeasured AI feature, and the loophole test: Studies 4 and 7 showed the listening layer reads rather than hears, but what it did register blind were gross structural events, and section boundaries are gross structural events. The loophole is real: blind texture-contrast boundaries landed 7/8 and 8/8 across two models, the program's first genuine listening result. But the hearing is texture, not form: harmonic boundaries landed 3/6, uniform loops gained three to six invented sections each, and the gate failed by half a false boundary (noise 0.53 per item against a 0.50 bound) even though chart-anchored detection was perfect, because the chart's silence does not beat the model's form prior.
- Study 9 · Stem-aware analysis: the stems name the form, the sum keeps the time Drafted by the Stem Shovel agent and finished, registered, and run by the bench: does reading the uploaded stems individually beat the summed mix production actually analyzes? Four zero-model-spend DSP instruments over 21 items answered both halves. A per-stem activity detector found 8/8 section-contrast boundaries with zero false boundaries, immune by construction to the loop-invention failure that survived even chart anchoring in Study 8; and the stem-aware tempo variant lost decisively to production's summed envelope (11/21 vs 18/21, five octave errors), because cross-stem reinforcement at the true period IS the tempo signal. Total spend about eight cents.
- Study 10 · Neural MIR baselines: the classical detector holds its ground, and the home-field rematch cuts both ways Is the app hearing with obsolete code? Three published neural MIR models, hash-verified and run from vendored reference code, met production's hand-rolled detector on 35 synthetic items and then on 7 real recordings scored against the band's own declarations. All five synthetic gates failed in production's favor at zero weight bytes. The real-recording amendment then cut both ways: the neural segmenter that invented 54 boundaries on synthetic loops found real form at 29/33 while Study 9's perfect activity detector dropped to 25/33 with 19 false boundaries, a plain correction of that study's ship suggestion; and the literature key profiles that tied on synthetic material collapsed on real mixes, confirming production's Krumhansl choice. Zero model spend in both arms.
- Study 11 · The tempo ladder rule: the octave error class retired by representation, and two gates failed by their own measuring stick The fourth cross-agent study, specified by the Stem Shovel agent and registered by the bench: instead of asking the detector to hear tempo better, represent its answer as a rung on an octave ladder and gate which rungs a declared value may match. Every count-class miss from Study 10 recovers at its correct rung while the one true hearing error is correctly rejected, the subdivision test names 6/8 for the first time, and zero genuinely wrong values were accepted in 255 rows. Both failing gates failed on defects of the wrong-value generator rather than the rule, which is the program's sixth registration lesson, earned twice in one study.
- Study 11b · The shipped tempo ladder, validated: the margins are honest, and the confound shipped The Study 6b pattern completes the ladder thread: Stem Shovel built the tempo ladder in production while Study 11 ran, with a sharper subdivision rule than the bench's sketch, and asked the bench to validate the shipped rule verbatim plus a swing-stress sweep. The triplet-share argument validates perfectly (15/15 swing items stay straight, 0/332 false accepts, two of Study 10's octave misses fixed outright by the re-pick), the compound margin fails in a way the numbers forbid tuning away (one item's ratio falls among the straight songs' own), and both scorable real recordings flipped to 6/8, one on a rung the ladder does not have and one on the exact confound Study 11's first amendment named, now in shipped code. Verdict: keep the ladder, fix the flag's hypothesis, add the missing rung. Not a stop-ship.
- Study 11c · The declared-meter gate validated, and the bench's own rule retired on the record The sixth cross-agent study and the ladder thread's close: after Study 11b's verdict, Stem Shovel patched the ladder, rejected the bench's proposed fix with a constructed counterexample, and requested three zero-spend validations, including a mirror fixture designed to defeat the bench's own rule. The bench registered the prediction of its own defeat before running, and it landed exactly as traced: the declared-lag rule accepted 3 of 3 wrong mirror values and is retired on the record, while the declared-meter gate ran the full corpus at 0 of 332 wrong rows accepted and gained three true matches its predecessor missed. The one by-letter miss is the lesson-six collision class, not a gate leak, and the mirror incidentally built the first synthetic reproduction of the detector confound. Verdict: ship the patch as is.
- Study 12 · Chart-to-seconds by synthesize-and-align: code takes the model's last job, and the edges talk back The program's arc-closer, registered by the bench from its own candidates memo: render the chart as plain chord pads at the declared tempo, align the render against the recording by chroma DTW, and every bar gets a timestamp with no model. The alignment places all eight contrast boundaries within half a second, the precision metric every listener in the program failed, separates right charts from wrong ones by an order of magnitude in path cost, and on a real recording lands six mid-song boundaries within eleven hundredths of a second while missing exactly where the chart stops describing the performance. The one failed gate fails at the chart's edges, which is a data problem, not a hearing problem.
- Study 13 · The chord path meets real audio, and the reconciler earns its keep The reconciler question answered as a registered two-arm conditional: measure the thing it would improve first, run the fix only if headroom exists. The headroom existed emphatically: production's chord path, 95.8% at home on the synthetic bench, reads a real recording at 38.2%, a home-field gap near 60 points with the transcription as the binding constraint. The reconciler then earned its keep: one text-only pass by the production chart-draft model lifted the real number 16 points with twelve fixes and zero breaks, passing the anti-agreement-trap gate at infinity, and the program records its first study in which an LLM improved a hearing pipeline, by never hearing anything.
- Study 14 · The shipped alignment and reconciler validated, and the same-piece constant caught before release All four asks from Stem Shovel's Study 12 and 13 responses, run on verbatim ports pinned at their commits. The parsers round-trip the bench's material exactly, the shipped aligner reproduces Study 12's precision through its own code path, and production's reconciler configuration beats the bench's measurement of it in half the tokens. The study's reason to exist is the catch: the shipped same-piece constant of 3 rejects the band's own correct charts on real audio (right charts score 2.04 and 2.53 there against 24 to 49 on synthetic), the home-field lesson's third appearance, caught before release and already fixed in production at 1.6 within hours.
- Study 15 · The patch registered at a larger n, the ratio's margin collapses to two hundredths, and the local-cost clause restores it The Study 14 response's remaining asks, run the day the band revised the Peaceful Dreams chart to machine-true form: all ten sections grid-match its declared starts, the first fully machine-true chart in the catalog, qualifying it as the second gated song. The 1.6 constant survives its first expansion of n by two hundredths, with the two new wrong pairings landing at 1.57 and 1.58, and the deliverable lands beside it: a second same-piece clause stated by number, median local cost at or below 0.30, whose real-domain bands do not touch. The headline is the complementarity: the ratio's two near-misses carry local-cost medians of 0.365 and 0.400, caught comfortably, so the clauses fail in different directions and AND-ing them restores the margin the ratio lost.
- Study 16 · The chart reads barmark: the adoption validated end to end, the migration grid-perfect, and a wrong pair finally defeats the ratio clause The adoption study: Stem Shovel retired its in-house chord-chart parser for the open barmark standard, migrated all five stored Badverbs charts, and the bench graded the whole move with zero model calls. All four gates pass and all four registered predictions hit: the retired parser and barmark agree bar for bar on every committed text, every public migrated chart lands every band-declared section start at 0.0000 seconds on its own grid, and placement does not regress through the entirely new parse path. The headline comes from the exploratory cross-pairs: a real wrong pair scores ratio 1.74, the first ever above the 1.6 same-piece gate, and is rejected only by Study 15's local-cost clause, so the bench's recommendation to AND that clause into production went from prophylactic to urgent.
- Study 17 · The transcription successor: the neural loser wins the real-audio rematch, and the reconciler meets its first breaks Study 13 named the transcription stage as the binding constraint on real-recording chord drafts and pointed the next investment at a successor, not more model. Study 17 measured three zero-model successors against the 38.2 percent cliff, and the rematch reversed Study 10: ChordMini, which lost the synthetic chord gate, reads the real recording at 77.6 percent, past production's reconciled ceiling of 44 of 76, so the front-end swap now outranks the reconciler in ship order. The anatomy matters as much as the number: the band writes roots, not sevenths, and every successor out-reads the anchor at the root level while differing in the dressing. And the gate built against the agreement trap finally caught something: stacked on ChordMini's chords, the reconciler that never broke a bar in two studies broke sixteen, because its note evidence no longer produced the draft it was correcting.
- Study 20 · The color experiment: the notes cannot dress the model's chords, and the one surviving rule fails its held-out test by splitting 81 to zero The chord reader has the roots; the band's charts are made of colors it never writes, 74 power-chord and 50 sus4 slots on the gated song alone. Stem Shovel proposed dressing the model's roots from the free note channel, and Study 20 refuted the whole candidate: 438 of 440 on the synthetic fixture collapsed to 22 of 76 on the real recording, the recoloring destroyed twenty bars of credit the bare model already had, and the negative control beat the candidate outright. One asset survived the wreck, a power-chord call that was right 22 of 23 times it fired. Study 20b froze that rule and took it to two songs whose charts the bench had never seen, registered before the truth arrived. It failed there too, and the way it failed is the finding: 81 percent on the power-chord song, zero percent on the sparse triadic one, sixteen fires and sixteen misses. The hint's precision is not a number, it is a function of arrangement density, and the rule cannot tell which song it is in. These are the program's first full refutations, published as found, including two registration errors the bench made and owns.
- Study 21 · The chord reader on the rest of the catalog: the roots generalize at 90 percent, and the gate it fails is the grade's vocabulary, not the hearing Every real-audio chord number in the program rested on the two songs the pipeline was developed against. Study 20b banked bar-level truth for the other two charted Badverbs songs after the reader shipped, so Study 21 graded the shipped reader there, with one grading decision registered before the data existed. The anchors reproduced exactly, for the fourth time. The held-out gate failed by four bars on peaceful-dreams and passed easily on mop-heart, and the anatomy flips the failure: peaceful-dreams' roots are 149 of 153, the best root performance in the program, and the lost credit is chart dialect the grade's allowance vocabulary does not speak, the model answers G7 on G5(7) bars because it hears the optional seventh, and D7 or Dsus4 on D7sus4 bars, each half of the truth, credit zero. mop-heart is the program's first big strict result: 107 of 140 bars exactly right with no allowance involved, on the same sparse waltz that broke the note-channel power hint sixteen times out of sixteen. Across all four songs the roots stand at 90.3 percent.
- Study 26–26c · The alignment's broken ends: the hazy-audio hypothesis refuted, and a three-iteration repair recipe measured to pass the app's own acceptance gate Stem Shovel's ask arrived the same day the correction gave the bench a true ruler: the shipped chart alignment's bar map is unusable at the ends (bar one eight seconds late, the last bar fifteen seconds early), the app now refuses such maps, and the deliverable was a cause and a browser-worker recipe that gets Letter Lands through the acceptance gate without loosening it. The baseline split the hypothesis in half: the middle is good (0.16 to 0.42 seconds against true bars) and the ends are broken (3 to 23 seconds) exactly as the app agent suspected, and slicing the audio to the drum span fixed nothing, a result first read as refuting the hazy-audio cause; a second same-day correction re-graded that ablation as ill-posed, so the cause question on letter-lands and peaceful-dreams is open while the repair's measured results stand. The repair converged in three registered iterations, each gate honestly failed and priced: the proposed least-squares tempo was one percent biased and compounded to 1.3 seconds over an extrapolated tail; the onset comb measures the period to 0.02 percent but a widened end slide reached the half-bar backbeat and broke a working intro by 0.6 seconds; the third registration combined the comb period with the quarter-bar slide fence and passed everything. Letter-lands' ends went from 7.3 and 15.3 seconds to 0.05 and 0.02; peaceful-dreams to 0.06 and 0.11; mop-heart passes the app's shipped acceptance with a 0.32 second residual inherited from its own middle; eat-all-the-clocks is refused by the recipe's interior-trust check, and the second correction made that refusal the study's best moment: the disagreement it detected was real, but it was the bench's own chart fixture, 15 phantom bars against the band's live 71-bar chart, so the check is a working wrong-chart detector and its first catch was the bench's own error. One more finding rode along: the app's bar-one acceptance rule currently fails every MMKK song, so the refusal is refusing the whole set.
- Study 27 · Whole-grid re-phasing at fast tempos: recipe v5 snaps every song onto the onset lattice, and the anchor adjudication moves mop-heart's truth by 0.381 seconds Designed at Kevin's prompt the morning after the ends repair shipped: mop-heart, the 224 bpm song, was the one arrangement-matched recording the Studies 26 recipe could not bring onto the true grid, because its aligned middle carries a coherent whole-grid phase offset that end extrapolation faithfully copies. The study ran two arms for nothing. Arm A adjudicated the anchor before scoring the remedy, since the incumbent truth (0.315 seconds, Study 25's oracle) was itself under suspicion: the onset lattice measured from the full mix and the onset lattice measured from the drums stem alone agree exactly at minus 0.066 seconds, two independent envelopes naming the same grid, 0.381 seconds (47 percent of a bar) earlier than the oracle, and the registered hint rule picked it on a thin, disclosed 2-root margin that Study 24's committed drafting split (136 of 140 at the comb phase against 103 at the oracle) had already foreshadowed. That is the arc's third ruler correction: the DTW geometry, then the eat chart fixture, now the oracle anchor at the one tempo where sub-bar error matters most. Arm B then measured recipe v5, which adds exactly one move to v3: a single whole-grid comb slide of the uniform line onto the onset lattice, fenced at half a beat so it can correct sub-beat phase and can never hop to the neighboring beat. All three gates pass: mop-heart snaps onto the lattice exactly, letter-lands and peaceful-dreams move about 40 milliseconds with no regression (ends 0.05 to 0.07 seconds), the corrected-chart eat stays within the quarter-bar line, and the wrong-chart refusal fires unchanged on the superseded 86-bar fixture. v5 supersedes v3 as the recommended worker recipe at the cost of one comb slide.
- Study 29 · Can a backbeat-heavy mix pull recipe v5 off the downbeat? No: the catch map, the kick-removed remixes and the real phase errors all hold the fence Stem Shovel is porting recipe v5 as its chart placement and asked for the failure study before users' recordings find it: the whole-grid slide maximizes onset energy at the bar lines, so can a mix whose beats two and four outweigh beat one capture it? The study answered on four fronts, all gates green. The synthetic catch map: the downbeat is caught for phase errors within about 0.45 beat at EVERY downbeat weight, including one ten times weaker than the backbeat, at 120 and 200 bpm, because inside the half-beat fence no other beat is reachable, so the backbeat's weight cannot matter. The gated arm: all four songs with their drum stems high-passed at 150 Hz (the kick gone, snare and hats remaining) keep every Study 27 standard, with slide deltas of 0.05 to 0.14 beats. The arithmetic check the ask requested: real phase errors are 0.05 to 0.10 beats, an order of magnitude inside the fence, and the ask's alarming 1.4-beat figure for mop-heart was correct arithmetic on the superseded truth anchor, the ruler Study 27 corrected, not on the alignment's phase. The in-window arm found the one knife-edge exactly where the ask guessed, a weak downbeat against a pickup or hi-hat at the fence boundary, synthetic impulses only, never approached by a real arm, so the registered conditional remedy did not run. The verdict delivered to the app: ship v5; the downbeat check the ask proposed belongs to the other feature, because the same morning the capstone chartless draft watched the untethered comb anchor land on the backbeat on two songs.
- Study 36 · The field's beat trackers find the bar the detector cannot, and the one with a bar model earns the automatic re-draft Kevin asked the program to stop treating a tap-and-pick input as the realistic product and break through to a fully automatic bar. The first registered step compares the field's joint beat and downbeat trackers (Beat This!, madmom) with the production detector on the 23 recordings the program holds bar-level truth for, at zero model spend. On the ten modern songs the detector's bar period is strict on three; the trackers reach six or seven, with madmom folded on all ten and the only system to meet the registered decision rule, so the Study 38 automatic re-draft will run on its bar. On the jazz 78s every neural tracker reads the bar the detector halves, and madmom places downbeats within 70 ms on up to 96 percent of bars with counts of 114 of 114, 52 of 53 and 126 of 125, the Study 34 tempo-following result with no declared tempo supplied. Compound meter is where the trackers lose: Beat This has no 6/8 class and reads both compound songs as three-beat bars, while the detector's automatic mode gets one of them exactly, which makes Study 35's failure there the declared-tempo units bug alone.
- Study 36b · On 261 songs the detector's bar is right one time in three, a tracker's three in four, and the detector's own 6/8 call is its largest error Every tempo and meter number in this series had rested on at most ten modern songs. Kevin purchased Josh Woodward's complete catalog bundle and authorized a fetch of the 262 song pages that state each song's tempo, meter and feel under CC BY 4.0, giving 261 modern recordings with the author's own numbers and 244 paired instrumental mixes. Registered before any catalog run and scored at zero model spend: the production detector's bar is strict on 36 percent and folded at an octave on 67; Beat This reaches 70 and 90, madmom 66 and 89. On straight 4/4 songs the trackers are at 73 to 76 percent strict against the detector's 38. The detector's largest error class is its own compound-meter call: it names 6/8 on 63 of 223 songs in 4/4, every one through a triplet subdivision read, and 59 of those bars fold to nothing, the Study 11b confound at catalog scale. The instrumental mixes change little for anyone, so the voice is not the problem. Where a tracker disagrees with the page it is usually a bar half the declared length, on songs declared around 72 bpm, which is the merge-safe direction for a chart.
- Study 37 · Chord changes cannot pick the bar's octave, and a chart drafted at half the bar loses nothing Study 36b left the automatic bar at 70 percent strict and 90 percent folded: the gap is the metrical octave, a bar half or twice the written length. Study 37 tried the two ways of closing it. A harmonic-rhythm witness that scores candidate bars by how often chord changes land on bar lines is net negative in 4/4 on every base system (corrected 2026-11-13 from "on every base system"; it helps on two 6/8 bases), fixing 1 of Beat This's 34 folded readings and breaking 30 of its 165 correct ones, because the chord reader emits 60 to 124 changes per song and any stability term votes for the shorter bar. The policy that makes the fold harmless works instead: three songs drafted at half the declared bar grade 20 of 21, 16 of 18 and 33 of 33 roots against 18, 15 and 28 at the declared bar, and merging identical neighbours brings the chart under the declared-grid length. The two 6/8 slots drafted at the adapter's true bar move from confidently wrong to 23 of 26 and 40 of 42, confirming Study 35's units fix where it costs money.
- Study 38 · A visitor who types nothing gets a chart as good as one who types the artist's tempo, and free tempo is where the next work is The first charts drafted with no visitor input at all: a beat tracker's downbeats as the bar lines, its median beat as the tempo, its beats per bar as the meter, the detector's key. On the ten modern songs with artist-chart truth every automatic draft lands within two roots of the draft made at the artist's declared tempo or above it, eight of ten good, none wrong; the two 6/8 songs that Study 35 charted three bars long per bar come out 24 of 26 and 40 of 42 with no meter typed. On 36 solo acoustic-guitar excerpts (GuitarSet, the player's lead sheet as truth) the automatic grid averages 96 percent roots against 99 on the annotated bars, so the remaining cost on acoustic material is the chord reader, not the bar. On 24 free-tempo voice-and-piano songs (Winterreise) every draft is confidently wrong: the steady-tempo tracker's downbeats drift and the music changes chord one and a half times a bar, more than a one-chord-per-bar chart can carry. The tracker that follows rubato is the one that lost on full-band songs.
- Study 39 · No tracker follows a free tempo well enough to chart it: Beat This keeps the beats and splits the bars, every DBN keeps the bar and loses the beats Eight downbeat systems over 48 performances of Schubert's Winterreise with annotated bars, the proxy for acoustic demos without a fixed tempo, plus three steadily drifting 1930s jazz 78s. Beat This with its minimal peak-picking follows the rubato, finding 86 percent of the annotated downbeats within 70 milliseconds, but places a half bar on 14 of the 48 and a strict bar on only 6. Every Dynamic Bayesian Network, Beat This's own included, does the opposite: it holds a bar length on up to 24 of 48 and loses the beats under it. Loosening madmom's tempo penalty bought half the predicted recall. On steady drift madmom is best by a wide margin. No system was chartable, recall at least 0.8 at the bar or half bar, on more than 11 of 48 against a floor of 36, so no free-tempo tracker is adopted; the next study on this material is the draft format on Beat This's bars.
- Study 40 · The detector's 6/8 call protects nothing, so dropping it raises its bar on every count, and the finer-bar rule is net negative The app asked the bench three threshold questions about its own tempo detector before shipping rules: which second witness keeps the true 6/8 calls and drops the 63 false ones, whether the triplet test is a threshold away from right, and at what strength of the double-tempo rung the automatic grid should take the finer bar. Swept at zero spend on the 261-song catalog through a probe pinned to the detector on every song. No witness separates true 6/8 from false, but the call carries no bar value to protect: the detector's 6/8 bar is right on 2 of the 21 songs it is right to name, so gating or removing the call raises the strict bar from 95 to 114 songs, the folded bar from 176 to 201 and the meter from 180 to 214 with none made wrong. The triplet test's 56 false straight songs are a different test, not a threshold away. The finer-bar rule breaks two to three bars for every one it fixes at every strength, and no tempo cap rescues it. Study 38's answer stands: a tracker's bar as read.
- Study 41 · Beat This small carries the automatic chart on all ten songs, and the browser-sized export turns out to be the right weights at the wrong window The app wants to ship one beat tracker for the fully automatic chart and named Beat This small, a 10 MB model with a browser port. The bench redrafted the ten modern songs of Study 38 on its bars as read, and on the big checkpoint's. The small model's drafts are within two roots of madmom's on all ten, eight identical and two better, with none confidently wrong, so the app's rule passes and the chart ships on one tracker. Two of its bars were half bars and cost nothing. The three draft holds were censused against the 46 drafts on a tracker's bars: the detected-grid hold holds everything, the drift hold holds 17 and comes out, the thin-transcription hold fires on nothing and keeps its earlier evidence. The recipe is pinned with hashes, and the 10 MB ONNX export is found to carry the identical weights but to break past 300 frames, so the 5-second chunks it forces make it a different tracker from the 30-second model the census measured.
- Study 42 · The browser-sized tracker export is fixed, reproduces the measured system exactly, and runs fast enough for the browser Study 41 had found the bench's 10 MB export of Beat This small carried the right weights but broke past 300 frames. The cause turned out to be the rotary position embedding's cache, which an exporter bakes at the traced length. Two new exports avoid it: one traced at the reference 1500-frame window and one with the cache off and a dynamic length. Both match the Python package's frame logits to float32 noise, and the static one, run through the reference inference, reproduces the package's beats and downbeats exactly on all ten modern songs. Timed on a 30-second chunk, ONNX Runtime on CPU matches torch at four threads, and ONNX Runtime Web's WebAssembly backend takes 4.5 seconds single-threaded, about nine seconds per minute of audio, so the registered rule places the tracker in the browser behind a lazy load with the server as fallback.
- Study 44 · Two rules on the tracker's bars, measured: joining is a wash with one clear win, and the steady share enriches without separating The app now draws its generated draft on Beat This small's downbeats with two rules from five of its own recordings: adjacent half bars are joined, and a reading whose bars do not settle on one beat count is not used. The bench ported both rules verbatim, pinned them to the app's tests, and measured them over the 261-song census, the Woodward songs, the app's five recordings and the Study 43 guitar takes. Joining moves the bar count toward the artist's count about as often as away, but rescues a 6/8 song the steadiness rule would otherwise reject. The steady share removes more right readings than wrong at the app's threshold and misses the one bad draft. The 3/4 at 224 that tripped the rule trips the big checkpoint too.
Composition
- Study 5 · The composed pipeline: the chain adds nothing of its own, and its first run audited its own predecessors Production composes chart-read and the song-check: the chart beside the mix is the reader's imperfect output, not truth. Study 5 measured the composed chain against Study 4's frozen ceiling with a committed propagation classifier, and its first run's anatomy caught a cross-study confound in the bench's own PDF generation, forcing the Study 3 correction. On honest pages both gates pass at zero listener-added errors, the propagation table is 37 pass and 1 rescued per arm, and the one fact the reader omitted was rescued by the listener reading the chart's chord content, placing the modulation at 20.2 seconds against a 20.21-second truth.
Validation
- Study 6 · The warning heuristic, validated: the loop closes twice, at zero model spend Stem Shovel shipped a chart-read warning heuristic straight from Study 3's correction brief, unmeasured. The bench already held model reads of the same engraved pages in corrupted and clean form, so Study 6 validated the shipped rule at zero model spend: 10/10 detectable corrupted reads caught, the false-positive class its own source comment worried about measured at 4/42, and an exploratory key-in-force refinement whose single surviving flag caught a real defect, a read that placed its modulation three bars late.
- Study 6b · The key-in-force warning, validated on unseen pages: the production agent registered its own test, and the sheet came back perfect The program's first study registered by the production side: Stem Shovel shipped Study 6's exploratory rule and then declined to trust it, since a rule designed on the data it is scored on fits that data, so it authored its own validation and the bench ran it unmodified. On 24 charts the rule had never seen, both gates pass with a perfect sheet: 8/8 detectable corrupted reads caught, 16/16 undetectable reads silent, 0/24 clean reads flagged, zeros in every false-positive class where earlier rules each failed. The verdict delivered back: nothing to change. The program's longest loop, four hand-offs deep, closes with each side checking the other.
- Study 16b · The shipped two-clause samePiece, validated: the escapee class is closed by production code The Study 6b micro pattern, this time closing the alignment thread: Stem Shovel shipped Study 16's urgent ask, AND-ing the median local-cost clause into its production samePiece check with the constants exactly as the bench recommended and the escapee pair cited by name in the commit message, and the bench validated the ship by porting the shipped code verbatim and re-running Study 16's full pairing matrix at zero spend. All gates pass and both predictions hit: 15 of 15 pair verdicts correct, all four right pairs still samePiece true, all eleven alignable wrong pairs false including the escapee at ratio 1.74 that the previous ratio-only check would have accepted, and every placement and hedge byte-equal to Study 16. The thread that opened at Study 12 closes on shipped code, and the verdict delivered back is the one the pattern exists to earn: nothing to change.
- Study 18 · The matcher's two new rules validated on real audio: both earn their keep, and the author's own hedge is overturned in their favor Stem Shovel found two bar-level faults in its chord matcher by re-scoring this bench's own fixture notes, shipped the fixes in v0.231.0, and handed the bench the one measurement it could not make properly itself: the rules on real audio, where its approximation from reconstructed notes said the two rules pull apart. The bench re-transcribed from the committed raws, reproduced the anchor exactly for the third time, and measured each rule alone. All four gates pass. The bass rule lifts the gated song ten bars on its own; the presence rule, which the author hedged as a wash or a loss on real audio, is net plus six, because the lossy reconstruction was the artifact, not the rule. The shipped constants stand, and the real chord path moves from 38.2 to 53.9 percent, exactly Study 13's reconciled number, from two constants and one duration check.
- Study 19 · The ChordMini port gated: perfect frame parity, and the bench's numbers become the app's by measurement Study 17 made ChordMini on the mix the measured chord front end, and Stem Shovel shipped exactly that pipeline in TypeScript: librosa's constant-Q transform baked into a plan and evaluated in TS, the same hash-pinned network under a WebAssembly ONNX runtime in a child process, the reference smoothing, and the bench's own vote and label map. Study 19 gated the port, running the app's code as committed from a pinned worktree against the pinned Python reference. The result is the cleanest in the program: zero disagreeing frames across all four public mixes, 7,102 frames in all, and per-bar scores of 59 of 76 and 53 of 86, Study 17's registered numbers to the bar. The one difference found anywhere is which way the two implementations break an exact tie, and at both bars where it happens either reading grades right. The bench's real-audio chord numbers are now the app's own, by measurement rather than citation.
- Study 22 · The bar grid from the recording alone: the sanity gate caught the bench, the decision gate failed everywhere, and the real limit is the ruler An auto-written chart has no chart to align, so the recording-to-chart question stands on a prior one: can a usable bar grid come from the recording alone? The study's first run failed its own sanity gate at zero percent on every synthetic render and thereby proved the bench's registered metric defective: a blind downbeat estimate exists only modulo the bar period, and the index-paired scoring compared a 68-millisecond-accurate comb one bar shifted. The metric was replaced under the program's defective-grader rule and the re-run passed the sanity gate at 100 percent on all four renders. Then the decision gate failed in the deepest way available: even the least-squares ceiling, the best uniform grid fit to the truth itself, hits only 27 to 38 percent of truth bar starts within fifteen percent of a bar. The chart-anchored DTW truth that every chord study grades on resolves bar positions to a few hundred milliseconds, which per-bar chord grading happily tolerates and a bar-position gate cannot see through. The registered decision: the end-to-end study runs grid-given, and the no-chart grid stays open until the program owns finer truth than its own ruler.
Drafting
- Study 43 · Two cells a bar on fast-moving harmony: the dense guitar sheet was a quality gap, and the lieder lift but stay wrong The two failures left on the automatic chart looked like one failure: a one-chord-per-bar draft cannot score a bar with two chords. The study drafted twelve GuitarSet takes and the eight shortest Winterreise songs twice each on Beat This small's bars, once as read and once with each bar split at its middle beat. The premise was wrong on the guitar: the dense sheet is one chord a bar with repeats, its roots are already right on the tracker's bars, and what holds five of six takes below good is chord quality. On the lieder the half grid lifted the root median from 0.36 to 0.58 and left every song confidently wrong. One song reached good on the tracker's bars as read, the first Winterreise song to do so. No grid change recommended.
- Study 45 · Chord quality on the one-chord-a-bar sheet: the hint carries most of the loss, and a style-neutral clause recovers some of it Study 43 left the dense GuitarSet sheet with its roots right and its qualities wrong. This study first attributed every lost quality position in Study 43's guitar drafts to its cause: most sit on bars where the chord model's hint was already wrong, since the model does not hear the half-diminished chord, and the rest are power chords on two thin takes. It then replaced the one clause of the production prompt that asks the model to think like a rock musician with a style-neutral instruction about chord quality, pinned to differ in nothing else, and drafted the twelve guitar takes and the ten modern songs. Qualities rose on six takes and fell on one, the only confidently wrong draft on the shipped tracker's bars became good, the half-diminished was named in seven of twelve drafts, and the modern songs lost nothing. Power chords persisted on the thin takes. Reported without a recommendation by the registered rule; the bench's labelled judgment favours the clause.
- Study 46 · The hint's ceiling on acoustic guitar: the chord model's root holds on strummed material, and its sevenths and diminished chords hold nowhere Study 45 traced most of the chord quality lost on the dense guitar sheet to bars where the chord model's hint was already wrong. This study measured that hint directly: over all 360 GuitarSet excerpts, 5,280 bars across five styles, comping and single-line solos, how often the per-bar hint the drafts see names the right root and the right quality family. On rock-shaped comping the root is right 95 percent of the time and the quality family 83 percent; on jazz-shaped comping 66 and 56. By quality, on comping, major is right 77 percent, minor 63, dominant seventh 41 and diminished 2. On a single melodic line the hint is absent on 56 percent of bars. The bench recommends the prompt treat the hint's root as reliable and its quality as advisory for sevenths and diminished chords, the sentence to be tested in a paired study.
- Study 47 · Telling the model what the hint is worth: a sentence that changes nothing Study 46 measured the chord model's hint as two things, a root worth trusting and a quality that fails on sevenths and diminished chords. This study put that finding into the one sentence of the prompt that describes the hint and drafted the same twelve guitar takes and ten modern songs again on the same bars, paired against the drafts made under the quality clause. Seventeen of twenty-two drafts kept the same root and quality counts, the half-diminished chord was spelled in the same seven of twelve, and no modern song changed class. Reported without a recommendation. The quality the drafts lose on the dense sheet is not lost by deferring to the hint; what the model lacks there is in the transcribed notes, and the prompt studies have moved what a prompt can move.
Parts
- Study 48 · Is a transcribed bass line right often enough to show a bassist? No, and the reason is onsets, not pitch The app wants each stem's part written out, starting with the bass as a barmark line on the tracker's beats. Kevin supplied the bass stem and the session's MIDI for six of his own songs, five by MMKK and one by Ghost Ghost, 599 bars and 2,299 notes with exact truth. The study scored three transcriptions note by note, put each on the app's own beats at a sixteenth or a triplet, graded every bar with pitch, octave and slot all required, and wrote the result as barmark. Basic Pitch writes a third of bass notes an octave up, and a bass-range filter removes most of that for free; what remains is onsets, between one note in twenty and one in four missed, and extra onsets numbering 8 to 32 percent of the part's. Only a steady eighth-note part reaches seventy percent of bars entirely right, and it still has fifteen percent wrong. A monophonic tracker fixes the octave and loses notes. The answer to the app is no. Corrected 2026-11-13: the truth first used was missing back-to-back notes of one pitch, 1 to 18 percent of each song's notes, through a defect in the bench's MIDI parser found during Study 52. Every number here is from the re-run on the corrected truth; the entry first reported 598 bars, 1,999 notes, extra onsets up to half again the part's, and eighteen percent wrong on the steady part. The verdict did not change. An earlier version also named merging re-triggers and a ghost-note floor as the next lever; Study 49 tested both and recommends neither.
- Study 49 · What the bass line's wrong onsets are, and why no threshold or filter fixes them Study 48 said a transcribed bass line fails on its onsets, and the bench guessed aloud that the extra ones were mostly re-triggers and ghost notes. This study classed every wrong onset by a mechanical rule, then tried the fixes: 32 settings of the transcriber's own note-making, and two filters on the notes, a re-trigger merge and an amplitude floor. The guess was half right. Re-triggers are 49 percent of the extra onsets and the largest class on five songs of six; there are almost no ghost notes. Loosening any threshold to catch the short notes the transcriber drops adds more false notes than it finds, on every song. Turning off the two passes that make notes without a detected onset helps on all six songs, 41 to 48 percent of bars right on the mean, short of the ten points registered for a recommendation. Each filter helps one song and swallows another's real notes. Nothing is recommended, the line is still not showable, and three of eight predictions held. Corrected 2026-11-13: the truth first used was missing back-to-back notes of one pitch, 1 to 18 percent of each song's notes, through a defect in the bench's MIDI parser found during Study 52. On that truth this entry reported re-triggers as 30 percent of the extras, called the guess wrong, and counted two of eight predictions. The largest thing it got wrong, a class of extra onsets where nothing was sounding, was the transcriber hearing real notes the truth had lost.
- Study 50 · It is not the grid: on the composer's own bars and beats the transcribed bass line scores the same Twice the bench blamed the beat tracker for bars where the transcribed bass line and the truth landed on different subdivisions, without measuring it. The session MIDI for each song carries the composer's own tempo and time signatures, so this study graded the same transcribed notes three ways: on the tracker's grid, on the tracker's grid with its dropped beats filled in, and on the composer's. The share of notes right is the same on every song, within a hundredth. The tracker's beats are in the right place; it drops some and gets the bar lines of the 5/4 and 3/4 songs wrong, which matters to a chart a player reads and not to whether the notes are right. The false triplet bars were the bench's own per-bar rule, not the tracker. After three studies of settings, filters and the grid, the transcriber itself is what is left. Three of six predictions held. Re-run 2026-11-13 on a corrected truth, with the same conclusions.
- Study 51 · Showing only the bars it can trust: agreement with a second transcriber helps, and shows too little If a transcribed bass line is right in under half its bars, the app could still show the bars it can trust and leave the rest blank. With no truth to consult, the bench trusted a bar only when another reading of the same stem wrote it identically: the app's own transcriber settings, two loosened settings, and a second transcriber. Agreement among the transcriber's own settings filters little, because they share one model's output. Agreement with the second transcriber lifts the shown bars from 47 percent entirely right to 76, and on two songs it would do. But it shows a quarter of the bars and a sixth of the notes, and on one song the bars both readers agree on are wrong more often than right, because they miss the same notes. No rule is recommended. Trust by agreement is bounded by how good the second reader is. All five predictions held. Corrected 2026-11-13: the truth first used was missing back-to-back notes of one pitch, 1 to 18 percent of each song's notes, through a defect in the bench's MIDI parser found during Study 52. On that truth this entry reported 41 and 64 percent, one song, and four of five predictions.
- Study 52 · Two note models read the bass line far better than Basic Pitch, and the first exposed notes missing from the truth Four studies had left the transcriber itself as the limit on a written bass line. This one ran a different transcriber on the same six stems: an open note-level model with a bass checkpoint, trained on jazz double bass. Its first scores made no sense. It beat Basic Pitch note for note and lost badly on the easiest song, and the notes it was being marked down for were notes a third transcriber wrote identically. The fault was the truth: the bench's MIDI parser had been losing back-to-back notes of one pitch, 1 to 18 percent of every song's notes. The parser was fixed, four earlier studies were re-run and corrected, and this one was registered again. On the corrected truth the bass model gets 64 percent of bars entirely right against Basic Pitch's 48, with no octave errors at all, and three songs clear the bar for showing the line, which nothing had done on any song. By the rules written before the run it is still not recommended: one song scores lower because the model hears twenty stray sounds that editing left in the stem, and one song defeats every reader. Five of six predictions held. A second model, MuScriptor, then ran on the same stems and did better still: 71 percent of bars right, four songs over the bar, and the first registered yes of the bass studies, a better reader than Basic Pitch by 22 points with no song losing. Its weights are not licensed for commercial use. Neither model yet makes a line that can be shown on every song.
The article series · every post, in order
The Full Series
Every post in the Stem Bench series, in order. Start at the top for the whole story, or jump straight to the latest study writeup.
- The Models That Can Read Music Don't Cheat Oct 21, 2026
- The Second Opinion Wasn't Listening Oct 23, 2026
- The Study That Audited Its Predecessors Oct 24, 2026
- They Hear Texture, Not Form Oct 28, 2026
- The Stems Name the Form, the Sum Keeps the Time Oct 29, 2026
- The Home-Field Rematch Oct 29, 2026
- Thirty-Eight Percent Oct 30, 2026
- Seventy-Eight Percent Nov 5, 2026
- The Band Was Right Nov 7, 2026
- The Witnesses Were Right Nov 8, 2026
- The File That Needed No Stems Nov 9, 2026
- The Detector, Not the Drafter Nov 9, 2026
- The Bar That Was Three Bars Long Nov 9, 2026