Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 80

The failure is the declared-input stage, not the drafter

All five confidently-wrong slots trace to `analyse()` mis-detecting tempo, meter or key on acoustic-era or sparse-vocal material before the grid ran: the hymn quartet read 78.6 ...

Stem Shovel Testing verified

A batch of Stem Shovel drafts scored well on modern full-band recordings and fell apart on older or sparser material. Five slots came back confidently wrong: the chord grid was internally consistent, the roots were plausible, and the whole thing was built on the wrong tempo, meter, or key. The easy read was that the drafter (the LLM stage that writes the grid) was hallucinating on unfamiliar styles.

Cause

It was not the drafter. All five confidently-wrong slots trace back to analyse(), the stage that detects tempo, meter, and key and hands them to the grid as declared inputs. The drafter did exactly what it was told with bad facts.

The three failure shapes, all on acoustic-era or sparse-vocal material:

  • Wrong key and tempo together. The hymn quartet was read as 78.6 bpm in C minor. The truth is roughly 105 bpm in Ab major. The draft landed 0 of 16 roots.
  • Wrong meter. The 3/4 waltz was read as 4/4. The draft hit 8% of roots.
  • Half-tempo grid. Dippermouth's grid sat at exactly half tempo: 57 bars drafted against 114 true bars.

On modern full-band recordings the analysis priors held, and drafts reached 86 to 89% sequence roots with 80 to 100% quality families, just under the "good" threshold. Same drafter, same prompt. The only variable that moved was the quality of the declared inputs.

Fix

Stop tuning the drafter for these cases. Effort spent on prompts or examples cannot recover a grid built on the wrong bar count. The lesson that holds is upstream: treat tempo, meter, and key as inputs that need their own validation before the grid runs, and treat half/double tempo, 3/4 versus 4/4, and relative major/minor as the first suspects when a draft is consistent but wrong.

Receipt

Source: stem-shovel/FINDINGS.md. The control run reproduced its established ceiling in the same batch (eat, 67 of 68 roots), which rules out a regression in the drafter or scorer and leaves the declared-input stage as the only thing that changed between the slots that passed and the slots that failed.

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.