Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 82

The app's `draftChart` verdict, read against the classes

Stem Shovel's draftChart hold check would have caught five bad chart drafts; the bench runner now logs data to prove it holds up.

Stem Shovel Browser Platform verified

Stem Shovel's draftChart feature returns a verdict on an AI drafted chart: accept it, or hold it back for review. Reading that verdict against how a batch of drafts were independently classified afterward showed which checks carry real signal.

Five charts in the study came back confidently wrong: the model was sure of a bad transcription. In every one of those five, the draft's bar count had drifted from the source grid's. That drift is a cheap symptom of the same upstream error that corrupted the content, since both numbers trace back to the same step. A changed bar count hold would have caught, and held, all five. The study's one genuine refusal, on a clean DAW export (interbellum), was the kind of error the model could diagnose and flag on its own. The changed bar count hold would have agreed with that refusal too, so it does not contradict the one judgment already trusted. A third candidate, a threshold on how much of a draft comes back unknown, could not be checked this run because the raw drafts were not kept.

The fix that shipped: the bench runner now records rawDraft, barCountChanged, and unknownShare on every run, not only the final verdict.

The general rule: when a cheap structural check and an expensive semantic judgment tend to fail together, check the cheap one first, and keep the raw material so the correlation can be proven later instead of assumed.

Receipt: Stem Shovel, draftChart verdict logic and the bench runner's rawDraft/barCountChanged/unknownShare instrumentation.

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.