Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 19

Two independent graders will disagree by one bar; explain it or trust neither

The product's re-score said 397 to 423; the bench's said 398 to 424. The one-bar gap was a silent bar counted correct under a registered equivalence. Chase edge disagreements to a documented rule.

Stem Bench LLM Operations verified Two independent implementations of the registered grading rules

After Stem Bench's study shipped fixes back to the product, both sides re-measured the same thing: the product's re-score said the matcher improved from 397 to 423 bars, the bench's said 398 to 424. Two independent implementations of the same grading rules, disagreeing by exactly one bar in both columns.

The disagreement turned out to be a definition, not a defect: one bar is silent, the ground truth and the matcher both call it "no chord," and the registered equivalence rules count that agreement as correct while the product's quick re-score skipped it. One sentence of documented rule resolved the discrepancy completely.

The general rule: when two graders agree to within a whisker, do not round the difference away. Chase it to a documented rule or trust neither number, because a one-unit mystery at the edge of agreement is exactly how a one-unit mystery at the center of a headline starts.

Receipt: Stem Bench Study 1, REPORT.md "Production response and Amendment 2," cross-checking note.

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.