Stem Bench

When a production app ships a feature built on a language model, the design decisions inside it are empirical claims: which model, which prompt scaffolding, whether a heuristic hint helps or hurts. Stem Bench is our open, pre-registered benchmark series measuring those decisions in Stem Shovel, Lightning Jar's music collaboration app: deterministic fixtures with ground truth by construction, production-pinned prompts and pipelines, mechanical grading, and spend fences.

Study 1 measured the chart-draft pipeline: when the model reads a noisy music transcription with a free template matcher's chord guess riding along as a hint, does it read the notes or copy the hint? 320 scored cells across five models. Hypotheses, corpora, and graders are committed before any scored run; results are published as found, failed gate letters included.

Methods & Provenance

One pre-registered study so far: 320 scored cells across five models at temperature 0 (October 2026). The study was registered by commit before its first scored run (hypotheses, corpus, synthesized fixture, and graders all committed first) and published as found, gate letters included. The fixture is 16 hand-authored chord progressions (440 bars, known chords by construction) rendered to audio by deterministic synthesis and transcribed by Spotify Basic Pitch with the production app's exact model files and thresholds, so the model reads the same noisy transcription the shipped feature reads. Prompt texts are the production app's verbatim, pinned by test; grading is mechanical (root pitch class plus canonical quality); and every call shape matches production, including two registered amendments: call-shape fixes for two models whose first run produced no scorable cells, and an 80-cell re-run of the shipped arm after Stem Shovel adopted the study's matcher fix in v0.142.0.

0/438

systematically wrong hints copied by the production draft model (claude-fable-5-1); the pilot model matched it, so the frontier tier copied zero roots in roughly 850 chances (Study 1)

69.1%

kimi-k2.6's flip rate: bars it answered correctly with no hint that it surrendered the moment a wrong hint appeared; haiku 4.5 flipped 49.5% (Study 1)

99.8%

bar accuracy of the shipped configuration (fable-5-1 reading notes plus the matcher's hint), 9.3 points over the free matcher alone at about $0.10 a draft (Study 1)

90.5% → 96.4%

the free template matcher's baseline as the study ran (v0.141) and after Stem Shovel shipped the study's missing-tone matcher fix in v0.142.0 the same day; under the better hint, haiku, kimi, and gpt-oss still score below the bare matcher in the shipped configuration (Study 1, Amendment 2)

11/16

songs where fable, shown wrong hints, flagged the disagreement in its notes field unprompted; kimi flagged zero (Study 1)

$8.70

what kimi-k2.6's default reasoning burned to produce no JSON in 62 of 64 cells before the registered amendment turned reasoning off, reproducing the production text-to-beat lesson (Study 1)

The study series

Every Study, One Page Each

Every study is pre-registered before its first scored run and gets its own page with charts, tables, and a verdict. Grouped by theme:

The chart-draft pipeline

Study 1 · The hint-following study: models that can read the notes don't copy the hint Stem Shovel's chart draft shows a model each bar's transcribed notes plus a free template matcher's chord guess. Across 320 cells on 440 seeded bars, the frontier models copied a systematically wrong hint zero times between them, while kimi-k2.6 surrendered 69% of the bars it could read correctly on its own; the shipped configuration (fable-5-1 plus the hint) scored 99.8%, 9.3 points over the free matcher alone, at about $0.10 a draft.