The Study That Audited Its Predecessors
Studies 3 and 4 of stem-bench measured two halves of a Stem Shovel feature separately. The score reader turns an uploaded lead sheet into a text summary. The listener takes a song's audio plus that summary and answers the question the app actually cares about: what changed, and at what second. In production they run as a chain, and the summary the listener receives is the reader's imperfect output, not the truth our benchmarks handed it.
Study 5 composed the real chain and classified every fact's path through it: pass, listener-added, propagated, rescued, or mutated. Twelve mixes, their matching lead sheets engraved by the same committed pipeline, the reader's actual output fed downstream, the frozen single-stage results as the registered ceiling.
The First Thing We Discovered
The biggest result arrived before the chain was even clean. The first run's anatomy showed the PDF-arm reader reporting chords with no flats in them, and the mechanism check found why: the font encoding in our PDF generation cannot represent the music accidental glyphs our engraver writes into chord text. Every flat and sharp in every committed PDF had silently printed as its natural. The pages were lying, in a way nobody had noticed across an entire published study.
That finding does not belong to Study 5 alone. It withdrew a headline from Study 3, where the "PDF ingestion is the bottleneck" mechanism and the rasterize-your-PDFs guidance both rested on corrupted pages; the published article now carries the correction, and the corrected numbers say frontier models read a clean PDF exactly as well as a clean image, 99.3 percent chord recall for the shipping model. A composition study auditing both of its predecessors within a day is the strongest argument we know for running composition studies.
The Chain Turned Up Clean
With honest pages, both gates passed at zero listener-added errors. Across 38 scored facts per arm the propagation table reads: 37 pass, 1 rescued, nothing propagated, nothing mutated, nothing added.
The one rescue is worth describing, because it is the whole series in one anecdote. The reader made exactly one error in the study: it omitted one modulation line from one summary. The listener recovered it, both arms, reporting the key change and placing it at 20.2 seconds against a truth of 20.21. But it did not hear the modulation: Study 4's blind arm already established that this model finds zero changes by ear. It read the chart's chord content, noticed the progression turning to C material, and did the arithmetic. Even the rescue is reading.
What a Chain Does With a Lying Page
We kept the confounded first run in the record, because it answers a question the clean run cannot: what does a two-model chain do with corrupted input? It transmits the damage, faithfully and confidently. A reader looking at a B-minor sheet whose F sharp seven printed as F seven reasonably called the key D major, and the listener propagated it one to one. A modulation the reader placed at measure twelve instead of measure nine became a confidently wrong second-placement downstream. No garbling, no compounding, no self-correction. Clean transmission of a lie.
Put the two runs together and the composition story is simple: the chain adds nothing of its own. It is exactly as good as the page its reader is shown.
The ship guidance follows directly. Since the song-check inherits chart-read's accuracy one to one, the protective move is to show the human the middle: surface the chart summary for correction, and everything downstream is protected. And treat the model's own unprompted inconsistency notes as a first-class signal, because when the key signature disagrees with the chord spellings, the production model already says so, and that note was the only witness to our corrupted pages.
The usual disclosure: the analyst model is Claude Fable 5, the reader under test is its sibling, every grade is mechanical and committed before any scored run, and the amendment recording the confound preceded the corrected run.
The registration, propagation classifier, both runs, and the full report live in the stem-bench repository; the Study 5 page is landing on the research dashboard. The two halves this study composed: Rasterize Your PDFs and The Second Opinion Wasn't Listening. The app is Stem Shovel.
