Rasterize Your PDFs
Stem Shovel has a feature where a model reads an uploaded score, a PDF or a photo of engraved sheet music, into a measure-by-measure text summary that everything downstream depends on. The model choice behind it rested on one informal test. Study 3 of stem-bench replaced the test with what we believe is the first registered benchmark of LLM notation reading, and the headline ship guidance is in the title.
How you Benchmark Sheet Music Reading
The trick is the same one this series keeps using: author the ground truth, then let the production pipeline make the artifact. We wrote eleven scores as MusicXML, so every key signature, meter change, repeat, first and second ending, segno, coda, slash chord, and no-chord bar was known by construction. Verovio engraved them into exactly the pages a user's PDF would contain, including two two-page scores. Models read the rendered pages under the production prompt, verbatim, and the grader compares their summaries against the deterministic summary the app itself makes of the source file: 268 readable measures, graded measure-exact per element class.
Plus one page rendered too small to read, where the registered correct answer is to say so.
The Answer is Yes
The shipping model reads engraved notation from a clean image at 98.8 percent exact chord recall, with perfect precision and zero measure-count drift, for about six cents a score. The informal anecdote that chose this model claimed that "read as text, the keys are plain." Under measurement it held to the decimal: we registered a 90 percent bar on key signatures, and the model landed at 91.7.
That sentence would have been worth the study by itself. It is not the most useful thing we learned.
The Bottleneck is the Plumbing
Send the identical page as a PDF instead of an image, and three of the four models lose twenty or more points of chord recall. The shipping model drops from 98.8 to 76.0. The first casualties are the dots on repeat barlines, which simply vanish; the volta brackets above them read fine everywhere.
Nothing about the notation changed. What changed is who rasterizes. A PDF handed to a provider gets rendered to pixels somewhere inside their pipeline, at a resolution and quality you do not control, and that internal step is lossy enough to erase small musical symbols. Opus, contrarian as ever, reads PDFs better than images, which settles the question of whether this is something about PDFs: it is per-provider plumbing, nothing more.
The ship guidance went to the app the same day: rasterize PDFs yourself and send pictures. It is one of the cheapest fixes this program has produced, and we would never have found it by testing prompts, because it is not a prompt problem.
The Fixture Fought Back
Two amendments landed before the matrix, both committed with their anatomy, both instructive.
First, our transparent PNGs reached the model as black on black. The first pilot carefully measured a void, and to its credit the model said so: "the provided image is completely black." Composite your transparency before you benchmark vision.
Second, the unreadable probe had to be destroyed twice. Vector PDFs stay crisp at any zoom, so our first "too small" page was not. Then the 150-pixel raster smear, which no human can read, turned out to be legible to a vision model: the pilot pulled seven of eight chords off it. The probe ended up at 80 pixels, and the calibration lesson is worth the whole amendment: unreadability has to be set by model behavior, not by eyesight.
The Gate Failed
Every clause gating the production model passed. The study gate still failed, because we wrote "zero fabricated lines, every model, both arms" into it, and the pilot model, shown the final 80-pixel smear, invented a reading: "m1 key C major 4/4," under a title it made up, "VIOLIN I," rather than say the page could not be read. All three Anthropic models declined honestly. This is the third time this program's registration wording has made the weakest subject the gate, and it is published as failed, per the house rule that gates are never re-thresholded after data exists.
The tier cliff deserves its own sentence: Haiku reads the key signatures, then writes a 120-token stub and stops, for 1.2 percent chord recall and a fifteen-measure drift. Study 1 found the cheap models copy answers they are handed; Study 3 finds that below some floor they cannot do the job at all. Some features have a minimum tier, and now this one's is measured.
The disclosure stands as always: the analyst model is Claude Fable 5, the matrix includes its sibling, and every grade is mechanical, with the ground-truth instrument being production's own summary code, ported verbatim.
The registration, corpus, amendments, and full report live in the stem-bench repository; the Study 3 page is landing on the research dashboard. The series so far: The Models That Can Read Music Don't Cheat and The Model That Ignored the Off Switch. The app is Stem Shovel.
