The Sentence That Changed Nothing

The first chapter of this series put a blunt question to Stem Shovel's chord-chart drafter: when the prompt sets a free guess at each bar's chord beside the transcribed notes, does the model read the notes or copy? In The Models That Can Read Music Don't Cheat the answer was clean: the frontier models never copied a planted wrong hint. The Second Opinion Wasn't Listening found the opposite habit in another feature, a listener that tracked the text beside the audio instead of the audio. After a run of studies on beat trackers and where the bar lines fall (the previous chapter), three studies came back to the hint with the program's oldest question turned around. Not whether the model trusts the hint too much, but whether the hint deserves the trust, and whether saying so out loud changes what the model writes.
What the Hint Is Now
In Study 1 the hint was a small template matcher. Since Study 19 it has been ChordMini, a chord-recognition model with a 170-label vocabulary, ported into the app and gated by the bench at perfect frame parity. Its per-bar vote rides in brackets beside each bar of transcribed notes, and the bench's port of the prompt still introduces it as a template matcher's guess: a hint, not an answer.
Where the Quality Went
Study 43's dense GuitarSet sheet, SS2-107-Ab, was never a format problem: on the Beat This small0 tracker's bars its roots were right on four of six takes, and what kept five of six below "good" was chord quality. The two SS2 sheets are ii-V-I progressions with a half-diminished seventh among their chords, and the drafts wrote "Bbm" or "Bb7" for the half-diminished, "Eb" for a dominant, and on two takes power chords throughout.
Study 45's first arm, at no cost, recomputed the hint for every bar of Study 43's 24 guitar cells, took the sheet's chord as truth, and classed every lost quality. On the dense sheet at one chord a bar there were 40 lost positions. On 25 of them the hint's quality family was already wrong: the model followed it on 14 and wrote some third family on 11. Thirteen were power chords on a single take. Only 2 were the model throwing away a right hint. P1, that the hint carries at least half the loss, was right.
The first write-up said the hint was wrong there because ChordMini's vocabulary has no half-diminished. That was false, caught the same day. The vocabulary carries hdim7 for every root, and the bench's own label map writes it as "dim", the family the grader shares with m7b5. The hint was wrong on those bars because ChordMini did not hear a half-diminished chord; on the A#:hdim7 bar it voted E or C#. The attribution numbers stand. The cause changed from vocabulary to accuracy, and accuracy can be measured.
A Clause Written for a Rock Band
The second arm went after a different suspect: production's system prompt tells the model to "think like a rock musician reading a chart" and that "power chords (5) and sus chords are common here." It was written for the rock band the app was built around, and on a jazz-flavoured acoustic sheet it leans against the very qualities the notes carry. Study 45 replaced that clause, and only that clause, pinned by test, with a style-neutral instruction to name the quality the notes support: "a seventh (7, m7, maj7, m7b5) when the seventh sounds, a plain triad when it does not, and a power chord (5) only when neither third sounds." The ten modern Josh Woodward songs rode along as the guard, because a clause that helps a jazz sheet and hurts a rock chart is not a recommendation.
On ten paired guitar takes, qualities rose on six and fell on one, and roots fell on none. The only confidently wrong draft on the Beat This bars, 04_SS2-107-Ab, came back good, every root right and the half-diminished named. Across all twelve takes, five drafts were good under the clause against three of the ten production drafts that answered, and the half-diminished was spelled in 7 of 12 drafts against production's 3 of 10. The Woodward songs lost nothing: no class lowered, one quality count up by one, one root count down by two, eight good drafts before and after. The rock clause was not carrying the rock charts.
Two predictions failed by letter. P2 wanted at least two more good takes among the pairs and got one. P5 wanted no power chord on the guitar takes; drafts carrying one went from three to five, so "only when neither third sounds" did not hold the line on thin takes. P1, P3 and P4 held. The registered rule recommends the clause only if P2 and P3 both hold, so the outcome is reported without a recommendation. Beside the rule, the bench's labelled judgment favoured the clause, and the quality clause is now the app's v0.262.1.
Unlooked-for: two guitar takes whose production drafts the gateway's content filter had stopped in Study 43 drafted normally under the new clause. Two cases; the bench recorded it and drew no causal line.
The Hint, Measured
Study 46 questioned the hint directly, without a single model call. Across all 360 GuitarSet excerpts, six players, five styles, comping and solo, 5,280 scored bars, how often does the per-bar vote the drafts see name the right root and the right quality family against the lead sheet the player was given?
On comping, the shape of the app's input, the root was right on 77.5 percent of bars, split sharply by style: 96.2 percent on Rock, 93.9 on Singer-Songwriter, 78.8 on Bossa Nova, 61.0 on Jazz, 57.8 on Funk. The quality family was right on 66.9 percent of comped bars, and family by family the picture sharpens: major 76.8 percent, minor 62.8, dominant seventh 40.8, diminished and half-diminished 1.7. Over the whole set the sheets held 240 diminished-family bars, and the hint named the family on four. On comped diminished bars it said minor 54 times, major 31 and dominant 21. ChordMini can name a diminished chord; across GuitarSet it almost never does.
On a single melodic line the hint is mostly absent: 56.2 percent of solo bars vote no chord at all, and the root is right on 11.1 percent. Against what the players actually performed, family accuracy rose by 2.3 points overall and 4.8 on comping, so the players' departures from the chart are a small part of the disagreement.
Six predictions, three right. P1 (comping roots at least 85 percent) failed at 77.5; P2 (family right on at least 70 percent overall) failed at 44.0, sunk by the solos; P6 (the performed truth at least three points higher) failed overall and held only on comping. P3 (diminished under 25 percent), P4 (dominant under 60) and P5 (solo roots at least ten points under comping) held. The registered decision: minor, dominant and diminished all fall under 50 percent over all excerpts, so the bench recommends treating the hint's quality as advisory for those families. On the labelled comp-only view, minor clears the line and dominant and diminished do not.
It is also a quiet audit of Study 1, whose trust in the hint's root was earned on a rock-shaped corpus and still holds there.
Telling the Model
Study 47 put Study 46's finding into the one sentence of the prompt that describes the hint. Where production's sentence calls the bracketed guess a hint, not an answer, the variant read:
A chord reader's guess follows each bar in brackets. Its root is reliable: keep it unless the notes plainly contradict it. Its quality is advisory: it hears major and minor well, hears a seventh as a plain triad more often than not, and almost never hears a diminished or half-diminished chord, so decide the quality from the notes and the key.
Nothing else changed, pinned by test against Study 45's prompt. The same 22 items were drafted again on the same bars with the same hints and paired against Study 45's quality-clause drafts.
The qualities the predictions were waiting for did not come. Guitar qualities rose on two takes and fell on one, against a registered five up. The half-diminished was spelled in 7 of 12 drafts, against a registered 10, and they were the same seven drafts as before. Roots fell on none. One saved draft, 03_SS2-88-F, crossed into good, so good drafts went from five to six, and drafts carrying a power chord went from five to four. On the Woodward songs no class changed and good stayed at eight; one song's root count fell a position, and another's rose two while losing a quality position. Seventeen of the twenty-two drafts kept their root and quality counts exactly. P1 and P2 failed; P3, P4 and P5 held. By S47-D1 the result is reported without a recommendation.
The bench's reading, labelled as a reading: the quality the drafts lose on this sheet is not lost by deferring to the hint. Study 45 counted 14 positions where a draft followed a wrong hint, but told in so many words that the hint almost never hears a half-diminished, the model still wrote it in the same seven of twelve drafts. That says the half-diminished it did not write is not in the transcribed notes it was given either, and that the remaining gap on this material is the evidence, Basic Pitch's notes on a strummed chord, not the prompt. The two prompt studies have moved what a prompt can move.
What It Does Not Say
Since Study 41 the app's prompt has moved past the bench's port: it hands the model a starting chart built from the chord model, with a sentence saying the chord model "hears roots and plain qualities (major, minor, 7ths, dim, sus) only". Study 47 tested the content of the bench's sentence in the bracketed structure Study 1 built, not the app's sentence, and it does not say the app's sentence is right about sevenths and diminished chords; Study 46 says it overstates the hint there. Describing the hint honestly avoids overstating it; it should not be expected to change a chart.
The Ledger
Study 45 cost $1.462 against a $2.50 fence, 322,311 tokens in and 54,483 out, 22 drafts and none refused. Study 46 cost $0 and never touched the gateway; it took 23 minutes, not the hour estimated. Study 47 cost $1.459 against a $1.50 fence, four cents to spare; the $0.75 estimate was half the real cost, because the Woodward songs' long prompts were under-counted.
The limits are disclosed. The gateway ignores temperature for this model, so draft-to-draft variance is unmeasured in both drafting studies, and a one-position move on a single take is within what a rerun might show. Study 45's guitar comparison has ten pairs, not twelve, because two production drafts were refused in Study 43. Study 46's comp-only and style cuts are post-hoc, labelled as such, and no recommendation rests on the solo excerpts.
Three corrections sit in the record. The vocabulary claim above was corrected the same day in the brief, in REPORT.md, on the dashboard and in the note to the app. Study 45's runner printed "recommend" under a looser condition than the registered rule; the brief's rule governs and corrects the printout. And Study 47's write-up first said nineteen of the twenty-two drafts kept their counts, which missed the two modern songs that moved; it was corrected in place on 2026-11-13, while this chapter was being checked.
The registrations, cached hints and every draft live in the stem-bench repository; the entries are Study 45, Study 46 and Study 47 on the research dashboard. Study 1 asked whether the model copies the hint, and the strong models did not. Now the hint is a chord-recognition model, its weaknesses are measured family by family, and the drafter has been told about them in plain words. The drafts barely noticed. On the bench's reading, the chord they miss is not in the notes they were given, and no sentence about the hint can put it there.