The Model That Ignored the Off Switch
There is a one-line comment in Stem Shovel's drum machine code that has been making a production decision for months: "a thinking model spends two or three minutes on a drum pattern and gets it no better." On the strength of that sentence, the Text-to-Beat feature calls its model with reasoning switched off. The sentence came from watching it happen a few times. Nobody had measured it.
Study 2 of stem-bench measured it. The production call was ported byte for byte: the same system prompt, the same validator with its exact error messages, the same retry that hands the model its own rejection reason. We registered 25 beat prompts with 49 mechanical checks, each tied only to something the prompt explicitly asks for, a stated bpm, a named drum, hits on named steps. No taste rubric anywhere; a benchmark should not have opinions about funk. Then we ran reasoning-off against reasoning-default across five models.
The Claim Held, and Then Some
For the shipping model, Kimi K2.6, reasoning off scored 95 percent. Reasoning default scored 44 percent, at twenty times the latency and thirty times the output tokens.
The comment said thinking "gets it no better." The measurement says it is catastrophically worse, and the mechanism is blunt: Kimi's thinking did not fit even an 8,000-token budget, more than five times production's cap, on 14 of 25 prompts. It burned the whole allowance deliberating and returned literally nothing. And the production retry cannot save an empty reply, because you cannot replay a conversation that contains an empty assistant message; the API refuses it. The one nuance worth keeping: on the eleven prompts where Kimi's reasoning actually finished, its checks were perfect. Completed thinking helps a little. It completes less than half the time.
This is the first gate in this research program to pass by letter, and it passed by 51 points.
The Off Switch is a Request, Not a Guarantee
The finding that outranks the gate came from the model we almost did not include. Gemini 3.5 Flash accepted the reasoning-off setting and ignored it. It thinks about three thousand tokens no matter what you ask. Inside production's token budget it burned to the cap, twice per request thanks to the retry, and produced nothing on 25 of 25 prompts. Given room to think, it was nearly perfect: 99 percent.
Stem Shovel's beat model is swappable by environment variable, deliberately, so models can be compared without a deploy. This measurement says that swap would silently kill the feature. Not degrade it: zero out of twenty-five, from a model that scores 99 when called the way it wants to be called.
Every provider answers the off switch differently. Anthropic's models shrugged: Fable and Haiku scored identically with reasoning on or off, with adaptive thinking costing almost nothing. gpt-oss-120b was the one model reasoning genuinely helped, from 77 to 92 percent, mostly by fixing its own sloppy first drafts before the validator saw them. And Gemini's broken off-arm was slower than its thinking arm, 26 seconds against 16, because burning to a token cap twice takes longer than thinking does. We registered a fallback for models that reject the setting with an error. We did not anticipate a model that accepts it and disobeys. Published as measured.
The Scorecard
Our registered predictions did not survive contact either. We predicted the reasoning arm would land within three points of the off arm for the production model; it landed 51 apart. We predicted 85 percent first-try validity in the reasoning arm; the measured figure was 40. Two corrections were made before any result was read, both dated in the brief: an analyzer scoring rule brought back to the registered wording, and the brief's own miscount of its committed corpus.
The ship guidance went to the app the same hour. The hard-coded reasoning-off choice is now a measured one for the shipped model. The env-var swap path needs either per-model output budgets or a one-request smoke test before any new model takes the feature. And any swap toward a reasoning-capable model should check the retry path against empty completions, which the current conversation-replay design cannot survive.
One disclosure we will keep repeating: the analyst model behind this series is Claude Fable 5 and the matrix includes its sibling; every grade here is computed by production's own validator, ported and pinned, with no judged scoring anywhere.
The registration, corpus, and full report live in the stem-bench repository, the Study 2 page is landing on the research dashboard, and the series opener is The Models That Can Read Music Don't Cheat. The app whose code comment started all this is Stem Shovel.
