The Cheapest Model Passed Every Gate
There is a lot of justified excitement about open-weight models right now, and our benchmark series had a blind spot: forty-two studies, every one of them run on closed-weight models. This week we backfilled. Three additions: kimi-k3 (the open-weight frontier flagship), gpt-oss-120b (open-weight, priced like a rounding error), and fable-5 (the closed frontier tier above the one our product actually ships on).
We did not re-run forty-two studies. We ran the instrument the series left behind for exactly this moment.
The Instrument
Every time a study in this series validated a shipped mechanism, it also left a pass/fail gate: a registered slice of tasks, a registered threshold, and a named piece of the product it protects. There are thirteen now. Anchored patches, focused views, search grounding, the ask-instead-of-guess hatch, the session memo, the last-edit echo, the precedence clause, and so on. Point the suite at any model id and about an hour later you get a report where any red line names the exact product surface that would break if you shipped that model. Thresholds are registered up front; a red gate gets investigated, never re-thresholded.
Cost per model: between 33 cents and 35 dollars, depending on the price tier. This is the cheapest form of due diligence we know.
The Cheap Model's Perfect Run
gpt-oss-120b passed all thirteen gates. Total spend: $0.32.
The implication deserves spelling out, because we designed for it and it still surprised us. The guardrails this suite tests are the ones we built after every study kept teaching the same lesson: hand the model what it needs instead of hoping it infers, give it a tool instead of a rule, and let the server own the invariants. A stack built that way should not depend on model brilliance. And apparently it does not: the cheapest model we have ever pointed at the suite cleared every gate, including slices where its raw unguided behavior is the worst we have measured. The guardrails are doing the work. That was the bet.
Two Frontiers, One Red Gate
kimi-k3 went 12 of 13. fable-5 went 12 of 13. Same red gate.
The gate is goal-safe eviction at the session memo's cap. Our chat surfaces let the agent keep a small structured memo of session facts, rules, and goals, capped at twenty notes. When a twenty-first arrives, the app runs a designed eviction: the oldest fact goes, never a goal, because an earlier study measured goals as the one thing only the memo carries. The design has a known boundary: it can only protect lists it receives. A model that trims its own list before sending is outside the app's reach.
Our shipped tier passes this slice by over-sending: it hands the app twenty-one notes and lets the server decide. Both new frontier models fail it the same way: they respect the cap. They quietly trim the list to twenty themselves, and the note they drop is an old goal. fable-5 did this in all three of its failing cells. The failure is a form of good behavior. The model followed the stated limit and sacrificed the one thing the limit exists to protect.
There is a grace note in the same data: in its passing cells, fable-5 sometimes answered the cap by consolidating twenty notes into fifteen denser ones, losing nothing. The disposition that breaks the guarantee and the disposition that transcends it live in the same model.
The Part Where We Grade Ourselves
Full disclosure: the agent that operates this benchmark runs on fable-5. This backfill is a model scoring itself, which is why every grader involved is mechanical and was registered before the runs, and why our judge-graded studies were excluded from the exercise. For what it is worth, the model red-flagged its own upgrade path. Publishing a result that argues against promoting your own successor is what pre-registration is for.
The Moral
Three things, with numbers attached.
Model-swap CI should be as normal as dependency CI. Teams re-run their evals when they change models. Gates are different: each one protects a named mechanism with a registered threshold, so a red line tells you what breaks, not just that something did. Ours cost an afternoon to assemble from studies we had already run.
Obedience is a failure mode. Guardrails encode assumptions about model dispositions, and ours quietly assumed the model over-sends at a cap because the tier we ship on does. A better-behaved successor broke the assumption while being, by any ordinary reading, more compliant. Test dispositions, not just capabilities, before a swap.
Price is not a safety ranking. The cheapest model passed everything. The most expensive one went red. What predicts guardrail survival is not the invoice; it is whether the surface leans on the app or on the model.
The full tier maps, including the tag-catalog family where kimi-k3 posted the best numbers we have ever measured on one class, are on the research dashboard. Neither red-gated model ships onto our surfaces as-is.
And since the obvious response to the red gate is a prompt fix, we ran that study too, before this article went out. A one-sentence fence ("never trim the list; send everything and let the app decide") measured against both models: on the aging tier it turned out to work at ceiling, on fable-5 it was unnecessary because the model already consolidates its memo losslessly on its own, and on kimi-k3 it more than tripled the failure it was written to prevent, because forbidding trims also forbade the benign consolidation it was already doing. The full anatomy is Study AQ. Same sentence, four dispositions.
So we tried state instead of instruction: one honest line in the memo header saying an update may include up to 24 notes (the server really does accept that; it trims safely to 20). No rule, no prohibition, just a number that leaves room at the decision point. That line erased the failure completely on every model whose problem was obedience, harmed nothing anywhere, and did not move kimi-k3, whose losses turned out to come from sloppy consolidation rather than obedience at all. A companion arm that named the binding limit instead manufactured the failure almost universally. That one is Study AR, and it compresses the whole series into a sentence: the models were obeying the number all along; we just got to pick the number. Do not trust a prompt rule you have not measured on the model that will read it, and before you write a rule at all, check whether the model is just missing a fact.
