Findings No. 13
The model's honest answers will find your grader's defects
A spaced '8 am' failed the literal '8am' matcher; a Unicode non-breaking hyphen failed another; correct sentinel-less refusals scored as misses. Every miss deserves classification before it becomes a headline number.
When a capable model scores worse than expected, some of the misses are yours. Auditing AEO Bench's graded "misses" decomposed them into three kinds: model failures, grader defects, and honest answers the grader was too rigid to accept. A model that answered "8 am" with a space failed the literal 8am matcher. A Unicode non-breaking hyphen in an otherwise correct answer failed another literal matcher. And one model wrote semantically correct refusals ("the site does not state this") without the registered sentinel phrase, which is formally non-compliant and arguably better behavior than the sentinel.
The fix respected the pre-registration: the grader corrections were registered as a follow-up (Study 1′), and the frozen response records were re-scored at zero new spend, with both readings published.
The general rule: grade the graders. Honest model output is an adversarial test suite for your matchers, and every "miss" deserves classification before it becomes a headline number.
Receipt: AEO Bench REPORT.md, the H3 fabrication-metric audit and Study 1′; docs/BRIEF-STUDY1-PRIME.md; the AEO Bench dashboard.