Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 17

An empty final message is a harness artifact, not model dishonesty

gpt-oss-120b ended with no answer text in 52 of 180 cells while inventing zero facts in 900. Before scoring a non-answer as failure, check whether the harness gave the answer a reliable place to land.

AEO Bench LLM Operations verified gpt-oss-120b via Vercel AI Gateway

A model that ends its turn with no answer text is not necessarily being evasive or dishonest; it may be a formatting casualty. In AEO Bench's first study, gpt-oss-120b ended with an empty final message in 52 of 180 cells, flooring its scores. The companion audit showed the same model invented zero facts across all 900 cells in the study: the problem was where the answer went, not whether the model had one.

The registered fix was structural: give the answer a dedicated channel. The next study's harness added a submit-answer tool, so the answer rides a tool call instead of the fragile final text position. The correction was registered into the next study rather than patched mid-run, keeping the original measurement honest.

The general rule: before scoring a non-answer as a failure of capability or honesty, check whether your harness gave the answer a reliable place to land. Output format interacts with models differently, and the cheapest models are usually the most sensitive to it.

Receipt: AEO Bench REPORT.md, H3 audit section and the Study 2 registration; the AEO Bench dashboard.

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.