Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 12

An LLM call can return an empty completion deterministically

One prompt returned 5,976 tokens in and zero out, twice in a row. An empty completion is a real output class distinct from errors; one retry is cheap insurance.

Stem Bench LLM Operations verified claude-fable-5-1 via Vercel AI Gateway

One prompt in Stem Bench's corpus returned 5,976 tokens of input and zero tokens of output, twice in a row. Not an error, not a timeout, not a refusal: a clean, reproducible, empty completion from a frontier model. Under the study's registered no-retry rule it scored zero on all 28 bars of that song, which is exactly what pre-registration is for, and exactly what production code cannot afford.

The production response shipped the same day: the chart-draft pipeline gained one retry on an empty completion (Stem Shovel v0.142.0), and the bench re-measured the shipped configuration under a registered amendment.

The lesson is that an empty completion is a real output class, distinct from errors, that deterministic inputs can deterministically produce. If your pipeline parses the completion, it needs a branch for the completion that does not exist, and a single retry is cheap insurance against the rare prompt that lands in this hole.

Receipt: Stem Bench Study 1, REPORT.md results footnote and "Production response and Amendment 2"; Stem Shovel v0.142.0.

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.