Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

Findings No. 59

To stop a recurring LLM output defect, name in the prompt the exact property your parser checks

A prompt already said 'no commentary', yet the top Sentry error was a translator prefixing correct output with a narrated line. Forbidding the line break itself fixed it.

CCRTA Website LLM Operations verified LLM translation, single-string prompt path

When a model keeps making one specific mistake despite a general instruction, restating the instruction more firmly rarely helps. Naming the exact property you check does.

The CCRTA website's translator has a single-string path, used as a last resort for a string that has already failed once. parseSingleResponse rejects a multi-line answer to a single-line source. That rejection was the top translator error in Sentry, with 23 events, mostly Chinese. The model was answering with a narrated line such as "This is already in Chinese, returning unchanged:" followed by the correct text. The prompt already said "no commentary", but the model did not treat that as commentary, and the prompt never forbade a line break, which was the thing actually being checked.

The fix changed the prompt to require exactly one line with no preamble, and to say explicitly what to do with text already in the target language: return it unchanged and nothing else. We did not bump the translation cache version, because existing translations were correct and re-translating all 162,000 cached strings would have cost exactly what the month's work was trying to save.

The general rule: if a parser rejects a specific structural defect, state that structure in the prompt. Tell the model the rule your code enforces, not a general category of bad output.

Receipt: CCRTA Website commit 7c4bef1 (parseSingleResponse; FINDINGS.md, "Translation pipeline").

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.