Study AJ · Confirmations & machinery
The error message didn't matter
The correction loop in isolation
Study Overview
A Commitment, Never Tested
Returning barkup's structured validation issues verbatim has been a design commitment in every arm of this series, the closing instruction of playbook guideline 01, and standing guidance in the production codebase, and it had never once been an experimental variable. Study AJ isolated it.
Seeding the Loop
Because correction loops rarely trigger naturally (first-pass validity runs 84 to 99 percent), the loop was seeded: the known-correct patch for each task was corrupted by a registered operator (dangling ids, missing fields, malformed op kinds, bad placement anchors, unknown attributes), injected as the assistant's own prior turn, and confirmed to fail the shipped applier. Then exactly one feedback message went back, the only variable in the study: the full structured issues, bare issue codes, or nothing but "the anchored patch was invalid."
The Result Is Parity
The result is parity. Opus recovered all 45 seeded failures in every arm, including from the bare sentence. Gemini scored an identical 42 of 45 in all three arms, and the three misses were the same cells every time, all in the one genuinely hard class (bad placement anchors); feedback quality changed nothing. Sonnet showed the only gradient, 45 to 44 to 42, not significant. Told a patch failed, models simply re-derive the correct edit from the task and the tree.
One Nuance
One nuance runs in bare feedback's favor: its extra failures stayed visibly invalid rather than becoming plausible wrong fixes, so thin feedback does not even push toward silent errors.
The Commitment Survives
The commitment survives with its rationale corrected: return the issues verbatim because it costs nothing and makes correction logs humane for the people debugging them, not because the loop depends on it. Disclosed caveats: seeded single-op corruptions are a constructed proxy for organic errors, and recovery was single-shot where the shipped loop allows three rounds, which would only narrow the gap further.