Findings No. 16
Temperature 0 is not determinism
A provider-side flake at temperature 0 during piloting: same input, same settings, different outcome. Treat temperature 0 as variance reduction and budget a noise floor into every conclusion.
Temperature 0 reduces variance; it does not deliver reproducibility. Barkup Bench observed a provider-side flake at temperature 0 during piloting: the same input, same model, same settings, and a different outcome on rerun. Providers do not promise bit-level determinism at temperature 0, and batching, hardware, and serving-stack details all leak into outputs.
The bench's response was to budget for it rather than deny it: effect sizes are sized against an assumed noise floor (about one task per cell), so a result has to clear the flake budget before it counts. Pre-registered predictions state margins, not exact values.
The general rule: treat temperature 0 as a variance-reduction knob and design like a scientist with a noisy instrument. If a conclusion would not survive one flaked task, it is not a conclusion yet. And if your production pipeline assumes identical inputs yield identical outputs, it is already wrong on some fraction of days.
Receipt: Barkup Bench REPORT.md, "Limitations"; the Barkup Bench dashboard.