Findings No. 56
A fail-open, cache-first system hides its own outage from the people running it
An AI Gateway 402 stopped CCRTA's translation for 14 days while cached strings kept the pages looking translated. Monitor real backend calls, not how pages look.
A system designed to degrade gracefully does exactly that when it breaks, which means nothing looks broken. Riders are protected, and so is the outage.
The CCRTA website translates every page into 14 languages through a cache-first pipeline that calls an LLM through Vercel's AI Gateway. In July the Gateway started answering 402 (budget exceeded), and fresh translation stopped for 14 days. Pages kept looking translated because cached strings, with a 90-day lifetime, kept serving. Only new text, such as a freshly published service alert, fell back to English. With that cache lifetime, the outage could have stayed invisible for months.
The error did reach Sentry, but as Error: gateway responded 402, in a queue of other alerts. We fixed that wording separately, so failures now state their impact and remedy. The deeper lesson was about what to measure. A count of actual backend translation calls drops to zero when translation is down, whatever the pages look like; the spend counter we built later records every call per service day and gives us exactly that signal.
The general rule: anything that fails open needs a health signal that does not depend on the degraded output looking wrong. Count real calls to the backend instead of sampling rendered pages.
Receipt: CCRTA Website commit e403c22 and the July 2026 maps cost sprint write-up (docs/2026-07-maps-cost-sprint.md, sections 4 and 6; FINDINGS.md, "Fail-open systems / observability").