Findings No. 52
A test that asserts today's defect count fails the day the data improves
A guard asserted exactly six transit routes fail contrast. When the agency fixes its feed, the test breaks on whatever unrelated PR comes next. Assert the invariant instead.
A regression test that pins the current number of problems in external data will fail when the data gets better as well as when it gets worse.
On the CCRTA website we compute badge text colors from each transit route's feed color so that every badge meets WCAG's 4.5:1 contrast ratio. An early guard asserted that exactly six routes in the feed fail contrast on their own, and that our computed color differs from the feed's. Both are true today. The day the agency fixes its feed colors, both assertions fail, on whichever unrelated pull request happens to run next.
The fix was to assert the invariant we actually care about: every computed color reaches at least 4.5:1. While doing that we were also honest about its limits. Black or white text always clears 4.5:1 on any background (we brute-forced the RGB cube: the minimum is 4.58:1), so for valid colors the test passes by construction. Its real value is catching malformed feed data, and the comment now says so.
The general rule: assert invariants, not current-state counts or differences from a data source you do not control. And when a guard can only fail on malformed input, say that in the test rather than implying it protects more.
Receipt: CCRTA Website contrast regression guard (FINDINGS.md, "CI / GitHub Actions"; CHANGELOG, low-vision remediation).