Parsing a Transit System into One Megabyte of JSON
Our transit website for CCRTA never parses GTFS at runtime. Every map, search box, route page, and departure board on the site reads from about a megabyte of JSON that is committed to the repository and regenerated by a scheduled job. This post is about that pipeline: a 481-line TypeScript script, a daily cron, and a handful of decisions that turned out to matter more than we expected.
The Shape of the Problem
GTFS, the open standard for transit data, is a zip of CSV files: routes, stops, trips, timetables, geometry. It is designed for interchange, not for serving. The raw feed for even a mid-size system is tens of megabytes, the useful facts are spread across joins (which routes serve this stop? which stops does this route pass, in order?), and the timetable file alone runs to hundreds of thousands of rows.
You can parse all that at runtime and cache it, or stand up a database and query it. We did neither. The pipeline runs in CI, does every join once, and emits four small JSON files: enriched stops, enriched routes, route geometry, and per-direction stop orderings. The production runtime reads committed files. There is nothing to warm, nothing to migrate, and a git diff shows exactly what changed in the transit system on any given day.
No GTFS Library, on Purpose
The script uses no GTFS library. It has a hand-written CSV parser (RFC 4180 quoting and CRLF handling included, because transit feeds exercise both) and a hand-written implementation of Google's encoded polyline format, matched by a small decoder on the client. That is less code than it sounds like, and it bought us three things: zero dependencies to audit in a security-sensitive public-sector project, output shaped exactly for our pages rather than a generic object model, and no library-version ceiling on a feed that occasionally does nonstandard things.
The precompute earns its keep in the joins. One example: a route page needs its stops in travel order, per direction. Deriving that at runtime meant scanning the timetable table by route, and an early version did exactly that: an unindexed query that took 35 seconds in production. The pipeline now walks the timetable once, keeps the longest ordered stop sequence per headsign, and writes it to a JSON file the page reads instantly. The comment memorializing the 35-second scan is still in the code as a warning to future maintainers.
The Longest-Shape Heuristic
Route geometry in GTFS comes as shapes: one polyline per trip variant. A single route can carry dozens of variants (short turns, school trippers, detours) and drawing them all produces a furry scribble. Drawing only one requires choosing which.
Our answer is deliberately simple: for each route and direction, keep the longest shape. The longest variant almost always traces the full corridor, and the minor variants are subsets of it. The whole network's geometry compresses to roughly 64KB of encoded polylines, small enough to ship to every visitor of the system map. Simple beat clever here; we tried nothing fancier because nothing fancier was needed.
Remembering Stops That GTFS Forgets
GTFS has no concept of a closed stop. When a stop is retired, it simply vanishes from the next feed, as if it had never existed. Normally that is fine. It is not fine when you have printed QR-code stickers bolted to physical poles all over the city, each pointing at that stop's departure page, as CCRTA does.
So the pipeline diffs. Each run compares the fresh feed against the previously committed stop list; stops that disappeared are appended to a retired-stops archive with a retirement date. The stop page checks the archive, and a scanned sticker at a decommissioned stop lands on a "this stop is no longer in service" page instead of a 404. The physical world updates on a slower cadence than a data feed, and the data pipeline is where we chose to absorb that difference.
CI Is the ETL
The automation is a GitHub Actions workflow on a daily schedule (07:23 UTC, off the hour because everyone's cron jobs fire at :00 and GitHub's queue congests). It downloads the feed, regenerates the JSON, and commits only if something actually changed; the commit triggers a deploy, so the site updates itself when the transit system does. A second script loads the full timetable into a Turso (SQLite) database for the departure boards, with a content-hash check that skips the write-heavy reload when the schedule is unchanged.
Two correctness details earned their scars. GTFS times routinely exceed 24:00:00 (a 1:15 a.m. trip on Saturday's service day is 25:15:00), so all time math happens in service-day space before formatting. And formatting is done with Intl.DateTimeFormat pinned to America/Chicago, because the serverless runtime's own clock is UTC and daylight saving is nobody's friend.
The Takeaway
The pattern generalizes well beyond transit: when the source data changes daily and the queries are known in advance, move the entire transform to CI and commit the output. The runtime gets simpler and faster, the diffs become documentation, and the database shrinks to the one table that genuinely needs indexed queries. Our ETL's load step is a git commit, and we mean that as a compliment.
