Skip to main content
Lightning Jar - Web Studio Lightning Jar Wordmark

When the Spec Is the Binary, Make the Binary Your Test Oracle

When the Spec Is the Binary, Make the Binary Your Test Oracle

Stem Shovel's converter writes Reaper project files. Reaper's .rpp is plain text, which makes it look easy: every project anyone has ever pushed to GitHub opens in an editor, and the blocks and keywords are guessable from their names. The writer came together quickly. The hard question came after: would Reaper read our file the way we meant it? Nobody but Cockos knows what Reaper does with a line, and we found no specification from Cockos. The community sources (the SWS extension, DawVert, the forum) tell you the keywords, not the semantics. The only authority on what an .rpp means is Reaper.

So we made Reaper the test. This is the first of two posts about the converter; the second, A File Converter That Never Uploads Your File, is about the product and the formats. This one is about a testing pattern we think applies far outside music software: when the only spec is an application's behavior, automate the application and make it the oracle.

A Format With No Spec

A Reaper project is a tree of blocks. A track is <TRACK through >, with lines like NAME "Bass", VOLPAN 0.8 -0.3 -1 -1 1 and MUTESOLO 0 0 0; an item is <ITEM with a POSITION, a LENGTH, a SOFFS start offset and a <SOURCE WAVE naming the file. Envelopes are blocks of PT <time> <value> <shape> points. The README on our corpus fixture is candid about where that knowledge comes from: "documented by its users (the SWS and DawVert sources, the Reaper forum); nothing here is Cockos's own."

The text answers "what is written" and leaves "what it means" open. Which direction does a pan envelope run? Is an envelope point's time in project seconds, item seconds, or something else? Does the time signature packed into a tempo point count, or is a flag after it required? Does a tempo ramp interpolate over seconds or over beats? You can guess at each of those from other people's files, and you will be wrong often enough to ship a project that opens cleanly and plays wrong. We wanted every one of those answers to come from Reaper itself, repeatably, on a machine with no screen.

Driving a DAW With No Screen

The development VM has no display and no sound card. Reaper has a Linux build and will run a ReaScript from its command line, so the harness (scripts/interop/rpp.ts) starts a virtual X display with Xvfb :98, then launches Reaper against it:

const reaper = spawn(
	bin,
	["-cfgfile", `${cfg}/reaper.ini`, "-nosplash", "-new", resolve("scripts/interop/rpp.lua")],
	{ env, stdio: ["ignore", "pipe", "pipe"] },
);

-cfgfile points Reaper at a reaper.ini of the run's own, so nothing the harness does touches the machine's settings. The Lua script runs at startup. It opens the project with reaper.Main_openProject("noprompt:" .. rpp) (the noprompt: prefix skips the save-changes question), walks Reaper's API, writes what it found to a result.json, saves the project again under the output folder, and quits with action 40004.

Reaper still raises dialogs on a bare display: no audio device, "create directory?", the render dialog. There is no window manager, so xdotool windowactivate fails, but windowfocus works. The runner polls every 1.5 seconds, finds any visible window whose name is not Reaper's own, focuses it and presses Return:

for (const id of ids) {
	const name = execFileSync("xdotool", ["getwindowname", id], quiet).trim();
	if (/REAPER v|^reaper$|^$/i.test(name) || /\.rpp/i.test(name)) continue;
	execFileSync("xdotool", ["windowfocus", "--sync", id], { env, stdio: "ignore" });
	execFileSync("xdotool", ["key", "--window", id, "Return"], { env, stdio: "ignore" });
}

That is the whole machinery: a virtual display, a config folder, a startup script and a loop that says yes to dialogs. It was built once, with Reaper 7.82, for the first check on October 21 (Reaper importing a bundle of timestamped clips and a MIDI file), and it has been reused for every probe since.

Ask the API, Then Judge

The harness does not test a hand-written file. It builds a fixture arrangement with everything the writer has to carry (a folder bus, a pre-fader send, volume, pan, mute and send envelopes, a master volume lane, a track color, a tempo ramp, two warped clips, a clip with a second take, a clip with a gain envelope) and pushes it through the same exportSong path the Export panel uses, with reaper: true. The zip that comes out is unpacked to a scratch folder and that is what Reaper opens.

The Lua side reads everything back through the API: GetMediaTrackInfo_Value(tr, "D_VOL") and "D_PAN" for the faders, I_FOLDERDEPTH for the folder structure, GetTrackSendInfo_Value for the receives, GetTrackEnvelopeByName for the lanes, GetMediaItemInfo_Value for item positions and lengths, GetTakeStretchMarker for the warps, MIDI_GetNote with MIDI_GetProjTimeFromPPQPos so MIDI notes come back in project seconds, GetTempoTimeSigMarker for the tempo map. The TypeScript side then judges the JSON against the arrangement with a plain check helper:

check(
	recv && recv.from === "Bass" && near(recv.volume, 0.5) && recv.mode === 3,
	`Drums receives Bass at 0.5, pre-fader (${recv ? `${recv.from} ${recv.volume.toFixed(3)} mode ${recv.mode}` : "none"})`,
);

Every line prints ok or NOT with what Reaper actually said in parentheses, so a failure reads as a diff, not a stack trace. After the API checks, the runner reads Reaper's re-saved .RPP back through our own reader (src/lib/utils/rppRead.ts) and judges that too, which is how the reader gets tested against a file Reaper wrote rather than one we wrote.

When the API Will Not Tell You, Render

Some questions the API cannot answer. The pan envelope was the first. We wrote a point of 0.6 and Reaper reported a point of 0.6, which proves nothing about which way the track pans when the envelope drives it. GetTrackUIVolPan looked like the tool for the job and turned out to return the knob position even in automation read mode.

So the script makes Reaper play. It puts the Bass track in read mode, solos it, sets the render bounds to two seconds through GetSetProjectInfo, and renders with Main_OnCommand(42230, 0), the action that renders with the most recent settings and closes the dialog:

reaper.SetMediaTrackInfo_Value(bass, "I_AUTOMODE", 1)
reaper.SetMediaTrackInfo_Value(bass, "I_SOLO", 1)
reaper.GetSetProjectInfo(0, "RENDER_STARTPOS", 2, true)
reaper.GetSetProjectInfo(0, "RENDER_ENDPOS", 4, true)
reaper.GetSetProjectInfo_String(0, "RENDER_PATTERN", "bass-pan", true)
reaper.Main_OnCommand(42230, 0)

The runner reads the WAV back, computes RMS per channel, and requires the right channel to be at least 1.5 times the left. The answer: Reaper stores the pan envelope the other way up from its knob. A <PANENV2 point of -0.6 pans the track to the right. Our writer negates the lane on the way out and the reader negates it back, and the render is the test that would catch either side drifting.

Three more facts came the same way. A mute envelope's 0 is muted and 1 is on (the render was silent where the envelope sat at 0), the opposite of our own lane, so the writer inverts it and writes every point square. The master's volume envelope lives at the project level, and a mix rendered across its drop to zero at six seconds came out with a silent second half, as it should; a hand-written <MASTERMUTEENV block, on the other hand, was simply dropped when Reaper saved the project, so the master has no mute envelope and ours does not travel. And a take's volume envelope needed the most careful probe of all.

The question there was what unit a point's position is in when the item plays at a different rate. A clip gain envelope on an item at play rate 0.8 could land in item seconds, in take time (seconds from the item's start times the rate), or be scaled twice by mistake. The API reports the number you wrote. So the script renders the item twice, once with the envelope on and once with it switched off through its state chunk (ACT 0), and the runner divides one by the other in 10 ms windows. The ratio is the envelope as Reaper plays it. The step we wrote at 1.2 showed up at 1.505 seconds into the item, where 1.5 was wanted (1.2 divided by 0.8), not 1.2 (item seconds, no rate) and not 1.875 (the rate applied twice). Positions are take time: a content second t of a clip at offset o goes out as (t - o) × rate.

What the Oracle Taught Us

A list of what we got wrong before Reaper corrected us, each now recorded in the project's FINDINGS.md with the probe that proved it, and each pinned by a unit test on the writer or the reader:

  • The time signature on a tempo point counts only with a flag after it. A point written as PT 8 100 1 262147 (the signature packed as (denominator << 16) | numerator) was kept in the file but reported back as -1/-1, and Reaper drew 4/4 bars on. With 0 1 after it, as Reaper's own files carry, it took 3/4.
  • A point that carries a time signature starts a new measure. With 3/4 from 8 seconds and a tempo point stamped 3/4 at 20 seconds, a third of a bar in, Reaper padded the partial bar and counted 34 quarter notes at 20 seconds instead of 32. The same point with no signature counted 32. The writer now stamps the signature only on the first point and where the meter changes.
  • Tempo ramps are linear in seconds. A ramp from 120 to 60 over 8 seconds gave 90 bpm at 4 seconds and twelve quarter notes at the end. A ramp linear over the beats, which is what a DAWproject or an Ableton Live set means by a ramp, would give 84.85 and 11.54. Dawn, Stem Shovel's multitrack recorder, now keeps a tempo ramp on one of two curves and lays the other curve down bar by bar when a writer needs it.
  • Reaper's tempo is quarter notes a minute in any time signature. At TEMPO 120 6 8, TimeMap2_timeToQN(0, 1.5) returns 3: a 6/8 bar is three quarter notes long. TimeMap2_GetDividedBpmAtTime returns 240 there, the figure Reaper's transport displays. A DAW-style tempo goes into an .rpp unscaled whatever the signature.

None of these is in any document we could find. Each is a fact about one program's behavior, and each would have shipped as a plausible wrong guess.

The General Lesson

We have written before about pinning a measurement to a published table so a plausible, wrong implementation cannot ship (Measuring Loudness in the Browser). This is the same discipline for the case where there is no table. When the only specification is what an application does, test against the application. Drive it headless, ask its API what it holds, and when the API reports your own numbers back to you, make it render and measure the output. The render is the one witness that cannot be fooled by a reader and a writer that share the same mistake: a round trip through our own rppRead.ts would have passed with the pan envelope backwards on both sides.

The cost is real but bounded. The run takes Reaper, Xvfb and xdotool, which CI does not have, so it is not a gate on every push. It runs on the VM before a release that touches the exchange formats, the way we run our other slow checks, and the unit tests carry what it taught in between. The same shape now covers Audacity (driven over its scripting pipe under Xvfb) and Sandyne (which inspects and renders a project from the command line with no display at all). Ardour's headless Lua turned out not to be able to import a file, so that one waits.

If you are writing a file for a program that owns its format, the question is not whether you can parse the examples. It is whether the program agrees with you, and the only way to know is to ask it.

Stem Shovel is at stemshovel.com; the converter is at stemshovel.com/convert.

headshot of Kevin Peckham
Kevin Peckham
Principal, Lightning Jar

Ask Eljay

Ask about the studio's work, research, packages, or writing. A few starters:

Answers come from this site's own content and link their sources. For anything that matters, email hello@lightningjar.com.