Skip to main content

Running the suite

The whole matrix takes many hours on an idle machine. It is never run at once. It is collected a budget at a time, and every run measures only what is due.

cd packages/bench

npm run bench -- status # the matrix: fresh, stale, missing, declined; time still due
npm run bench -- next --budget 20m # measure the due cells that fit in 20 minutes, then publish
npm run bench -- next --budget 2h --wait # wait for an idle machine instead of stopping
npm run bench -- next --dry-run # print the batches it would run
npm run bench -- report --scenario patchColumn

next picks cells in order (missing and stale first, the default point before wide and small, scenarios in report order), packs as many as the budget allows into batches, and estimates each from the batch that last measured it (or from the scenario's probes). It always runs at least one batch. An interrupted run loses only the batch in flight: every finished batch is already stored.

Narrowing​

Every command takes the same filters:

npm run bench -- next --point default --scenario patchColumn,patchNote
npm run bench -- run --adapter oimdb --point small # re-measure, due or not
npm run bench -- next --retry # also re-measure failed and flagged cells
npm run bench -- verify --adapter mobx # correctness only, nothing timed or stored
npm run bench -- plan --scenario resortWindow # forget a sampling plan; the next batch re-calibrates
npm run bench -- memory # heap held per participant, stale or missing ones
npm run bench -- list # points, scenarios, participants

How much to measure​

Three knobs decide how much a run costs, and they combine:

knobwhat it cutswhat the cell says afterwards
--budget 20mhow many batches run; the rest stay due for the next runnothing — an unmeasured cell is simply still due
--point, --scenario, --adapterwhich cells are considered at allnothing
--processes 1processes per cell below the point's own (default 3, wide and small 2)flagged shallow: shown with its number, never counted as finished, and next --retry measures it again at full depth
npm run bench -- next --point small --processes 1 --budget 15m # a quick first look at everything
npm run bench -- next --retry --budget 2h # later: deepen the shallow cells

Every measured cell records how many timed runs stand behind its number (processes × samples per process × runs per sample) and the batch's wall time. The report prints it as 3p×12s×40r=1440, and the tables on this site carry it in the timed runs column (hover for the breakdown). A batch logs the same count before it starts timing.

Leaving it to run​

The measurement window is exclusive: no test suite, no type-check, no build, no indexing editor, no other benchmark. --wait checks the load gate before every batch and sleeps until the machine is quiet, so an overnight run is:

npm run bench:night --workspace @oimdb/bench # caffeinate -i … next --budget 6h --wait

To have it start by itself every night at 01:00 on macOS, save this as ~/Library/LaunchAgents/org.oimdb.bench.plist (adjust the two paths) and load it with launchctl load ~/Library/LaunchAgents/org.oimdb.bench.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>org.oimdb.bench</string>
<key>WorkingDirectory</key><string>/path/to/oimdb/packages/bench</string>
<key>ProgramArguments</key>
<array><string>/bin/zsh</string><string>-lc</string><string>npm run bench:night</string></array>
<key>StartCalendarInterval</key><dict><key>Hour</key><integer>1</integer><key>Minute</key><integer>0</integer></dict>
<key>StandardOutPath</key><string>/path/to/oimdb/packages/bench/results/night.log</string>
<key>StandardErrorPath</key><string>/path/to/oimdb/packages/bench/results/night.log</string>
</dict>
</plist>

launchctl unload the same file stops it. Every night picks up where the last one stopped, because only due cells are measured.

Where the numbers go​

pathwhat
packages/bench/results/cells/<point>--<scenario>.jsonthe latest record of every cell: fingerprint, plan, batch, load, outcome
packages/bench/results/plans/the stored sampling plans
packages/bench/results/memory/<point>.jsonthe heap each participant held, with the fingerprint it was weighed at
packages/bench/results/batches.jsonlone line per batch ever run, for timing estimates and history
website/src/data/bench/results.jsonwhat this site renders — written by next and by npm run bench -- publish
website/src/data/bench/audit.jsonthe audit cards — written by npm run audit -- publish

All of it is committed: a run writes the files, and the commit after it carries the cells and the page together. Nothing commits by itself. A checkout on another machine sees every cell as stale with reason environment until it is measured there; only results/night.log is ignored.

Keeping the audit current​

npm run audit -- status # current / stale / unstamped / missing, per participant and scenario
npm run audit -- stamp mobx # after re-reading audit/participants/mobx.md against the adapter
npm run audit -- publish

Changing an adapter makes its card stale; changing a scenario's script makes that scenario's card stale. The card format is in packages/bench/audit/README.md.