Accuracy and reliability

Raw readings against hourly and daily aggregates

Every night at 11:04 the local day two days back is checked: settled, and its raw readings still in the hot store. Every place with readings or hourly aggregates around that day is ranked by the SHA-256 of "<date>:<place id>", and the lowest 8 per category, 40 in all, are checked.

For each, a separate implementation that shares no code with the production rollup or the history API recomputes every hour and the whole day from the raw readings. A reading holds until the next reading, until its source's freshness limit runs out, or until the end of the hour, whichever comes first. A posted range counts at its midpoint, and an empty queue posts no value. avg is the time-weighted mean while open. median and p90 are the smallest values held for at least 50% and 90% of the open time; for a day, they weigh the hour averages by open time. n counts the posted updates.

Each hour is compared with the stored hourly aggregate on avg, min, max, n, open seconds, upper-bound seconds and the local date and hour. Minutes must agree to within 0.000001, and counts exactly. Each hour and day is also compared with GET /v1/places/{id}/history on avg, median, p90, min, max, n and open share. The API rounds minutes to 0.1, so a difference over 0.05 is a mismatch. A bucket on one side only is a mismatch too. Max difference is the largest absolute difference, after rounding the way the API does.

The archive is checked at 11:06. The 10 newest archived files and 10 older ones, ranked by the SHA-256 of "<date>:<file>", are read back by a separate reader. Each must match its manifest and the archive index: row count, size, both SHA-256 checksums, gzip members that tile the file exactly, and the line count of every member.

Reliability

Every minute, the public URLs are probed from Cloudflare's network, as an agent reaches them. REST API: /health and a /v1/places query. MCP server: tools/list. Website: the homepage and /_health. Accounts: the account app.

Every hour, these numbers are computed for the trailing 24 hours, 7 days and 30 days. Checks: the probes in the window. Error rate: the share of probes that got no HTTP answer (a timeout or network error) or a status outside 2xx. A 429 counts as a success: the keyless daily limit answering shows the route works. p50, p95 and p99: nearest-rank percentiles of every probe's response time in milliseconds, meaning the smallest time that at least that share of probes took no longer than. A timeout counts at its 5-second limit.

The probes run inside Cloudflare, so an outage of Cloudflare itself can't be recorded here.

Forecast accuracy

At 5 past every hour, the forecasts for one hour ahead (lead 60) and three hours ahead (lead 180) are logged as they are made. Each is scored once its hour is over. The actual is the hour's time-weighted posted mean, and an hour with under 30 open minutes isn't scored. The served model and the early model (baseline_hod_v0, logged but never served) are shown separately.

MAE is the mean absolute difference between actual and forecast, in minutes. Within ±5 min is the share of forecasts off by 5 minutes or less. Two baselines are scored on the same rows. Persistence assumes the wait stays what it was when the forecast was made; rows with no live wait then are left out of that comparison. Usual for the hour is the same model with no live adjustment. Skill is 1 − MAE of the model ÷ MAE of the baseline: above 0 beats the baseline, below 0 loses to it.

Weeks are ISO weeks (Monday to Sunday) of the forecast hour. A week, category and lead shows numbers only once 100 forecasts are scored. Until then it shows how many are scored, never an estimate. The JSON carries the sums behind every number, so each one can be recomputed.

Data honesty

Every night at 11:08, live production answers are fetched through the public API: over REST with a monitoring key when one is set, otherwise over keyless MCP with its own session. They are checked against six rules. A place with no reading returns no number, with a reason. A place whose last reading is past its freshness limit is not usable. ER lists say coverage is partial. A point outside the US gets no places and a US-only warning. A list mixing sources that measure differently says to compare minutes only within one source. A category Waity doesn't track gets no places and says so.

The places asked about are picked from the data that night, by the same SHA-256 ranking. A check with nothing to test, such as a night with no stale feed, or an answer cut off by a rate limit, is marked skipped, never passed.

Kept for good

Every run, check and snapshot is stored for good and served as JSON at /status/accuracy.json. This page reads from the site's own database, so it stays up when the API is down.