Replaying a Spatial Incident Against a New Detector

A corrective action that adds a detector is a hypothesis: had this rule existed, the incident would have been caught in minutes instead of hours. Most teams ship the hypothesis and never test it, which is how a review can close with a new alert in place that would still not have fired — the threshold too loose, the window too long, the label set missing the dimension that isolated the fault. Replaying the incident against the new rule converts the hypothesis into a measured detection interval you can put in the review document. This guide covers how to build the replay from data you already have, how to read the result, and what to do when the replay says the new detector is no better than the old one. It belongs to post-incident review for geospatial data in the spatial incident response and tooling program.

Replay harness reconstructing incident conditions and evaluating a candidate rule against them Two inputs feed a replay harness. The first is the incident's retained telemetry, replayed at its original timestamps. The second is the affected batch itself, re-run through the ingestion checks in a sandbox so that detectors computed from data rather than from metrics also produce values. The harness evaluates the candidate rule over the reconstructed window and outputs three results: whether the rule fires at all, the timestamp at which it first crosses its threshold, and the resulting detection interval compared against the original. Replay = original telemetry + the original batch, evaluated by the candidate rule Retained telemetry counters and gauges at their original timestamps The affected batch re-run through ingestion checks in a sandbox schema Replay harness candidate rule evaluated over the reconstructed window Does it fire at all? a rule that never crosses is not a fix When does it first cross? absolute timestamp, not relative New detection interval compared against the original

Problem framing: what a replay must reproduce

A spatial detector reads one of two kinds of input, and a replay has to reproduce whichever kind the candidate rule uses.

Metric-derived detectors read counters and gauges — features ingested, topology errors, freshness age, publish attempts. Replaying these means re-evaluating the rule expression over the retained series for the incident window. Provided your retention covers the incident, this is straightforward and needs no data at all.

Data-derived detectors compute a value from the features themselves — a distinct-projection count, a coverage extent, an attribute distribution. These have no historical series to query, because the value did not exist as a metric during the incident. Replaying them means re-running the affected batch through the check in a sandbox and observing what value it would have produced.

Most useful corrective actions land in the second category, because the whole point is usually to compute something nobody was computing before. That makes batch retention the practical prerequisite for meaningful replay: if the affected batch is gone, the strongest test available is a synthetic reconstruction, which is weaker evidence but far better than nothing.

The replay must also reproduce the timing, not just the values. A rule with for: 15m behaves very differently from one with for: 2m against a defect that appeared in a single batch and then paused until the next nightly run. Evaluating the expression at a single point in time tells you nothing about whether the sustained-condition clause would ever have been satisfied.

Implementation: evaluating a candidate rule over the incident window

For metric-derived detectors, evaluate the candidate expression as a range query bounded by the incident window and look for the first timestamp at which it crosses.

# Evaluate a candidate rule expression across the incident window and report the
# first crossing. Step matches the rule's evaluation interval so `for:` clauses
# can be assessed against the same sample spacing production would have used.
INCIDENT_START='2026-04-11T22:00:00Z'   # first bad batch, from the exposure bisect
DETECTED_AT='2026-04-12T04:40:00Z'      # when the ORIGINAL detector fired

curl -sG http://prometheus:9090/api/v1/query_range \
  --data-urlencode 'query=
     count by (layer) (
       count by (layer, srid) (gis_spatial_feature_srid_present{layer="parcels_authoritative"})
     ) > 1' \
  --data-urlencode "start=$INCIDENT_START" \
  --data-urlencode "end=$DETECTED_AT" \
  --data-urlencode 'step=60s' \
| jq -r '.data.result[]?.values[0][0] // "never crossed"'

The output is one number: the epoch timestamp of the first crossing, or a statement that the candidate never fired. Subtract the incident start and you have the new detection interval; subtract it from the original detection interval and you have the improvement figure that belongs in the review.

For data-derived detectors, the equivalent is to load the retained batch into a sandbox and run the check against it, recording the value the detector would have seen.

-- Replay a data-derived detector against the retained batch.
-- The sandbox mirrors the served layer's schema so the check is the real one.
CREATE SCHEMA IF NOT EXISTS replay;
CREATE TABLE replay.parcels (LIKE prod.parcels INCLUDING ALL);

COPY replay.parcels FROM PROGRAM
  'zcat /var/archive/parcels/batch_2026-04-11T22.csv.gz' WITH (FORMAT csv, HEADER true);

-- The candidate check: does the batch contain more than one projection?
SELECT ST_SRID(geom)  AS srid,
       COUNT(*)       AS features,
       MIN(ingested_at) AS first_seen
FROM replay.parcels
GROUP BY 1
ORDER BY 2 DESC;
-- A candidate rule of "distinct SRIDs > 1 halts the load" fires on this batch
-- at first_seen — record that timestamp as the replayed detection point.

Run both kinds against the same incident window so the resulting intervals are directly comparable, and record the candidate rule’s full definition — expression, for: window, threshold and labels — next to the result. A replay whose rule text is not recorded cannot be reproduced when someone later loosens the threshold.

Four replay outcomes and the action each one implies A two by two grid classifies replay outcomes. Fires much earlier means ship the rule and record the improved interval. Fires but only marginally earlier means the threshold or window is still too loose and needs tightening before shipping. Fires at the same time as the original means the new rule adds no detection value and the real gap is elsewhere, usually in routing or in the data itself. Never fires means the hypothesis was wrong and the detector must be redesigned around a different signal. Read the replay result before shipping the rule Fires much earlier ship it · record both intervals in the review e.g. 6 h 40 m → 4 m on the same batch add a regression test that pins the threshold Fires marginally earlier tighten before shipping usually the `for:` window or a rate denominator that averages the spike away Fires at the same time the gap is not detection look at routing, acknowledgement, or the fact that the defect only became visible late Never fires the hypothesis was wrong the chosen signal does not move for this defect redesign around one that does

Verification: make the replay repeatable

A replay that lives in someone’s shell history is worth little. Two small investments make it durable.

Check the candidate rule and its replay assertion into the same repository, as a test. The assertion is simple: given this fixture, the rule must cross its threshold within N minutes of the fixture’s start. Anyone who later relaxes the threshold then breaks a test rather than silently undoing the corrective action.

Retain the fixture. For metric-derived rules, that is an exported slice of the relevant series over the incident window; for data-derived rules, it is the batch file or a redacted subset of it. Fixtures are small, and the alternative is that the next threshold change is made without evidence.

Finally, run the replay against a clean window as well as the incident window. A rule that fires on the incident and also fires on three ordinary Tuesdays is not a detector, it is a future muted alert. This false-positive check is the half of the verification that teams routinely skip, and it is the reason many post-incident rules get disabled within a month.

Replayed detection interval against the original, for five shipped detectors Five detectors shipped as corrective actions are compared on the detection interval they achieve when replayed against the incident that produced them. Four reduce the interval substantially, from hours to minutes. The fifth reduces it only marginally and is marked as needing a tighter threshold before it can be considered a fix. The note states that this is exactly the case the replay exists to catch, and that it would otherwise have shipped as a completed action. One of these five is not yet a fix — and only the replay says so projection contract check 6 h 40 m → 4 m per-source validity rate 9 h 10 m → 32 m coverage ratio baseline 14 h → 1 h 15 m reporting-region count 5 h → 34 m tile size distribution 22 h → 19 h · tighten it

Gotchas

Retention shorter than the detection interval. If your telemetry retains fifteen days and the incident’s silent phase ran for three weeks, the replay cannot see the beginning. Extend retention for the specific series that participate in correctness detection rather than globally; they are usually low-cardinality and cheap.

Replaying the resolved state. Running the candidate check against the repaired layer proves nothing — the defect has been removed. The replay must use the retained batch or a snapshot taken before the repair.

Ignoring the for: clause. Evaluating the raw expression tells you when the condition became true; the rule fires only after it has been true continuously for the for: duration. For a defect that appears in one nightly batch and then pauses, a fifteen-minute sustained clause may never be satisfied at all.

Treating a single replay as proof. One incident is one sample. Where the failure class has recurred, replay against every occurrence — a rule that catches the most recent instance and misses the two before it is not yet the right rule.

FAQ

What if the affected batch was not retained?

Reconstruct a synthetic batch carrying the same defect: take a current clean batch and apply the transformation that caused the incident — reproject a subset, corrupt a fraction of geometries, shift the timestamps. The replayed interval is then approximate, but the fire/no-fire answer remains sound, and that is the most important half of the result.

Should the replay run against production alerting configuration?

No. Run it against an isolated evaluation so a candidate rule cannot page anyone or pollute alert history. The point is to measure the rule, not to exercise the notification path — that is a separate test, described in the routing guidance in alert routing and on-call design for spatial pipelines.

How do I replay a detector that needs a baseline the incident predates?

Build the baseline from data preceding the incident window and freeze it. Using a baseline computed after the fact leaks the incident into the baseline and makes the defect look normal — the same contamination problem that affects distribution-drift baselines generally.

Does replay apply to threshold changes as well as new rules?

It applies especially well. A threshold change is the cheapest corrective action available and the easiest to get wrong, and a replay answers directly whether the new value would have caught the incident and whether it would have fired spuriously on ordinary days.

Where do the replayed intervals belong in the review?

In the timeline section, alongside the original four intervals, labelled as replayed rather than observed. Stating “detection was 6 h 40 m; the shipped detector replays at 4 m on the same batch” is the single most convincing line a review can contain, and it is checkable.