Post-Incident Review for Geospatial Data

Most incident reviews for data platforms ask the wrong first question. They ask “why did the pipeline break”, when the question that matters for spatial data is “how long were consumers given wrong answers, and which answers”. A geospatial incident has a property that a service outage does not: the damage persists in the data after the system recovers. A projection fault that ran for six hours leaves six hours of mis-located features sitting in a published layer, feeding joins, tiles and compliance extracts long after the alert resolved and the loader went green. Reviewing such an incident by uptime alone declares victory while the wrong answers are still in circulation.

This topic covers how to run a review that measures data-correctness exposure rather than service downtime, how to reconstruct a spatial incident timeline from the signals you already collect, and how to close the loop so a detector genuinely improves. It sits under spatial incident response and tooling, downstream of the spatial pipeline incident runbooks that produce the incidents and the alert routing and on-call design whose quality it measures.

Incident timeline showing the gap between service recovery and data recovery A timeline runs from the first bad batch through detection, acknowledgement, mitigation, service recovery and finally data recovery. Two bars are drawn above it. The service-impact bar starts at mitigation and ends at service recovery and is short. The data-exposure bar starts at the first bad batch, well before detection, and ends only at data recovery when the affected features have been repaired and downstream extracts reissued, and it is several times longer. Four intervals are marked beneath the axis: time to detect, time to acknowledge, time to mitigate and time to repair. Service recovery is not data recovery — the review must measure the longer bar Data exposure — wrong answers reachable by consumers starts at the first bad batch, ends when features are repaired and extracts reissued Service impact the only bar an uptime review sees first bad batch detected acknowledged mitigated service ok data repaired time to detect to ack to mitigate time to repair — usually the largest and the least measured A review that reports only the service bar will keep shipping the same detection gap.

The four intervals worth measuring

Every spatial incident decomposes into four intervals, and each one has a different owner and a different fix. Reporting them separately is what turns a narrative into a set of actionable numbers.

Time to detect runs from the first bad batch to the first true alert. This is the interval most spatial incidents lose the most time in, because the defect is silent by construction: an invalid geometry, a wrong datum, or a stale layer all pass row counts and health checks. Long detection intervals are always a detector problem, never a responder problem, and the corrective action belongs in the alert rules.

Time to acknowledge runs from the alert to a human picking it up. This measures the routing, not the engineer. A long acknowledgement on a correctly-severe alert means the page went somewhere wrong, arrived amid noise, or was inhibited by an over-broad rule.

Time to mitigate runs from acknowledgement to stopping the bleeding — usually halting ingestion for the affected layer or failing over to a fallback source. This measures runbook quality. If the responder spent twenty minutes deciding whether to halt, the runbook did not make the containment decision explicit.

Time to repair runs from mitigation to the data being correct again: bad features quarantined or fixed, derived layers rebuilt, extracts reissued, caches invalidated. This interval is routinely omitted from reviews and is routinely the largest. It measures how well the platform can undo a bad load, which is a design property rather than an operational one.

Interval What it measures Typical corrective action
Detect Detector coverage and sensitivity New or tightened rule; lower threshold on the specific source
Acknowledge Routing, severity, noise level Fix matchers, severity tier, or inhibition scope
Mitigate Runbook decisiveness Make the halt/continue decision explicit; pre-authorise the halt
Repair Reversibility of the pipeline Retain pre-load snapshots; make derived layers rebuildable on demand

Reconstructing the spatial timeline

The reconstruction problem is that the incident began before anyone was watching. You need to establish when the first bad feature landed, which is a data question, not a monitoring one — and the answer usually lives in the data itself.

The reliable technique is to bisect on an ingestion timestamp. Every landed feature should carry ingested_at and a batch identifier; with those, the first appearance of the defect is one query away.

-- When did the defect first land? Bisect on ingestion time, not on alert time.
SELECT date_trunc('hour', ingested_at)          AS hour,
       COUNT(*)                                  AS features,
       COUNT(*) FILTER (WHERE NOT ST_IsValid(geom))              AS invalid,
       COUNT(*) FILTER (WHERE ST_SRID(geom) <> 27700)            AS wrong_srid,
       MIN(batch_id)                             AS first_batch
FROM prod.parcels
WHERE ingested_at >= now() - interval '3 days'
GROUP BY 1
HAVING COUNT(*) FILTER (WHERE NOT ST_IsValid(geom) OR ST_SRID(geom) <> 27700) > 0
ORDER BY 1
LIMIT 1;

That first hour is the true start of the exposure interval, and it is frequently hours or days before the alert. The gap between it and the alert timestamp is the detection interval, and it is the single most useful number the review produces.

Two supporting reconstructions are worth doing routinely. The first is a downstream reach query: which derived layers, extracts and tile pyramids consumed the affected features between the first bad batch and the repair. Without lineage this is guesswork; with the lineage graph described in how to map geospatial data lineage for observability it is a traversal. The second is a near-miss check: which detectors were trending before the one that fired. Very often a reason-labelled counter was already climbing hours earlier, and lowering its threshold is a cheaper fix than inventing a new detector.

Three evidence sources combined to reconstruct a spatial incident Three evidence sources feed a reconstruction step. The data itself supplies the first bad batch via an ingestion-time bisect. The telemetry store supplies detector histories including the near-miss signals that were trending before the alert fired. The lineage graph supplies the set of downstream layers, extracts and tile pyramids that consumed the affected features. The reconstruction step combines them into a single timeline carrying the four intervals and a named blast radius, which then feeds the corrective actions. Reconstruction takes three sources, not one The data ingested_at bisect → first bad batch The telemetry detector history → near-miss signals The lineage graph traversal → downstream reach Reconstruction one timeline four intervals named blast radius Detector change tighten the signal that nearly caught it Runbook change make the containment call explicit Reversibility change snapshot retention, rebuildable layers

Exposure accounting: how much wrong data, for how long

The headline number of a geospatial review is not minutes of downtime. It is feature-hours of exposure: how many features were wrong, multiplied by how long they were reachable. It is a crude measure and it is enormously more useful than uptime, because it scales with the thing that actually harms consumers.

Computing it needs only the affected feature count and the exposure interval, both of which the reconstruction already produced. Reporting it consistently across incidents lets you compare a six-hour projection fault on eleven thousand parcels against a twenty-minute topology spike on four hundred, and allocate engineering effort accordingly — a comparison that raw incident counts cannot support.

Alongside it, record the consumer reach: the named downstream systems that read the affected features during the exposure window. This is the field that turns a technical review into an organisational one, because it identifies who needs to be told and what needs reissuing. A compliance extract generated during the window has to be regenerated whether or not anyone noticed, and that decision belongs in the review, not in someone’s memory.

Where exposure cannot be bounded — no ingestion timestamps, no lineage, no snapshot to diff against — record that fact as the primary finding. “We could not determine how many features were wrong or who read them” is a more important outcome than any narrative, and it generates the most valuable corrective action available: make the next incident measurable.

Running the review itself

Keep the meeting short and the document specific. A workable structure for spatial incidents:

  1. The timeline, with the four intervals as numbers, reconstructed from data rather than memory.
  2. The exposure, in feature-hours and named consumers.
  3. The detection gap, stating explicitly which signal could have fired earlier and why it did not.
  4. The containment decision, stating what was halted, when, and whether the runbook made that call obvious.
  5. The repair, including what was rebuilt and what remains unrepaired.
  6. Corrective actions, each with an owner, each of which must change a detector, a runbook, or a reversibility property.

That last constraint is the one that keeps reviews honest. A corrective action that is neither a detector change, a runbook change, nor a reversibility change is almost always “be more careful”, which changes nothing. If a review produces no such action, the correct conclusion is usually that the incident was already handled well — which is a fine outcome to record explicitly rather than inventing busywork.

Two cultural rules matter more in spatial work than elsewhere. First, treat the upstream data provider as part of the system, not as an external excuse; a vendor changing an export format is a foreseeable event and the corrective action is a contract check, not a complaint. Second, resist the pull toward a single root cause. Spatial incidents are typically a chain — a format change, an unvalidated assumption, a detector scoped too coarsely, a runbook that did not say to halt — and naming only the first link leaves the other three in place.

Reviewing incidents whose cause sits outside your platform

A large share of geospatial incidents originate with a data provider you do not control: a municipality re-publishes its parcel export with a different projection, a satellite vendor changes tile naming, a sensor operator’s firmware update starts emitting timestamps in local time. The temptation is to record the cause as external and close the review. That is a mistake, and it is the reason the same class of incident recurs on a different feed six months later.

The productive framing treats the provider boundary as a trust boundary your platform is responsible for defending — precisely the boundary described in defining spatial data trust boundaries. The review question is then not “why did the vendor change the export” but “why did the change reach the published layer instead of being rejected at the boundary”. That question has an answer you own, and the corrective action is a contract check rather than an email.

Three checks cover the overwhelming majority of provider-caused incidents, and a review should verify each was in place before it accepts an external cause as unavoidable. A projection contract asserts the declared and actual spatial reference identifier of every incoming batch against the registry value, rejecting the batch rather than reprojecting silently. A structural contract asserts the column set, types and nullability against a pinned fingerprint, which is the mechanism covered by schema and attribute drift detection. A volumetric contract asserts row count and extent within a tolerance of the previous accepted batch, catching truncated exports and partial transfers that pass every structural check.

Where a contract existed and the batch still landed, the review has found something valuable: the contract was scoped too loosely, or it warned instead of halting. Where no contract existed, the corrective action writes itself. Either way the finding is actionable inside your own repository, which is the property that distinguishes a useful review from a grievance.

It is worth recording the provider-facing action separately and without expectation. Telling a municipality that their export changed is courteous and occasionally effective, but a review that depends on an external party changing their behaviour has no enforceable corrective action at all. Defend the boundary first; notify second.

Tracking corrective actions and proving they worked

Corrective actions decay. They are written under the vivid impression of an incident, assigned to whoever is in the room, and then compete with roadmap work for six months. The practices that keep them alive are unglamorous and worth being strict about.

Give every action a single named owner and a due date measured in days, not quarters. Actions that cannot be completed in weeks are usually projects wearing an action’s clothing; split them, and let the review own only the part that closes the specific gap this incident exposed. An action such as “build a full data-contract framework” will not ship; “add an ST_SRID contract check to the parcels loader that halts on mismatch” will.

Then validate the action against the incident that produced it. The validation is a replay: reconstruct the conditions — the same batch, or a synthetic one with the same defect — and confirm that the changed detector fires, that it fires earlier than the original one did, and that it routes to the intended receiver. Record the new detection interval next to the original one in the review document. This converts an assertion (“we added a check”) into evidence (“detection went from 6 h 40 m to 4 m on a replay of the same batch”), and it catches the common case where a new rule is technically present but its threshold is still too loose to have caught the incident it was written for.

Finally, watch the aggregate. A handful of numbers reviewed monthly tells you whether the process is working: median time to detect across incidents, the fraction of incidents where exposure could be bounded at all, the fraction of corrective actions completed by their due date, and the count of repeat incidents in the same failure class. A rising repeat rate in one class is the clearest possible signal that reviews in that area are producing narratives rather than changes.

Those aggregate numbers also feed back into routing. A steadily falling acknowledgement interval usually means noise reduction is working; a rising one, even with unchanged headcount, is early evidence of alert fatigue and a prompt to revisit the severity model and inhibition scope described in the alert routing guide before people start muting pages.

Corrective action completion by action type Five corrective action types are compared by the share completed within their due date across a year of reviews. A threshold change on an existing detector completes most often because it is a small edit. Adding a new detector completes less often. A runbook change completes less often again. A snapshot-retention change completes rarely because it needs storage approval. A platform-wide framework completes almost never. The note advises splitting large actions until they look like the first two types. Split every action until it looks like one of the first two rows threshold change on an existing detector 94% completed on time new detector added 73% runbook change 58% snapshot retention change 27% platform-wide framework 6%

Failure modes of the review process

Measuring uptime instead of exposure. The pipeline was green fourteen minutes after the alert; the wrong parcels sat in the published layer for two days. A review that reports the first number and not the second will conclude the response was excellent.

Reconstructing from chat logs. Human memory of an incident compresses badly and systematically underestimates the detection interval. Reconstruct from ingested_at, batch identifiers and detector histories; use the chat log only for decisions, not for times.

Corrective actions with no detector attached. “Add monitoring for this” is not an action. “Add a per-source ST_SRID contract check on the parcels feed with a for: 10m window, owned by the geo-platform rotation” is.

Skipping the near-miss analysis. The cheapest improvement available is nearly always tightening a signal that was already moving. Reviews that jump straight to new instrumentation build more surface area than they need.

No re-review of the corrective action. An action that ships and is never validated against a synthetic repeat of the incident is a guess. Replaying the incident’s conditions against the new detector is a ten-minute test that either confirms the fix or reveals that the threshold is still too loose. The procedure is set out in replaying a spatial incident against a new detector, and its output — an original interval beside a replayed one — is the most checkable line a review can contain.

Reviewing only the incidents that paged. A defect caught by a pre-ingestion gate never reaches a human, and a review culture that only examines paged incidents never learns which gates are carrying the load. Sampling a handful of caught-at-the-boundary rejections each month tells you which contracts are earning their keep and which have quietly stopped matching the data they were written for.

Review checklist

  1. Reconstruct the first bad batch from ingestion timestamps, not from the alert.
  2. Compute all four intervals and record them as numbers in the document.
  3. Compute feature-hours of exposure and list the named downstream consumers.
  4. Identify the near-miss signal that was trending before the firing detector.
  5. State whether the runbook made the containment decision explicit, and fix it if not.
  6. List what was repaired, what was rebuilt, and what remains unrepaired.
  7. Write corrective actions that each change a detector, a runbook, or a reversibility property, with owners.
  8. Replay the incident conditions against the changed detector to confirm it would now fire earlier.