Alert Routing and On-Call Design for Spatial Pipelines

A spatial pipeline generates a different alert shape than a web service. Its failures are slow, geographically scoped, and frequently correlated: one bad coordinate reference system in a source feed produces a projection error, a coverage-extent collapse, a topology validity spike, and a tile-render timeout, all within the same ten minutes, all technically true, all pointing at one upstream change. Route them naively and the on-call engineer receives four pages describing one incident, then learns to mute the noisiest of the four — which is usually the one that would have caught the next incident earliest. This guide covers how to route, group, suppress and escalate alerts so that a spatial incident produces one actionable page with the right context attached, and it sits under the broader spatial incident response and tooling program alongside the spatial pipeline incident runbooks those pages resolve into.

The audience is the data engineer or SRE who owns the pager rotation for a geospatial platform: the person who decides whether a freshness lag on a nightly cadastral refresh is worth waking someone at 03:00, and who has to defend that decision at the next review.

Why spatial alerts need their own routing rules

Three properties of spatial data make generic alert routing produce the wrong outcome.

The first is correlation by geography rather than by service. A conventional alert graph groups by host, service, or endpoint. A spatial failure groups by layer and source: the unit that broke is a dataset, and the symptoms surface in whatever services happen to read it. A parcel layer whose upstream export switched from EPSG:2263 to EPSG:4326 will trip a projection detector in the ingestion worker, a coverage detector in the nightly quality job, and a latency detector in the tile renderer — three services, one cause. Routing keyed on service scatters that incident across three rotations.

The second is latency of consequence. A failing checkout endpoint hurts within seconds. A spatial freshness breach on a weekly-updated administrative boundary layer may not hurt anyone for a day. The severity of a spatial alert is a function of the dataset’s update cadence and its downstream contracts, not of the size of the numeric deviation. A ten-minute lag is catastrophic on a live vehicle feed and meaningless on a cadastral refresh, so the same metric needs different thresholds and different routing per layer — the concern the tracking spatial data freshness SLAs topic works through in detail.

The third is partial-region failure. Spatial systems fail in patches. A tile pipeline can be perfectly healthy in one region and stalled in another; a replication link can lag for one bounding box while the global aggregate looks fine. An alert that aggregates away the region label hides the failure until it is global, which is exactly too late. Routing rules therefore have to preserve region and layer labels all the way to the notification, which in turn means the label set has to be bounded — the cardinality problem addressed in bounding spatial metric tag cardinality.

Severity model: what earns a page

Severity is a routing decision, not a description of how bad the number looks. Fix the meaning of each tier in writing and hold to it, because the tier is what determines whether a human loses sleep.

The critical tier deserves the strictest gate. Ask two questions before assigning it: does a human action exist that materially shortens the damage, and will waiting until morning make it worse. If the answer to either is no, it is not critical. A nightly batch that failed at 02:00 and cannot be re-run until the upstream export lands at 06:00 fails both tests — page nobody, open a ticket, and let the morning rotation handle it with full context.

The inverse mistake is more common and more expensive: leaving a genuinely corrupting failure at warning because it fires often. Frequency is a signal that the detector needs tuning, not that the failure class is unimportant. Corrupt geometry entering a published layer is critical every time; if that pages twice a week, the fix is a pre-ingestion gate, not a demotion.

Grouping and suppression that respects spatial causality

Correlation is where most of the noise reduction happens, and the grouping key is the whole design. Group by layer and source, never by alert name. Grouping by alert name gives you one notification per detector — the opposite of what you want. Grouping by layer gives you one notification per broken dataset, with every firing detector listed inside it, which is exactly the payload an engineer needs to start triage.

# alertmanager.yaml — routing for a spatial platform
route:
  receiver: spatial-daily-queue
  # The grouping key is the dataset, not the detector. Five detectors firing on
  # one layer become one notification listing all five.
  group_by: ['layer', 'source']
  group_wait: 90s          # long enough for correlated spatial detectors to arrive
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical", data_domain="spatial"]
      receiver: spatial-pager
      group_wait: 30s      # correctness incidents wait less
      continue: false
    - matchers: [severity="baseline"]
      receiver: spatial-review-channel
      group_wait: 10m      # distribution drift is never urgent

inhibit_rules:
  # A projection break upstream explains the topology, coverage and tile
  # symptoms downstream. Silence the consequences, page for the cause.
  - source_matchers: [alertname="CrsContractBreak"]
    target_matchers: [alertname=~"TopologyErrorRateSpike|CoverageExtentShrink|TilePublishLag"]
    equal: ['layer']
  # An in-flight full reload legitimately empties and refills a layer.
  - source_matchers: [alertname="LayerReloadInProgress"]
    target_matchers: [alertname=~"FreshnessSlaBreach|RowCountDelta|CoverageExtentShrink"]
    equal: ['layer']

The group_wait of ninety seconds is deliberate and spatial-specific. Correlated spatial detectors do not fire simultaneously: the projection check runs at ingestion, the coverage sweep runs on a five-minute schedule, and the tile latency detector needs a full scrape window. A thirty-second wait sends the first alert alone and the rest as a noisy follow-up. Ninety seconds collects the family.

The inhibition rules encode causality that the metrics themselves cannot express. When a coordinate reference system contract breaks, the resulting topology errors and coverage collapse are symptoms; paging on them wastes attention and, worse, sends the engineer to the wrong runbook. Suppressing them and paging only on the cause is the single highest-leverage rule in a spatial alerting stack — and it is the routing counterpart of the CRS cascade failure runbook, which walks the same causal chain from the responder’s side.

Enriching the notification so triage starts immediately

A page that says TopologyErrorRateSpike firing costs the responder five minutes of orientation before any thinking begins. A page that arrives already carrying the layer, the source, the failure reason breakdown, the runbook link and the last successful run timestamp starts triage at second zero. The annotations block is where that context is assembled, and every field in it should be one the responder would otherwise have to look up.

groups:
  - name: spatial-routing
    rules:
      - alert: CrsContractBreak
        expr: |
          min by (layer, source) (gis_spatial_crs_contract_match) == 0
        for: 5m
        labels:
          severity: critical
          data_domain: spatial
          owning_team: geo-platform
        annotations:
          summary: >-
            CRS contract broken on {{ $labels.layer }} from {{ $labels.source }}
          # Everything below removes a lookup the responder would otherwise do.
          runbook_url: "/spatial-incident-response-and-tooling/spatial-pipeline-incident-runbooks/crs-cascade-failure-runbook/"
          expected_srid: '{{ with query "gis_spatial_expected_srid{layer=\"parcels_authoritative\"}" }}{{ . | first | value }}{{ end }}'
          last_clean_run: '{{ with query "gis_etl_last_success_timestamp_seconds{layer=\"parcels_authoritative\"}" }}{{ . | first | value | humanizeTimestamp }}{{ end }}'
          blast_radius: "downstream: tiles, analytics extracts, boundary joins"

Two annotation habits are worth adopting permanently. First, always carry runbook_url and make it resolve to a real page — an alert whose runbook link 404s trains people to stop clicking it. Second, always carry a blast radius field naming the downstream consumers of the affected layer. Spatial datasets fan out unusually widely, and the responder frequently cannot know from the layer name alone whether a compliance extract depends on it. Naming the consumers in the page converts a technical alert into a business decision the responder can actually make.

On-call rotation design for a geospatial platform

The rotation itself needs shaping to the workload. Three patterns work well.

Split the pager by data domain, not by system. A geospatial platform typically spans a database, a tile server, an ingestion fleet and a collector pipeline. Splitting on-call by component means every spatial incident starts with a hand-off, because the symptom appears in one component and the cause lives in another. A single rotation that owns the data across those components, backed by component specialists as secondary, resolves incidents faster.

Give the rotation an explicit quiet contract. Write down which alert classes may page overnight, publish it, and route everything else to the morning queue. On a spatial platform the overnight-eligible set is usually small: correctness breaks on authoritative layers, live-feed staleness, and replication divergence past the correctness bound. Batch freshness, index health, and distribution drift almost never belong on it. A rotation with a published quiet contract can defend itself against alert creep; one without it accumulates alerts until people mute the pager.

Measure the rotation, not just the pipeline. Track pages per shift, the fraction of pages that resulted in an action, and time-to-acknowledge. A rotation where fewer than half of overnight pages lead to an action has a routing problem, and the fix is upstream in this document — better inhibition, stricter severity, longer for: windows — not more stoicism from the on-call engineer. Those numbers are also the raw material for the review process covered in post-incident review for geospatial data.

Ownership labels: getting the alert to the team that can fix it

Routing by severity decides whether to page. Routing by ownership decides whom. On a geospatial platform those two axes are genuinely independent, because the same layer can break for reasons owned by very different teams: a vendor export changed shape (data stewardship), the load worker ran out of memory on a dense boundary set (platform engineering), the tile renderer fell behind (serving), or a projection definition was edited in the catalogue (spatial data governance). Sending all four to one rotation guarantees three of them start with a hand-off.

The workable pattern is a two-label scheme carried on every spatial rule: owning_team, which names the rotation that receives the page, and data_steward, which names the person or group accountable for the dataset’s content. The first drives routing; the second drives the follow-up conversation and appears in the notification body rather than in the route tree. Both should be derived from a single layer registry rather than hand-written into rules, because hand-written ownership drifts the moment a team reorganises and nobody notices until an incident lands in an empty inbox.

Deriving them mechanically is straightforward. Keep a small table mapping each layer to its owning team, steward, update cadence and pipeline class; export it as a set of metric series at scrape time; then join against it inside the alerting expression so the labels ride along automatically. The join costs nothing at evaluation time and means a re-org is a one-row edit rather than a rule sweep.

# Ownership labels come from a registry series, not from hand-written rules.
# gis_layer_registry_info{layer, owning_team, data_steward, pipeline_class,
#                         update_cadence_seconds} == 1
(
  sum by (layer, source) (rate(gis_spatial_topology_error_total[5m]))
    /
  clamp_min(sum by (layer, source) (rate(gis_etl_features_ingested_total[5m])), 1)
  > 0.02
)
* on (layer) group_left(owning_team, data_steward, pipeline_class)
  gis_layer_registry_info

There is a second, less obvious benefit. Once ownership is a label rather than a convention, you can measure it: count pages per owning team, and count how often a page was re-assigned after triage. A team that receives many pages it immediately hands off has a routing problem — usually a layer whose registry entry is stale, or a failure class that genuinely belongs elsewhere. Reassignment rate is one of the few alerting metrics that points directly at a fixable configuration error rather than at a vague sense that the pager is noisy.

Ownership routing also needs a defined default. Every route tree should end in a catch-all receiver that goes somewhere a human actually reads, because the alert most worth seeing is the one whose labels you did not anticipate. A catch-all that discards is how a new layer silently loses coverage for months.

Testing the routing before you trust it

Routing configuration is code that only executes during an incident, which is the worst possible time to discover a typo in a matcher. Three cheap tests catch nearly everything.

Fire a synthetic alert through the real path. Push a series with the exact label set your rule produces, let the rule evaluate, and confirm the notification arrives at the intended destination with the runbook link intact. Do this for one alert per severity tier after every routing change.

Verify inhibition with a paired injection. Inject the cause and the symptom together and confirm only the cause notifies. Inhibition rules fail silently when the equal: label is missing from one side — a symptom alert that omits the layer label will never be inhibited, and no amount of reading the configuration reveals it.

Audit the runbook links. A scheduled job that requests every distinct runbook_url in the rule set and reports non-200 responses takes an hour to write and prevents the slow rot that makes responders stop trusting annotations.

Failure modes of the routing layer itself

The grouping key is too coarse. Grouping by source alone merges unrelated layers from the same vendor into one notification, and a genuine second incident gets appended silently to an open one. Always include layer.

The for: window is shorter than the scrape interval. Spatial detectors that run on a five-minute quality sweep with a for: 2m window fire on a single sample. One sample is noise on a coverage or freshness metric; require at least two evaluation intervals.

Inhibition suppresses a real second incident. If a projection break on a layer inhibits topology alerts on that layer for hours, a genuinely new topology fault during the same window is invisible. Bound the inhibition with the cause alert’s own resolution, and keep the suppressed alerts visible in the review channel even while they are not paging.

Runbook links point at removed pages. The response degrades quietly: responders learn the link is unreliable and stop opening it, losing the whole benefit of enrichment.

Severity assigned by metric magnitude. A rule that escalates to critical because a number is large rather than because the consequence is severe produces pages nobody can act on. Severity belongs to the consequence, always.

Troubleshooting checklist

1. Confirm the alert actually fired: query the rule’s expression directly over the incident window rather than trusting the notification history.
2. Check the label set on the firing series — a missing layer or severity label routes to the default receiver and looks like a routing bug.
3. Verify the matchers in the route tree match the labels exactly, including value case; a matcher on severity="Critical" never matches critical.
4. Inspect active inhibitions before concluding an alert was lost — a suppressed alert is present in the alert state but absent from notifications.
5. Confirm group_wait has elapsed; a notification that seems missing is often still batching.
6. Test the receiver independently with a synthetic payload to separate a routing fault from a delivery fault.
7. Re-check the runbook link resolves and names the same failure mode the alert describes.

Related

• Spatial incident response and tooling — the parent section covering tooling choice, runbooks and review.
• Spatial pipeline incident runbooks — the procedures a routed page hands the responder.
• Post-incident review for geospatial data — how routing quality gets measured and corrected after the fact.
• Tracking spatial data freshness SLAs — where the per-layer thresholds that drive severity come from.