Suppressing Alerts During Planned Layer Reloads

A full reload of a spatial layer looks exactly like a catastrophic failure to every detector you own. Row count drops to zero, the coverage extent collapses, freshness age spikes, the geometry-validity sample returns nothing, and the tile cache starts serving stale tiles because nothing has been published for the new state yet. Every one of those alerts is technically correct and every one of them is worthless, because the layer is being rebuilt on purpose. This guide covers how to declare a reload window that suppresses the right alerts for the right layer for exactly as long as the reload runs — without opening a hole through which a real failure escapes. It belongs to alert routing and on-call design for spatial pipelines within the spatial incident response and tooling program.

Row count and coverage during a truncate-and-load reload, with the suppression window overlaid Two measured series run across a reload. Row count starts at full, drops to zero when the truncate executes, and climbs back to full as the load progresses. Coverage extent follows the same shape one step behind. A shaded suppression window opens when the reload job signals its start and closes only after a post-load verification check passes, which is later than the load finishing. Outside the window both series are monitored normally. The reload signature is indistinguishable from total loss — unless the window says otherwise suppression window · layer=parcels_authoritative full zero row count coverage extent job signals start load complete verification passes — window closes The window closes on a verified state, never on a timer, or the first post-load minute alerts on a half-built layer.

Problem framing: why a timer is the wrong instrument

The instinct is to schedule a silence: “mute this layer’s alerts from 02:00 to 03:00 every night”. It fails in both directions, and both failures are expensive.

It fails long when the reload finishes early. A silence that runs to 03:00 keeps suppressing after the load committed at 02:20, so a genuine failure in that forty-minute gap goes unpaged. On a layer that feeds compliance extracts, forty silent minutes is the difference between catching a bad load and shipping it.

It fails short when the reload runs long. A source that doubled in size, a lock wait, a retry — any of these push the load past 03:00, the silence expires, and the pager erupts with a full family of alerts describing a load that is progressing normally. The on-call engineer now has to distinguish “still loading” from “broke while loading” using exactly the metrics that a reload makes uninformative.

The fix is to make the suppression state-driven rather than time-driven: the reload job itself publishes a signal while it is running, an inhibition rule keys on that signal, and the signal clears only after a post-load verification passes. The window then has exactly the duration of the reload, whatever that turns out to be, and it closes on evidence rather than on a clock.

Implementation: a reload signal the alerting stack can see

The signal is a metric, not a configuration change. Publishing it from the job means no human has to remember to open or close a silence, and it works identically for a scheduled reload and an operator-initiated one.

# reload_signal.py — publish a reload-in-progress gauge for the duration of a load.
from contextlib import contextmanager
from opentelemetry import metrics

meter = metrics.get_meter("gis.etl")
_state: dict[tuple[str, str], int] = {}

# An observable gauge is the right instrument: the collector reads current state
# on every scrape, so a crashed job stops publishing and the window self-closes
# instead of pinning the layer silent forever.
def _observe(options):
    for (layer, phase), value in _state.items():
        yield metrics.Observation(value, {"layer": layer, "phase": phase})

meter.create_observable_gauge("gis.etl.reload_in_progress", callbacks=[_observe])


@contextmanager
def reload_window(layer: str):
    """Hold the reload signal for the load, then for verification, then clear."""
    _state[(layer, "load")] = 1
    try:
        yield
        # The load finished, but the layer is not trustworthy until verified —
        # keep the window open across verification so the half-built state
        # between commit and check never pages anyone.
        _state[(layer, "load")] = 0
        _state[(layer, "verify")] = 1
        verify_layer(layer)          # raises if the reload produced a bad state
    finally:
        _state.pop((layer, "load"), None)
        _state.pop((layer, "verify"), None)

Using an observable gauge rather than a set-and-forget counter matters. If the loader process dies mid-reload, it stops being scraped, the series goes stale, and the suppression lapses on its own — which is the behaviour you want, because a dead loader is a real incident. A signal written once into a durable store would keep the layer muted indefinitely.

The corresponding inhibition rule turns the signal into suppression. Note that it suppresses only the alert classes a reload legitimately triggers, and leaves everything else — projection breaks, topology corruption, disk pressure — fully live.

# Publish the signal as an alert so it can act as an inhibition source.
groups:
  - name: reload-windows
    rules:
      - alert: LayerReloadInProgress
        expr: max by (layer) (gis_etl_reload_in_progress) == 1
        for: 0m
        labels: { severity: none, data_domain: spatial }
        annotations:
          summary: "Reload running on {{ $labels.layer }} — volume alerts suppressed"

inhibit_rules:
  - source_matchers: [alertname="LayerReloadInProgress"]
    # Only the alerts a truncate-and-load legitimately trips.
    target_matchers:
      - alertname=~"FreshnessSlaBreach|RowCountDelta|CoverageExtentShrink|TilePublishLag"
    equal: ['layer']          # never suppress a different layer

The equal: ['layer'] clause is the whole safety property. Without it, a reload of one layer suppresses volume alerts on every layer in the platform — a mistake that is invisible until the night a second layer fails during the first one’s reload.

Which alert classes a reload window may suppress and which must stay live Two columns divide the detector set. The suppressed column contains freshness age, row-count delta, coverage extent shrink and tile publish lag, each labelled as an expected consequence of a truncate and load. The always-live column contains projection or coordinate reference system contract breaks, topology corruption, schema contract breaks, disk and lock pressure, and loader-process absence, each labelled as a fault that a reload does not explain. A dividing rule states the test: suppress only what the reload itself causes. Suppress only what the reload itself causes Suppressed inside the window freshness_age_seconds row_count_delta_ratio coverage_extent_ratio tile_publish_lag_seconds truncate sets age to the reload start count is zero until the load commits extent rebuilds region by region nothing published until load ends expected consequences · not evidence of a fault Never suppressed crs_contract_match topology_error_rate schema_fingerprint_match loader_heartbeat_absent a reload never changes the datum corrupt input is corrupt either way a contract break is still a break a dead loader is the incident faults the reload does not explain Scoping test: a window on one layer must not silence another Two layers are shown during the same period. Layer one is inside a declared reload window, so its volume alerts are suppressed. Layer two is not in any window and has a genuine coverage failure at the same moment; its alert must page. A third row shows the outcome when the equal-on-layer clause is missing from the inhibition rule: layer two's alert is also suppressed, and the failure goes unnoticed for the duration of layer one's reload. The equal-on-layer clause is the whole safety property layer 1 · reloading window open · volume alerts suppressed layer 2 · real failure coverage collapse must page without equal: [layer] layer 2 suppressed too — failure invisible Test it by opening a window on one layer and firing a suppressed-class alert on another.

Verification: confirm the window opens, closes and scopes correctly

Three assertions are worth automating, because each corresponds to a failure that is silent in production.

First, confirm the window opens. Run a reload in staging and query the signal series during it; a window that never opens produces a nightly alert storm that people learn to ignore rather than report.

Second, confirm the window closes on verification, not on load completion. Query the signal at a timestamp between the load’s commit and the verification’s completion — it must still read 1. This is the assertion that catches the most common regression, which is someone moving the signal clear inside the load function.

-- Post-load verification the window waits on. All three must hold before the
-- layer is trustworthy again; any failure keeps the reload marked unhealthy.
SELECT
  COUNT(*)                                              AS feature_count,
  COUNT(*) FILTER (WHERE NOT ST_IsValid(geom))          AS invalid_geoms,
  COUNT(DISTINCT ST_SRID(geom))                         AS distinct_srids,
  ST_Area(ST_Extent(geom)::geometry)                    AS extent_area
FROM prod.parcels;
-- Expect: feature_count within 5% of the pre-reload baseline,
--         invalid_geoms = 0, distinct_srids = 1, extent_area within 2%.

Third, confirm the scoping. Open a window on one layer and fire a suppressed-class alert on a different layer; it must page. A missing equal: clause is otherwise undetectable until it costs you an incident.

Gotchas

The window outlives a crashed loader. If the signal comes from a durable store rather than a live scrape, a crashed job leaves the layer muted forever. Publish it as a scraped gauge so staleness closes the window automatically, and add a companion alert on the reload exceeding its p99 duration so a stuck load pages on its own terms.

Suppressing the wrong severity tier. It is tempting to inhibit everything on the layer during a reload. That silences topology corruption arriving in the new data, which is precisely when you most want to hear about it — the failure class the topology corruption incident runbook exists to handle. Keep correctness detectors live through the window.

Verification that only counts rows. A reload that loads the right number of features with the wrong projection passes a count check and fails everything downstream. The verification query above deliberately checks count, validity, projection and extent together, mirroring the gate described in coordinate reference system validation.

No record that the window existed. When a post-incident review asks why nothing paged between 02:00 and 02:40, the answer needs to be in the record. Emit the reload signal as an annotated event so review can see the window on the same timeline as the alerts.

FAQ

Should the window suppress or merely downgrade the alerts?

Downgrading — routing suppressed-class alerts to a review channel instead of dropping them — is strictly better where the tooling supports it. The alerts stay visible for anyone actively watching the reload, and the post-incident timeline keeps a complete record, while nobody is paged. Full suppression is acceptable but loses that evidence.

What if a reload legitimately takes hours?

Long reloads need a progress signal in addition to a state signal: publish the fraction of the layer rebuilt, and alert when progress stalls rather than when the layer looks empty. Stalled progress is the real failure mode of a long reload, and it is invisible to volume detectors either way.

Can I reuse this for schema migrations?

Yes, with a different target set. A migration legitimately trips schema-fingerprint and attribute-drift detectors, so those move into the suppressed column while the volume detectors stay live — the mirror image of the reload case. Keep the two window types distinct rather than making one permissive window that covers both.

How does this interact with the grouping window?

They compose cleanly. Inhibition is evaluated before grouping, so a suppressed alert never joins a notification group at all. If you have tuned a ninety-second grouping window as described in tuning alert grouping windows for batch GIS jobs, the reload window simply removes members from the family before it is assembled.

Does an incremental load need a window at all?

Usually not. An incremental upsert does not empty the layer, so the volume detectors never trip. Windows are for operations that destroy and rebuild state — truncate-and-load, partition swap, full re-projection — and applying one to an incremental job hides real regressions for no benefit.