Setting Error Budgets for Spatial Freshness

A freshness objective without a budget is a wish. “The parcels layer should be under four hours old” tells nobody what to do when it is four and a half hours old at 02:00 on a Sunday, or whether the third such event this month is normal or alarming. An error budget answers both: it converts the objective into a quantity of allowed staleness per window, spends it as breaches occur, and gives the team an unambiguous policy trigger when it runs low. This guide covers how to size a spatial freshness budget from the layer’s update cadence rather than from a copied web-service number, how to compute it correctly for batch layers, and what policy to attach when it depletes. It belongs to spatial data contracts and SLO design under geospatial observability architecture fundamentals.

Freshness age against its bound, with the shaded area above the bound consuming the budget A sawtooth series shows layer age rising between refreshes and dropping to near zero at each successful load. A horizontal bound line marks the contracted maximum age. Three excursions rise above the bound and the area of each excursion is shaded, representing budget consumed. A side panel converts the total shaded area into minutes and compares it against the monthly allowance derived from the objective target, showing roughly two thirds of the allowance spent. Budget is spent by area above the bound, not by the count of breaches age contracted bound · 4 h 32 m 88 m 21 m 30-day window Budget ledger target 99.5% of minutes allowance 219 min / 30 d spent 141 min remaining 78 min (36%) below 25% triggers a change freeze Three breaches, very different costs: counting incidents would rank them equally.

Problem framing: minutes, not incidents

The instinct is to budget breaches — “no more than three freshness breaches per month”. It ranks a two-minute overshoot equally with a nine-hour outage, which is exactly backwards, and it creates a perverse incentive to batch problems into fewer, longer events.

Budget the time out of bound instead. The objective states a fraction of minutes during which age must be under the bound; the allowance is the complement of that fraction over the window; each breach spends its own duration. A 99.5% target over thirty days allows 216 minutes of staleness, and a nine-hour outage spends two and a half times the entire month’s allowance in one night — which is the correct and useful conclusion.

The second framing decision is what “age” means for a batch layer, and it is where most spatial freshness objectives go wrong. A layer refreshed nightly is, by construction, up to twenty-four hours old just before its next run. Measuring raw age against a four-hour bound puts the layer permanently in breach. The bound has to be expressed relative to the layer’s cadence: age is measured from the source generation time of the newest accepted batch, and the objective asserts that this age stays under the cadence plus a tolerance, not under some absolute constant.

For a nightly layer generated at 22:00 and loaded by 23:00, a sensible bound is “age under 26 hours” — one cadence plus a two-hour tolerance for the load. Breaching it means a night was missed, which is exactly the event worth budgeting.

Implementation: computing the budget from cadence

Derive the bound and the allowance from the registry entry rather than hand-writing them per layer, so a cadence change updates the objective automatically.

# Age of the newest accepted batch, measured from SOURCE generation time.
# gis_etl_source_generated_timestamp_seconds is stamped by the loader from the
# source's own metadata, not from when we happened to ingest it.
(time() - max by (layer) (gis_etl_source_generated_timestamp_seconds))

# In-bound indicator: age under (cadence + tolerance), both from the registry.
(
  (time() - max by (layer) (gis_etl_source_generated_timestamp_seconds))
  <=
  on (layer) group_left()
  (
    max by (layer) (gis_layer_registry_cadence_seconds)
    + max by (layer) (gis_layer_registry_tolerance_seconds)
  )
)

Record that indicator as a recording rule so attainment is a simple average over the window rather than a nested expression evaluated repeatedly.

groups:
  - name: freshness-slo
    interval: 1m
    rules:
      # 1 when the layer is inside its freshness bound, 0 otherwise.
      - record: layer:freshness_in_bound
        expr: |
          clamp_max(
            (
              (
                on (layer)
                (max by (layer) (gis_layer_registry_cadence_seconds)
                  + max by (layer) (gis_layer_registry_tolerance_seconds))
              )
              >= bool
              (time() - max by (layer) (gis_etl_source_generated_timestamp_seconds))
            ), 1)

      # Attainment over the rolling window.
      - record: layer:freshness_attainment_30d
        expr: avg_over_time(layer:freshness_in_bound[30d])

      # Fraction of the error budget still available.
      # target comes from the registry so each layer can differ.
      - record: layer:freshness_budget_remaining
        expr: |
          1 - (
            (1 - layer:freshness_attainment_30d)
            /
            clamp_min(1 - on (layer) group_left() max by (layer) (gis_layer_slo_target), 0.0001)
          )

      - alert: FreshnessBudgetLow
        expr: layer:freshness_budget_remaining < 0.25
        for: 30m
        labels: { severity: warning, data_domain: spatial }
        annotations:
          summary: >-
            {{ $labels.layer }} has {{ $value | humanizePercentage }} of its
            freshness budget left — freeze non-essential changes

      - alert: FreshnessBudgetExhausted
        expr: layer:freshness_budget_remaining <= 0
        for: 30m
        labels: { severity: critical, data_domain: spatial }

The clamp_min on the denominator guards against a registry entry with a target of exactly 1.0, which would otherwise divide by zero. Freshness should not carry a 100% target — that is reserved for correctness clauses — but registries acquire strange values and a rule that produces NaN silently stops alerting.

Freshness bound derived from each layer's cadence rather than a shared constant Four layers with different cadences are shown as rows. A live vehicle feed with a thirty-second cadence gets a two-minute bound. A tile publish pipeline with a fifteen-minute cadence gets a forty-five minute bound. A nightly cadastral refresh with a twenty-four hour cadence gets a twenty-six hour bound. A quarterly boundary set with a ninety-day cadence gets a ninety-five day bound. Each row shows the cadence as a solid bar and the tolerance as a lighter extension, and a note states that applying one constant bound across these layers would keep two of them permanently in breach. The bound is cadence plus tolerance — never a shared constant live vehicle feed tile publish nightly cadastral quarterly boundaries cadence 30 s cadence 15 min cadence 24 h cadence 90 d bound 2 min bound 45 min bound 26 h bound 95 d cadence tolerance A single four-hour bound would leave the nightly and quarterly layers permanently in breach and the vehicle feed unmonitored. Thirty-day and seven-day attainment diverging after a bad week Two attainment series are plotted. The thirty-day rolling figure declines gently and remains close to the target, because a bad week is diluted by three good ones. The seven-day figure drops sharply during the bad week and recovers afterwards. The divergence between the two is highlighted as the signal that something changed recently, which the thirty-day figure alone would smooth away. Publish both windows: the divergence is the recent-change signal target 99.5% 30-day — barely moves 7-day — shows the bad week plainly day 0day 15day 30

Verification: check the budget arithmetic against a known outage

The arithmetic is easy to get subtly wrong, so validate it against an event whose duration you know.

Pick a past outage of known length — say ninety minutes — and confirm that the budget consumed over the containing window equals ninety minutes of allowance. If it reads a different number, the usual causes are an evaluation interval that does not divide the window evenly, a recording rule whose bool comparison inverted the sense, or gaps in the underlying series being treated as in-bound rather than as unknown.

That last one deserves a specific test. Stop the loader’s metric export entirely for ten minutes and confirm the budget treats the gap conservatively rather than crediting it. A series that simply vanishes during an outage — because the exporting process died along with the pipeline — will otherwise make the worst outages invisible to the budget, which is a spectacular way to report perfect attainment through a total failure.

Gotchas

Budgeting breach counts. Ranks a two-minute overshoot with a nine-hour outage and rewards batching failures together. Budget minutes.

A constant bound across layers. Guarantees permanent breach for slow-cadence layers and no coverage for fast ones. Derive the bound from cadence.

Measuring age from ingestion. A pipeline that promptly ingests two-day-old data reports excellent freshness while consumers get stale answers. Measure from source generation time, as the contract clause in the parent topic specifies.

Missing series credited as healthy. When the exporter dies with the pipeline, absence must count against the budget. Use an explicit absence rule or a staleness-aware recording rule.

Rolling windows that hide a bad week. A thirty-day window smooths a severe recent week into an acceptable average. Publish the seven-day attainment alongside the thirty-day figure; a large divergence between them is the signal that something changed recently.

FAQ

What target should a new layer start with?

Start at 99.5% for a batch layer and 99.9% for anything serving interactive traffic, then adjust after a month of measurement. The first month’s attainment is more informative than any prior guess, and moving a target down once with an explanation is far healthier than leaving an aspirational target permanently breached.

Should planned reloads consume budget?

No, provided the reload is inside its declared window and the layer’s consumers were told. Exclude declared maintenance from the in-bound indicator by the same signal that suppresses the alerts, described in suppressing alerts during planned layer reloads. A reload that overruns its window should start consuming budget at the moment it overruns.

How does the budget interact with paging?

They serve different purposes and should not be conflated. Paging answers “is something broken now”; the budget answers “have we spent our allowance”. A budget alert is a warning to the owning team about change policy, never an overnight page — the severity model in alert routing and on-call design places it firmly in the daily queue.

What policy should a depleted budget trigger?

Freeze non-essential changes to the layer’s pipeline until attainment recovers, and require the owning team to name the corrective action. The freeze is the point of the budget: it converts reliability from an opinion into a constraint on the work queue.

How do I budget a layer whose source is only available during business hours?

Restrict the window rather than loosening the target. If the source cannot possibly refresh overnight, minutes outside the source’s own availability window are not minutes the pipeline failed, and including them just inflates the denominator until the objective becomes meaningless. Compute attainment over the layer’s declared active window — expressed in the contract alongside the cadence — and state the window explicitly in the objective so nobody later reads 99.5% as covering the whole month. The same reasoning applies to seasonal feeds that legitimately pause: budget the periods during which the layer is expected to be current, not the calendar.

Can one budget cover several layers?

Only if consumers treat them as one product. Layers with different cadences and different consumers need separate budgets, because merging them lets a healthy high-volume layer mask a chronically late one — the same averaging failure that partial-region alerting exists to avoid.