Publishing Layer Status for Downstream Jobs

A downstream job that reads a spatial layer has no way to tell a healthy layer from a broken one. It issues a query, gets rows back, and proceeds — whether those rows are today’s parcels, yesterday’s, or a half-loaded set covering two thirds of the country. The result is the most expensive failure mode in a data platform: not an error, but a confident wrong answer produced downstream and discovered weeks later. A published layer status document fixes this by making health machine-readable, so a job can decide to skip, degrade, or proceed before it spends an hour computing something worthless. This guide covers what the status document must contain, how to serve it cheaply, and how consumers should act on it. It belongs to spatial data contracts and SLO design under geospatial observability architecture fundamentals.

Consumer decision path with and without a status check before reading a layer Two paths are compared. The upper path shows a job reading the layer directly, computing a result, and publishing it, with an annotation noting that a stale or partial layer produces a confidently wrong output that is discovered weeks later. The lower path shows the same job first fetching the layer status document, then branching three ways: proceed when the layer is healthy, degrade to the last known-good snapshot when freshness is breached, and skip with a clear signal when coverage or projection is breached. Check status first, or compute confidently on data you never inspected job starts read layer compute publish wrong answer found weeks later job starts fetch status one cached request healthy → proceed freshness breached → use snapshot coverage or CRS breached → skip outcome is explicit and attributable to the layer

Problem framing: what a consumer actually needs to decide

Consumers do not need your dashboards. They need enough information to make one of three choices — proceed, degrade, or skip — and they need it in a form a script can evaluate in a few milliseconds.

That rules out several tempting designs. A link to a monitoring dashboard is useless to a scheduled job. A single boolean “healthy” flag is too coarse, because the right response differs by which clause broke: a freshness breach usually means “use the previous snapshot”, while a projection breach means “do not use this data at all”. And a full metrics endpoint puts the burden of interpretation on every consumer, guaranteeing that each one interprets differently.

The useful shape is a small document per layer that states, per contract clause, whether it currently holds; the timestamp and identity of the newest accepted batch; the current objective attainment and remaining budget; and a recommended action. The recommendation is what lets a naive consumer do the right thing without understanding spatial data at all, while a sophisticated one can ignore it and reason from the clauses.

Implementation: a small, cacheable status document

Keep it flat, keep it stable, and version it alongside the contract.

{
  "layer": "parcels_authoritative",
  "contract_version": "v8",
  "generated_at": "2026-08-11T09:41:02Z",
  "recommendation": "degraded",
  "newest_accepted_batch": {
    "batch_id": "2026-08-11T02:14Z-parcels-full",
    "source_generated_at": "2026-08-10T22:00:00Z",
    "feature_count": 12844091
  },
  "last_known_good": {
    "batch_id": "2026-08-10T02:11Z-parcels-full",
    "snapshot_ref": "s3://geo-snapshots/parcels/2026-08-10T02:11Z/"
  },
  "clauses": {
    "projection":  { "holds": true,  "detail": "EPSG:27700 on 100% of features" },
    "structure":   { "holds": true,  "detail": "fingerprint parcels_v7 matched" },
    "validity":    { "holds": true,  "detail": "0 invalid geometries" },
    "coverage":    { "holds": true,  "detail": "extent ratio 0.997" },
    "freshness":   { "holds": false, "detail": "age 11h42m exceeds the 4h bound" },
    "domains":     { "holds": true,  "detail": "no unmapped land_use codes" }
  },
  "objectives": {
    "freshness":  { "attainment_30d": 0.9931, "target": 0.995, "budget_remaining": -0.38 },
    "coverage":   { "attainment_30d": 0.9997, "target": 0.999, "budget_remaining": 0.70 }
  }
}

Three properties make this work in practice.

It is generated by the evaluator, not hand-maintained. The clause results come from the gate’s own decisions and the attainment figures from the same recording rules that drive alerting, so the document cannot disagree with the monitoring.

It is cacheable for a short interval. A sixty-second cache lifetime means thousands of consumers cost one generation per minute. The document should be small enough that consumers can fetch it unconditionally rather than trying to be clever about when to check.

It names a last known good snapshot. This single field is what makes “degrade” an actionable recommendation rather than advice. Without a pointer to the previous good state, a consumer facing a stale layer has no alternative but to proceed or fail.

On the consumer side the check is short enough that there is no excuse for skipping it.

# consumer_guard.py — decide before reading, not after computing.
import httpx

STATUS = "https://geo-platform.internal/status/{layer}.json"

class LayerUnusable(Exception):
    pass

def resolve_source(layer: str, tolerate_stale: bool = True) -> str:
    status = httpx.get(STATUS.format(layer=layer), timeout=5.0).json()
    clauses = status["clauses"]

    # Correctness clauses are non-negotiable: wrong data is worse than no data.
    for name in ("projection", "structure", "validity", "coverage", "domains"):
        if not clauses[name]["holds"]:
            raise LayerUnusable(f"{layer}: {name} breached — {clauses[name]['detail']}")

    # Freshness is degradable: fall back to the last good snapshot if allowed.
    if not clauses["freshness"]["holds"]:
        if not tolerate_stale:
            raise LayerUnusable(f"{layer}: stale and no stale tolerance configured")
        return status["last_known_good"]["snapshot_ref"]

    return f"warehouse://{layer}"

The asymmetry in that function mirrors the objective design in the parent topic: correctness clauses cause a hard stop, degradable clauses cause a fallback. A consumer that inverts this — tolerating a projection breach because “some data is better than none” — is producing exactly the confidently wrong answers the status document exists to prevent.

Mapping from breached clause to recommended consumer action A two-column table maps each contract clause to the action a consumer should take when it is breached. Projection, structure, validity and domain breaches all map to skip, on the reasoning that the data is wrong rather than late. Coverage breach maps to skip for area-wide analysis but proceed for point lookups inside the covered region. Freshness breach maps to using the last known-good snapshot. A footnote states that the recommendation field in the status document encodes the strictest applicable action so a naive consumer needs no logic of its own. Breached clause → consumer action clause action reasoning projection structure validity domains coverage freshness skip skip skip skip skip if area-wide use snapshot features are somewhere else on the planet a cast or join will misalign silently spatial predicates return empty or throw categorical rollups drop or misclassify point lookups inside the covered area stay valid late is degraded, not wrong Game-day result: which downstream jobs honoured a degraded status Eight downstream jobs are listed with their behaviour when a layer status was set to report a coverage breach. Three skipped correctly. Two fell back to the last known-good snapshot. Three proceeded regardless and are marked as unguarded consumers. A note states that the unguarded list is the actionable output of the exercise and is usually longer than the team expects. Game day: the unguarded list is the output that matters skipped correctly · 3 nightly risk model · compliance extract · boundary join fell back to snapshot · 2 operations dashboard · public map tiles proceeded regardless · 3 ad-hoc analyst notebook · partner export · legacy ETL job

Verification: prove consumers actually honour the status

Publishing the document is the easy half. Confirming that consumers read it — and behave correctly when it says something unwelcome — is where the value is realised.

Run a game-day: set the status for a non-production layer to report a coverage breach and observe which downstream jobs skip and which proceed regardless. Every job that proceeds is an unguarded consumer, and the list is usually longer than expected.

Then test the degrade path specifically. Report a freshness breach with a valid last_known_good pointer and confirm consumers fall back to the snapshot rather than failing outright. A fallback path that has never been exercised is a fallback path that does not work, and the snapshot reference is exactly the sort of field that rots quietly.

Finally, verify the failure mode of the status service itself. If the status endpoint is unreachable, consumers must fail closed — treating an unknown status as unusable — or the guard evaporates precisely during a platform-wide incident.

Gotchas

A status document that can disagree with monitoring. If clause results are recomputed for the document rather than read from the gate’s decisions, the two drift and consumers learn to distrust both. Generate from the same source.

No last-known-good pointer. Reduces every recommendation to proceed-or-fail and makes the degrade path unusable. Retain snapshots and reference them.

Consumers caching the status too long. A status cached for an hour means an hour of jobs proceeding against a layer that broke at minute two. Sixty seconds is generous; align it with the evaluator’s own interval.

Failing open when status is unavailable. Turns the guard off exactly when it matters. Treat an unreachable status as a breach.

Publishing status without publishing the contract. A consumer cannot interpret "coverage": {"holds": false} without knowing what coverage was promised. Serve the contract alongside the status, versioned together as described in versioning spatial data contracts without breaking consumers.

FAQ

Should the status document include per-region detail?

For layers served or built per region, yes — a summary that hides a single failed region repeats the averaging mistake described in alerting on partial-region failures. Add a per-region clause block, and let the top-level recommendation reflect the worst region.

Is a status endpoint not just a duplicate of the metrics endpoint?

They serve different audiences. Metrics are dimensional time series for operators; the status document is a point-in-time decision aid for programs. Trying to make one serve both produces something a scheduled job cannot evaluate in a millisecond.

What should the recommendation be during a declared reload?

degraded, with the freshness clause failing and the last-known-good pointer live. A reload is a legitimate period during which the layer is not current, and consumers should fall back rather than read a half-built state — the same window used to suppress alerts in suppressing alerts during planned layer reloads.

How do I get existing consumers to adopt this?

Ship the guard as a small shared library rather than as documentation, and make it the default path in whatever job template teams copy. Adoption follows the path of least effort; a five-line import will be used, a page of instructions will not.

Does this replace alerting?

No. The status document tells consumers what to do; alerting tells the owning team to fix it. A platform with a status endpoint and no alerting degrades gracefully forever and never recovers.

Should consumers record which status they acted on?

Yes, and it costs almost nothing. Have each job log the contract_version, batch_id and recommendation it read, and stamp the same values onto whatever it produces. When a downstream result is later found to be wrong, that stamp answers immediately whether the job ran against a degraded layer or against a healthy one — which is otherwise a multi-hour reconstruction. It also supplies the consumer-version signal that the retirement gate for old contracts depends on, so one field serves two purposes.