Architecting Fallback Routing for Spatial Feature Servers

When a primary spatial feature server degrades or fails, the cascading impact on downstream ETL pipelines, web map clients, and compliance audit trails can escalate within minutes. Architecting a deterministic fallback routing layer is not an optional luxury; it is a core reliability requirement for any production-grade geospatial platform. This guide provides a hands-on, step-by-step configuration and debugging workflow for implementing resilient fallback routing across spatial feature servers, with explicit threshold values, circuit-breaker logic, and observability hooks engineered to slash MTTR.

1. Establishing Routing Baselines and Trust Boundaries

Before implementing routing logic, you must define the operational perimeter of your spatial services. Every fallback decision hinges on understanding which data layers are authoritative, which are derived, and how they map to compliance requirements. When evaluating fallback candidates, cross-reference your telemetry collection strategy with the Observability Scoping Rules for Vector Data to ensure that metrics aggregation does not inadvertently mask degraded geometry precision or topology errors during failover.

Fallback routing must never silently downgrade coordinate reference system transformations or strip mandatory attribute constraints. Configure your ingress proxies to enforce strict schema validation on fallback responses using JSON Schema or Protobuf definitions aligned with your feature catalog. If a secondary server returns vector payloads with mismatched SRID values, missing mandatory attributes, or invalid GeoJSON FeatureCollection structures, the routing layer must treat it as a hard failure rather than a degraded success. This strict validation preserves lineage integrity and ensures compliance teams can audit exactly which data tier served a request during an outage.

2. Tiered Fallback Chain Configuration

A robust fallback architecture relies on a tiered routing chain rather than a simple primary/secondary toggle. Deploy a three-tier topology: Tier 1 (primary regional cluster), Tier 2 (cross-region read replica with cached feature tiles), and Tier 3 (static snapshot or degraded vector cache). The following Envoy-compatible YAML configuration enforces the exact thresholds required for automatic, deterministic failover.

# fallback_routing_config.yaml
static_resources:
  clusters:
  - name: tier1_primary
    type: STRICT_DNS
    connect_timeout: 2s
    lb_policy: ROUND_ROBIN
    outlier_detection:
      consecutive_5xx: 3
      interval: 10s
      base_ejection_time: 120s
      max_ejection_percent: 50
      enforcing_consecutive_5xx: 100
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 3
      healthy_threshold: 2
      http_health_check:
        path: /healthz
        expected_statuses:
        - start: 200
          end: 299

  - name: tier2_replica
    type: STRICT_DNS
    connect_timeout: 3s
    lb_policy: ROUND_ROBIN
    health_checks:
    - timeout: 5s
      interval: 10s
      unhealthy_threshold: 3
      http_health_check:
        path: /healthz

  - name: tier3_snapshot
    type: STATIC
    load_assignment:
      cluster_name: tier3_snapshot
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 10.0.5.10
                port_value: 8443

  # Aggregate cluster: deterministic priority failover tier1 -> tier2 -> tier3.
  - name: spatial_failover
    connect_timeout: 2s
    lb_policy: CLUSTER_PROVIDED
    cluster_type:
      name: envoy.clusters.aggregate
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.clusters.aggregate.v3.ClusterConfig
        clusters:
        - tier1_primary
        - tier2_replica
        - tier3_snapshot

  listeners:
  - name: spatial_gateway
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: spatial_ingress
          route_config:
            name: local_route
            virtual_hosts:
            - name: spatial_api
              domains: ["*"]
              routes:
              - match:
                  prefix: "/ogc/features"
                route:
                  cluster: spatial_failover
                  timeout: 1.5s
                  retry_policy:
                    retry_on: "5xx,reset,connect-failure,retriable-status-codes"
                    num_retries: 3
                    per_try_timeout: 0.5s

3. Circuit Breaker State Machine and Traffic Probing

Implement a deterministic half-open circuit breaker state after 120 seconds of Tier 1 isolation. During this phase, allow exactly 10% of traffic to probe recovery before fully restoring the primary route. The following logic governs state transitions:

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: latency p95 high or error budget burned
  Open --> HalfOpen: after 120s isolation
  HalfOpen --> Closed: probe success rate recovers
  HalfOpen --> Open: probe failures persist

Closed State: 100% traffic to Tier 1. Monitor p95 latency and error budgets.
Open State: Triggered when p95 exceeds 1,500 ms for three consecutive 30-second windows, or when the rolling 60-second error rate exceeds 3.5%. All traffic routes to Tier 2.
Half-Open State: Activated after 120s isolation. Route exactly 10% of requests to Tier 1. If success rate > 98% over a 60-second evaluation window, transition to Closed. If failure rate > 2%, revert to Open and extend isolation by 240s.

To enforce spatial-specific error tracking, configure your proxy to classify 400 Bad Request responses containing InvalidGeometry or CRSMismatch as circuit-breaking events. Reference the OGC API - Features Specification for standardized error payload structures that your routing layer can parse deterministically.

4. Observability Hooks and Telemetry Validation

Ground your monitoring stack in the Geospatial Observability Architecture & Fundamentals to ensure telemetry accurately reflects spatial routing states. Inject OpenTelemetry resource attributes into every feature request to preserve routing lineage:

# otel-collector-config.yaml
processors:
  resource:
    attributes:
    - key: spatial.routing.tier
      value: "1"
      action: insert
    - key: spatial.crs
      value: "EPSG:4326"
      action: insert
    - key: spatial.topology.validated
      value: "true"
      action: insert

Deploy the following PromQL alert rules to trigger automated runbooks when thresholds breach:

# prometheus_alert_rules.yml
groups:
- name: spatial_fallback_routing
  rules:
  - alert: SpatialFeatureServerLatencyDegradation
    expr: >
      histogram_quantile(0.95,
        sum(rate(http_request_duration_seconds_bucket{path=~"/ogc/features.*"}[30s]))
        by (le)) > 1.5
    for: 90s
    labels:
      severity: critical
      routing_action: "trigger_tier2_fallback"
    annotations:
      summary: "p95 latency exceeds 1500ms for 3 consecutive 30s windows"

  - alert: SpatialFeatureServerErrorRateSpike
    expr: >
      sum(rate(http_requests_total{status=~"5..|400",path=~"/ogc/features.*"}[60s]))
      / sum(rate(http_requests_total{path=~"/ogc/features.*"}[60s])) > 0.035
    for: 60s
    labels:
      severity: warning
      routing_action: "initiate_circuit_breaker"
    annotations:
      summary: "Rolling 60s error rate exceeds 3.5%"

For distributed tracing, ensure your GIS pipelines propagate traceparent headers across fallback hops. Consult the OpenTelemetry Semantic Conventions for standardized span naming conventions that map cleanly to spatial routing states.

5. Incident Playbook and Debugging Workflow

When fallback routing activates, execute the following debugging workflow to isolate spatial metric lag or misrouted geometry:

Verify Fallback Activation: Query your observability dashboard for spatial.routing.tier attribute shifts. Confirm Tier 2 or Tier 3 received traffic.
Validate Geometry Integrity: Run a targeted PostGIS validation query against the fallback endpoint response:
```
SELECT ST_IsValid(geometry), ST_SRID(geometry)
FROM fallback_feature_cache
WHERE feature_id IN (SELECT id FROM recent_fallback_requests LIMIT 100);
```
If ST_IsValid returns false or SRID deviates from baseline, halt fallback routing and escalate to the data engineering team.
Trace Circuit Breaker State: Inspect Envoy cluster stats via the admin interface:
```
curl -s http://localhost:9901/stats | grep -E "cluster.tier1_primary.outlier_detection|circuit_breakers"
```
Verify ejection_time aligns with the 120s half-open threshold. If max_ejection_percent caps prematurely, adjust outlier detection parameters.
Audit Compliance Lineage: Cross-reference request logs with your audit trail. Ensure every fallback request carries the spatial.routing.tier span attribute and an X-Data-Lineage-ID header. Missing lineage metadata indicates proxy misconfiguration and requires immediate rollback to Tier 1 with manual health validation.
Recovery Validation: Once Tier 1 metrics stabilize (p95 < 1,000ms, error rate < 1%), manually transition the circuit breaker to Closed. Monitor for 15 minutes before decommissioning Tier 2 routing.

By enforcing strict schema validation, deterministic state transitions, and spatially-aware telemetry, your platform will maintain operational continuity during server degradation while preserving data integrity and compliance auditability.