Multi-Region Deployment & Geo-Routing

Running an agent fleet in a single cloud region is an availability bet that most enterprises cannot afford to take. A regional outage, whether a cloud provider incident, a network partition, or a botched deployment, takes your entire platform offline. Multi-region deployment turns a single point of failure into a graceful degradation, but it introduces a new class of problems around state consistency and data residency.

Active-Active vs Active-Passive

The first architectural decision is whether both regions serve live traffic simultaneously (active-active) or whether one region stands by until the primary fails (active-passive).

flowchart TD
    C["Client Request"] --> DNS["Latency-Based DNS\n(Route 53 / Cloudflare)"]
    DNS --> USE1["us-east-1\nAgent Fleet + DB Primary"]
    DNS --> EUW1["eu-west-1\nAgent Fleet + DB Primary"]
    USE1 <-->|"PostgreSQL\nLogical Replication"| EUW1
    style C fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style DNS fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style USE1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style EUW1 fill:#f0fdf9,stroke:#0D9488,color:#0F172A

Active-active runs full agent fleets in both regions simultaneously. Every region serves real traffic, so you get both disaster recovery and performance benefits (users connect to the nearest region). The cost is double the infrastructure and significant complexity in handling state conflicts when two regions write to the same logical record concurrently.

Active-passive keeps one region as primary and the other as a warm standby that only accepts writes after promotion. It is cheaper and simpler, but the standby region is wasted capacity during normal operation, and failover requires a promotion step that adds minutes to your RTO.

For enterprise agent platforms, active-active is preferable when your workload is regionally partitionable (EU tasks go to EU agents, US tasks go to US agents). Active-passive suits workloads where tasks cannot be partitioned and cross-region write conflicts are unacceptable.

Latency-Based DNS Geo-Routing

Latency-based DNS routing evaluates the actual network latency from the requester’s resolver to each regional endpoint and returns the IP of the fastest region. In AWS Route 53, you create one record per region with the same DNS name, enable latency routing, and attach a health check to each record.

A 30-second health check interval with a failure threshold of 3 means Route 53 detects an outage within 90 seconds. Set DNS TTL to 60 seconds so that once a region is marked unhealthy, clients pick up the change within 60 seconds of cache expiry. Total worst-case failover time: approximately 150 seconds, well within a 5-minute RTO.

Data residency constraints (GDPR) require that you pin certain client segments to specific regions regardless of latency. Use geolocation routing rules in Route 53 to ensure EU-origin requests always resolve to EU endpoints, overlaid on top of latency routing for non-restricted segments.

Cross-Region State Synchronisation

sequenceDiagram
    participant US as us-east-1 Agent
    participant USDB as us-east-1 PostgreSQL (primary)
    participant EUDB as eu-west-1 PostgreSQL (primary)
    participant EU as eu-west-1 Agent

    US->>USDB: INSERT agent_state (task_id=T1)
    USDB-->>EUDB: Logical replication slot\n(async, ~200ms lag)
    EU->>EUDB: INSERT agent_state (task_id=T2)
    EUDB-->>USDB: Logical replication slot\n(async, ~200ms lag)
    Note over USDB,EUDB: Conflict: both regions wrote\nto the same task_id row
    USDB-->>USDB: Conflict resolution:\nlast-write-wins by updated_at timestamp

PostgreSQL logical replication propagates row-level changes between regional primaries. Configure each region as both a publication source and a subscription target. The replication is asynchronous, typically 150–300 ms cross-region, which sets your practical RPO floor.

# WHY use a dedicated replication user with restricted permissions rather
# than the application user: the replication slot requires REPLICATION
# privilege, which is powerful. Least-privilege limits blast radius if
# credentials are compromised.
REPLICATION_DSN = (
    "postgresql://replicator:secret@us-east-1-db:5432/agentdb"
    "?application_name=eu-west-1-replica&replication=database"
)

# WHY store updated_at as microsecond-precision UTC rather than the
# application server's wall clock: cross-region clocks drift. Using
# PostgreSQL's now() at write time (server-side) gives a consistent
# reference point within each region. Conflict resolution compares
# the two servers' timestamps and accepts the later one.
CREATE_TABLE_DDL = """
CREATE TABLE agent_state (
    task_id     UUID PRIMARY KEY,
    agent_id    TEXT NOT NULL,
    state       JSONB NOT NULL,
    region      TEXT NOT NULL,          -- originating region tag
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- Conflict rule: last-write-wins by updated_at
-- Applied in the subscription's conflict_action setting
"""

Failover Automation and RTO/RPO Targets

RTO (Recovery Time Objective) is how long the business can tolerate an outage. RPO (Recovery Point Objective) is how much data loss is acceptable. These two numbers directly dictate your architecture budget.

flowchart TD
    F["Region Failure Detected\nt=0"] --> HC["Route 53 Health Check\nmarks endpoint unhealthy\nt=0s to t=90s"]
    HC --> DNS2["DNS TTL expires\nat clients/resolvers\nt=90s to t=150s"]
    DNS2 --> TR["Traffic routes\nto healthy region\nt~150s"]
    TR --> PR["Replica promotes\nto primary (if active-passive)\nt~180s"]
    PR --> SV["Service restored\nt~180s < RTO 5min"]
    style F fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style HC fill:#fff7ed,stroke:#f59e0b,color:#0F172A
    style DNS2 fill:#EEF0F7,stroke:#6366F1,color:#0F172A
    style TR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style PR fill:#f0fdf9,stroke:#0D9488,color:#0F172A
    style SV fill:#f0fdf9,stroke:#0D9488,color:#0F172A

A 5-minute RTO means your entire detection-routing-promotion chain must complete in 300 seconds. A 30-second RPO means replication lag must never exceed 30 seconds during normal operation; monitor pg_stat_replication.write_lag and alert when it crosses 10 seconds.

Multi-region deployment doubles infrastructure cost but reduces expected annual downtime from hours (single-region p99 availability) to seconds. For enterprise platforms processing financial transactions or healthcare workflows, that cost is not optional infrastructure spend, it is a contractual SLA requirement.

In the next module you will build on this resilient, scaled, multi-region foundation by adding self-healing capabilities that detect and recover from failures automatically, without human intervention.