Cross-Region Event Processing Platform

0) Problem Restatement

Design a platform (asked at Capital One) that ingests events from producers in multiple regions, stores them durably, and delivers them to multiple consumers for processing. For example, transaction events consumed by fraud detection, notifications and analytics. It must survive a whole region failing without losing events, keep ordering per key (e.g., per account), and let consumers replay past events.

1) Requirements

• Producers send events continuously from several regions.
• Durable storage: no acknowledged event is lost, even if a region goes down.
• Multiple independent consumer groups, each tracking its own progress.
• Ordering per key (per account ID).
• At-least-once delivery, with support for consumers to deduplicate.
• Replay from a point in time. Low latency (sub-second in region).

2) Architecture

flowchart LR
    PA["Producers - Region A"] --> KA[("Kafka cluster A")]
    PB["Producers - Region B"] --> KB[("Kafka cluster B")]
    KA <-->|"async replication - MirrorMaker / cluster linking"| KB
    KA --> CA["Consumers in A - fraud, notifications"]
    KB --> CB["Consumers in B"]
    CA --> DED[("Dedup store / idempotent sinks")]
    CB --> DED
    REG["Schema registry"] --- PA
    REG --- PB

• Per-region Kafka clusters (or a managed equivalent) with replication factor 3 across availability zones in the region, and acks=all so a write is on multiple brokers before being acknowledged.
• Cross-region replication copies topics between regions (asynchronously) so the other region has the data if one fails.
• Partitioning by key (account_id) keeps per-key order within a partition.
• Schema registry for event formats and compatibility.

3) Design Choices

3.1 Active-passive vs active-active

• Active-passive: all producers write to Region A, and it's replicated to B. On failure, switch producers and consumers to B. Simpler ordering, but a failover step is needed, and events not yet replicated can be lost (RPO = replication lag).
• Active-active (our choice for availability): producers write to their local region. Each region has "local" topics plus "mirrored" topics from the other region. Consumers that need everything read both. Per-key ordering is kept by routing each key to a home region (e.g., an account's events always go to its home region's topic), with failover of the home region when needed.

3.2 Durability vs latency

• Synchronous cross-region writes (a "stretch cluster") give RPO = 0 but add cross-region latency (tens of ms) to every write. Use this only for critical events, like money movement.
• Asynchronous replication is faster, with a small RPO (seconds). State it and choose per topic.

4) Delivery Semantics

• At-least-once: consumers commit offsets after processing. Crashes lead to re-delivery.
• Deduplication: every event has a unique event_id. Consumers use idempotent writes (upsert by event_id) or a dedup store with a TTL.
• Exactly-once within Kafka (transactions) exists for read-process-write inside Kafka, but external side effects still need idempotency.

5) Region Failover

1. Region A goes down. Producers switch to B (clients configured with both endpoints, or through DNS/traffic management).
2. Consumers in B continue from their offsets. For the mirrored copy of A's topics, offsets differ between clusters, so use offset translation (checkpointing the mapping, as MirrorMaker 2 does) or resume by timestamp, accepting some duplicates (dedup handles them).
3. When A returns, reverse-sync, then fail back gradually.

6) Operations

• Monitor producer error rates, replication lag per topic (this is effectively your RPO), consumer lag per group, and broker health.
• Keep retention long enough for replays (e.g., 7 days), with tiered storage to object storage for longer.
• Run regular failover drills.

7) Wrap-Up

Run a replicated Kafka cluster per region (acks=all across zones), partition by key for ordering, and replicate topics across regions: asynchronously by default (small RPO) and synchronously only where zero loss is required. Prefer active-active with a home region per key. Deliver at-least-once with event IDs and idempotent consumers, handle failover with offset translation or timestamp resume, and monitor replication and consumer lag while regularly practicing failover.