Pub/Sub System with Replay (Seek Back in Time)

0) Problem Restatement

Google asked: design a pub/sub system that supports replay. Besides normal real-time delivery (publishers send, subscribers receive new messages), a subscriber must be able to rewind: "re-deliver everything since yesterday 9:00" (e.g., after fixing a bug that processed messages wrongly), or seek to a specific position. This is like Google Pub/Sub "seek" or Kafka's offset reset.

1) Key Idea: A Durable Log, Not a Queue

A classic queue deletes messages once acknowledged, so there's nothing to replay. Instead, store messages in an append-only log kept for a retention period (e.g., 7 days). Each subscriber just tracks its position (offset) in the log. Replay = move the position back.

2) Architecture

flowchart LR
    PUB["Publishers"] --> BR["Brokers - topic partitions"]
    BR --> LOG[("Partition logs - segments on disk")]
    LOG -->|"old segments"| TIER[("Tiered storage - object store")]
    BR --> TIDX[("Time index - timestamp to offset")]
    SUB["Subscribers / consumer groups"] -->|"fetch from offset"| BR
    SUB --> OFF[("Committed offsets per subscription")]
    ADM["Seek API"] --> OFF

• Topics → partitions: messages with the same key go to the same partition, which keeps order per key.
• Each partition is a log of segments (files). Each message has an offset and a timestamp.
• Time index per segment: maps timestamps to offsets, so "replay from 9:00" is a fast lookup.
• Subscriptions store committed offsets per partition.

3) Replay Flow

1. The operator calls seek(subscription, timestamp=yesterday 09:00) (or an offset or a snapshot).
2. For each partition, find the first offset with timestamp ≥ 09:00 using the time index (binary search), and set the subscription's position there.
3. Consumers resume fetching from the new position. Old segments may come from tiered storage (slower, but cheap).
4. Messages are re-delivered in order within each partition.

4) Important Details

• Retention: time-based (7 days) or size-based. Replay only works within retention, so choose retention per topic by business needs. Tiered storage keeps old segments in object storage cheaply, allowing weeks or months.
• Snapshots: save a subscription's positions (all partitions) under a name before a risky deploy, so you can later "seek to snapshot".
• Isolation: a big replay reads lots of old data. Serve historical reads from tiered storage or separate read paths, and apply per-subscription rate limits, so live consumers and publishers aren't slowed.
• Consumers must handle re-processing: replay re-delivers messages that may already have had effects. Consumers should be idempotent (dedupe by message ID, or upsert results), or write to a new output table and then switch.
• Ordering across partitions isn't guaranteed (and isn't needed for most use cases). Mention it.

5) Wrap-Up

Store every topic as partitioned, append-only logs with offsets, timestamps and a time index, keep them for a retention window (extended cheaply with tiered storage), and let each subscription track its own committed positions. Replay is a seek: map a timestamp or snapshot to per-partition offsets and resume fetching, in order per partition. Protect live traffic with read isolation and rate limits, and require idempotent consumers so reprocessing is safe.