0) Problem Restatement
Design a system (asked at Atlassian) that takes a high-throughput stream of events (e.g., product usage events), validates and transforms them, computes aggregations over time windows (e.g., active users per workspace per minute), and writes the results to downstream sinks (a database, dashboard store or another topic). It must handle out-of-order and late events, recover from crashes without losing or double-counting, and scale out.
1) Requirements
- Ingest 100K+ events/sec.
- Validate (schema, required fields), drop or quarantine bad events.
- Transform and enrich (e.g., add workspace plan from a lookup).
- Windowed aggregations (tumbling 1-minute, sliding 5-minute) per key.
- Output within seconds. Correct counts after crashes.
- Scale horizontally.
2) Architecture
Architecture Diagram
flowchart LR
SRC["Producers"] --> K[("Kafka - partitioned by key")]
K --> V["Validate + parse"]
V -->|"bad"| DLQ[("Quarantine topic")]
V --> E["Enrich - cached lookups"]
E --> W["Windowed aggregate - keyed state"]
W --> SINK[("Sink - DB / OLAP / topic")]
W --> CK[("Checkpoints - object storage")]Built on a stream framework like Flink or Kafka Streams, which provides keyed state, windows, watermarks and checkpoints.
3) Key Concepts (in simple words)
- Partitioning: events are keyed (e.g., by workspace_id). All events for a key go to the same partition and the same processing task, so per-key order holds and state is local.
- Event time vs processing time: event time = when it happened on the device, and processing time = when we see it. Aggregating by event time gives correct results even if events arrive late.
- Watermark: the processor's estimate that "all events up to time T have arrived" (e.g., the max seen event time minus 30 seconds). A window closes when the watermark passes its end. Later events can update the result (allowed lateness) or go to a side output.
- State: running counts per key and window, stored in the operator (e.g., RocksDB on local disk).
- Checkpointing: periodically snapshot all state plus the Kafka offsets together. After a crash, restore the snapshot and re-read from those offsets. No loss, no double counting inside the processor.
- Exactly-once to the sink: use idempotent upserts (key = window + key) or transactional sinks committed with checkpoints.
4) Flow Example
Event {workspace: w1, user: u9, ts: 12:00:42} → validated → enriched with plan = "premium" → added to the w1 window [12:00, 12:01) set of users → at watermark 12:01:30 the window emits {w1, 12:00, active_users: 57} → upsert into the metrics DB.
5) Scaling and Backpressure
- Parallelism = number of Kafka partitions per operator. Add partitions and tasks for more throughput.
- Hot keys (one giant workspace): pre-aggregate with a random sub-key, then combine.
- Backpressure: if the sink is slow, the framework slows reading from Kafka. Data waits safely in Kafka (lag grows) instead of being dropped. Alert on consumer lag.
- Enrichment lookups: cache them locally, or load the reference data as a broadcast stream, instead of calling a DB per event.
6) Operations
- Monitor lag, throughput, checkpoint duration, late-event counts and quarantine rates.
- Reprocessing: to fix a bug, deploy a new job version that reads from an earlier Kafka offset (or from the data lake), and writes to a new table version, then switch.
- Schema evolution: a schema registry with backward-compatible changes.
7) Wrap-Up
Read keyed events from Kafka, validate (quarantining bad ones), enrich with cached lookups, and aggregate in event-time windows closed by watermarks. Keep state in the operators with periodic checkpoints of state plus offsets for crash-safe exactly-once processing, write idempotently to sinks, scale by partitions (pre-aggregating hot keys), and rely on backpressure and lag monitoring to stay healthy.