User Behavior Tracking (Clickstream Analytics)

0) Problem Restatement

Design a system that records what users do in web and mobile apps: page_view, click, app_install, add_to_cart, purchase. Product teams across several products use it to answer questions like "how many daily active users do we have?", "where do users drop off between signup and purchase?" (a funnel), or simply "how many people viewed this product page?"

Asked many times at Rippling, and at Uber as "product page view tracking".

1) Requirements

1.1 Functional

• A client SDK to send events with properties (user, session, device, page, product ID).
• Collect events from many products with a shared schema.
• Real-time counts (e.g., views of a product page in the last hour) and historical analysis (DAU, retention, funnels).
• Dashboards and ad-hoc queries.

1.2 Non-Functional

• High ingest throughput and no slowdown for the app itself.
• Low data loss: a small loss (under 0.1%) is acceptable for analytics, but not large gaps.
• Freshness: real-time panels within ~1 minute, and batch reports within hours.
• Privacy compliance: consent, and deleting a user's data on request (GDPR).

1.3 Scale Estimates

• 50M daily users × 100 events = 5B events/day ≈ 60K/sec, peak ~200K/sec.
• ~500 bytes each → 2.5 TB/day raw, less after columnar compression (~5–10x).

1.4 API Design

• SDK: track("add_to_cart", { product_id: 991, price: 1299 }) → batched.
• Collector: POST /v1/events with [{ event_id, event_name, user_id, anonymous_id, session_id, ts, properties }]
• Query: GET /v1/metrics/page-views?product_id=991&window=1h and SQL access for analysts.

2) High-Level Architecture

2.1 Overview

• Client SDK: batches events (every 10 seconds or 50 events), stores them locally when offline, retries with backoff, and attaches a unique event_id.
• Collector: a thin, stateless HTTP service that validates, adds server time and IP-based geo, and writes to Kafka.
• Schema registry: defines allowed events and properties per product. Unknown or bad events go to a "quarantine" topic instead of polluting data.
• Stream processing (Flink): real-time counters (page views per product per minute) → Redis or an OLAP store.
• Data lake (S3 + Parquet files): all events, partitioned by date and event name.
• Warehouse / OLAP (BigQuery, Snowflake, ClickHouse): DAU, funnels and retention via batch jobs and ad-hoc SQL.

2.2 Architecture Diagram

flowchart LR
    APP["Web / Mobile SDK - batch, retry"] --> COL["Collectors"]
    COL --> SR["Schema check"]
    SR --> K[("Kafka - events")]
    SR -->|"invalid"| QU[("Quarantine topic")]
    K --> FL["Stream jobs - real-time counts"]
    FL --> RT[("Redis / OLAP - live metrics")]
    K --> LAKE[("Data lake - Parquet by date")]
    LAKE --> ETL["Batch jobs - DAU, funnels"]
    ETL --> WH[("Warehouse")]
    DASH["Dashboards"] --> RT
    DASH --> WH

3) Data Model

Event (common envelope):
  event_id (UUID), event_name, product, user_id (nullable), anonymous_id, session_id,
  client_ts, server_ts, platform, app_version, geo, properties (JSON)

Data lake layout:
  s3://events/product=checkout/event_name=page_view/date=2026-09-19/hour=10/part-*.parquet

Partitioning by product, event and date means a query like "page views yesterday" reads only a small slice.

4) Key Flows

4.1 Tracking an event

1. The app calls track(). The SDK adds event_id, timestamps and session info, and queues the event locally.
2. It flushes a batch to the collector. On failure, it keeps the batch and retries later (bounded by disk space).
3. The collector validates the batch against the schema and writes to Kafka, then returns 200.

4.2 Product page view count

Flink reads page_view events, deduplicates by event_id, and counts per product_id per minute. It writes rolling totals to Redis. The product page reads "1,203 people viewed this in the last hour" from Redis (cached).

4.3 Daily metrics

Hourly or daily jobs build clean tables (sessions, daily active users, funnel steps) from the data lake into the warehouse. Dashboards query those tables.

5) Deep Dive A — Data quality

• Duplicates: retries send the same batch twice. Deduplicate by event_id in streaming (a short window) and in batch (exact).
• Clock skew: phone clocks are wrong. Keep both client_ts (event order within a session) and server_ts (trusted time), and use server time for daily partitioning.
• Late events: offline phones upload hours later. Batch jobs reprocess the last 2–3 days so late events are counted.
• Schema drift: teams rename properties. The schema registry enforces versions and rejects breaking changes.
• Identity: before login, events have only anonymous_id. On login, link anonymous_id → user_id so funnels span the whole journey.

6) Deep Dive B — Privacy and cost

• Consent: the SDK doesn't send (or sends only strictly necessary) events until the user consents.
• PII: don't put emails or names in properties. Hash or drop them at the collector.
• Deletion requests: keep a user → partitions index, or rewrite affected partitions in the lake periodically to remove a deleted user's events. Warehouse tables are rebuilt from the cleaned data.
• Cost: keep raw data for 13 months and aggregates longer, sample very high-volume low-value events (e.g., scroll events at 10%), and compress with columnar formats.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you compute a funnel?" For each user, order their events by time and check whether they did step 1 → step 2 → step 3 within a window (e.g., 1 day). Warehouses have functions for this, or you can precompute it daily.
• "How do you count unique viewers cheaply?" Use HyperLogLog sketches per page per hour. They merge across hours with about 1% error.
• "Ad blockers drop events?" Use a first-party collector domain, and accept some loss for web analytics.

9) Wrap-Up

A client SDK batches and retries events with unique IDs, collectors validate them against a schema registry and write to Kafka. Stream jobs produce real-time counters, like page views per product, while everything lands in a partitioned data lake that batch jobs turn into DAU, funnel and retention tables. Deduplicate by event ID, reprocess late data, and build in consent, PII handling and deletion from the start.