Data Engineering for a Short-Video App (TikTok-scale)

0) Problem Restatement

Apple asked a data engineer to design the data systems behind a TikTok-like app. Billions of interaction events every day (views, watch time, likes, shares, comments, follows) must be ingested, processed in streaming and batch, and turned into data products: recommendation features, creator analytics, business dashboards and ad reporting. The focus: scalable ingestion, processing orchestration, partitioning, and failure handling.

1) Requirements

• Ingest ~50B events/day (~600K/sec average, much more at peaks).
• Real-time aggregates (views per video in the last minute, trending) within seconds.
• Daily and hourly tables for analytics (DAU, watch time, creator stats) with correct numbers.
• Features for recommendation models (fresh and historical).
• Handle late and duplicate events, backfills, and data quality problems.

2) Architecture

flowchart LR
    APP["Apps + backend services"] --> COL["Collectors"]
    COL --> K[("Kafka - events by type")]
    K --> FL["Flink - real-time aggregates, features"]
    FL --> RT[("Real-time stores - Redis, OLAP")]
    FL --> FS[("Online feature store")]
    K --> RAW[("Lakehouse raw - Iceberg/Delta tables")]
    RAW --> ORCH["Orchestrator - Airflow DAGs"]
    ORCH --> SPK["Spark batch - clean, dedupe, model"]
    SPK --> CUR[("Curated tables - facts, dimensions")]
    CUR --> WH["Query engines - Trino / Spark SQL"]
    CUR --> OFF[("Offline features / training sets")]
    WH --> DASH["Dashboards, creator analytics"]
    DQ["Data quality checks"] --> CUR

3) Layers (the "medallion" idea, explained simply)

• Raw (bronze): events exactly as received, partitioned by date/hour/event_type, stored in columnar files (Parquet) in a table format (Iceberg/Delta) that supports schema changes, time travel and safe rewrites.
• Clean (silver): deduplicated (by event_id), validated, enriched (user country, video metadata), with bad rows quarantined.
• Curated (gold): business tables such as fact_video_views_daily (video_id, date, views, watch_seconds, unique_viewers_hll), dim_video, dim_creator, and daily active users.

4) Streaming Path

• Flink jobs read Kafka and compute windowed aggregates (views per video per minute, trending scores) and real-time features (a user's last 50 interactions, a video's recent completion rate) into the online feature store for recommendations.
• Checkpointing gives exactly-once state within Flink, and outputs are idempotent (upserts).

5) Batch Path and Orchestration

• Airflow DAGs run hourly and daily: raw → clean → curated → aggregates → exports. Each task depends on its inputs being complete (sensors on partitions).
• Late data: re-process the last 2–3 days' partitions every run (overwrite the partition), so late events are included.
• Backfills: when logic changes, re-run the DAG for past dates. Idempotent partition overwrites make this safe.
• Partitioning and file sizes: partition by date (and hour for big tables), and compact small files into ~512 MB files, since many tiny files make queries slow.

6) Data Quality and Reliability

• Checks on every curated table: row counts vs yesterday, null rates, uniqueness of keys, and value ranges. Failures stop downstream tasks and alert the owner.
• Data contracts with producer teams (schema registry, and no breaking changes without a version bump).
• SLAs: e.g., "daily creator stats ready by 06:00 UTC", with lag monitoring.
• Cost: tiered storage, TTLs on raw data (e.g., 90 days raw, curated longer), and sampling for exploratory data.

7) Wrap-Up

Ingest events through collectors into Kafka. A Flink streaming path produces real-time aggregates and online features, while the raw stream lands in a lakehouse (Iceberg/Delta) that Airflow-orchestrated Spark jobs refine from raw to clean to curated tables. Partition by date, compact files, re-process recent partitions for late data, backfill idempotently, and gate every table with data-quality checks and SLAs for the dashboards, analytics and recommendation models downstream.