IoT / Sensor Telemetry Pipeline

0) Problem Restatement

Design a system that collects readings from millions of devices, such as temperature sensors in stores, GPUs in a data center, or smart meters. Each device sends a reading every few seconds. Users want a dashboard that shows the current values in near real time, and charts of history over days or months.

Two things make this harder than it sounds:

• Devices go offline. When they come back, they upload a batch of old readings, so data arrives late and out of order.
• Sometimes the backend must ask a device to do something, such as "send detailed logs for the last hour".

1) Requirements

1.1 Functional

• Ingest readings: { device_id, metric, value, timestamp }.
• Show the latest value per device (live dashboard).
• Show history charts for any time range.
• Alerts, e.g., "freezer temperature above -10°C for 5 minutes".
• Send commands to devices (collect logs, change sampling rate).

1.2 Non-Functional

• Scale: millions of devices.
• No data loss, even for readings that arrive days late.
• Live view within a few seconds; history queries in about a second.
• Secure: only real devices can send data.

1.3 Scale Estimates

• 5 million sensors × 1 reading every 10 seconds = 500,000 readings/sec.
• Each reading ≈ 50 bytes → 25 MB/sec, about 2 TB/day raw. Time-series compression brings that down about 10x.
• Keep raw data for 30 days, 1-minute averages for 1 year, and hourly averages forever.

1.4 API Design

• Device → cloud: MQTT publish to telemetry/{device_id} or HTTPS POST /v1/telemetry with a batch of readings.
• Dashboard: GET /v1/devices/{id}/latest, GET /v1/devices/{id}/series?metric=temp&from=&to=&step=1m.
• Cloud → device: POST /v1/devices/{id}/commands { type: "collect_logs", params }.

MQTT is a lightweight messaging protocol made for small devices on unreliable networks.

2) High-Level Architecture

2.1 Overview

• Device gateway (MQTT broker or HTTPS endpoint): authenticates each device with its own certificate and accepts batches.
• Kafka: buffers all readings, partitioned by device_id, so one device's data stays in order.
• Stream processor: validates readings, drops duplicates, updates the "latest value" store, and checks alert rules.
• Latest-value store (Redis): the current reading per device, for the live dashboard.
• Time-series DB (TimescaleDB, InfluxDB, or Cassandra with time buckets): stores history.
• Rollup jobs: compute 1-minute and 1-hour averages.
• Command service: stores commands and delivers them when the device is online.

2.2 Architecture Diagram

flowchart LR
    D["Devices / Sensors"] -->|"MQTT / HTTPS batches"| GW["Device Gateway - cert auth"]
    GW --> K[("Kafka - by device_id")]
    K --> SP["Stream processor - validate, dedupe"]
    SP --> R[("Redis - latest value")]
    SP --> TS[("Time-series DB")]
    SP --> AL["Alert engine"]
    TS --> RU["Rollups 1m / 1h"]
    UI["Dashboard"] --> API["Query API"]
    API --> R
    API --> TS
    CMD["Command Service"] -->|"deliver when online"| GW

3) Data Model

Time-series table (partitioned by day, clustered by device):
  device_id, metric, ts, value
  primary key ((device_id, day), metric, ts)

Latest value (Redis hash):
  latest:{device_id} → { temp: -18.2, ts: 1726740000 }

Commands:
  command_id, device_id, type, params, status (pending/sent/done), created_at

The (device_id, ts) key makes each reading unique, so writing the same reading twice just overwrites it. Deduplication comes for free.

4) Key Flows

4.1 Normal reading

1. The device sends a batch every 10–60 seconds (batching saves battery and network).
2. The gateway checks the device certificate and puts the batch on Kafka.
3. The processor writes readings to the time-series DB and updates Redis only if the reading is newer than the stored one.

4.2 Device was offline for 6 hours

1. The device stored readings locally and now uploads them in batches, oldest first.
2. The pipeline writes them into the correct past time slots. Nothing special is needed, since each row has its own timestamp.
3. Redis is not overwritten with old values (the "only if newer" check).
4. Rollups for those past hours are marked "dirty" and recomputed.

4.3 Sending a command

The command is saved as pending. When the device connects (or already is), the gateway pushes it. The device replies "done" with a result. If the device never comes back, the command expires.

5) Deep Dive A — Late data and alerts

• For live alerts, use event time (when the reading was taken), not arrival time. A watermark lets the processor wait a short time (e.g., 30 seconds) for stragglers before deciding "the average for 12:00–12:05 is final".
• Readings that arrive hours late should not trigger old alerts. Mark them late and send them only to storage and rollup recomputation.
• A device that has been silent for longer than expected is itself an alert ("sensor offline").

6) Deep Dive B — Storage and cost

• Time buckets: partition data by device and day. A query for "device X, last week" reads 7 small partitions.
• Downsampling: after 30 days, keep only 1-minute averages (with min and max so spikes are not hidden).
• Compression: time-series stores keep only the difference from the previous value, which works very well for slowly changing sensor data.
• Bad data: validate ranges (a freezer cannot be +500°C). Quarantine obviously wrong data, often caused by firmware bugs, instead of storing it.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "What if all devices reconnect at once after an outage?" Kafka absorbs the burst. Devices also wait a random delay (jitter) before uploading, so they don't all send at the same second.
• "How do you secure devices?" Each device gets a unique certificate at the factory. If one is stolen, revoke that one certificate only.
• "Real-time and batch analytics?" Send the Kafka stream to both the stream processor (live) and a data lake (batch jobs such as monthly reports).

9) Wrap-Up

Authenticate each device, buffer batched readings in Kafka by device, and store them in a time-series DB keyed by device and timestamp, which also removes duplicates. Keep the latest value in Redis, handle late uploads with event-time processing and rollup recomputation, downsample old data, and deliver commands through the same gateway when devices are online.