Metrics Monitoring and Alerting System

0) Problem Restatement

Design an internal monitoring platform like Datadog or Prometheus. Every server and service sends numbers over time, called metrics: CPU usage, request count, error rate, latency. Engineers look at these on dashboards and get alerts when something goes wrong (for example, "error rate above 5% for 5 minutes").

Each metric has a name and tags (labels), such as http_requests{service=checkout, region=us-east, status=500}. The system must take in a huge number of data points, store them cheaply, and answer queries fast.

1) Requirements

1.1 Functional

• Collect metrics from agents on every host and from application libraries.
• Store time series and query them by name, tags and time range, with functions like sum, average and 99th percentile.
• Dashboards with charts that refresh automatically.
• Alert rules that notify on-call engineers through PagerDuty, Slack or email.

1.2 Non-Functional

• Ingest a lot: millions of data points per second.
• Fast queries: a dashboard over the last hour should load in under a second.
• Very reliable: when production is on fire, monitoring must still work.
• Cheap long-term storage: keep data for months, just with less detail.

1.3 Scale Estimates

• 100,000 hosts × 500 metrics each, sent every 10 seconds → 5 million data points/sec.
• Each point is a timestamp + value (16 bytes raw). Time-series compression brings it down to about 1.5 bytes, which is about 650 GB/day.
• Retention: raw 10-second data for 15 days, 1-minute rollups for 3 months, 1-hour rollups for 2 years.

1.4 API Design

• POST /v1/metrics (from agents, batched): [{ name, tags, timestamp, value }, ...]
• GET /v1/query?q=avg(cpu{service=checkout}) by (region)&from=-1h&step=60s
• POST /v1/alerts with { query, condition: "> 0.05", for: "5m", notify: ["pagerduty:checkout"] }

2) High-Level Architecture

2.1 Overview

• Agent on each host: collects metrics, adds up counters locally for 10 seconds, and sends batches. It buffers on disk if the backend is unreachable.
• Ingestion gateway: authenticates and validates data, then writes it to Kafka.
• Kafka: a buffer that protects storage from spikes and lets several consumers read the same data.
• Time-series DB (TSDB): stores the data, sharded by series. Examples include Prometheus/Thanos, M3, VictoriaMetrics and InfluxDB.
• Rollup jobs: build the 1-minute and 1-hour summaries.
• Query service: reads from the TSDB shards and merges the results.
• Alert evaluator: runs every alert query on a schedule and sends notifications.

2.2 Architecture Diagram

flowchart LR
    A["Agents on hosts"] --> G["Ingestion Gateway"]
    G --> K[("Kafka")]
    K --> W["TSDB Writers"]
    W --> T[("Time-series DB - sharded")]
    K --> RU["Rollup jobs"]
    RU --> CS[("Long-term store - object storage")]
    D["Dashboards"] --> Q["Query Service"]
    Q --> T
    Q --> CS
    AE["Alert Evaluator"] --> Q
    AE --> N["Notifier - PagerDuty, Slack"]

3) Data Model

A series = metric name + a unique set of tags. Each series gets an ID, and its points are stored together in time order.

series index:  series_id 981 → cpu_usage{host=web-12, service=checkout, region=us-east}
inverted index: service=checkout → [981, 982, 1044, ...]   (to find series by tag)
data blocks:   series 981, 12:00–14:00 → compressed [(t, v), (t, v), ...]

4) Key Flows

4.1 Write path

1. The agent sends a batch every 10 seconds.
2. The gateway puts it on Kafka, partitioned by hash(series).
3. TSDB writers append points to an in-memory block for 2 hours and write a write-ahead log (a file we append to first, so nothing is lost if the writer crashes). Then they flush a compressed block to disk or object storage.

4.2 Query path

1. The query service uses the inverted index to find matching series (service=checkout).
2. It fetches blocks from the right shards: recent data from memory, older data from disk or rollups.
3. It computes the aggregation (e.g., average by region) and returns points for the chart.

4.3 Alert path

Every 30–60 seconds, the evaluator runs each rule's query. If the condition is true for the whole for period, it fires once and sends a single notification (not one per evaluation). When the condition clears, it sends "resolved".

5) Deep Dive A — High cardinality

"Cardinality" means the number of unique series. If someone adds user_id as a tag, one metric can turn into 100 million series and crash the index.

• Set limits per metric and per team (e.g., 100K series), and reject or drop tags beyond them.
• Show teams their cardinality usage so they can fix it.
• Guide people to use logs or traces for per-user data, not metrics.

6) Deep Dive B — Keeping monitoring up at 99.99%

• Separate failure domains: monitoring must not share databases or clusters with the systems it watches. Otherwise the outage it should report also takes monitoring down.
• Replicate writes to 2 TSDB replicas. Queries can use either one.
• Agents buffer locally so a short backend outage creates a gap that fills in later instead of losing data.
• Meta-monitoring: a tiny, separate system watches the monitoring system ("no data from region X for 2 minutes" is itself an alert).
• Old clients with different metric names: map known naming variants to one canonical name at ingestion, using a rename table, so dashboards stay correct.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you compute p99 latency across 100 hosts?" You cannot average percentiles. Send histograms (counts per latency bucket), add the buckets together, then compute p99 from the merged histogram.
• "How do you avoid alert spam?" Group alerts (one page per service, not per host), add a for duration, and silence alerts during planned maintenance.
• "Multi-tenant?" Add a tenant ID to every series, enforce per-tenant limits, and keep query costs isolated.

9) Wrap-Up

Agents batch metrics and send them through a gateway into Kafka. Writers store compressed time series in a sharded TSDB, and rollup jobs keep cheap long-term summaries. A query service answers dashboards, and an alert evaluator runs rules on a schedule. Control cardinality, use histograms for percentiles, and keep monitoring in its own failure domain so it works when everything else is broken.