Centralized Logging System (ELK / Splunk)

0) Problem Restatement

Design a platform where thousands of services send their logs to one place, and engineers can search them within seconds. For example: "show all ERROR logs from the checkout service in the last 15 minutes that contain 'timeout'". Logs must not be lost when there are traffic spikes, old logs should be kept cheaply, and access to sensitive logs must be controlled.

1) Requirements

1.1 Functional

• Collect logs from apps, containers and hosts.
• Parse logs into fields (time, level, service, message, trace ID).
• Search by time range, fields and keywords, and tail logs live.
• Keep logs for a set period (e.g., 7 days fast, 90 days cheap, 1 year archive).
• Per-team access control.

1.2 Non-Functional

• Durable: no lost logs, even during spikes.
• Fresh: searchable within ~10 seconds.
• Fast search for recent data (seconds).
• Cost-efficient: log volume is huge.

1.3 Scale Estimates

• 10,000 services, 2 million log lines/sec at peak.
• 300 bytes per line → about 50 TB/day raw. With compression (~10x), about 5 TB/day stored.
• 7 days hot = 35 TB on fast disks. 90 days in object storage.

1.4 API Design

• Agents send batches: POST /v1/logs (or the Kafka protocol directly).
• Search: GET /v1/search?q=service:checkout AND level:ERROR AND "timeout"&from=-15m&limit=500
• Live tail: a WebSocket stream for a query.

2) High-Level Architecture

2.1 Overview

• Agents (Fluent Bit, Vector) on each host: read log files, batch them, compress them, and send them. They buffer on local disk if the backend is slow.
• Kafka: a large durable buffer. If indexing falls behind, logs wait here instead of being dropped.
• Processors: parse text into fields, add metadata (host, region, team), and mask secrets such as passwords or card numbers.
• Indexers: write logs into a search engine (Elasticsearch/OpenSearch), using one index per day per tenant.
• Object storage: cheap long-term storage for older logs.
• Query service: searches hot indexes, and older data when asked.

2.2 Architecture Diagram

flowchart LR
    S["Services + Agents"] --> K[("Kafka - buffered")]
    K --> P["Parse, enrich, mask secrets"]
    P --> IX["Indexers"]
    IX --> ES[("Search cluster - hot, 7 days")]
    P --> OS[("Object storage - compressed, 90 days+")]
    ES -->|"age out"| OS
    U["Engineers / UI"] --> Q["Query Service"]
    Q --> ES
    Q --> OS

3) Data Model

Each log becomes a document:

{ "ts": "2026-09-19T10:15:02.331Z", "level": "ERROR", "service": "checkout",
  "host": "web-17", "region": "us-east", "trace_id": "ab12...", "message": "payment timeout after 3000ms" }

• Indexes are split by time (e.g., logs-checkout-2026.09.19). Most searches are about recent time, so we only search a few indexes, and deleting old data just means dropping a whole index.
• Fields like service and level are exact-match (keyword) fields. message is full-text indexed.

4) Key Flows

4.1 Ingest

1. The agent reads new lines, batches them for up to 1 second or 1 MB, compresses the batch, and sends it.
2. If Kafka is unreachable, the agent writes to a local disk buffer and retries.
3. Processors turn raw text into fields (JSON logs are easy, and plain text uses patterns), then send documents to indexers and a compressed copy to object storage.

4.2 Search

1. The query service picks the indexes that match the time range and the user's allowed teams.
2. It runs the query on all shards in parallel and merges the newest results first.
3. For data older than 7 days, it scans the compressed files in object storage. That is slower but cheap.

5) Deep Dive A — Surviving spikes (backpressure)

During an incident, error logs can jump 10x, exactly when people need them most.

• Kafka absorbs the burst. Indexers catch up later, so logs are delayed, not lost.
• Per-service quotas: a noisy service that logs in a loop is sampled or capped so it can't slow down everyone else. Its logs are still kept in object storage.
• Agents apply backpressure: they slow down and buffer instead of using all network bandwidth.

6) Deep Dive B — Cost control

Full-text indexing everything is expensive (the index can be as big as the data).

• Tiering: hot (fast SSD, 7 days) → warm (cheaper disks, 30 days) → cold (object storage, months).
• Index less: index only key fields and store the message compressed. This is the Grafana Loki approach: much cheaper, with slower keyword search.
• Drop noise: filter DEBUG logs in production, and sample repetitive INFO logs.
• Retention per team: teams pay for (or justify) longer retention.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you link logs to a request?" Put a trace_id in every log line, so one search shows the whole journey of a request across services.
• "How do you protect sensitive data?" Mask secrets at the processor, restrict indexes by team, and audit who searched what.
• "Alerts on logs?" Run saved searches every minute (e.g., more than 100 "payment failed" in 5 minutes), or turn log patterns into metrics.

9) Wrap-Up

Agents batch and buffer logs into Kafka, processors parse, enrich and mask them, and indexers write time-based indexes for fast recent search, with compressed copies in object storage for cheap retention. Kafka and per-service quotas protect the system during spikes, and tiering plus selective indexing keeps costs under control.