Analytics Dashboard for an AI Chat Product

0) Problem Restatement

Design the backend of an internal analytics system for a ChatGPT-like product (asked at Salesforce). Product managers and engineers want a dashboard showing things like:

• daily and monthly active users (DAU/MAU), conversations and messages per day,
• response latency percentiles (time to first token, total time),
• token usage and cost per model, errors and timeouts,
• user feedback (thumbs up or down),
• all sliced by time range, model, region, plan and app version.

The dashboard UI is out of scope. Focus on instrumentation, pipelines, storage and query serving.

1) Requirements

• Collect events from the chat service (and clients).
• Near-real-time panels (errors, latency, traffic) within ~1–2 minutes.
• Historical analysis (trends, retention, cost) with ad-hoc filters.
• Consistent metric definitions ("an active user is someone who sent at least one message").
• Privacy: no raw conversation text in analytics by default.

1.1 Scale Estimates

• 20M messages/day → with request, response and feedback events, about 100M events/day (~1.2K/sec average, 10K/sec peak). This is moderate volume, and the challenge is flexibility and trust in the numbers.

2) What to Log (instrumentation)

Define a small set of well-structured events:

message_sent      { event_id, ts, user_id(hashed), conversation_id, plan, region, app_version, client }
response_complete { event_id, ts, conversation_id, message_id, model, input_tokens, output_tokens,
                    ttft_ms, total_ms, status (ok/error/timeout/filtered), gpu_cluster }
feedback          { event_id, ts, message_id, rating (up/down), reason_code }
session_start     { event_id, ts, user_id(hashed), client, app_version }

• The chat backend emits response_complete, because server-side timing and token counts are the trustworthy ones.
• No prompt or response text. Optionally keep a small, consented, sampled dataset in a separate locked-down store for quality review.

3) Architecture

flowchart LR
    CS["Chat service + clients"] --> K[("Kafka - analytics events")]
    K --> ST["Stream jobs - 1-min rollups"]
    ST --> OL[("OLAP store - ClickHouse / Druid")]
    K --> LAKE[("Data lake - raw events")]
    LAKE --> BATCH["Daily batch - DAU/MAU, retention, cost"]
    BATCH --> WH[("Warehouse tables")]
    ML["Metrics layer - shared definitions"] --> OL
    ML --> WH
    DASH["Dashboard"] --> QS["Query service + cache"]
    QS --> ML

• Real-time path: stream jobs aggregate per minute per (model, region, plan): message counts, error counts, latency histograms and token sums → OLAP store.
• Batch path: daily jobs compute exact DAU/MAU (distinct users), retention cohorts and cost per model (tokens × price, or GPU-hours) → warehouse.
• Metrics layer: one place that defines each metric (the SQL and filters), used by both the dashboard and analysts, so numbers match everywhere.
• Query service: translates dashboard filters into queries, and caches popular panels for 1 minute.

4) Key Metrics and How to Compute Them

• Latency p50/p95/p99: store histograms per minute (counts per latency bucket), not averages. Percentiles are computed from merged histograms for any time range and filter.
• DAU/MAU: exact distinct counts in batch. For real-time "active users today", use HyperLogLog sketches (about 1% error).
• Tokens and cost: sum tokens per model, and multiply by a price table (versioned, since prices change).
• Satisfaction: thumbs-up rate = ups / (ups + downs), shown with the number of ratings so small samples aren't misread.
• Error rate: errors / total responses, split by error type (timeout, overloaded, safety filter).

5) Data Quality and Freshness

• Deduplicate by event_id. Late events are included by re-running recent partitions.
• Checks: row counts vs expected, null rates, and sudden drops (a broken client release that stops sending events). Alert the data team.
• Show "data as of HH:MM" on every panel, so people know how fresh the numbers are.
• Backfills: when a metric definition changes, recompute history from the raw data lake.

6) Trade-offs & Alternatives

7) Wrap-Up

Emit a small set of structured, content-free events (message, response with tokens and latency, feedback) into Kafka. Build minute-level rollups with latency histograms in an OLAP store for live panels, and compute exact DAU/MAU, retention and cost in daily batch jobs from a raw data lake. Put a shared metrics layer in front so every dashboard uses the same definitions, and add dedup, quality checks and "data as of" freshness labels.