0) Problem Restatement
Design a platform (asked at Airbnb) that lets product teams run experiments. A team defines an experiment ("new checkout button"), with variants (control and treatment) and who is eligible. Users are consistently assigned to a variant (the same user always sees the same one). The app shows the assigned variant, the platform logs exposures (the user actually saw it) and outcomes (bookings, clicks), and then computes results with statistical significance.
1) Requirements
- Create, start, ramp (1% → 50%), stop and roll out experiments.
- Targeting: country, platform, new vs existing users.
- Consistent, fast assignment in services and apps.
- Log exposures and outcomes, and compute metrics per variant with confidence intervals.
- Many concurrent experiments without interfering with each other.
- Guardrails: auto-alert or stop if key metrics (errors, revenue) get worse.
2) Architecture
Architecture Diagram
flowchart LR
UI["Experiment UI"] --> CFG["Config Service"]
CFG --> DB[("Experiment configs")]
CFG -->|"push config"| SDK["Assignment SDK in services / apps"]
SDK -->|"exposure events"| K[("Kafka")]
APP["Product events - bookings, clicks"] --> K
K --> LAKE[("Data lake")]
LAKE --> MET["Metrics pipeline - join exposures + outcomes"]
MET --> STAT["Stats engine"]
STAT --> DASH["Results dashboard"]
STAT -->|"guardrail breach"| ALERT["Alerts / auto-stop"]3) Assignment (the core idea)
bucket = hash(experiment_salt + user_id) % 1000.- The config says: buckets 0–499 = control, 500–999 = treatment (for a 50/50 split at 100% traffic).
- Deterministic: the same user always gets the same bucket, and no database lookup is needed.
- Local evaluation: the SDK has the config in memory and assigns in microseconds, with no network call per request.
- Ramp-up: first expose only buckets 0–9 (1% of users). Increasing to 50% adds buckets, and users already in stay in (no switching).
- Layers: experiments in different layers use different salts, so they're independent. Experiments that might conflict (two tests changing the same button) go in the same layer and get non-overlapping buckets.
4) Logging and Metrics
- Log exposure only when the user actually sees the variant (not just when assigned), otherwise results are diluted.
- The event contains
{ user_id, experiment_id, variant, ts }. Deduplicate to the first exposure per user. - The metrics pipeline (daily plus intraday) joins exposures with outcome events after the exposure time, per user. Then it aggregates per variant: conversion rate, revenue per user, etc.
- Metric definitions are shared and reviewed, so every experiment computes "booking rate" the same way.
5) Statistics (explain simply)
- Compare treatment vs control with a t-test or z-test and report the difference with a 95% confidence interval. If the interval doesn't include 0, the result is significant.
- Power / sample size: decide beforehand how many users are needed to detect the effect you care about (e.g., +1% booking rate).
- Variance reduction (e.g., CUPED): using each user's pre-experiment behavior makes results significant faster.
- Pitfalls to mention:
- Peeking: checking every day and stopping when it looks good inflates false positives. Use fixed durations or sequential testing methods.
- Sample ratio mismatch (SRM): if a 50/50 test has 52/48 users, something is broken (e.g., a bug dropping exposures). Check automatically and block results.
- Novelty effects: run at least 1–2 weeks to cover weekly patterns.
- Multiple metrics: correct for many comparisons, or pick one primary metric up front.
6) Trade-offs & Alternatives
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Assignment | Hash-based, local SDK | Consistent, zero latency | Assignment service call: latency, dependency |
| Unit | User ID (or device before login) | Consistent experience | Per request: inconsistent, noisy |
| Isolation | Layers with salts | Many experiments at once | One at a time: too slow for many teams |
| Analysis | Batch pipeline + stats engine | Accurate, auditable | Live counters only: no proper stats |
7) Wrap-Up
Assign users deterministically with hash(salt + user_id) into buckets, evaluated locally by an SDK that receives pushed configs, with layers for independent experiments and bucket-based ramp-up that never reshuffles users. Log real exposures, join them with outcomes in a metrics pipeline using shared metric definitions, and report differences with confidence intervals, while guarding against peeking, sample ratio mismatch and harm to guardrail metrics.