Ads Audience Targeting System

0) Problem Restatement

Advertisers want to show ads to specific audiences: e.g., "users in the US who watched sci-fi in the last 30 days", "people on our customer list" (uploaded by the advertiser, matched privacy-safely), or "users similar to our best customers" (lookalikes). Netflix asked this. At ad-serving time, for each request, the system must quickly find which campaigns target this user, which means knowing which segments the user belongs to, in a few milliseconds.

1) Requirements

• Define segments: rule-based (behavior, demographics, geography), advertiser lists, and lookalikes.
• Compute and refresh membership (daily for most, near real time for some).
• At serving time: get_segments(user_id) → the list of segment IDs, and then which campaigns target those segments (include and exclude rules).
• Estimate audience size when an advertiser builds a segment ("~2.3M users").
• Respect privacy and consent (opt-outs, data retention, minimum audience sizes).

1.1 Scale Estimates

• 200M users, 50K active segments. The average user is in ~200 segments.
• Ad requests: 200K/sec → 200K membership lookups/sec at under ~3 ms.

2) Architecture

flowchart LR
    EV["User events + profiles"] --> LAKE[("Data lake")]
    ADV["Advertiser lists (hashed)"] --> MATCH["Privacy-safe matching"]
    LAKE --> SEG["Batch segment builder - Spark"]
    MATCH --> SEG
    K[("Real-time events")] --> RTS["Streaming segment updater"]
    SEG --> MEM[("Membership store - user to segments")]
    RTS --> MEM
    SEG --> BM[("Segment bitmaps - sizes, overlaps")]
    AS["Ad server"] -->|"get_segments(user)"| MEM
    AS --> IDX["Campaign index: segment to campaigns"]
    UI["Audience builder UI"] --> BM

3) Membership Storage (the key decision)

Two views of the same data, each optimized for one question:

• User → segments (for serving): a key-value store (e.g., Aerospike/Redis/DynamoDB) keyed by user_id, with a compact list of segment IDs (sorted ints, delta-encoded). One lookup per ad request. 200M users × ~200 segments × 2 bytes ≈ 80 GB, which fits in a memory-heavy cluster.
• Segment → users (for sizing, overlaps and building): compressed bitmaps (Roaring bitmaps) per segment, with user IDs mapped to integers. Size = bitmap cardinality. "Sci-fi AND US AND NOT existing-customers" is a bitmap AND/ANDNOT, taking milliseconds, even for 100M users.

4) Building Segments

• Batch (daily): Spark jobs evaluate each rule-based segment over the data lake, produce bitmaps, then invert them into per-user lists and bulk-load the serving store (write a new version, then switch, so nothing is half-updated).
• Streaming (for fast segments like "viewed a car ad in the last hour"): a stream processor updates membership for affected users within seconds, with a TTL.
• Advertiser lists: the advertiser uploads hashed emails or phone numbers. We match them to our users with the same hashing (in a secure environment), and never reveal which users matched. Enforce minimum sizes (e.g., at least 1,000 matched) to prevent targeting individuals.
• Lookalikes: train a model or use embeddings to find users similar to a seed segment, and take the top N by score.

5) Serving: Matching Campaigns

1. The ad request arrives with user_id → fetch the user's segment list (one KV read, cached locally for a few minutes for repeat requests).
2. A campaign index maps segment_id → campaigns that include it, plus exclusion rules. Candidate campaigns = the union over the user's segments, minus those whose exclusion segments the user is in.
3. Combine with other targeting (geo, device, time), frequency caps and pacing, then run the auction or selection.

6) Privacy and Correctness

• Consent: users who opted out of personalized ads are excluded at build time and checked at serve time.
• Deletion: removing a user removes them from all segments at the next build, with streaming removal for opt-outs.
• Freshness labels: each segment has a "last built" time, so advertisers know how fresh it is.

7) Trade-offs & Alternatives

8) Wrap-Up

Keep two views of membership: a per-user segment list in a fast key-value store for ad serving, and compressed bitmaps per segment for sizing and boolean combinations. Build most segments in daily batch jobs (swapped in atomically), update fast segments with streaming and TTLs, and match advertiser lists privately with minimum audience sizes. At serve time, fetch the user's segments once and use a segment → campaign index (with exclusions) to find eligible campaigns, respecting consent throughout.