Publisher Configuration Rules and Safe Global Rollout

0) Problem Restatement

Netflix asked about the supply side of an ad platform: publishers own many websites, apps, channels and ad placements, and each needs custom configuration rules, e.g.:

• blocked ad categories (no alcohol ads on a kids' channel),
• floor prices (minimum price per impression),
• max ads per break, frequency limits, allowed ad formats.

Rules can be set at the publisher level and overridden lower down (site, then placement). Ad servers all over the world must apply the right rules on every ad request, in microseconds. A related question asked about config rollout: changes must reach servers globally within minutes, safely, because a bad config can break ad serving everywhere.

1) Requirements

• CRUD for rules at publisher, site and placement levels, with inheritance and overrides.
• Compute the effective config for a placement.
• Distribute changes to all ad servers within a few minutes.
• Validation, versioning, audit log, staged rollout and instant rollback.

2) Data Model

entities:  entity_id, type (publisher|site|placement), parent_id
rules:     entity_id, key (e.g. "blocked_categories", "floor_price_cents"), value (JSON),
           mode (set | append | remove), version, updated_by, updated_at

3) Architecture

flowchart LR
    UI["Publisher / admin UI"] --> CS["Config Service - validate, version, audit"]
    CS --> DB[("Rules DB")]
    CS --> BLD["Snapshot builder - effective configs"]
    BLD --> OS[("Versioned snapshots - object storage")]
    CS --> RO["Rollout controller - stages, health"]
    RO -->|"notify version N"| AS1["Ad servers - canary"]
    RO -->|"then"| AS2["Ad servers - region 1"]
    RO -->|"then"| AS3["Ad servers - all regions"]
    AS1 -->|"fetch snapshot"| OS
    AS2 --> OS
    AS3 --> OS
    AS1 -->|"metrics"| RO

4) Key Flows

4.1 Changing a rule

1. Validate: schema (types, ranges, e.g., a floor price can't be negative), references (the category exists), and a dry-run diff showing which placements' effective config will change.
2. Save it as a new version with an audit record (who, what, why).
3. The snapshot builder computes effective configs for affected placements, and writes a new versioned snapshot (the full config, or a delta) to object storage.

4.2 Rolling it out

1. The rollout controller tells canary servers (e.g., 1%) to load version N.
2. It watches metrics: ad fill rate, errors, revenue per request, latency. If they're healthy after a few minutes, it moves to one region, then all.
3. If metrics degrade → automatic rollback: tell servers to go back to version N-1 (still cached locally).

4.3 Ad server side

• Keeps the effective config in memory, keyed by placement ID. The lookup takes microseconds.
• Loads new snapshots in the background and swaps atomically, so requests never see half-applied config.
• If it can't fetch updates, it keeps serving with the last good version (fail static), rather than failing.

5) Push vs Pull

• Pull (servers poll for the latest version every 30 s): simple and robust.
• Push (notify servers when a version is ready, and they fetch it): faster.
• A common mix: push a small notification ("version 812 available"), and servers pull the snapshot from storage/CDN. It's fast, and servers that miss the push still catch up on the next poll.

6) Trade-offs & Alternatives

7) Wrap-Up

Store rules at publisher, site and placement levels with clear override semantics, validate every change (including a dry-run of affected placements), and version and audit it. Build versioned snapshots of effective configs, and roll them out in stages (canary → region → global) guarded by health metrics with automatic rollback. Ad servers keep the effective config in memory, swap versions atomically, and keep serving the last good version if updates fail.