Ad Creative Relationship Processing with an External Model API

0) Problem Restatement

TikTok asked this. Advertisers submit creative plans: a set of ad videos and a set of keywords (or audiences, product categories). The platform needs to know which keywords relate to which videos (e.g., relevance scores), so ads can be matched to searches and contexts. Computing a relationship requires calling an external model API (an ML service owned by another team or vendor) that is slow (seconds per call), rate-limited (N requests/sec), and sometimes fails. Design the processing system that turns plans into stored relationship results.

1) Requirements

• Accept a plan: { plan_id, advertiser_id, videos: [...], keywords: [...] }.
• For each needed (video, keyword) pair, or each video with a batch of keywords, call the model API and store the result { video_id, keyword, score, model_version }.
• Show plan progress and status. Results should be usable by ad serving as soon as they're ready.
• Stay under the API's rate limit, retry failures, never lose work, and avoid duplicate paid calls.

1.1 Scale Estimates

• 50K plans/day × 20 videos × 50 keywords = 50M pairs/day. If the API accepts a video + up to 50 keywords per call, that's 1M calls/day ≈ 12 calls/sec, under a quota of, say, 30/sec. Spikes (a big advertiser uploads 10K videos) must be smoothed.

2) Architecture

flowchart LR
    ADV["Advertiser upload"] --> API["Plan API"]
    API --> DB[("Plans + tasks DB")]
    API --> PL["Planner - split into tasks"]
    PL --> Q[("Task queue - priority by advertiser tier")]
    Q --> W["Workers - concurrency limited"]
    W --> RL["Shared rate limiter - API quota"]
    RL --> EXT["External model API"]
    W --> RES[("Relationship results store")]
    W --> DB
    W -->|"permanent failures"| DLQ[("Dead-letter queue")]
    RES --> SERVE["Ad serving / indexing"]

3) Key Design Points

• Task granularity: one task = one video + a batch of keywords (as many as one API call accepts). Fewer calls, less overhead.
• Deduplicate before calling: the same video with the same keywords (e.g., a re-submitted plan) is looked up in the results store by (video_hash, keyword, model_version) and skipped. This saves money and quota.
• Shared rate limiter: all workers take a token from a central token bucket (e.g., Redis) before calling, so the total stays under the vendor's limit no matter how many workers run.
• Concurrency limit: since calls are slow (seconds), each worker runs several calls in parallel, capped to avoid piling up timeouts.
• Retries: on timeouts or 5xx, retry with exponential backoff and jitter. On 429 (rate limited), back off globally. On a bad-input error (4xx), send the task to the dead-letter queue, since retrying won't help.
• Circuit breaker: if the API is failing heavily, pause calling for a while. Tasks wait in the queue (nothing is lost).
• Idempotent writes: results are keyed by (video_id, keyword, model_version), so a retried task overwrites with the same data.

4) Flow

1. The plan is saved as processing. The planner creates tasks and enqueues them.
2. Workers lease tasks (visibility timeout), get a rate-limit token, call the API, and write results.
3. A task done → the plan's completed counter goes up. When all tasks are done or failed, the plan becomes ready (or partially_ready with a list of failures).
4. Ad serving can use results as they arrive (per video), and doesn't wait for the whole plan.

5) Fairness, Priority and Model Updates

• Priority queues: new plans from live campaigns first, and backfills and re-processing later.
• Fair share per advertiser: a single advertiser uploading 10K videos can't block everyone. Round-robin across advertisers within a priority.
• Model version changes: when the vendor ships a new model, re-process in the background at low priority. Keep both versions until the new one is complete, then switch serving to it.

6) Trade-offs & Alternatives

7) Wrap-Up

Split each plan into video-level tasks that batch keywords, queue them by priority with fair sharing across advertisers, and let workers call the slow external model API through a shared token-bucket rate limiter with bounded concurrency, backoff retries, a circuit breaker and a dead-letter queue. Skip already-computed pairs using a content-hash + model-version cache, write results idempotently so serving can use them immediately, and re-process in the background when the model version changes.