0) Problem Restatement
Design a service that processes images, for example resizing, applying filters, or running an ML model (classification, captioning). Clients submit jobs (one image or a batch of millions) and get results later. Anthropic asked it as "start with one worker, then evolve safely to many concurrent processors" and "how would you scale batch image pipelines?".
1) Requirements
- Submit a job (one image or a batch manifest), and check status and progress.
- Process each image with one or more steps, and store the outputs.
- No lost images and no duplicate outputs.
- Handle both interactive requests (fast) and huge batch jobs (throughput).
1.1 Scale Estimates
- Interactive: 200 images/sec, target a few seconds each.
- Batch: jobs of 10M images, finished within hours → ~1–3K images/sec for that job.
- Processing: 50–500 ms CPU per image (or GPU for ML).
2) Stage 1: One Worker
A single process reads jobs from a DB table, processes them, and writes results. It's simple and fine for low volume. But when we add a second worker, both may take the same job, and if a worker crashes mid-job, the job is stuck forever. We need a queue with proper claiming.
3) Stage 2: Queue + Many Workers
Architecture Diagram
flowchart LR
C["Client"] --> API["Job API"]
API --> DB[("Jobs + tasks DB")]
API --> SPL["Splitter - batch to tasks"]
SPL --> QI[("Interactive queue")]
SPL --> QB[("Batch queue")]
QI --> W["Worker pool - autoscaled"]
QB --> W
IN[("Input images - object storage")] --> W
W --> OUT[("Outputs - object storage")]
W --> DB
W -->|"failed 5 times"| DLQ[("Dead-letter queue")]- Job API: creates a job. For batches, the client uploads a manifest (a list of image URLs) to object storage.
- Splitter: turns a batch into tasks (one per image, or small groups of ~100 for efficiency) and enqueues them.
- Queues: interactive and batch are separate, so a 10M-image batch doesn't delay a user waiting for one image.
- Workers: pull a task, download the input, process it, upload the output, and mark the task done.
- Progress: count completed tasks per job (e.g., an atomic counter), and show
done / total.
4) Making Many Workers Safe
- Visibility timeout / lease: when a worker takes a task, the queue hides it for N minutes. If the worker finishes, it deletes (acks) the task. If it crashes, the task reappears and another worker takes it. That's at-least-once processing.
- Idempotent outputs: since a task can run twice, write outputs to a deterministic key like
outputs/{job_id}/{image_id}/{step}.jpg. Running twice overwrites with the same result, and the task's "done" update is conditional (WHERE status != 'done'). - Retries with backoff, and after 5 failures move the task to a dead-letter queue (a broken or huge image shouldn't block others) and mark it failed in the job report.
- Heartbeats for long tasks (e.g., big images or GPU inference) to extend the lease.
- Concurrency inside a worker: download and upload are I/O-bound, while processing is CPU or GPU-bound. Use a small pipeline (download threads → process pool → upload threads) to keep the CPU busy.
5) Scaling Batch Jobs
- Autoscale workers on queue depth (and on GPU availability for ML steps).
- Batch tasks: groups of ~100 images reduce queue overhead. The worker processes them and reports per-image results.
- Throughput limits: object storage request rates and network bandwidth can be the bottleneck. Spread keys (prefix by hash) and keep workers in the same region as the storage.
- Fairness: several batch jobs share capacity (round-robin across jobs), and interactive work is always served first.
- Cost: use spot/preemptible machines for batch (it's retry-safe anyway, thanks to leases and idempotent outputs).
6) Trade-offs & Alternatives
| Decision | Choice | Why | Alternative |
|---|---|---|---|
| Delivery | At-least-once + idempotent outputs | Simple and safe | Exactly-once: very hard, not needed |
| Queues | Separate interactive and batch | Batch can't starve users | One queue: long waits for small jobs |
| Task size | ~100 images per batch task | Less overhead | One per image: queue overhead; huge tasks: slow retries |
| Failures | Retries + dead-letter queue | One bad image doesn't block a job | Retry forever: stuck workers |
7) Wrap-Up
Move from a single worker to a queue-based design where workers lease tasks with visibility timeouts, write outputs to deterministic keys so re-runs are harmless, and retry failures before sending them to a dead-letter queue. Split big batches into grouped tasks, keep interactive and batch queues separate, track progress by counting completed tasks, and autoscale workers (spot machines for batch) on queue depth.