Text-to-Video Generation Service (Sora)

0) Problem Restatement

Design a service like OpenAI's Sora. A user types a prompt ("a corgi surfing at sunset"), and the system generates a video. Generation takes minutes on expensive GPUs, and GPUs are scarce and sometimes fail. So the request can't be synchronous. It is an asynchronous job: the user submits, sees progress, can cancel, and downloads the video when it's ready.

Paid tiers get higher priority and larger quotas. The interviewer usually cares less about the ML model and more about jobs, scheduling and reliability around scarce compute.

1) Requirements

1.1 Functional

• Submit a prompt with options (length, resolution) → get a job_id.
• See status and progress: queued (with position), running (percent), done or failed.
• Cancel a job.
• Download or stream the finished video, and see history.
• Quotas and priority per tier (free, plus, pro).

1.2 Non-Functional

• No lost jobs: a submitted job eventually completes or fails clearly.
• Efficient GPU use: GPUs should rarely sit idle, and work isn't done twice.
• Fairness: free users still make progress, and paid users wait less.
• Cost control: stop runaway usage and abuse.

1.3 Scale Estimates

• 1M videos/day ≈ 12 jobs/sec on average, with peaks of 50/sec.
• Each video needs ~2 GPU-minutes on average → 2M GPU-minutes/day ≈ 1,400 GPUs busy on average, more at peak. That's why queues form.
• Output: ~20 MB per video → 20 TB/day to object storage, served via CDN.

1.4 API Design

• POST /v1/videos (Idempotency-Key) { prompt, duration_s: 10, resolution: "1080p" } → { job_id, status: "queued" }
• GET /v1/videos/{job_id} → { status, progress, queue_position, video_url? }
• POST /v1/videos/{job_id}/cancel
• Optional webhook or SSE for progress updates.

2) High-Level Architecture

2.1 Overview

• Job API: validates, checks quota and moderation, creates the job and returns immediately.
• Job DB: the durable job state (source of truth).
• GPU Scheduler: keeps priority queues per tier and assigns jobs to free GPU workers.
• GPU workers: run the model. They send heartbeats and progress, and save checkpoints.
• Post-processing workers (CPU): encode, make thumbnails, add a watermark.
• Object storage + CDN: final videos and intermediate checkpoints.
• Notification: tells the user when the job is done.

2.2 Architecture Diagram

flowchart LR
    U["User"] --> API["Job API - quota, moderation"]
    API --> DB[("Job DB")]
    API --> SCH["GPU Scheduler - tier queues"]
    SCH -->|"lease job"| G1["GPU worker"]
    SCH -->|"lease job"| G2["GPU worker"]
    G1 -->|"heartbeat, progress"| SCH
    G1 --> CK[("Checkpoints - object storage")]
    G1 --> PP["Post-process: encode, thumbnail"]
    PP --> OS[("Videos - object storage")]
    OS --> CDN["CDN"]
    SCH --> DB
    PP --> N["Notify user"]
    U -->|"poll status / download"| API

3) Data Model

jobs:
  job_id, user_id, tier, prompt, params, status (queued, running, postprocessing, succeeded,
  failed, cancelled), priority, attempt, worker_id, lease_until, progress, checkpoint_key,
  output_key, gpu_seconds_used, created_at, started_at, finished_at
usage:
  user_id, period, gpu_seconds_used, videos_count      -- for quotas

4) Key Flows

4.1 Submit

1. Check the prompt against content policy, and check the user's quota (e.g., 50 videos/month) and number of concurrent jobs.
2. Create the job as queued (the idempotency key stops double submits) and add it to the scheduler queue for the user's tier.
3. Return job_id. The client polls every few seconds or listens on SSE.

4.2 Run

1. When a GPU worker is free, the scheduler picks the next job (see fairness below) and leases it to the worker: lease_until = now + 60s.
2. The worker heartbeats every ~15 seconds with progress, extending the lease.
3. It saves checkpoints every few minutes (e.g., finished segments), so a crash doesn't lose all the work.
4. When done, it uploads raw output, and post-processing encodes it. The job becomes succeeded and the user is notified.

4.3 Cancel

Mark cancelled. The worker sees it in the next heartbeat response, stops and frees the GPU.

5) Deep Dive A — Scheduling scarce GPUs

• Priority queues per tier with weighted fair sharing: e.g., out of every 10 free GPU slots, 6 go to pro, 3 to plus and 1 to free. Free jobs are slower but never starve.
• Per-user limits: max 2 running jobs per user, so one heavy user can't take the whole fleet.
• Right-size placement: long or high-resolution jobs need bigger GPUs or multiple GPUs, so match the job type to the worker pool.
• Preemption (optional): a pro job can pause a free job, which later resumes from its checkpoint.
• Queue position and ETA: show "~6 minutes", computed from queue depth and average job time. This reduces the urge to cancel and re-submit.

6) Deep Dive B — Failures without duplicate expensive work

• Lease expiry: if heartbeats stop (the worker crashed), the scheduler re-queues the job with attempt + 1, starting from the last checkpoint rather than from zero.
• Fencing: a "zombie" worker from attempt 1 that wakes up late can't overwrite the result, because writes include the attempt number, and only the current attempt is accepted.
• Retry limits: after 3 failed attempts, mark failed, refund the user's quota, and alert if many jobs fail (a bad model version or bad nodes).
• Idempotent outputs: results are written to videos/{job_id}/attempt-{n}.mp4, and the job record points to the winner.
• Cost controls: hard caps on duration and resolution per tier, per-user daily GPU-second budgets, and alerts on unusual spending.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you plan capacity?" Track queue wait time per tier. When pro waits exceed the target, add GPUs (slow, since reserved capacity is bought ahead) or temporarily lower free-tier limits.
• "Multi-stage pipelines?" Model stages (draft → upscale → audio) as separate tasks in a small DAG, each with its own queue and checkpoint.
• "Abuse?" Moderate prompts before queueing and outputs before release, and rate-limit new accounts.

9) Wrap-Up

Make video generation an asynchronous job: validate and store it, queue it by tier, and let a GPU scheduler lease jobs to workers with heartbeats and checkpoints. Use weighted fair queues and per-user limits to share scarce GPUs, fencing and attempt numbers to stop duplicate work, and quotas and cost caps to control spending. Store results in object storage behind a CDN and report progress through polling or SSE.