Large File Upload and Processing Pipeline

0) Problem Restatement

Design a service where users upload large files (videos, datasets, product images, documents up to many GB), and the system processes them afterwards: scans for viruses, checks the format, makes thumbnails or runs an analysis, and finally shows a result or publishes the file. Examples include sellers uploading product images, users uploading a file for analysis, or an internal upload portal.

The main ideas: don't push big files through our own servers, make uploads resumable, and do the processing asynchronously so users aren't waiting on a slow request.

1) Requirements

1.1 Functional

• Upload large files, and resume after a network drop.
• Show upload and processing progress.
• Validate, scan and process each file (thumbnails, analysis, moderation).
• Make the result or approved file available for download or display.

1.2 Non-Functional

• Reliable: no lost uploads and no stuck jobs.
• Secure: only allowed users can upload or read, and malware never reaches other users.
• Scalable: thousands of uploads at the same time.
• Cheap bandwidth: our API servers should not carry file bytes.

1.3 Scale Estimates

• 1M uploads/day, average 50 MB → 50 TB/day into storage.
• Peak ~50 uploads starting per second, with thousands in progress at once.
• Processing: average 30 seconds of CPU per file → about 350 CPU cores busy on average, more at peak.

1.4 API Design

• POST /v1/uploads with { file_name, size, content_type } → { upload_id, part_size, part_urls: [...] }, where each URL is a pre-signed URL.
• The client uploads each part with PUT straight to object storage.
• POST /v1/uploads/{id}/complete with the list of uploaded parts.
• GET /v1/uploads/{id} → { status: uploading | processing | ready | rejected, progress, result_url }

A pre-signed URL is a link from object storage (like S3) that lets the holder upload or download one specific file for a short time, without any other credentials.

2) High-Level Architecture

2.1 Overview

• Upload API: checks the user's permission and quota, creates an upload record, and hands out pre-signed URLs.
• Object storage (S3/GCS): receives the file parts directly from the client.
• Upload events: when the upload completes, storage (or our API) publishes an event.
• Processing queue + workers: scan, validate, process and write results.
• Metadata DB: upload state, ownership and results.
• CDN: serves approved files and results fast.

2.2 Architecture Diagram

flowchart LR
    C["Client"] -->|"1. create upload"| API["Upload API"]
    API --> DB[("Upload metadata DB")]
    API -->|"pre-signed part URLs"| C
    C -->|"2. PUT parts directly"| S3[("Object storage - raw bucket")]
    C -->|"3. complete"| API
    API --> Q[("Processing queue")]
    Q --> W["Workers: scan, validate, process"]
    W --> S3P[("Processed bucket")]
    W --> DB
    W -->|"failed 3 times"| DLQ[("Dead-letter queue")]
    S3P --> CDN["CDN"]
    CDN --> V["Viewers"]

3) Data Model

uploads:
  upload_id, owner_id, file_name, size, content_type,
  status (initiated, uploading, uploaded, processing, ready, rejected, failed),
  storage_key, parts_done, checksum, result_key, error, created_at, updated_at

4) Key Flows

4.1 Uploading (multipart and resumable)

1. The client asks to start an upload. The API checks the file type and size, and the user's quota.
2. The file is split into parts (e.g., 16 MB each), and each part has its own pre-signed URL.
3. The client uploads parts in parallel. If the network drops, it asks which parts are done and uploads only the missing ones. That is what makes it resumable.
4. The client calls complete. Storage joins the parts into one file, and we verify the checksum.

4.2 Processing

1. The API sets status to uploaded and puts a job { upload_id } on the queue.
2. A worker downloads the file, runs a virus scan, and checks that the content really matches the type (a ".jpg" that is actually an executable is rejected).
3. It processes the file (thumbnails, transcoding, analysis) and writes results to the processed bucket.
4. It sets status to ready (or rejected with a reason) and notifies the user by webhook, WebSocket or email.

5) Deep Dive A — Reliability of processing

• At-least-once + idempotent workers: a job may run twice if a worker crashes. Results are written to a key that depends only on upload_id (e.g., processed/{upload_id}/thumb.jpg), so running twice gives the same result.
• Visibility timeout: while a worker is processing, the job is hidden from other workers. If the worker dies, the job becomes visible again and someone else picks it up.
• Retries and dead-letter queue: retry with backoff. After 3 failures, move the job to a dead-letter queue for a human to look at, and mark the upload failed.
• Stuck uploads: a cleanup job deletes uploads that never completed after 24 hours, and aborts their unfinished multipart uploads, which otherwise keep costing storage.

6) Deep Dive B — Security

• Scoped tokens: pre-signed URLs work only for one file, one method (PUT), and a few minutes. Download URLs for private files are short-lived too.
• Two buckets: raw uploads go to a private "quarantine" bucket. Only files that pass scanning are copied to the public/processed bucket.
• Authorization for large transfers: use a token (e.g., a JWT) with claims such as user_id, upload_id and max_size, and check them on the server. Never trust the client's size or type fields alone.
• Limits: max file size, a per-user daily quota and rate limits to stop abuse.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "Files of 100 GB?" Use bigger parts (e.g., 100 MB) and let processing work on chunks in parallel (e.g., video segments).
• "How does the client show a progress bar?" Upload progress comes from the client itself (bytes sent). Processing progress comes from the worker updating progress in the DB.
• "Users far away?" Use storage transfer acceleration or regional buckets, so users upload to the nearest region.

9) Wrap-Up

Hand out short-lived pre-signed URLs so clients upload big files in resumable parts straight to object storage. Then queue a processing job that idempotent workers pick up to scan, validate and process the file, with retries and a dead-letter queue. Keep raw files in a private quarantine bucket, publish only approved results through a CDN, and track every upload's state in a metadata DB.