Online Judge and Coding Contest Platform (LeetCode)

0) Problem Restatement

Design a platform like LeetCode or Codeforces. Users browse problems, write code in the browser in many languages, and submit it. The system runs the code against hidden test cases and returns a verdict: Accepted, Wrong Answer, Time Limit Exceeded, Runtime Error or Compilation Error. Contests add a timer, a burst of submissions at the start and end, and a live leaderboard.

The hardest part is that we run untrusted code from strangers, so it must be isolated and limited.

1) Requirements

1.1 Functional

• Browse and search problems, and view statements and examples.
• Run code against sample tests, and submit against hidden tests.
• Get a verdict with runtime and memory.
• Contests: registration, timed window, scoring and a live leaderboard.
• Submission history.

1.2 Non-Functional

• Security: user code can never reach the network, other users' data or our servers.
• Fairness: same code gives the same verdict and similar timings.
• Fast feedback: verdict in a few seconds normally.
• Handle spikes: contest start or end can bring 10x traffic.

1.3 Scale Estimates

• 5M daily users, 2M submissions/day ≈ 25/sec average.
• Contest with 50K participants: in the last 10 minutes, maybe 1,000 submissions/sec.
• Each submission runs ~50 tests, taking 2–10 seconds of CPU in total. At 1,000/sec, that is 2,000–10,000 CPU cores busy, so runners must autoscale.

1.4 API Design

• POST /v1/submissions { problem_id, language: "python3", code, contest_id? } → { submission_id, status: "queued" }
• GET /v1/submissions/{id} → { status, verdict, runtime_ms, memory_kb, failed_test? } (or a WebSocket push)
• GET /v1/contests/{id}/leaderboard?page=1

2) High-Level Architecture

2.1 Overview

• Problem Service: problems, statements and sample tests (cached heavily and served via CDN).
• Submission Service: saves the submission and puts a job on the queue.
• Queue: separate queues for contest submissions (priority) and practice.
• Runner fleet: workers that compile and run code inside a sandbox against test cases.
• Test case store: hidden test files in object storage, cached on runners.
• Result delivery: updates the DB and pushes the verdict to the user.
• Leaderboard Service: updates contest rankings from accepted results.

2.2 Architecture Diagram

flowchart LR
    U["User browser"] --> API["Submission Service"]
    API --> DB[("Submissions DB")]
    API --> Q[("Job queue - contest priority")]
    Q --> R1["Runner - sandbox"]
    Q --> R2["Runner - sandbox"]
    TC[("Hidden test cases - object storage")] --> R1
    TC --> R2
    R1 -->|"verdict"| RES["Result handler"]
    R2 -->|"verdict"| RES
    RES --> DB
    RES -->|"push"| U
    RES --> LB["Leaderboard Service"]
    LB --> RD[("Redis sorted set")]

3) Data Model

problems:     problem_id, title, statement, difficulty, time_limit_ms, memory_limit_mb, tests_version
submissions:  submission_id, user_id, problem_id, contest_id, language, code_ref,
              status (queued, running, done), verdict, runtime_ms, memory_kb, created_at
contest_scores: contest_id, user_id, score, penalty, solved_count, last_accepted_at

4) Key Flows

4.1 Submit

1. Save the submission (code in object storage or DB) with status queued.
2. Push a job { submission_id } to the queue. Return right away.
3. A runner takes the job, loads the code and the problem's tests (cached locally by tests_version), compiles if needed, and runs each test inside a sandbox with limits.
4. It stops at the first failing test (as most judges do), and returns the verdict with time and memory.
5. The result handler saves the verdict and pushes it to the user over WebSocket (or the client polls every second).

4.2 Leaderboard update

When an accepted verdict arrives during a contest, update the user's score (points, plus a penalty for time and wrong tries) in a Redis sorted set, a structure that keeps items ordered by score. Reading the top 100 is then instant.

5) Deep Dive A — Running untrusted code safely

Each run happens in a fresh, locked-down sandbox:

• Isolation: containers with gVisor, or lightweight VMs like Firecracker, so a kernel bug inside is much harder to exploit.
• No network and a read-only file system, except a small temporary folder.
• Limits: CPU time (the time limit), memory (e.g., 256 MB via cgroups), number of processes (stops "fork bombs"), and output size.
• Unprivileged user, dropped Linux capabilities, and seccomp filters that block dangerous system calls.
• Destroy the sandbox after each submission, or reset it to a clean state.

6) Deep Dive B — Fair timing and contest spikes

• Fair timing: measure CPU time, not wall-clock time. Pin each run to a dedicated CPU core, avoid running too many jobs on one machine, and use the same machine type for all runners. For borderline results, re-run a few times and take the minimum.
• Autoscaling: pre-warm extra runners before a contest starts, because they take minutes to boot. Also autoscale on queue length.
• Priority: contest jobs go first. Practice submissions may wait a little longer during contests.
• Test cache: runners keep test files on local disk, keyed by tests_version, so there's no download per submission.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you support 20 languages?" One runner image per language (or one image with all toolchains), each with language-specific time multipliers (e.g., Python gets extra time).
• "How do you detect plagiarism?" After the contest, compare submissions with token-based similarity tools (like MOSS) and flag suspicious pairs for review.
• "How do you avoid leaking hidden tests?" Don't return full failing inputs for hidden tests in contests. Only show the test number.

9) Wrap-Up

Save each submission and queue it, and let an autoscaled runner fleet execute code in isolated sandboxes (gVisor or Firecracker) with no network and strict CPU and memory limits. Measure CPU time on dedicated cores for fairness, push verdicts back over WebSockets, and keep contest rankings in a Redis sorted set. Pre-warm runners and prioritize contest jobs to survive spikes.