CI/CD Platform (GitHub Actions / Jenkins)

0) Problem Restatement

Design a CI/CD system. CI (continuous integration) means every code push is built and tested automatically. CD (continuous delivery or deployment) means passing builds are deployed. When a developer pushes code, the system creates a pipeline: a set of jobs like build, unit tests, integration tests and deploy, where some jobs depend on others. Jobs run on a pool of worker machines, and users watch live status and logs.

Interviewers often focus on: scheduling jobs with dependencies, handling stuck jobs (a worker dies silently), and build caching to make builds fast.

1) Requirements

1.1 Functional

• Trigger pipelines from pushes, pull requests or a schedule.
• Run jobs in dependency order (a DAG), in parallel where possible.
• Match jobs to workers with the right capabilities (Linux, macOS, GPU).
• Stream live logs, and show status per job.
• Retry, cancel and re-run jobs.
• Cache dependencies and build outputs.

1.2 Non-Functional

• Reliable: no job stuck forever, and no job silently lost.
• Fast: start jobs within seconds, and use caches to shorten builds.
• Isolation: one team's build can't read another's secrets.
• Scale: thousands of repos and hundreds of thousands of jobs per day.

1.3 Scale Estimates

• 500K jobs/day, peaking during work hours at ~30 job starts/sec.
• Average job: 5 minutes → about 2,000 jobs running at once, so 2,000+ workers at peak.
• Logs: 1 MB per job average → 500 GB/day of logs.

1.4 API Design

• Webhook from Git: POST /v1/hooks/git { repo, commit, branch }
• GET /v1/pipelines/{id} → jobs and status
• GET /v1/jobs/{id}/logs?follow=true (streaming)
• POST /v1/jobs/{id}/retry, POST /v1/pipelines/{id}/cancel
• Worker protocol: POST /v1/workers/lease → a job; POST /v1/jobs/{id}/heartbeat; POST /v1/jobs/{id}/complete

2) High-Level Architecture

2.1 Overview

• Trigger Service: receives webhooks, reads the pipeline config file (e.g., YAML in the repo) and creates the pipeline and its jobs.
• Scheduler: tracks the job DAG. When all of a job's parents succeed, the job becomes "ready" and goes to the queue that matches its labels (e.g., linux-large).
• Workers / runners: ask for jobs they can run (pull model), run each job in a fresh container or VM, heartbeat, and upload logs and artifacts.
• Log Service: receives log chunks and streams them live to browsers, then stores them in object storage.
• Cache Service: stores dependency caches and build outputs by key.
• State DB: pipelines, jobs and their states.

2.2 Architecture Diagram

flowchart LR
    GIT["Git host"] -->|"webhook"| TR["Trigger Service"]
    TR --> DB[("Pipelines and Jobs DB")]
    SCH["DAG Scheduler"] --> DB
    SCH -->|"ready jobs by label"| Q[("Job queues")]
    W["Workers - fresh container per job"] -->|"lease job"| Q
    W -->|"heartbeat, status"| SCH
    W -->|"log chunks"| LS["Log Service"]
    LS --> OS[("Object storage - logs, artifacts")]
    W <-->|"get/put cache"| CS["Cache Service"]
    UI["Web UI"] --> LS
    UI --> DB

3) Data Model

pipelines: pipeline_id, repo, commit_sha, trigger, status, created_at
jobs:      job_id, pipeline_id, name, labels, depends_on[], status
           (pending, ready, leased, running, succeeded, failed, cancelled),
           attempt, worker_id, lease_until, started_at, finished_at
artifacts: job_id, name, storage_key, size

4) Key Flows

4.1 Running a pipeline

1. A push arrives. The trigger reads the config at that commit and creates jobs with their dependencies.
2. The scheduler marks jobs with no parents as ready and enqueues them.
3. A worker with matching labels leases a job (lease_until = now + 60s), pulls the code, restores caches, runs the steps and streams logs.
4. On finish, the worker reports the result. The scheduler marks children ready once all their parents have succeeded. If a parent fails, its children are skipped.

4.2 Live logs

The worker sends log chunks every second. The log service appends them to a short-term buffer (e.g., Redis streams) that browsers read over SSE/WebSocket. When the job finishes, the full log is written to object storage.

5) Deep Dive A — Stuck jobs

Workers crash, lose network or hang. We handle each case:

• Heartbeats + leases: a running worker renews its lease every 15 seconds. A reaper finds jobs with lease_until < now, marks the attempt as failed ("lost worker"), and re-queues it if retries remain.
• Timeouts: every job has a maximum runtime (e.g., 60 minutes). The scheduler kills jobs that exceed it.
• Safe final state: the state machine only allows one terminal state. A late "success" from a zombie worker, after the job was already retried, is rejected because its attempt number is old. This is called fencing.
• Cancellation: set cancelled. Workers see it on the next heartbeat response and stop the container.

6) Deep Dive B — Build caching

Rebuilding everything on every push is slow. There are two kinds of cache:

• Dependency cache: e.g., node_modules or the Maven repo. The key is a hash of the lockfile (hash(package-lock.json)). If the lockfile didn't change, restore the saved archive.
• Build output cache: compiled outputs keyed by a hash of the inputs (source files + compiler version + flags), as Bazel does. The same inputs give the same output, so we can skip the step entirely.
• Container layer cache: reuse Docker image layers that didn't change.

Never let untrusted pull requests (e.g., from forks) write to the shared cache, or they could poison other builds.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you deploy safely?" Deploy jobs roll out gradually (canary → 10% → 100%), watch health metrics, and roll back automatically on errors. For AI services, also run an evaluation suite before promoting a model.
• "How do you handle secrets?" Store them in a vault, inject them only into jobs of that repo and branch, and mask them in logs.
• "Flaky tests?" Track pass/fail history per test, auto-retry known flaky tests once, and report flakiness to owners.

9) Wrap-Up

Turn pushes into a DAG of jobs, enqueue ready jobs by worker label, and let workers lease jobs and heartbeat while running them in fresh containers. Leases, timeouts and fenced attempt numbers catch stuck and zombie jobs. Stream logs through a buffer to the UI and store them in object storage, and speed builds with content-hash keyed caches that untrusted builds can't write to.