GPU Cluster Control Plane (Host Health, Repair and Job Dispatch)

0) Problem Restatement

Design the control plane for a fleet of about 100,000 GPU servers (asked at Oracle and NVIDIA). Each host constantly reports its health: heartbeats, GPU temperature, memory errors (ECC), NVLink/network status and disk health. The control plane must:

• keep an up-to-date view of every host,
• detect sick or dead hosts,
• stop sending new jobs to them and move work away (drain),
• trigger repair workflows (reboot, reimage, open a hardware ticket), and
• return fixed hosts to service.

GPUs are expensive, so both idle broken hosts and jobs failing on bad hosts cost a lot of money.

1) Requirements

1.1 Functional

• Ingest heartbeats and health metrics from all hosts.
• A health state per host, with history.
• Automated detection rules (and ML later), plus manual overrides.
• Repair workflows with multiple steps and retries.
• An API for the job scheduler: "which hosts are healthy and free?"

1.2 Non-Functional

• Scale: 100K hosts × a report every 10 seconds = 10K reports/sec, plus detailed metrics.
• Fast detection: under a minute for dead hosts.
• Few false positives: don't drain healthy hosts running expensive training jobs.
• Safety: automation must never drain a large part of the fleet at once by mistake.

1.3 Scale Estimates

• 10K heartbeats/sec, and ~1M metric points/sec (100 metrics per host per 10s).
• Hardware failures at this size are frequent: maybe hundreds of hosts per day need attention.

1.4 API Design

• Host agent: POST /v1/hosts/{id}/heartbeat { ts, gpu: [...], ecc_errors, nvlink_ok, running_jobs }
• Scheduler: GET /v1/hosts?state=healthy&gpu_type=H100&free=true
• Operators: POST /v1/hosts/{id}/drain, POST /v1/hosts/{id}/cordon, GET /v1/hosts/{id}/history

2) High-Level Architecture

2.1 Overview

• Host agent: collects GPU and system health locally (e.g., via NVIDIA DCGM), runs quick self-tests, and sends heartbeats.
• Ingestion: heartbeats to a state service. Detailed metrics go to a time-series DB.
• Host state store: the current state per host in a strongly consistent store (e.g., etcd or a DB with conditional updates), partitioned by host ID.
• Health evaluator: rules like "no heartbeat for 60s", "ECC uncorrectable errors > 0", "GPU fell off the bus", "NVLink down".
• Remediation engine: a workflow engine (e.g., Temporal) that runs repair steps.
• Job scheduler integration: reads healthy capacity and receives drain requests.

2.2 Architecture Diagram

flowchart LR
    H["Host agents - 100K GPUs"] -->|"heartbeats"| ING["Ingestion"]
    H -->|"metrics"| TS[("Time-series DB")]
    ING --> ST[("Host state store")]
    EV["Health evaluator - rules"] --> ST
    EV --> TS
    EV -->|"unhealthy"| RE["Remediation workflows"]
    RE -->|"drain"| SCH["Job scheduler"]
    RE -->|"reboot / reimage / ticket"| H
    SCH -->|"only healthy hosts"| ST
    OPS["Operators UI"] --> ST

3) Host State Machine

healthy → suspect → draining → repairing → validating → healthy
                        ↘ (repair failed 3x) → broken (hardware ticket / RMA)

• suspect: one signal looked bad. Stop placing new jobs, but don't kill running ones yet. Wait for confirmation (e.g., a repeated check or a second signal).
• draining: ask the scheduler to checkpoint and move running jobs, or wait for them to finish, up to a deadline.
• repairing: run the workflow (reset GPU → reboot → reimage → hardware ticket).
• validating: run burn-in tests (GPU stress, NCCL bandwidth test) before returning to service.

Every transition is a conditional update (only if current state = X and version = V), so two evaluators or an operator can't make conflicting changes.

4) Key Flows

4.1 A host stops heartbeating

1. The evaluator notices no heartbeat for 60 seconds.
2. Before declaring it dead, check whether its whole rack or switch went silent. If many hosts vanish together, it's probably a network problem, so raise a single incident instead of draining hundreds of hosts.
3. For a single host: mark it suspect, then draining. The scheduler reschedules its jobs from their last checkpoint.
4. Remediation tries a remote power cycle (through the out-of-band management controller, BMC). If the host comes back and passes validation, it returns to healthy.

4.2 GPU memory errors

Uncorrectable ECC errors mean results may be wrong. Drain immediately (don't let a training run continue on bad memory), reset the GPU, and if errors repeat, open a hardware replacement ticket.

5) Deep Dive A — Avoiding false positives

• Require two independent signals, or a repeat over time, for disruptive actions.
• Hysteresis: a host must be healthy for a while (and pass burn-in) before returning to service, so it doesn't flap in and out.
• Distinguish host problems from job problems: if the same job fails on many hosts, it's the job, not the hosts.
• Track the precision of each rule: how often did a drained host turn out fine? Tune or disable noisy rules.

6) Deep Dive B — Safety rails for automation

• Rate limits: never drain more than, e.g., 2% of the fleet (or 10% of any one cluster) per hour automatically. Beyond that, page a human.
• Blast radius checks: detect correlated failures (same rack, same firmware, same driver version) and pause automation for that group.
• Idempotent workflows: every step can be retried safely, and workflows persist their progress so a control plane restart resumes them.
• Capacity awareness: repair teams and spare parts are limited, so queue hardware tickets by priority (e.g., hosts in the biggest training clusters first).

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you handle 100K heartbeats?" Partition hosts across evaluator instances by host ID, and keep last-seen times in memory with a periodic flush.
• "Predict failures?" Train a model on metric history (rising correctable ECC errors, temperature trends) to proactively drain hosts before they fail.
• "Firmware or driver rollouts?" Treat them like deployments: canary a few hosts, validate, then roll out in waves, with automatic pause if failure rates rise.

9) Wrap-Up

Agents send heartbeats and GPU health, a consistent state store tracks each host through a clear state machine, and an evaluator with confirmation and correlation checks decides when a host is really sick. A durable workflow engine drains, repairs, validates and returns hosts, while the scheduler only places work on healthy capacity. Rate limits and blast-radius checks keep the automation from ever taking out a large part of the fleet.