0) Problem Restatement
A new model version is ready: its weights are 500 GB (or more). It must be copied to 1,000+ GPU hosts as fast and reliably as possible so they can start serving it. The source (object storage or a seed machine) has limited bandwidth, and each host has a network link of limited speed (e.g., 25 Gbps shared for upload and download). Anthropic asked this in several forms: "stream a large file to 1,000 hosts fastest", "peer-to-peer under a shared link cap", "deploy a 500 GB model to GPU workers".
Asked at: Anthropic — 4 candidate reports between Mar 2026 and Aug 2026.1) Requirements
- Deliver the identical file (or set of files) to N hosts.
- Minimize the total time until all hosts have it.
- Verify integrity (no corrupted weights).
- Survive host and network failures, and resume without starting over.
- Switch hosts to the new version safely (no half-loaded models).
2) Bandwidth Math (do this out loud)
- File F = 500 GB = 4,000 Gb (gigabits). Link per host = 25 Gbps.
- One host downloading at full speed needs 4,000 / 25 = 160 seconds at the very least.
- Naive: every host downloads from one source with a 100 Gbps link. Total data = 1,000 × 4,000 Gb = 4,000,000 Gb → 40,000 seconds (11 hours). The source is the bottleneck.
- With peer sharing: once hosts have pieces, they upload to others. Total upload capacity grows with the number of hosts, so the ideal time approaches F / link speed, i.e., about 160 seconds plus overhead. That's why the answer is chunking + peer-to-peer.
3) Approaches
Tree (fan-out): the source sends to 10 hosts, each sends to 10 more, and so on. The depth is log(N), but each parent splits its upload between children, and a slow node delays its whole subtree. Pipeline / chain: split the file into chunks. Host 1 forwards chunk k to host 2 while receiving chunk k+1, and so on. Every link is busy at full speed, and total time ≈ F / bandwidth + (N × chunk time). It's great in theory, but one slow or failed host breaks the chain (so use several chains, and repair around failures). BitTorrent-style swarm (our choice): split the file into chunks (e.g., 64 MB). Hosts download different chunks from the source and from each other, preferring rarest chunks first so every chunk spreads quickly. A tracker or coordinator tells hosts who has what. It handles slow and failed peers naturally.4) Architecture
Architecture Diagram
flowchart LR
REG["Model Registry - version, manifest, checksums"] --> CO["Distribution Coordinator / tracker"]
OS[("Object storage - seed")] --> H1["GPU host - rack A"]
OS --> H2["GPU host - rack B"]
H1 <-->|"chunks"| H3["GPU host - rack A"]
H2 <-->|"chunks"| H4["GPU host - rack B"]
H1 <-->|"chunks"| H2
CO -->|"who has which chunks"| H1
CO --> H2
CO --> H3
CO --> H4- Manifest: the list of chunks with a SHA-256 for each. Hosts verify every chunk before sharing or using it.
- Coordinator: tracks chunk availability and suggests peers. Prefers peers in the same rack (fast, cheap links) and limits cross-rack or cross-zone traffic.
- Seeds: object storage plus a few "super-seed" hosts that get the file first and have full upload capacity.
5) Key Details
- Shared link cap: with one link used for both upload and download, balance them. Each host uploads roughly as much as it downloads, and the coordinator limits parallel connections per host.
- Topology awareness: seed at least one host per rack first, then spread within racks. Cross-rack links are often oversubscribed.
- Failures: a failed download of a chunk is retried from another peer. A host that dies just drops out. Progress is saved per chunk, so a restarted host resumes where it stopped.
- Integrity: verify per-chunk hashes and a final whole-file hash before loading.
- Loading into GPUs: many hosts can start loading completed shards (e.g., per tensor-parallel shard) before the full file arrives, if the model is split into shard files.
6) Rolling Out the New Version
- Pre-stage: distribute the new weights to local NVMe while the old version keeps serving.
- Verify the checksums on every host.
- Switch in waves: drain a subset of servers, load the new model, run a quick health and eval check, and put them back into rotation. Watch error and latency metrics before the next wave.
- Keep the previous version on disk for fast rollback.
- Garbage-collect old versions later, keeping N versions.
7) Trade-offs & Alternatives
| Approach | Good | Bad |
|---|---|---|
| Single source | Simple | Source bandwidth bottleneck (hours) |
| Tree fan-out | log(N) depth | Slow node delays its subtree, uneven link use |
| Pipeline chain | Near-optimal bandwidth use | Fragile to slow or failed hosts |
| P2P swarm (chosen) | Scales with hosts, robust | Needs a coordinator, more moving parts |
8) Wrap-Up
Do the math first: a single source takes hours, while peer sharing approaches file size ÷ link speed. Split the weights into hashed chunks, seed one host per rack, and let hosts swap chunks rarest-first with a topology-aware coordinator, verifying every chunk and resuming after failures. Pre-stage weights on local disk, then switch serving to the new version in health-checked waves, keeping the old version for rollback.