Distributed File System (GFS / HDFS)

0) Problem Restatement

Design a distributed file system (asked at Databricks) that stores very large files (GBs to TBs) across thousands of cheap machines, like Google File System or HDFS. It offers a normal hierarchy (/logs/2026/09/19/part-001) and operations like create, read, append, delete and list. Disks and machines fail every day, so the system must keep data safe and available automatically, and give high throughput for big sequential reads and writes (batch analytics).

1) Requirements

• Namespace: directories and files with permissions.
• Large sequential reads and writes, and appends (random overwrites are rare).
• Survive disk, machine and rack failures without data loss.
• Petabytes of data, thousands of clients reading in parallel.

2) Architecture

flowchart LR
    C["Client library"] -->|"1. where are chunks of file X?"| M["Metadata master (NameNode)"]
    M -->|"2. chunk IDs + server locations"| C
    C -->|"3. read/write data directly"| CS1["Chunk server - rack 1"]
    C --> CS2["Chunk server - rack 2"]
    C --> CS3["Chunk server - rack 3"]
    CS1 -->|"heartbeats + chunk reports"| M
    CS2 --> M
    CS3 --> M
    M --> LOG[("Metadata edit log + snapshot")]
    SB["Standby master"] --> LOG

• Separate metadata from data: one master (with a standby) holds the namespace (the tree of files) and the mapping file → chunks → chunk servers, all in memory for speed. Clients get locations from it, then move data directly with chunk servers, so the master never becomes a data bottleneck.
• Large chunks (64–128 MB): fewer chunks to track (less metadata) and efficient sequential I/O.
• 3 replicas per chunk, placed rack-aware (e.g., one in the local rack, two in another rack), so a rack failure doesn't lose all copies.

3) Read Path

1. The client asks the master for the chunk locations for a byte range (and caches the answer).
2. It reads from the closest replica (same rack if possible) and verifies checksums on each block. On a mismatch, it reads another replica and reports the corruption.

4) Write / Append Path

1. The client asks the master for a new chunk. The master picks 3 servers and grants a lease to one of them (the primary) for this chunk.
2. The client pushes data to all replicas through a pipeline (client → A → B → C), so each link carries the data once.
3. The client tells the primary to commit. The primary assigns the order of mutations and tells the secondaries to apply in the same order, then replies success when all have written.
4. If a replica fails mid-write, the client retries. Appends are at-least-once (possible duplicate records), so apps use record IDs to deduplicate. HDFS simplifies this by allowing only a single writer per file.

5) Failures and Healing

• Chunk servers send heartbeats and chunk lists to the master. If a server is silent for a while, the master marks it dead and schedules re-replication of its chunks from surviving copies, prioritizing chunks with only 1 copy left.
• A background scrubber verifies checksums, and corrupted replicas are replaced.
• Balancer: moves chunks to even out disk usage.

6) Master Scalability and HA

• Metadata durability: every namespace change is written to an edit log (replicated, e.g., to a quorum of journal nodes) before being applied, with periodic snapshots (checkpoints).
• High availability: a hot standby master replays the same edit log and takes over on failure (with fencing so the old one can't keep writing).
• Scale limits: all metadata in one master's RAM (~150 bytes per file or chunk) limits the file count. Fixes: federation (several masters, each owning part of the namespace), or a distributed metadata store. Avoid millions of tiny files: pack them into bigger container files.
• Cheaper durability: erasure coding (e.g., 6 data + 3 parity) for cold data instead of 3 full copies, which cuts storage from 3x to 1.5x.

7) Wrap-Up

Split files into large chunks stored with rack-aware 3-way replication on many chunk servers, and keep the namespace and chunk map in memory on a metadata master backed by a replicated edit log and a hot standby. Clients get locations from the master but read and write data directly, writes use pipelines, a leased primary for ordering and checksums. Heartbeats drive automatic re-replication. Scale metadata with federation, and cut cost with erasure coding for cold data.