How to Generate Unique IDs at Scale

Every entity in a distributed system needs a unique ID : tweets, orders, messages, likes. The naive answer, auto-increment, collapses at scale because it requires a single sequence. This tutorial walks the ID generation approaches interviewers expect, from auto-increment to snowflake, with the trade-offs of each.

1. Requirements for an ID System

• Uniqueness across the whole system, with no coordination between writers
• Availability and scale: millions of IDs per second, no central bottleneck
• Index friendliness: roughly increasing, small footprint, no random-key index bloat
• No information leak: do not reveal total counts or creation rate

2. Option A: Auto-Increment (Single DB)

Works up to one database. Beyond that, replicas and multiple masters fight for the same sequence or produce collisions. Rule it out explicitly when the system is distributed, and say why.

3. Option B: UUID / GUID

Generate 128-bit random IDs anywhere with essentially zero collision probability.

• Pros: no coordination, universally unique, hides volume, trivial implementation.
• Cons: 128 bits (double the width), random ordering in B-trees causes index fragmentation and cache misses, and leaking none of the ordering can matter for audit trails.

4. Option C: Redis INCR / Database Ticket Server

A single Redis counter serves IDs via INCR. Simple and fast, but Redis becomes a single point of failure unless replicated, and a Redis outage stops ID issuance. Batch-number tricks (grab a range once, dish out locally) remove the round trip.

-- Reserve a range of 1000 IDs atomically
local start = redis.call('INCRBY', KEYS[1], 1000)
local end = start + 999
return { start, end }

5. Option D: Snowflake (the FAANG Favorite)

A 64-bit ID where each part encodes time, machine, and sequence. Time-origin = epoch; every node and millisecond is distinct.

graph LR
    subgraph "64-bit snowflake"
        B["1 bit
unused"]
        T["41 bits
timestamp (ms)"]
        M["10 bits
machine ID"]
        S["12 bits
sequence"]
    end
    style T fill:#D97A2B,stroke:#B86418,color:#fff
    style M fill:#FAF6EE,stroke:#E8DFC8
    style S fill:#FAF6EE,stroke:#E8DFC8

function snowflake(machineId: number, seq: { v: number }) {
  const elapsed = BigInt(Date.now() - EPOCH_MS);
  if (seq.v >= 4095) {                       // 12-bit sequence exhausted
    // wait for next millisecond (spin can be optimized with sleep)
    while (BigInt(Date.now() - EPOCH_MS) <= elapsed) { /* busy wait */ }
    seq.v = 0;
  }
  const id = (elapsed << 22n) | (BigInt(machineId) << 12n) | BigInt(seq.v++);
  return id;                                  // 1 + 41 + 10 + 12 = 64 bits
}

Properties: numerically time-ordered, 69 years of range with 41 bits, up to 4,096 IDs per millisecond per machine, and no coordination : each node gets a unique machine ID from a config service. The 10-bit machine field covers 1,024 workers, and the split is tunable.

6. Option E: Base62 / Short Codes

For user-facing short links you want compact IDs, so map a snowflake/UUID to base62. This is exactly how the URL shortener tutorial turns a 64-bit ID into a 7-character slug.

7. Choosing: Decision Table

8. Common Interview Mistakes

• Answering "just use auto-increment" in a distributed system without addressing the bottleneck.
• Picking UUID and ignoring index fragmentation when asked about performance.
• Not knowing the snowflake bit layout : name all four fields.
• Forgetting clock-skew handling for snowflake (wait, reject, or NTP discipline).
• Not connecting ID generation to the specific feature (short URL vs tweet vs order).

9. Summary: Key Decisions

Open with why auto-increment fails at scale, propose snowflake as the default for ordered 64-bit uniqueness, mention UUID where coordination-free or confidential counts are needed, and show the Redis/range trick when the interviewer wants a cache-integrated alternative.

Frequently Asked Questions

Related Tutorials

• URL Shortener : base62 short codes built from a 64-bit ID.
• Caching Strategies : Redis INCR as the ticket server.
• Database Sharding : why coordinated sequences die when you shard.
• System design case studies : ID schemes inside real architectures.