Why message queues?Core idea
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","actorBkg":"#e7f5ff","actorBorder":"#1e1e1e","noteBkgColor":"#fff9db","noteBorderColor":"#1e1e1e"}}}%%
graph LR
P["Producer<br/>(order service)"] -->|"order created"| Q["Message queue"]
Q --> C1["Payment"]
Q --> C2["Inventory"]
Q --> C3["Email"]
class P blue
class Q orange
class C1,C2,C3 green
classDef orange fill:#ffd8a8,stroke:#1e1e1e,color:#1e1e1e
classDef blue fill:#a5d8ff,stroke:#1e1e1e,color:#1e1e1e
classDef green fill:#b2f2bb,stroke:#1e1e1e,color:#1e1e1e
classDef red fill:#ffc9c9,stroke:#1e1e1e,color:#1e1e1e
classDef violet fill:#d0bfff,stroke:#1e1e1e,color:#1e1e1e
classDef yellow fill:#ffec99,stroke:#1e1e1e,color:#1e1e1e
classDef grey fill:#e9ecef,stroke:#1e1e1e,color:#1e1e1eDecoupling
Services don’t need to know about each other. Add a new consumer without touching the producer.
Buffering
If messages arrive faster than they can be handled, the queue absorbs the burst.
Async processing
Slow work (image processing, sending email) runs in the background instead of blocking the request.
Fan-out
One message can be read by several services at once (the publish/subscribe or “pub/sub” pattern).
Order processing exampleExample
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","actorBkg":"#e7f5ff","actorBorder":"#1e1e1e","noteBkgColor":"#fff9db","noteBorderColor":"#1e1e1e"}}}%%
sequenceDiagram
participant Client
participant OrderSvc as Order service
participant Q as Kafka
participant Pay as Payment
participant Inv as Inventory
participant Mail as Email
Client->>OrderSvc: place order
OrderSvc->>Q: publish order-created
OrderSvc-->>Client: order confirmed
Q->>Pay: process payment
Q->>Inv: reserve inventory
Q->>Mail: send confirmation
Note over Pay,Mail: all three consume independentlyKafka vs RabbitMQCompare
Apache Kafkaevent streaming
Messages are stored in topics, split into partitions for parallelism, and kept for a set time even after they are read. Consumers read at their own pace. Best for event sourcing, log aggregation, real-time analytics and high-throughput pipelines.
RabbitMQmessage broker
A traditional broker: messages are deleted once consumed. It supports rich routing (topic and header exchanges), priorities and TTLs. Best for task queues, RPC-style requests and notifications.
| Feature | Kafka | RabbitMQ |
|---|---|---|
| Model | Event streaming | Message broker |
| Retention | Configurable (days or weeks) | Until consumed |
| Ordering | Within a partition | Per queue |
| Throughput | Millions per second | Tens of thousands per second |
| Use case | Log aggregation, analytics, event sourcing | Task queues, RPC, notifications |
Delivery guaranteesHow it works
At-most-oncemay lose
Send once, never retry. Fast, but messages can be lost. Fine for logs and metrics.
At-least-oncemay duplicate
Retry until the consumer acknowledges. Nothing is lost, but duplicates happen, so the consumer must be idempotent. The most common choice.
Exactly-oncehard
Each message takes effect once. Very hard in general; Kafka offers it within one session using its transactional APIs.
Message orderingHow it works
Kafka keeps messages in order within a partition, not across partitions. If one user's messages must be handled in order, send them all to the same partition by using the user ID as the partition key.
producer.send({
topic: "order-events",
key: userId, // same userId → same partition → in order
value: { orderId, action: "created" },
});Dead-letter queuesDesign
A dead-letter queue (DLQ)catches messages that still fail after the maximum number of retries. Without one, a single bad message (a “poison pill”) can block the whole queue.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","actorBkg":"#e7f5ff","actorBorder":"#1e1e1e","noteBkgColor":"#fff9db","noteBorderColor":"#1e1e1e"}}}%%
graph LR
Q["Main queue"] --> C["Consumer"]
C -->|"max retries exceeded"| DLQ["Dead-letter queue"]
DLQ --> A["Alert the team"]
class Q orange
class C green
class DLQ red
class A yellow
classDef orange fill:#ffd8a8,stroke:#1e1e1e,color:#1e1e1e
classDef blue fill:#a5d8ff,stroke:#1e1e1e,color:#1e1e1e
classDef green fill:#b2f2bb,stroke:#1e1e1e,color:#1e1e1e
classDef red fill:#ffc9c9,stroke:#1e1e1e,color:#1e1e1e
classDef violet fill:#d0bfff,stroke:#1e1e1e,color:#1e1e1e
classDef yellow fill:#ffec99,stroke:#1e1e1e,color:#1e1e1e
classDef grey fill:#e9ecef,stroke:#1e1e1e,color:#1e1e1eCommon interview mistakesPitfalls
In the interviewIn the interview
Frequently asked questionsFAQ
When should you add a message queue to a design?
Add a queue when you need to decouple producers from consumers, smooth traffic bursts, or run work that can be handled later. The canonical cases are async jobs like email, heavy fan-out, and pipeline stages where a slow stage must not block the request path.
What is the difference between at-least-once and exactly-once delivery?
At-least-once retries every message until it is acknowledged, so consumers must be idempotent to handle duplicates. Exactly-once removes duplicates but needs transactional or deduplicated consumers and costs throughput. Most systems pick at-least-once plus consumer-side dedup, which behaves like once in practice.
How does a queue keep ordering across multiple partitions?
Systems like Kafka keep messages ordered within a partition, not globally across all partitions. If global order matters, use a single partition or route related messages to the same partition with a shared key; otherwise accept per-partition ordering to gain parallelism.
Now explain message queues to me out loud
I'll give you a design problem and push on the trade-offs, like a real interviewer.