Design a Distributed Message Queue

Kafka-style Event Streaming System Design

Overview

A distributed message queue moves streams of events between services reliably. One service publishes events — say an Orders service emitting an OrderPlaced event for every checkout — and several other services consume them: a billing service, an inventory service, an analytics pipeline. The publisher does not call each consumer directly; it writes events to the queue once, and every consumer reads them at its own pace. This decouples producers from consumers: a slow or crashed consumer never blocks the producer, and you can add a new consumer later without touching the producer.

It is "Hard" because the core is an ordered, durable, replayable log that scales horizontally while surviving broker crashes. The system must accept a high write rate, keep events durable when a machine dies, let many independent consumers read the same stream, preserve ordering where it matters, and — the part candidates most often get wrong — offer honest delivery semantics (at-most-once, at-least-once, and the exactly-once effect) rather than promising a guarantee that does not exist.

In this walkthrough you'll scope the system, size it, model the append-only log and its offsets, design the produce/consume API, draw the produce and consume paths across a broker cluster, and go deep on partitioning & ordering, replication & durability, and consumer groups & delivery semantics.

Out of scope (state it): the exact wire protocol, cross-datacenter mirroring, schema registries, and stream-processing operators (joins, windows). We keep the transport — reliably moving ordered event streams from producers to consumer groups — not the processing layer on top.

Functional requirements

• Publish events to a topic. A producer appends an event (optionally with a key) to a named topic; the write is durable once acknowledged.
• Consume events from a topic. A consumer reads events in order from where it left off and moves forward. Reading does not delete the event — many consumers can read the same stream independently.
• Consumer groups for parallelism. A group of consumers splits a topic's work among its members so throughput scales by adding consumers.
• Track progress (offsets). Each consumer group records how far it has read, so it resumes after a restart instead of re-reading from the start or skipping ahead.
• Retain and replay. Events are kept for a retention window (time or size based), so a new or recovering consumer can replay history from any past position.

Out of scope (state it): deleting individual events by id (a log is append-only, not a key-value store), priority ordering, and exactly-once delivery as a wire guarantee (it does not exist — deep dive 3). Naming these keeps interview time on the log, partitioning, and semantics.

Non-functional requirements

• High throughput. The system must absorb a large, bursty write rate and fan it out to many consumers — millions of events per second in aggregate — by adding partitions and brokers, not by making one node faster.
• Durability. An acknowledged event must survive broker crashes. Data is replicated across brokers; losing one machine loses no acknowledged event.
• Ordering where it matters. Events with the same key (e.g. the same order_id) are delivered in the order they were produced. There is no global total order across the whole topic — only per-partition order (deep dive 1).
• Durability-tunable delivery. The producer chooses how strong an acknowledgment it waits for, trading latency for durability. The consumer chooses when it records progress, trading duplicates for loss (deep dive 3).
• Horizontal scalability. Throughput and storage scale by adding partitions and brokers. No single broker holds a whole topic.
• Replayability. Consumers can rewind to an earlier offset and reprocess, because the log is the durable source of truth, not the consumer's memory.

Estimations

State assumptions; the goal is to justify partitioning, replication, and retention sizing — not to be exact.

Rounded assumptions

• Write rate: ~1M events/sec produced across all topics at peak. High write QPS — this drives partitioning.
• Event size: ~1 KB average (an order event with ids, amounts, timestamps).
• Fan-out: each event is read by ~5 consumer groups (billing, inventory, search, analytics, audit). Reads ≈ 5× writes ≈ ~5M reads/sec, so the read path must be cheaper than the write path.
• Replication factor: 3 copies of every partition, so an acknowledged event survives losing 1–2 brokers.
• Retention: keep 7 days of events so a consumer can be down for a long weekend and still catch up, and a new consumer can replay a week of history.

How the numbers affect the design

Storage. 1M events/sec × 1 KB ≈ ~1 GB/sec written. Over 7 days that is ~605 TB of raw log, and at replication factor 3 it is ~1.8 PB on disk. That is far past one machine, which is exactly why a topic is partitioned across many brokers and retention is bounded — the log trims old segments as they age out.

Core entities / data model

The heart of the system is the partition: an append-only, offset-ordered log. Everything else — topics, offsets, consumer groups — is organized around it.

A partition is the core mental model. Picture a single file you can only append to. Each appended record gets the next integer offset — 0, 1, 2, 3, … — its position in that partition. Offsets are per-partition, so "offset 42" only means something once you also name the partition. A record is immutable: you never update or delete a single record, you only append new ones and let old ones age out under retention.

flowchart LR
    subgraph P0["Partition 0 (append-only log)"]
      direction LR
      R0["offset 0"] --> R1["offset 1"] --> R2["offset 2"] --> R3["offset 3<br/>(latest)"]
    end
    Producer["Producer"] -->|"append"| R3
    C1["Consumer group A<br/>reads at offset 2"] -.->|"poll"| R2
    C2["Consumer group B<br/>reads at offset 0"] -.->|"poll"| R0

Two consumer groups can sit at different offsets in the same partition and neither disturbs the other — reading is just "give me records from offset X onward," which does not mutate the log. This is why fan-out is cheap: the log is written once and read many times, and each group tracks its own committed offset (its bookmark).

API design

The interface is small: produce, consume (poll), commit an offset, and manage topics. A message queue exposes a client contract through a client library, not chatty REST — but we describe each call as an operation.

Produce (append an event)

POST /produce
Body: { "topic": "orders", "key": "order_123", "value": { ... }, "acks": "all" }
Returns: { "partition": 2, "offset": 84512 }   (the durable position the event landed at)

Append an event to a topic. If a key is supplied, the broker routes it to a partition by hash(key) % partition_count, so every event for order_123 lands in the same partition and stays ordered (deep dive 1). acks chooses the durability level — 0, 1, or all (deep dive 2). The response returns the (partition, offset) where the event was durably written.

Consume / poll (read forward)

GET /poll
Body: { "group": "billing", "topic": "orders", "max_records": 500 }
Returns: { "records": [ { "partition": 2, "offset": 84510, "key": "order_123", "value": {...} }, ... ] }

Fetch a batch of records for a consumer group, starting from each assigned partition's current fetch position. That position is initialized to the group's committed offset on (re)start or after a rebalance, then advances with each poll independently of commits — so the consumer keeps fetching ahead of what it has committed. Consumption is pull-based — the consumer asks for records when it is ready, so it controls its own rate and a fast broker never overwhelms a slow consumer (deep dive 3, pull vs push). The broker only returns records up to the partition's high-water mark (data replicated up to that offset — deep dive 2).

Commit offset (record progress)

POST /commit
Body: { "group": "billing", "topic": "orders", "partition": 2, "offset": 84510 }
Returns: { "committed": true }

Record that this group has processed up to offset in that partition, so a restart resumes from there. When you commit — before or after processing — decides your delivery semantics (deep dive 3). The commit is itself stored durably (in an internal offsets topic), so progress survives a consumer crash.

Create topic

POST /topics
Body: { "topic": "orders", "partitions": 12, "replication_factor": 3, "retention": "7d" }
Returns: { "topic": "orders", "partitions": 12 }

Create a topic with a fixed partition count, a replication factor, and a retention policy. Partition count is the parallelism ceiling for a consumer group (deep dive 1) and is hard to change later without breaking key ordering — so choose it with headroom.

High-level architecture

Two paths matter: a produce/append path where a producer writes to a partition leader and it replicates, and a consume path where a consumer group reads records replicated up to the high-water mark from partition leaders. We'll walk each, then combine them across a broker cluster.

1. Produce / append path

A produce request is routed to the right partition, appended to the leader's log, and replicated before it is acknowledged.

flowchart LR
    Producer["Producer<br/>(Orders service)"] -->|"1. batch + pick partition<br/>hash(key) % N"| Leader["Partition 2 Leader<br/>(broker 1)"]
    Leader -->|"2. append to log<br/>assign offset"| Log[("Append-only log<br/>+ page cache")]
    Leader -->|"3. replicate to ISR"| F1["Follower<br/>(broker 2)"]
    Leader -->|"3. replicate to ISR"| F2["Follower<br/>(broker 3)"]
    Leader -->|"4. ack once ISR has it (acks=all)"| Producer
    style Leader fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e

1. The producer batches several events and picks a partition: with a key, hash(key) % N; without a key, round-robin across partitions. Batching amortizes network and disk cost — one bigger sequential write instead of many tiny ones.
2. The request goes to that partition's leader broker, which appends the batch to its log and assigns each record the next offset. Appending is a sequential disk write, which is fast (deep dive 2).
3. The leader replicates the records to its follower replicas in the in-sync replica (ISR) set.
4. Depending on the producer's acks setting, the leader acknowledges: immediately (acks=1) or only after the ISR has the records (acks=all). Only now is the write durable to the level requested (deep dive 2).

2. Consume path

A consumer group is assigned partitions; each member polls its partitions' leaders for records past its committed offset and commits progress as it goes.

flowchart LR
    Coord["Group Coordinator"] -->|"1. assign partitions to members"| C1
    C1["Consumer 1<br/>(group billing)"] -->|"2. poll from committed offset"| L0["Partition 0 Leader"]
    C1 -->|"2. poll"| L1["Partition 1 Leader"]
    L0 -->|"3. return records <= high-water mark"| C1
    C1 -->|"4. process, then commit offset"| Off[("Offsets store<br/>(internal topic)")]
    style Coord fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e

1. A group coordinator assigns each partition of the topic to exactly one member of the group (deep dive 3). If members join or leave, it rebalances the assignment.
2. Each consumer polls its assigned partitions' leaders, asking for records from its last committed offset onward. Pull-based, so the consumer sets its own pace.
3. The leader returns records only up to the high-water mark — the highest offset known to be replicated across the ISR — so a consumer never sees an event that could still be lost on failover.
4. The consumer processes the batch and commits the new offset to the durable offsets store. On restart it resumes from the committed offset. Whether it commits before or after processing sets its delivery semantics (deep dive 3).

3. Combined architecture

Putting it together — producers append to partition leaders spread across a broker cluster; each partition is replicated ×3; consumer groups read records replicated up to the high-water mark in parallel and commit their offsets; a controller manages leadership and the metadata.

flowchart LR
    P["Producers"] -->|"produce (routed by key)"| BC
    subgraph BC["Broker Cluster"]
      direction TB
      B1["Broker 1<br/>P0 leader, P2 follower"]
      B2["Broker 2<br/>P1 leader, P0 follower"]
      B3["Broker 3<br/>P2 leader, P1 follower"]
    end
    BC -->|"replicate partitions across brokers (ISR)"| BC
    BC -->|"consume (per consumer group)"| CG1["Consumer group: billing"]
    BC -->|"consume"| CG2["Consumer group: analytics"]
    Ctrl["Controller<br/>(leader election, metadata)"] -.->|"assign leaders, track ISR"| BC
    Off[("Offsets topic")] -.->|"per-group committed offsets"| BC

The broker cluster holds all partitions; each broker leads some partitions and follows others, so leadership and load spread evenly. Every partition is replicated across brokers, and the ISR tracks which replicas are caught up. A controller (a coordination role backed by a consensus layer such as ZooKeeper or Raft) elects partition leaders and tracks membership, but it is off the hot data path. Each consumer group reads the whole topic independently at its own offset, and committed offsets live in an internal offsets topic so progress is as durable as the data itself.

Deep dives

1. Partitioning & ordering — where throughput and ordering come from

A single append-only log on one machine caps out at that machine's write rate. To scale, a topic is split into partitions, each an independent append-only log that can live on a different broker. This is the single source of both throughput and ordering, and the tension between them is where candidates slip.

Partitioning gives throughput. With 12 partitions spread over the cluster, 12 leaders accept writes in parallel and up to 12 consumers in a group read in parallel. Add partitions (and brokers) and the topic goes faster. A partition is the unit of parallelism.

A partition gives ordering — but only within itself. Records inside one partition are strictly offset-ordered. Across partitions there is no global total order: partition 0 and partition 1 advance independently, and there is no meaning to "which came first" between an event at P0/offset 5 and one at P1/offset 5.

Key-based partitioning buys per-key ordering. Route each event by partition = hash(key) % N. All events for order_123 hash to the same partition, so they are consumed in the exact order produced. Different orders spread across partitions for parallelism, while each individual order stays ordered. Choose the key so that "things that must stay ordered" share a key (e.g. key by order_id if per-order order matters).

key routing, N = 4 partitions:
  hash("order_123") % 4 = 2  → all order_123 events → partition 2 (ordered)
  hash("order_777") % 4 = 0  → all order_777 events → partition 0 (ordered)
  → per-order ordering preserved; the 4 partitions run in parallel

  all writes → partition 0 (one broker) → capped at one machine's append rate
  one consumer per group can read it → no parallel consumption
  → a single hot log becomes the whole system's bottleneck

flowchart LR
    P["Producer"] -->|"hash(order_id) % 12"| Router{"route by key"}
    Router --> Pa["Partition 3<br/>order_123 (ordered)"]
    Router --> Pb["Partition 7<br/>order_777 (ordered)"]
    Router --> Pc["... 12 partitions,<br/>drained in parallel"]
    style Pa fill:#dcfce7,stroke:#16a34a,color:#14532d
    style Pb fill:#dcfce7,stroke:#16a34a,color:#14532d

Producer batching drives throughput too. Producers group many records into one request per partition before sending. One larger sequential write beats thousands of tiny ones — it amortizes network round-trips and lets the broker do one big append. Batching (with a small linger delay to fill the batch) is a primary throughput lever alongside partition count.

2. Replication, durability & leader election — how a write becomes safe

A single copy of a partition means one dead disk loses acknowledged events. So each partition has a leader replica and one or more follower replicas on other brokers. The leader takes all reads and writes; followers copy the leader's log. The question is when a write is safe and what happens when the leader dies.

The ISR is the set of caught-up replicas. The in-sync replica (ISR) set is the leader plus every follower that is fully caught up to the leader's log. A follower that falls too far behind is dropped from the ISR until it catches up. Only ISR members are eligible to become the next leader — which is what makes failover lossless.

acks picks the producer's durability. The producer chooses how strong an acknowledgment to wait for:

acks=all is only as safe as min.insync.replicas. acks=all waits for the current ISR — which is not a fixed size. If followers fall behind and the ISR shrinks to just the leader, then with min.insync.replicas=1 the leader acknowledges on itself alone, and a leader crash right after loses that acknowledged write. Durability is governed by min.insync.replicas, not momentary ISR size: set it to ≥ 2 so the broker rejects writes when the ISR is too small rather than silently acking on one replica. Only then does acks=all survive RF − min.insync.replicas broker losses without data loss.

The high-water mark is what consumers can see. The high-water mark is the highest offset that every ISR member has. Consumers can only read up to it. This is the crucial safety rule: a consumer never sees a record that isn't yet safely replicated, so a leader failover can never "un-deliver" an event a consumer already processed. The producer ack and the high-water-mark advance are driven by the same replication, but the consumer-visible high-water mark advances a fetch round-trip later — the leader only learns the followers hold the record on their next fetch — so a just-acked record can briefly be durable but not yet consumable.

becoming durable with acks=all, ISR = {leader, f1, f2}:
  leader appends offset 84512               → its log has it
  f1 copies 84512, f2 copies 84512          → ISR all caught up
  leader acks the producer                  → producer's write is safe
  high-water mark advances to 84512         → visible to consumers a fetch later
  (if the leader dies now, f1 or f2 — both have 84512 — takes over losslessly)

Leader election on failure. If a leader broker dies, the controller (backed by the consensus layer — ZooKeeper or Raft) elects a new leader from the ISR. Because every ISR member is caught up to the high-water mark, the new leader already has every acknowledged event — no committed data is lost. A replica that was not in the ISR is not eligible (electing a lagging replica would silently drop acknowledged events).

Why the append-only log is fast (concept-first). A partition is written by appending to the end of a file, which is a sequential disk write — far faster than random writes because the disk head (or SSD controller) does not seek around. Recently written data stays in the operating system's page cache (RAM), so consumers reading near the tail are served from memory. And sending log data to a consumer can use zero-copy — the OS ships bytes from the page cache to the network socket without copying them through the application. Sequential append + page cache + zero-copy is why a log-based queue sustains such high throughput on ordinary hardware.

3. Consumer groups & delivery semantics — the headline

This is where interviews are won or lost. Two ideas: how a consumer group parallelizes reading, and what delivery guarantee you can honestly offer.

A consumer group divides a topic's partitions among its members. Each partition is consumed by exactly one consumer in the group at a time — never two, so per-partition order is preserved and work isn't duplicated within the group. With 12 partitions and 4 consumers, each consumer owns 3 partitions.

flowchart LR
    subgraph T["Topic: orders (12 partitions)"]
      P0["P0..P2"]
      P1["P3..P5"]
      P2["P6..P8"]
      P3["P9..P11"]
    end
    P0 --> C1["Consumer 1"]
    P1 --> C2["Consumer 2"]
    P2 --> C3["Consumer 3"]
    P3 --> C4["Consumer 4"]
    Note["Each partition → exactly one consumer in the group.<br/>Different groups each get the whole topic independently."]

Partition count caps a group's parallelism. A group can usefully run at most N consumers for N partitions — a 13th consumer sits idle because there's no unassigned partition. This is why partition count is the parallelism ceiling from deep dive 1.

Rebalancing on join/leave. When a consumer joins or crashes, the group coordinator rebalances — reassigning partitions across the surviving members so every partition still has exactly one owner. In the classic (eager) protocol a rebalance is group-wide and stop-the-world: every consumer revokes all its partitions and pauses, then the group resumes from each partition's committed offset. A partition revoked after its records were processed but before that progress was committed is reprocessed by the new owner — the at-least-once duplicates you see at handoff. And if a consumer takes so long between polls that it exceeds the max poll interval, the coordinator assumes it died and rebalances, which can cascade into repeated rebalances that starve progress.

Offset commits track progress. Each group stores, per partition, the offset it has processed up to. On restart or reassignment, a consumer resumes from the committed offset. The subtle question — the whole ballgame — is when you commit relative to when you process.

Now the delivery-semantics ladder. Be precise: exactly-once delivery over a network does not exist; what you can build is an exactly-once effect.

  read offset 84510 → COMMIT 84511 → crash before processing 84510
  restart → resume from 84511 → offset 84510 is never processed → LOST

  read offset 84510 → process (send invoice) → crash before COMMIT
  restart → resume from 84510 → process AGAIN → duplicate invoice

  process offset 84510 (order_123) → write "order_123 handled" (unique key) → crash before COMMIT
  restart → reprocess 84510 → INSERT order_123 → already exists → NO-OP
  → the effect happened exactly once, even though delivery happened twice

Retention: the log is the source of truth, consumers are cheap. Records are kept for a retention window — by time (e.g. 7 days) or by size (e.g. 1 TB per partition) — and old log segments are deleted as they age out, regardless of who has read them. Because the durable log holds history, a consumer is just a moving offset (a bookmark), so adding a new consumer group that replays a week of history costs nothing on the write path — it reads the existing log from offset 0. The log, not the consumer, is authoritative.

Consumer lag & backpressure: the queue absorbs bursts. Consumer lag is how far behind the tail a group is — latest offset minus committed offset. When a burst arrives faster than a consumer processes, the extra events pile up in the durable log, and the consumer drains them when it catches up. The producer is never blocked by a slow consumer. Monitor lag as your health signal: rising lag means scale the group (up to N partitions) or speed up processing. Because consumption is pull-based (below), the consumer sets its own rate and simply falls behind rather than being overwhelmed.

Pull vs push: consumers control the rate. Consumers pull batches when ready rather than having the broker push records at them. Pull lets each consumer set its own pace, batch efficiently, and fall behind gracefully under load (the log buffers the backlog) instead of being flooded. The cost is a little latency and some empty polls when there's nothing new — a good trade for a system whose whole point is decoupling fast producers from slower, independent consumers.

Tradeoffs & bottlenecks

• Per-key order vs global order. You get strict order within a partition (per key), never a global total order across the topic. A single-partition topic buys global order at the cost of all parallelism. Order per key, not per topic.
• Partition count is a hard commitment. More partitions = more parallelism and throughput, but raising the count later re-maps keys and breaks ordering, and too many partitions add per-partition overhead (open files, replication streams). Pick with headroom.
• Durability vs latency (acks). acks=all (with min.insync.replicas ≥ 2) survives broker loss but waits for the ISR; acks=1 is faster but can lose data on leader failover; acks=0 can lose data with no crash. Choose per topic by how precious the data is.
• At-least-once vs at-most-once. Commit-after-process risks duplicates; commit-before-process risks loss. Most systems pick at-least-once + idempotent consumers. Exactly-once is an effect, not a delivery guarantee.
• Hot partition (key skew). One very hot key (e.g. a single huge account) overloads its one partition while others idle. May need a composite key or per-key sub-partitioning.
• Rebalance pauses. Adding/removing consumers triggers a group-wide (eager) rebalance that stops consumption for the whole group, and partitions revoked before their offset commit are reprocessed by the new owner (handoff duplicates). Frequent membership churn (flapping consumers, or slow processing that exceeds the max poll interval) can cascade into repeated rebalances that starve progress.
• Retention vs storage. Longer retention enables longer replay windows but multiplies storage (×replication factor). Bound it and let segments age out.

Extensions if asked

If the interviewer wants to push beyond the core, add only the extension that changes the design discussion. (These are standalone topics; there's no dedicated guide for them yet.)

• Dead-letter topic. After N failed processing attempts, route a poison-pill record to a separate topic so one bad event doesn't block a partition forever.
• Compacted topics. Retain only the latest value per key (log compaction) so the topic acts as a durable changelog / snapshot rather than a full history.
• Tiered / cheaper storage. Offload old log segments to object storage so retention can be long without keeping everything on broker disks.
• Cross-datacenter replication. Mirror topics to another region for disaster recovery, accepting async lag and the ordering caveats it introduces.
• Schema registry. Enforce a versioned schema on event values so producers and consumers evolve the payload safely.
• Transactions / exactly-once pipelines. Atomic read-process-write across topics for a full exactly-once effect between stages.

What interviewers look for & common mistakes

What interviewers usually reward:

• The partition as the core unit — an append-only, offset-ordered log that is both the unit of ordering and of parallelism, with a topic split across brokers for throughput.
• Honest ordering — per-partition (per-key) order, and the explicit statement that there is no global total order across a topic.
• Replication and the ISR — leaders + followers, acks as the durability knob, the high-water mark as the consumer read ceiling, and leader election from the ISR so failover is lossless.
• Consumer groups — one partition per consumer in a group, rebalancing on join/leave, offsets as durable progress.
• Precise delivery semantics — at-most-once vs at-least-once by when you commit, and exactly-once as an effect (idempotency or transactions), never as a delivery guarantee.
• Log economics — retention makes the log the source of truth, consumers are cheap bookmarks, lag/backpressure let the queue absorb bursts, and pull lets consumers set their rate.

Before you finish, do a quick mistake check:

• Did you split the topic into partitions across brokers for throughput, not lean on one big log?
• Did you say there is no global total order — only per-partition/per-key order — and route by hash(key) % N?
• Did you flag that changing partition count breaks key ordering, so it's a hard up-front choice?
• Did you replicate each partition, explain acks (0/1/all) durability, and elect leaders only from the ISR?
• Did you use the high-water mark so consumers never read un-replicated data?
• Did you give each partition to exactly one consumer in a group and handle rebalancing?
• Did you tie delivery semantics to when you commit, and describe exactly-once as an effect (idempotent consumer / transactions), not a delivery guarantee?
• Did you cover retention, consumer lag / backpressure, and pull vs push?