Design ChatGPT

LLM Inference Service System Design

Overview

An LLM inference service takes a user's prompt, runs it through a large language model, and streams back a generated reply — this is the serving backend behind ChatGPT, Claude, Gemini, and any "chat with an AI" product. A user types a question; the system must produce an answer that appears word by word, fast, for millions of concurrent users, while the expensive machines doing the work stay busy.

This is not about training the model (that's a separate, offline, weeks-long job) and not about the model's internal architecture. It's about serving an already-trained model: getting a prompt in, generating tokens out, and streaming them to the user.

It's labeled "Hard" because the scarce resource is unusual. In most systems the bottleneck is storage, network egress, or database throughput. Here it is GPUs — the specialized processors that run the model. They are expensive, supply-limited, and the single largest cost in the system. A correct design is one that keeps those GPUs as busy as possible while still feeling fast to each user. Two ideas dominate: streaming the answer token by token so the user sees output immediately, and batching many users' requests onto one GPU at once so no compute is wasted.

In this walkthrough you'll scope the service, size it (the GPU and token numbers drive everything), model the data, define the streaming response contract, draw the request flow, and then go deep on token streaming, GPU scheduling with continuous batching, and tiered rate limiting.

Functional requirements

These are the things the system must do. Keep the list short and confirm scope with the interviewer first.

• Submit a prompt and get a reply. A user sends a prompt (plus the prior conversation as context) and receives a model-generated completion.
• Stream the reply incrementally. The answer arrives token by token as it is generated, not in one final blob — this is the defining user experience.
• Enforce per-user limits. Free-tier users are capped (by requests and by tokens); paying users get higher limits and priority.
• Stop / cancel a generation. A user can stop a response mid-stream, and the system must free the GPU work immediately.

Out of scope (state it): training or fine-tuning the model, the model's internal architecture, retrieval-augmented generation (RAG), and tool/function calling. We mention RAG and tools as extensions, but the core problem is streaming inference + GPU scheduling + rate limiting.

Non-functional requirements

These are the qualities the system must have. For an inference service the dominant ones are:

• Low time-to-first-token (TTFT) and steady streaming. The user should see the first word quickly (target a few hundred milliseconds at the p99) and then a smooth flow of tokens. TTFT matters more than total time, because a response that starts fast feels fast even if the full answer takes several seconds.
• High GPU utilization. GPUs are the scarce, expensive resource. Idle GPU time is wasted money. The whole scheduling design exists to keep them full.
• Massive concurrency. Millions of users, each holding a long-lived streaming connection that stays open for the whole multi-second generation. The connection model must support many simultaneous open streams.
• Graceful degradation under load. When demand exceeds GPU capacity, the system must queue, prioritize paid traffic, and reject excess politely (a 429), never crash or hang.
• Fairness across tiers. A flood of free-tier traffic must not starve paying customers.

Estimations

State your assumptions out loud; the goal is to show how the numbers force the architecture (batching, queueing, autoscaling the GPU pool), not to divide perfectly.

Rounded assumptions

• Users: ~100M daily active, sending ~1B messages/day → on the order of ~10K–50K prompts/sec at peak.
• Output length: an average reply is ~500 tokens; prompts (with conversation history) average ~1K tokens of context.
• A single modern GPU running one large model generates on the order of ~50–100 tokens/sec per active sequence; a multi-GPU server (e.g. 8 GPUs with a large model sharded across them) with batching produces on the order of a few thousand tokens/sec aggregate across many concurrent sequences.
• A GPU (or multi-GPU server) is expensive and supply-constrained — the dominant cost line by far.

Token throughput (the number that drives the design)

• ~30K prompts/sec × ~500 output tokens each = ~15M output tokens/sec generated at peak.
• If one multi-GPU server yields, say, a few thousand tokens/sec aggregate, you need on the order of tens of thousands of GPUs running continuously just to keep up — and that is the entire cost of the system.
• Therefore every percentage point of GPU utilization is money. A design that leaves GPUs idle waiting for slow requests is a design that costs far more for the same output.

Concurrent streams

• ~30K prompts/sec, each streaming for ~5–10 seconds → ~150K–300K simultaneously open response streams at any instant.
• Each is a long-lived connection (not a quick request/response). The gateway and connection layer must hold hundreds of thousands of open streams cheaply.

How the numbers affect the design

Core entities / data model

Inference is largely stateless per request — the heavy "state" is the model weights loaded on the GPUs, not rows in a database. The data model is small and mostly supports limits, conversation history, and observability.

A few points worth saying out loud:

• Conversation history is the prompt. A chat model is stateless between calls — it doesn't "remember" anything. To continue a conversation you resend the prior messages as context each time. That's why prompts average ~1K tokens: they carry the history. Storing Message rows lets you reconstruct that context, but the model itself holds no per-user memory.
• Usage counters live in a fast in-memory store, not the main database. Every request reads and updates them, so they belong in something built for high-rate atomic counters (the same kind of store a rate limiter uses), with a per-window TTL — not a SQL row you lock on each call.
• The model weights are the real "data," and they live on the GPUs. Loading a large model onto a GPU (or sharding it across several) takes time and memory; you do it once at worker startup, not per request. This is why a GPU worker is a long-lived, warmed-up process, not something you spin up per call.

API design

The one endpoint that matters is submit a prompt and stream the completion. Unlike a normal REST call that returns one JSON body, this response streams many small chunks over a single long-lived connection, one per token (or small group of tokens), until the model finishes.

Create a streaming completion

POST /v1/chat/completions
Body: {
  "conversation_id": "c_123",
  "messages": [ {"role": "user", "content": "Explain TTFT"} ],
  "model": "default-large",
  "stream": true,
  "max_tokens": 800
}
200 OK
Content-Type: text/event-stream     ← Server-Sent Events: a stream of chunks, not one body

Returns (a sequence of SSE events, sent as they are generated):
  data: {"request_id": "r_789"}     ← sent first, so the client can cancel this request
  data: {"delta": "Time"}
  data: {"delta": " to"}
  data: {"delta": " first"}
  data: {"delta": " token"}
  ...
  data: {"delta": ".", "finish_reason": "stop"}
  data: [DONE]

The client opens the request and then reads events as they arrive instead of waiting for a complete body. The first event carries the request_id — the client needs it to target the cancel endpoint below, since the streaming response otherwise returns no request handle. Each subsequent data: line carries a delta — the next token(s) of text. A terminal event (finish_reason plus a [DONE] marker) tells the client the stream is complete. Because the connection stays open for the whole multi-second generation, this is fundamentally different from a normal request (deep dive 1). The transport is SSE (Server-Sent Events) over plain HTTP, or chunked transfer encoding — both let the server push pieces as they're ready.

Cancel a generation

POST /v1/chat/completions/{request_id}/cancel
200 OK
Returns: { "request_id": "r_789", "status": "CANCELLED" }

A user who stops the response (or closes the tab) must free the GPU work — those tokens cost real compute. Cancellation signals the scheduler to drop the sequence from the active batch so the GPU stops generating for it. A dropped SSE connection is detected lazily — when the server's next token write to the closed socket fails, not as an instant push — and that failed write triggers the same cleanup: evict the sequence and free its GPU slot and KV-cache memory.

High-level architecture

A request flows through a gateway, a rate-limit check, a queue, a scheduler, the GPU workers, and then the generated tokens stream back along the same connection. We'll walk the request path, then the token-return path, then combine them.

1. Request admission path

This is everything that happens before a GPU touches the request: authenticate, rate-limit, and enqueue.

flowchart LR
    User[User] -->|"1. POST /chat/completions (stream)"| GW[API Gateway]
    GW -->|"2. check tier limits"| RL[Rate Limiter]
    RL -->|"3. over limit: 429 + Retry-After"| User
    RL -->|"4. allowed: enqueue request"| Q[Request Queue - priority by tier]
    Q -->|"5. scheduler pulls when GPU has room"| Sched[GPU Scheduler]

1. The user opens a streaming request at the API gateway, which holds the long-lived connection open.
2. The gateway calls the rate limiter, which checks the user's tier limits (by requests and by tokens — deep dive 3).
3. If the user is over the limit, return 429 Too Many Requests with Retry-After immediately — no GPU work.
4. If allowed, the request is placed on a queue in front of the GPU pool. The queue is priority-ordered so paid traffic is served ahead of free traffic under load.
5. The scheduler pulls requests from the queue whenever a GPU has spare capacity in its current batch.

2. Generation + token-return path

Once the scheduler admits a request, the GPU generates tokens autoregressively and each token streams straight back to the waiting connection.

flowchart LR
    Sched[GPU Scheduler] -->|"1. add to running batch"| GPU[GPU Workers - model loaded]
    GPU -->|"2. generate next token (one step)"| GPU
    GPU -->|"3. emit token"| Stream[Streaming Layer]
    Stream -->|"4. SSE chunk per token"| GW[API Gateway]
    GW -->|"5. data: {delta} to the open connection"| User[User]
    GPU -.->|"6. repeat until stop/EOS, then free slot"| Sched

1. The scheduler adds the request to a GPU's running batch alongside other concurrent requests.
2. Each GPU step generates one more token for every sequence in the batch at once (the model is autoregressive — it produces output one token at a time, each depending on all prior tokens).
3. The new token is emitted to the streaming layer as soon as it exists.
4. The streaming layer wraps it as an SSE chunk.
5. The gateway pushes the chunk down the user's already-open connection.
6. Steps 2–5 repeat until the model emits an end-of-sequence token or hits max_tokens; the sequence then leaves the batch and frees a slot for a queued request.

3. Combined architecture

Putting it together — admission on the left, the GPU pool and its scheduler in the middle, the token stream flowing back to the user:

flowchart LR
    User[User] -->|"stream request"| GW[API Gateway]
    GW --> RL[Rate Limiter]
    RL -.->|"counters"| Counters[(Usage Store - in-memory)]
    RL -->|"429 if over"| User
    RL -->|"allowed"| Q[Request Queue - priority by tier]
    Q --> Sched[GPU Scheduler]
    Sched -->|"assign to batch"| GPU[GPU Worker Pool - model loaded]
    GPU -->|"token stream"| GW
    GW -->|"SSE deltas"| User
    GPU -.->|"utilization / queue depth"| Auto[Autoscaler]
    Auto -.->|"add/remove workers (slow)"| GPU
    GW -.->|"persist messages async"| MsgDB[(Message DB - sharded)]

The admission side (gateway → rate limiter → queue) is fast and cheap. The expensive side is the GPU pool. At fleet scale that pool is split into many replicas (each a small group of GPUs with the model loaded), and work assignment happens at two levels: a fleet-level router / load balancer places each request on a replica (by load, queue depth, or session affinity), and a per-replica scheduler owns continuous batching for the GPUs in that one replica. There is no single scheduler across thousands of GPUs — that would be a bottleneck; the single-point-of-assignment property holds only within a replica. The queue decouples incoming demand from finite GPU supply: bursts wait in the queue instead of crashing the workers. An autoscaler watches utilization and queue depth and adds replicas — but slowly, because loading a multi-gigabyte model onto a fresh GPU takes minutes, so the queue (not instant scaling) absorbs short spikes.

Deep dives

1. Token streaming — why and how

Why the response streams at all. A language model is autoregressive: it generates the answer one token at a time, and each token is computed from the prompt plus every token it has already produced. It physically cannot produce a 500-token answer in one shot — it must loop 500 times, producing one token per step. That has a direct latency consequence:

• Time to first token (TTFT) is the delay until the first token appears. It is dominated by the prefill step — the model reading and processing the whole prompt once before it can produce anything.
• Total latency is TTFT plus ~500 more decode steps. At, say, 50 tokens/sec that's ~10 seconds.

If you waited for all 500 tokens before responding, the user stares at a blank screen for ten seconds. By streaming each token the moment it's produced, the user starts reading after a few hundred milliseconds. Same total time, hugely better experience. Streaming optimizes the latency the user actually feels.

The transport. A normal API call is one request, one response. Here the server must push ~500 small pieces over several seconds on one connection. Two standard choices:

 t=0.0s  client opens POST /chat/completions (stream=true)
 t=0.01s server writes: data: {"request_id":"r_789"}            ← lets the client cancel
 t=0.3s  prefill done → server writes: data: {"delta":"Time"}    ← TTFT
 t=0.32s data: {"delta":" to"}
 t=0.34s data: {"delta":" first"}
   ...   one chunk every ~20ms as tokens are produced
 t=9.8s  data: {"delta":".", "finish_reason":"stop"}
 t=9.8s  data: [DONE]   → server closes the response

 client → server: audio chunks (user speaking)
 server → client: token deltas (assistant replying)
 client → server: "stop" (user interrupts mid-reply)   ← needs the back-channel
 server → client: closes that generation

2. GPU scheduling and continuous batching — the heart of it

This is where the question is won or lost. The GPU is the scarce, expensive resource, and the entire job of this layer is to keep it busy without making any single user wait too long.

Why batch at all. A GPU is a massively parallel processor. Running the model for one request uses a tiny fraction of its compute — most of the chip sits idle, and you've paid for the whole thing. If you instead run many requests through the model together (a batch), the GPU produces one token for every request in roughly the same time it took for one. Throughput multiplies; cost per token plummets. So you batch to turn an underused expensive GPU into a fully-used one. Decode is memory-bandwidth-bound: each step's slow part is streaming the model weights out of GPU memory, not doing math. At small batch sizes the full weight matrix is re-read from memory every step regardless of how many sequences ride along, so adding sequences amortizes that fixed read over more tokens — which is why batching sharply raises decode throughput. But it isn't free: as the batch grows it eventually becomes compute- or KV-bound, and per-token latency starts to rise — the throughput-vs-latency tension again.

This creates the core tension: throughput vs latency. Bigger batches = higher total tokens/sec (better GPU utilization, lower cost) but each individual request can wait longer. The scheduler's job is to keep batches as full as possible without starving anyone.

Static batching (the naive version).

 step:  req A (needs 10 tokens) ████████░░  done at step 10, then IDLE ░░░░░░░░
        req B (needs 12)        ██████████  done at step 12, then IDLE ░░░░░░
        req C (needs 80)        ██████████████████████████████  still going
        req D (needs 15)        ████████████  done at step 15, then IDLE ░░░
                                ▲ A,B,D finished but their GPU slots sit empty
                                  until C finishes at step 80

Continuous (in-flight) batching — the real answer.

 batch slot 1: A(10) ████████ │ E(new) ███████████████...   ← E joins the instant A finishes
 batch slot 2: B(12) █████████ │ F(new) ██████████...        ← F joins when B finishes
 batch slot 3: C(80) ██████████████████████████████████████ ← long one keeps running
 batch slot 4: D(15) ████████████ │ G(new) ████████...       ← G joins when D finishes
               ▲ no idle gaps — freed slots are refilled every step

KV cache — the real limit on batch size. As the model generates, it keeps a per-sequence cache of intermediate state for every prior token so it doesn't recompute the whole context each step — the KV cache. This cache lives in GPU memory and grows with the number of sequences in the batch and the length of each sequence. Per token it's roughly 2 × num_layers × hidden_size × bytes (a key and a value vector across every layer), so it adds up fast — which is exactly why it, not compute, caps the batch size. So the limit on how many requests you can batch is usually not GPU compute — it's GPU memory for the KV cache. A long conversation (big context) eats more KV-cache memory, so fewer such requests fit per GPU.

 GPU memory budget:
 ┌───────────────────────────────────────────────┐
 │ model weights (fixed, loaded once)             │  ← large, constant
 ├───────────────────────────────────────────────┤
 │ KV cache: seq A │ seq B │ seq C │ ... free?     │  ← grows per active sequence
 └───────────────────────────────────────────────┘
   batch size is capped when KV cache fills the remaining memory

Prefill vs decode — admitting a new sequence stalls the stream. Refilling a freed slot isn't free: a new sequence must run its prefill (a compute-heavy parallel pass over the whole prompt) before it can decode. Prefill is compute-bound and bursty, while decode is steady and memory-bandwidth-bound (one token per step). Naively injecting a long prefill into a running decode batch monopolizes the GPU for that step and stalls everyone else's token stream — a visible TTFT/streaming hiccup for all the active users. The standard mitigations: split a long prefill into smaller pieces interleaved with decode steps (chunked prefill), or run prefill and decode on separate GPU pools so neither blocks the other (disaggregated prefill/decode).

The scheduler is the single point of assignment within a replica. Within one replica, a single logical scheduler decides which requests are in that replica's GPU batch at each step. This matters: because it's the single place that assigns work for those GPUs, it can enforce priority (paid before free), respect KV-cache limits, and refill freed slots immediately. It's the concurrency-control center of the replica — the GPUs just execute what it hands them, step by step. This is not a fleet-wide singleton: a separate fleet-level router (below) spreads requests across many replicas, each with its own scheduler.

3. Rate limiting and fairness across tiers

Free users must be capped; paying users get more and shouldn't be starved by free-tier floods. Two twists make this harder than a generic rate limiter.

Limit by requests and by tokens. A normal API limiter counts requests. Here, requests are not the real cost — tokens are. One request asking for 4,000 tokens costs far more GPU time than one asking for 10. So you enforce two kinds of limit per user: requests-per-window (e.g. free tier: 20 messages/3 hours) and tokens-per-window (e.g. free tier: 40K tokens/day). Token limits are what actually protect the GPU budget.

 free tier:  20 requests / 3h   AND   40K tokens / day
 paid tier:  unlimited-ish requests   AND   higher token budget + priority

Count input and output tokens. A token limit (and billing) must charge prompt/prefill tokens plus generated output tokens, not output alone. Otherwise a flood of huge prompts that each generate only a few tokens consumes large amounts of prefill compute — the most compute-heavy phase — while barely moving an output-only counter, letting the user evade the limit that is supposed to protect your GPU budget. In practice you debit the known input tokens up front, then count output tokens as they stream (since output length isn't known until generation finishes), and stop a generation that would blow the budget — returning a partial result with a "limit reached" marker.

The atomic counter under concurrency. A user can have several requests in flight at once (multiple tabs, retries). If each request does "read the count, check it's under the limit, write the new count," two requests can read the same old value, both think they're allowed, and both proceed — the user slips over the limit. This is a read-modify-write race.

The fix is to make "increment and check the limit" a single atomic step the datastore performs as one indivisible unit, so no two requests can both act on a stale count. (Concretely, an in-memory store like Redis does this with an atomic increment — INCR — returning the new total, optionally wrapped in a small atomic script that also checks the limit and sets the window's expiry.) Lead with the concept — one indivisible increment-and-check — the specific command is just the tool that provides it.

Fairness: priority queueing. Rate limits cap individual users, but the system-wide bottleneck is total GPU capacity. When demand exceeds supply, which admitted requests run first? Use a priority queue in front of the scheduler: paid-tier requests are dequeued ahead of free-tier ones, so a surge of free traffic waits while paying customers keep flowing. You can also reserve a fraction of GPU capacity for paid traffic so it's never fully blocked, and apply per-user fair scheduling within a tier so one heavy free user can't hog the free-tier share.

Graceful rejection. When a user is over their limit, or the queue is too deep to admit them in reasonable time, return 429 Too Many Requests with a Retry-After header telling them when to try again — never a 500 or a silently hung connection. Well-behaved clients then back off instead of retrying immediately and amplifying the overload.

Tradeoffs & bottlenecks

• Throughput vs latency (batch size). Larger batches raise GPU utilization and lower cost per token but increase per-request latency; smaller batches are snappier but waste GPU. Continuous batching gets most of the throughput benefit while keeping latency reasonable — but the batch-size knob is still a deliberate tradeoff.
• GPU memory vs concurrency. The KV cache caps how many sequences fit per GPU. Long contexts (big conversation histories) cut concurrency. Techniques like KV-cache paging help, but memory is the ceiling.
• Prefill vs decode (TTFT vs throughput). Prefill is compute-bound and bursty; decode is steady and memory-bandwidth-bound. Injecting a long prefill into a running decode batch stalls everyone's token stream, so heavy prefill batching can hurt first-token latency and streaming smoothness for active users. The mitigations are to interleave prefill in chunks with decode (chunked prefill) or to run prefill and decode on separate GPU pools (disaggregated prefill/decode) — both add scheduler complexity in exchange for protecting the decode stream.
• Autoscaling is slow. You can't scale GPUs instantly — loading a large model onto a fresh GPU takes minutes, and GPU supply is limited. The queue absorbs short spikes; for sustained load you provision ahead of demand and over-provision somewhat, accepting idle cost as insurance against rejecting users.
• Cost vs experience. Bigger or higher-quality models cost more GPU per token. Offering a smaller/faster model to free users and the large model to paid users is both a cost lever and a tiering lever.
• Fairness vs utilization. Reserving capacity for paid traffic protects them but can leave GPUs idle when paid demand is low. Tune the reserved fraction to balance protection against waste.

Extensions if asked

If the interviewer wants to push beyond the core, add only the extension that changes the design discussion. These are standalone topics, not covered in depth here:

• Retrieval-augmented generation (RAG). Before generating, fetch relevant documents from a vector database and prepend them to the prompt as context. This adds a retrieval step and grows the prompt (more prefill, more KV cache) but doesn't change the streaming/scheduling core.
• Tool / function calling. The model can emit a structured request to call an external tool (search, code execution), the system runs it, and feeds the result back for continued generation. This turns one logical turn into multiple inference passes.
• Prompt / prefix caching. Many requests share a common prefix (a long system prompt, or a reused conversation head). Caching the KV state for that shared prefix skips recomputing prefill for it, cutting TTFT and GPU work.
• Speculative decoding. A small, cheap model drafts several tokens ahead and the large model verifies them in one step, raising tokens/sec per sequence. An optimization on the generation loop, transparent to the client.
• Multi-region routing. Route users to the nearest GPU region for lower TTFT, with capacity-aware load balancing across regions when one region's GPUs are saturated.

What interviewers look for & common mistakes

What interviewers usually reward:

• Naming the GPU as the scarce, design-driving resource — and orienting the whole design around keeping it busy, rather than around storage or egress.
• Token streaming — explaining the model is autoregressive, that TTFT is the latency that matters, and that tokens stream over SSE/chunked HTTP on a long-lived connection.
• Continuous (in-flight) batching — describing how the scheduler swaps finished sequences out and new ones in every step, versus naive static batching, and that KV-cache memory caps batch size.
• A queue + scheduler in front of the GPU pool — decoupling demand from finite GPU supply, with priority by tier.
• Token-based rate limiting + atomic counting — limiting by tokens (the real cost), making increment-and-check atomic, and keeping paid traffic from being starved.

Before you finish, do a quick mistake check:

• Did you center the design on GPU utilization, not database/egress?
• Did you stream tokens incrementally over a long-lived connection (SSE/chunked), not return one JSON blob?
• Did you describe continuous batching and why static batching wastes GPU?
• Did you name KV-cache memory as the real limit on batch size?
• Did you put a queue + single scheduler in front of the GPUs, with tier priority?
• Did you rate-limit by tokens (not just requests) and make the counter update atomic?
• Did you return 429 + Retry-After for over-limit users and free GPU work on cancellation?