Design a Voice AI Agent

Real-Time Voice Assistant System Design

Overview

You talk to an AI and it talks back — out loud, in real time, like a phone call. You ask a question, it answers within a beat, and if it starts rambling you can cut it off mid-sentence and it stops to listen. This is OpenAI's Advanced Voice Mode, an AI phone agent, a voice assistant, an AI interviewer. What makes it feel like talking to a person isn't any one model — it's that the whole loop happens fast enough, and handles interruption naturally enough, that you forget there's a pipeline behind it.

And there is a pipeline. Your microphone audio streams over a real-time connection to a server, where it is transcribed to text (STT), the text is fed to an LLM that generates a reply, and the reply is synthesized back into speech (TTS) and streamed back to your speaker.

It is labeled "Hard" because a text chatbot answers a request and forgets you, while a voice agent has to do four things a chatbot never does. It must hold a live, low-latency audio connection open for the whole conversation. It must respond in under a second so the reply feels like a person's — which means you cannot run transcribe → think → speak one after another; you overlap them. It must handle turn-taking: know when you've stopped talking, and let you interrupt it (barge-in). And it runs as a long-lived, stateful session pinned to a worker — the opposite of a stateless web request — which makes scaling and cost their own problem. A strong answer treats the design as orchestration around three streaming systems, not as any single model.

In this walkthrough you'll scope the agent, size it, model the session, define the client contract (the real-time media channel, not plain REST), draw the setup and turn loops, then go deep on the latency budget, turn-taking and barge-in, and transporting and scaling stateful sessions.

Out of scope (say so): the internals of LLM serving (covered in the LLM inference breakdown), the ML of the STT/TTS models themselves, video, and the telephony/PSTN gateway (we'll note where a phone number plugs in). We keep the core: a real-time spoken conversation with an LLM, with natural turn-taking and interruption.

Functional requirements

• Hold a spoken conversation. The user speaks; the agent replies out loud in real time, back and forth, for the length of a session.
• Detect end-of-turn and respond. The agent notices when the user has finished speaking and only then takes its turn — it doesn't talk over a mid-sentence pause.
• Barge-in (interruption). The user can start talking while the agent is speaking; the agent stops immediately and listens.
• Maintain conversation context. The agent remembers the dialogue across turns (and can call tools/functions mid-conversation).

Out of scope (state it): video, the telephony gateway, voice cloning, and offline/on-device inference.

Non-functional requirements

• Latency. The headline property. From the moment the user stops speaking to the moment the agent starts speaking should feel conversational — target < ~800 ms. The metric that matters is time-to-first-audio, not total reply length.
• Scalability. Each conversation is a stateful session held open for minutes, consuming a media slot plus streaming STT/LLM/TTS the whole time. The system scales by concurrent live sessions, not by requests/sec — and each session-minute costs real money, so cost-efficiency is part of scaling.
• Availability. A dropped or frozen call is extremely visible — worse than a slow web page. Sessions must survive component blips and support reconnect.
• Fault tolerance. STT/LLM/TTS are separate (often third-party) services that can be slow or error mid-turn; a failure in one stage must degrade gracefully, not kill the call.
• Security & privacy. Audio is sensitive personal data. It must be encrypted in transit, gated by consent, and governed by a clear retention policy — you are recording people's voices.

Estimations

State assumptions; the goal is to justify the streaming pipeline and the stateful-session capacity, not to be exact.

Rounded assumptions

• Scale by concurrent sessions. Say ~1M daily users × ~5 minutes of talk time = ~5M session-minutes/day. Average concurrent = 5M / 1,440 min ≈ ~3.5K, and peaks run ~5–10× → ~30K concurrent live sessions at peak. This number — not requests/sec — sizes the fleet.
• The latency budget is tight. Target < 800 ms from end-of-speech to first agent audio. The pieces: end-of-turn detection (~300–500 ms), LLM time-to-first-token (~200–400 ms), TTS time-to-first-audio (~100–200 ms), network + jitter buffer (~100–200 ms). They overlap, so the budget is a critical path, not a sum (Deep Dive 1).
• Bandwidth is small; compute is the cost. Compressed voice (Opus) is ~16–64 kbps each way, so 30K sessions is only a few Gbps of media — trivial. The real cost per minute is inference: STT + LLM tokens + TTS, roughly $0.05–0.15/session-minute, dominated by the LLM and TTS.
• State is small but pinned. A session's live state — the conversation context and pipeline connections — is a few KB, but it lives on one worker for the whole call and can't be moved mid-conversation.

How the numbers affect the design

Core entities / data model

Four things are worth naming: the session (one conversation), the turn (one utterance in it), the conversation context (what the LLM sees), and optionally the recording.

The live conversation state — the context the LLM sees, plus the open STT/LLM/TTS streams — lives in memory on the assigned worker for the duration of the call. It has to: the agent runtime touches it many times a second, and reaching a database on every audio frame would blow the latency budget.

But in-memory state disappears if the worker crashes or the user drops. So you append each finalized turn to a durable transcript (a Turn row) as it completes — the same "keep a durable copy independent of the live session" pattern a chat app uses for messages. The durable transcript is the source of truth for history, analytics, and recovery; the in-memory context is a fast, rebuildable working copy.

One subtlety that Deep Dive 2 returns to: an agent turn is only committed to the context once it's actually been spoken. If the user barges in halfway through the agent's reply, the half that never played must not enter the conversation history — otherwise the LLM believes it said things the user never heard.

Client contract

A voice agent is not a REST API for its core loop. Setting up and tearing down a session are ordinary request/response calls, but the conversation itself is a continuous, two-way media stream — the client can't "request" the agent's next sentence, and the server can't wait to be asked before speaking. So the contract has two parts: a small signaling API, and the real-time media channel that carries the actual audio.

1. Set up a session (signaling)

POST /api/sessions
Body:    { "persona_id": "interviewer", "consent_recording": true }
201 Created
Returns: { "session_id": "...", "media_url": "wss://media-7.example.com", "token": "<join token>" }

The API creates the Session, picks a media worker with spare capacity, and returns the address and a token the client uses to open the media connection. This is signaling — the setup handshake, separate from the media.

2. The media channel (WebRTC)

Client ⇄ media worker over WebRTC:
  audio track  (client → server):  the user's microphone, streamed continuously
  audio track  (server → client):  the agent's synthesized voice
  data channel (both ways):        JSON events, e.g.
    { "type": "transcript", "role": "user",  "text": "...", "final": false }
    { "type": "agent_speaking", "state": "start" | "stop" }
    { "type": "interrupted" }
    { "type": "session_ended", "reason": "..." }

The client publishes its microphone as a live audio track and subscribes to the agent's audio track; a side data channel carries structured events (live captions, "agent is speaking," barge-in, end). The audio never rides on a plain request — it flows continuously, both directions, for the whole call.

3. End the session

DELETE /api/sessions/{session_id}   → 204   (or the client simply disconnects)

Ends the call, flushes the final transcript, and frees the worker slot. A disconnect (tab close, network loss) triggers the same teardown after a short grace window.

High-level architecture

Two loops matter: setting up a session (signaling → connect), and a single conversation turn (the audio round-trip). We'll describe the turn as a straightforward pipeline first, then Deep Dive 1 makes it fast.

1. Session setup (signaling)

flowchart LR
    Client["Client"] -->|"POST /sessions"| API["Session API"]
    API -->|"pick a worker with capacity"| Reg[("Worker registry / capacity")]
    API -->|"create Session row"| DB[("Session DB")]
    API -.->|"media_url + token"| Client
    Client -->|"WebRTC connect"| MW["Media Worker + Agent Runtime"]

1. The client asks the Session API to start a call.
2. The API picks a media worker with spare capacity and records the assignment (worker_id).
3. It returns the worker's address and a join token; the client opens a WebRTC connection straight to that worker.

2. A conversation turn — the pipeline

The agent runtime on the worker is the brain of the session. Here is one turn, described the simple (sequential) way first — we'll overlap it in Deep Dive 1.

flowchart LR
    Mic["User mic"] -->|"audio (WebRTC)"| MW["Media Worker"]
    MW --> VAD["VAD / endpointing<br/>(is the user speaking? done?)"]
    VAD -->|"audio while speaking"| STT["STT (streaming)"]
    STT -->|"final transcript"| Agent["Agent Runtime<br/>(context + LLM call)"]
    Agent -->|"reply tokens"| LLM["LLM (streaming)"]
    LLM -->|"text"| TTS["TTS (streaming)"]
    TTS -->|"agent audio"| MW
    MW -->|"audio (WebRTC)"| Spk["User speaker"]

1. The user's audio arrives continuously. A voice-activity detector (VAD) decides when the user is talking and, crucially, when they've stopped (endpointing).
2. While the user speaks, audio streams to STT, which emits a transcript.
3. On end-of-turn, the agent runtime appends the user's text to the conversation context and calls the LLM.
4. The LLM's reply text goes to TTS, whose audio is streamed back through the media worker to the user's speaker.

3. Combined architecture

flowchart LR
    Client["Client"] -->|"signaling"| API["Session API"]
    API --> DB[("Session DB")]
    Client <-->|"WebRTC: audio + data"| MW["Media Worker + Agent Runtime"]
    MW <-->|"stream"| STT["STT service"]
    MW <-->|"stream"| LLM["LLM service"]
    MW <-->|"stream"| TTS["TTS service"]
    MW -->|"append finalized turns"| TX[("Transcript store")]
    MW -.->|"optional, consent"| Rec[("Recording blob store")]
    API -->|"assign / drain"| Pool[("Media worker pool")]

The Session API is the stateless control plane (create/assign/end). Each media worker is a stateful data-plane node that owns a set of live calls, runs the agent runtime, and talks to the streaming STT/LLM/TTS services. Transcripts are appended durably per turn; audio is optionally recorded under consent. Everything about a call lives on its one worker until it ends.

Deep dives

1. The latency budget — streaming and overlapping the pipeline

The whole game is the gap between "user stops talking" and "agent starts talking." Above ~1 second it feels like a laggy robot; the target is under ~800 ms. You get there by never doing one stage at a time. Three levels, worst to best.

user stops → wait full STT      (~300 ms)
           → wait full LLM reply (~1,200 ms for a sentence or two)
           → wait full TTS audio (~600 ms)
           → start playing       ⇒ ~2,100 ms before the user hears anything

user stops (t=0)
  endpoint detected            ~300 ms   ← the dial you tune (Deep Dive 2)
  STT final transcript         +~20 ms   (was streaming already)
  LLM first tokens             +~250 ms
  TTS first audio from them    +~150 ms
  network + jitter buffer      +~150 ms
  ⇒ first audio at ~850 ms — just over the ~800 ms target; trimming the endpoint delay is how you push under it

2. Turn-taking — endpointing, VAD, and barge-in

This is the part a text chatbot never faces and the part interviewers dig into. Two questions: when has the user finished their turn? and what happens when they interrupt the agent?

Knowing the user stopped — endpointing. A VAD marks each audio frame as speech or silence. The naive rule "respond the instant you hear silence" fails, because people pause mid-thought. So you wait for a short window of continuous silence — the endpointing delay — before declaring the turn over. That delay is a direct tradeoff:

endpointing delay too SHORT (e.g. 100 ms):
    agent jumps in every time the user pauses to think → feels pushy, cuts people off
endpointing delay too LONG (e.g. 1,500 ms):
    agent feels slow and unsure → awkward dead air after every sentence

There is no universal right value — it depends on the product. A casual assistant wants it snappy; a setting where people think out loud (dictation, an interview where the candidate reasons and pauses) wants it longer, deliberately trading a little responsiveness so the agent doesn't interrupt a thinking pause. Make it tunable, not hard-coded.

Filtering false triggers. Keyboard clicks, breaths, and background speech can trip a naive VAD into thinking the user is talking — which either interrupts the agent wrongly or starts a bogus turn. Raise the VAD's confidence threshold and require a minimum speech duration (a click is too short to be a word) so noise doesn't count as a turn.

Barge-in — letting the user interrupt. While the agent is talking, the user should be able to just start speaking and have the agent stop. This is what makes it feel human, and it's the trickiest correctness problem in the system.

flowchart LR
    Speaking["Agent is speaking"] -->|"VAD detects user speech"| Cancel["Barge-in"]
    Cancel --> A["1. stop TTS playback now"]
    Cancel --> B["2. abort the in-flight LLM stream"]
    Cancel --> C["3. commit only what was ACTUALLY spoken<br/>to the conversation context"]
    A --> Listen["Agent listens to the new turn"]
    B --> Listen
    C --> Listen

When the VAD fires while the agent is speaking, three things must happen together:

1. Stop TTS playback immediately — cut the audio going to the user's speaker.
2. Abort the in-flight LLM generation — you don't want to keep paying for, or later speak, a reply the user talked over.
3. Commit only what was actually spoken to the context. This is the subtle one. Say the LLM generated three sentences and the user interrupted after the first was voiced. The conversation history must record only that first sentence — what the user actually heard — not all three. If you commit the full generated reply, the LLM's memory diverges from reality: it "remembers" saying things the user never heard, and the conversation drifts. Track how much audio actually played and truncate the agent's turn to match.

• Naming barge-in as a first-class feature, and getting the "commit only what was spoken" detail right, is exactly the depth a strong candidate shows here.

3. Transport and scaling stateful sessions

Two structural choices sit under everything above: how the audio travels, and how you run tens of thousands of long-lived sessions.

Why WebRTC, not a WebSocket. Real-time voice needs the audio-optimized transport, not a general two-way pipe.

Scaling stateful sessions. A voice session is the opposite of a stateless web request. It's pinned to one worker — that worker holds the WebRTC connection, the conversation context, and the open STT/LLM/TTS streams — for the entire call, which can run minutes. You can't load-balance a call mid-sentence to another machine. That has three consequences:

• Capacity is measured in concurrent sessions, and placement is sticky. The Session API routes a new call to a worker with spare slots (by session count / CPU headroom), and that assignment holds for the call's life. Autoscale on concurrent-session count, not requests/sec.
• Draining, not hard cutover. To deploy or retire a worker you must stop sending it new sessions and let its live calls finish (or migrate them, which is expensive) — you can't just kill it without dropping live conversations. This is the same "stateful nodes can't be restarted freely" property a chat app's connection servers have.
• The connection lifecycle is where calls die. Networks drop. WebRTC first tries a fast resume (ICE restart) that repairs the existing session transparently; if that fails, the client does a full reconnect — a brand-new connection that must be re-attached to the same session's state. A real hazard: a grace timer that ends the session before a slow reconnect lands strands the user in a dead call. Size the reconnect grace window to outlast a full reconnect (not just a resume), and on reconnect re-check session state so a returning user rejoins their call instead of a torn-down one.

Cost control. At ~$0.10/session-minute, idle time is pure waste. Tear down on prolonged silence (a user who wandered off), right-size the models for the use case, and — because you stream — you stop paying for LLM tokens and TTS the instant a barge-in cancels a reply the user didn't want.

Failure modes

The live session is ephemeral and pinned; the durable transcript is the backstop. Design each failure around that.

• A slow or failing STT/LLM/TTS stage. Each is a separate streaming service. Apply per-stage timeouts and a graceful fallback — if the LLM stalls past budget, play a short filler ("one moment") or a spoken apology rather than dead air; retry idempotently or fail the turn, not the call.
• Media worker crashes mid-call. The in-memory session dies, but the durable transcript up to the last finalized turn survives. The client detects the drop and reconnects; the Session API places it on a healthy worker and rebuilds context from the transcript. Turns finalized before the crash are intact.
• Network degradation. WebRTC's jitter buffer and packet-loss concealment absorb short blips; a real drop triggers resume, then full reconnect (Deep Dive 3). The user hears a brief artifact, not a stall.
• LLM latency spike. Time-to-first-audio blows past budget. Because the pipeline streams, you can start speaking as soon as the first tokens arrive; if even those are late, a filler phrase covers the gap while generation catches up.
• User silently abandons. No teardown signal, but the meter keeps running. A silence/idle timeout ends the session and frees the slot, and the transcript is flushed — the same reason you tear down on prolonged silence for cost.

The through-line: finalize each turn to a durable transcript as it completes, so a crash or reconnect loses at most the turn in flight, never the conversation.

Tradeoffs & bottlenecks

• Overlapped pipeline vs. simplicity. Streaming-and-overlapping hits sub-second latency but forces you to juggle three concurrent streams and cancel all of them on a barge-in. A sequential pipeline is trivial and unusable. There's no cheap middle for production.
• Endpointing delay: responsiveness vs. interruption. A short delay feels snappy but talks over thinking pauses; a long one is polite but laggy. It's a per-product dial, not a constant.
• WebRTC vs. WebSocket. WebRTC gives loss-tolerant, echo-cancelled, NAT-traversing audio at the cost of a media server and TURN relays. A WebSocket is simpler but stalls on packet loss — fine for a demo, not for quality.
• Stateful sessions vs. stateless scaling. Pinning a call to a worker is unavoidable (the media and context live there) but costs you free load-balancing, cheap deploys, and simple failure recovery — you pay with draining, sticky placement, and an explicit reconnect path.
• Buy vs. build the models. Third-party STT/LLM/TTS are fast to integrate and offload scaling, at higher per-minute cost and less control over latency; self-hosting (see LLM inference) trades ops burden for cost and latency control.
• Pipeline vs. speech-to-speech. A single speech-to-speech model (below) removes stage boundaries and can be faster and more natural, but you lose the ability to inspect/modify the transcript, swap components, and reuse a text LLM's tooling.

Extensions if asked

Add only the extension that changes the design discussion (standalone; no dedicated guide for each yet unless linked):

• Speech-to-speech models. A single model that takes audio in and emits audio out (e.g. a realtime multimodal model) collapses the STT→LLM→TTS pipeline into one hop — lower latency and more natural prosody, but you give up the intermediate transcript, per-stage swapping, and text-tool reuse. Worth contrasting directly with the pipeline.
• Tool / function calling mid-conversation. The agent pauses to call an API (check an order, book a slot) and speaks the result; you must cover the call latency with a filler and fold the result into context.
• Telephony (PSTN). A phone number lands on a SIP gateway that bridges the call into the same media-worker pipeline — the agent design is unchanged; only the ingress differs.
• Multilingual and code-switching. Language detection feeds STT/TTS voice selection; mid-sentence language switches are the hard case.
• Recording, compliance, and redaction. Consent capture, encryption at rest, retention windows, and redacting sensitive spans from stored transcripts/audio.

What interviewers look for & common mistakes

What interviewers usually reward:

• Streaming and overlapping the pipeline — starting the LLM on the final transcript, TTS on the first sentence, and playback on the first audio chunk — with time-to-first-audio, not reply length, as the metric.
• Turn-taking as a first-class problem — VAD + an endpointing delay with its responsiveness/interruption tradeoff named.
• Barge-in with correct cancellation — stop TTS, abort the LLM, and commit only what was actually spoken to the context.
• WebRTC for the media transport, and being able to say why (UDP loss-tolerance, jitter buffer, echo cancellation, NAT) over a WebSocket.
• Scaling stateful sessions — pin to a worker, size by concurrent sessions, drain instead of kill, and an explicit reconnect path.
• Cost and durability awareness — tear down on silence, and finalize each turn to a durable transcript.

Before you finish, do a quick mistake check:

• Did you stream and overlap STT/LLM/TTS, instead of running them fully in sequence?
• Did you handle end-of-turn detection (endpointing) and name its responsiveness-vs-interruption tradeoff?
• Did you support barge-in, and commit only the spoken portion of an interrupted reply to context?
• Did you use WebRTC for audio and justify it over a plain WebSocket?
• Did you treat sessions as stateful and pinned — scaling by concurrent sessions, draining workers, and reconnecting — not as stateless requests?
• Did you address per-minute cost (teardown on silence) and durable transcripts for recovery?