Building Voice Agents: The Real-Time Speech Stack

Talking to a voice agent and typing to a chatbot feel like different products, but underneath they share a brain. The difference is everything wrapped around it: turning speech into text, text back into speech, and doing it fast enough that the conversation feels alive rather than like a walkie-talkie with a delay. Get the model right and a laggy, uninterruptible voice agent still feels broken.

Building one well is mostly a latency-and-turn-taking problem, not a modeling one. People expect a reply within a few hundred milliseconds, and they expect to be able to cut the agent off mid-sentence the way they would a person. Those two expectations drive nearly every design decision in the stack.

The three stages of a voice turn

The classic voice agent is a cascade: speech comes in, gets transcribed, the model thinks, and the answer is spoken back out. Each stage is a model of its own, and each adds latency you have to fight.

• Speech-to-text (STT). Transcribe the user's audio into words the model can read — accurately and, crucially, incrementally as they speak.
• The LLM. The agent reasons over the conversation in text, can call tools, and produces a reply — just like a text chatbot.
• Text-to-speech (TTS). Turn the reply back into natural-sounding audio, ideally streaming it out as it is generated rather than after.
• Turn-taking glue. Voice activity detection to know when the user stopped, and barge-in to stop talking the instant they start again.

Latency is the whole game

A natural conversation has gaps of a few hundred milliseconds between turns. A naive cascade blows right past that: wait for the full transcript, then the full LLM reply, then the full audio, and you are seconds behind. The fix is to stream every stage — start transcribing while the user talks, start generating once you have enough, start speaking the first words before the last are written.

Turn-taking is the other half. The agent needs to detect when the user has actually finished (endpointing, usually via voice activity detection, not just silence) and to handle barge-in — when the user starts speaking while the agent is talking, it must stop immediately and listen. Newer speech-to-speech models collapse the cascade into one model, cutting latency further and preserving tone, but the same conversational rules apply.

The loop, in code

Conceptually, one turn of the classic stack is short — the engineering is in streaming and interrupting each stage:

# The classic voice-agent loop: speech in -> text -> LLM -> text -> speech out.
def voice_turn(audio_in, stt, llm, tts, history):
    text = stt.transcribe(audio_in)              # speech-to-text: the user's words
    history.append({"role": "user", "content": text})
    reply = llm.chat(history)                     # the agent thinks in text, can call tools
    history.append({"role": "assistant", "content": reply})
    return tts.speak(reply)                       # text-to-speech: the agent's voice

# In production every stage STREAMS, and a barge-in stops the TTS the moment the user talks.

That readable version hides the real work: each call is a stream, voice activity detection decides when the user is done, and an interrupt path tears down the current response when they speak over it. Those details are what make it feel like a conversation instead of a transaction.

Users forgive a voice agent that is occasionally wrong. They will not forgive one that is slow or that cannot be interrupted — because that is not how talking works.

What separates good from painful

• Stream everything. Transcribe, generate, and speak incrementally; never wait for one full stage before starting the next.
• Nail endpointing. Use voice activity detection to tell when the user truly finished, so the agent neither interrupts nor leaves dead air.
• Support barge-in. The instant the user speaks, stop the agent's audio and listen — talking over someone is the rudest possible bug.
• Consider speech-to-speech models. A single realtime model cuts latency and keeps tone and emotion that a transcribe-then-synthesize cascade loses.
• Handle tools mid-conversation. Let the agent call a function and keep the conversation flowing, with a filler line so the silence is not awkward.

The bottom line

A voice agent is speech-to-text, an LLM, and text-to-speech, wired together under a hard latency budget with real turn-taking. The model is rarely the bottleneck — streaming, endpointing, and barge-in are. Build those in from the start, or consider a speech-to-speech model that handles them natively, and the agent stops feeling like a phone tree and starts feeling like a conversation.

Everything that makes voice hard is about time and turns, not intelligence. Respect how humans actually talk — fast replies, easy interruptions, no awkward gaps — and the same model that powers your chatbot becomes something people are happy to talk to out loud.