Voice Agents Are a Latency Budget in Disguise: The Numbers That Decide If Yours Feels Human

Search 'how to build a voice agent' and you get the same picture forty times: a box labeled speech-to-text, an arrow into a box labeled LLM, an arrow into a box labeled text-to-speech, and a caption that says 'just stream everything. ' It is not wrong. It is just useless the moment you plug in a real microphone and the thing feels like a walkie-talkie with a bad connection.

The advice everyone repeats — stream each stage, detect when the user stops, let them interrupt — is correct and about as actionable as 'to lose weight, eat less. ' The interesting question is never whether to stream. It is where your milliseconds are actually going, which line item to cut first, and what breaks when you cut it. So let me give you the model I wish these posts led with.

The mental model: a turn is a latency budget you spend down

Here is the reframe. A voice turn is not a pipeline. It is a budget. A human conversation runs on gaps of roughly 200 to 400 milliseconds between turns — that is the rhythm your users unconsciously expect. Cross about 700 to 800 ms and the pause reads as 'is it broken? ' Everything you build is you spending down that budget, one component at a time, until either the reply comes out or the illusion breaks.

Once you see it as a ledger, the design stops being about diagrams and becomes arithmetic. Every stage is a line item: some you can shrink, some you can overlap with another item so it costs zero wall-clock time, and some are fixed taxes you cannot avoid. So let us write the ledger down. Treat these as rough rules of thumb, not measurements from any specific vendor — the point is the shape of the bill, not the exact cents:

• Network in (user's audio reaching your server): say 20 to 80 ms, and worse on mobile.
• Endpointing delay — how long you wait after the user goes quiet before you decide they are DONE: often 300 to 700 ms, and this is usually the single biggest, most overlooked line item.
• STT finalizing the last words: small if you stream, because most of it already happened while they talked.
• LLM time-to-first-token: maybe 300 to 800 ms depending on model and prompt size.
• TTS time-to-first-audio-chunk: often 100 to 400 ms — and this is the one people forget entirely.
• Network out plus client playback buffer: another small tax you pay every turn.

Add the naive version up and you are well past a second before a syllable comes out — and notice the LLM, the part everyone obsesses over, is not even the largest line. Endpointing and TTS-first-chunk quietly eat more. That is the whole reason 'the model is the easy part' is true, and also why repeating it does not help you: it tells you where the cost is NOT, without telling you where it is.

A worked turn makes it concrete. Say a user asks your support agent, 'do I still have the annual plan or did it switch me to monthly? ' Walk that one turn through the ledger.

The user stops speaking. Your endpointer is set to a safe-feeling 600 ms of silence before it declares the turn over — because you got burned by cutting people off mid-sentence. That 600 ms is pure dead air on every turn, and you have spent more than half your 'feels human' budget before the LLM sees a single token. Then the honest answer requires a tool call to look up the subscription, so the real sequence is: the model decides to call a function, you hit your billing API, the result comes back, the model resumes and produces text, and only THEN does TTS start buffering its first chunk. Stack those serially and you are two seconds deep — long enough that the user says 'hello? ', which your agent hears as a brand-new turn and gets confused by.

The fix is not a faster model. It is spending the budget differently. Drop endpointing to 300 ms but pair it with a cheap check that the utterance sounds grammatically complete, so you cut fast on 'switch me to monthly? ' and wait longer on a trailing 'um. ' The instant the LLM commits to the tool call, play a short filler line — 'let me check that for you' — so the API round-trip happens under cover of speech instead of under cover of silence. Now the same two seconds of work feels like a person thinking out loud instead of a frozen app.

You do not make a voice agent feel human by making it faster. You make it feel human by making sure the user is never staring at silence — which is a scheduling problem, not a model problem.

The loop the tutorials refuse to show you

The original version of this explainer had a code block and then admitted 'that readable version hides the real work. ' That is exactly backwards — the real work is the part worth showing. The hard bit of a voice loop is not the happy path; it is the interrupt path, where the user talks over the agent and you have to tear the current response down cleanly. Here is the shape that matters, with the ordering that actually bites you:

// One live turn. The subtlety is the interrupt path, not the happy path.
async function runTurn(session) {
  const userAudio = session.mic; // streaming chunks in
  const stt = session.stt.openStream();

  // Stream the user's speech into STT as it arrives.
  userAudio.on('chunk', (buf) => stt.push(buf));

  // Endpointing decides when the user is actually DONE talking.
  const transcript = await waitForEndpoint(stt, {
    silenceMs: 300,
    requireLooksComplete: true, // do not fire on a trailing 'um'
  });

  const llm = session.llm.stream(transcript);
  const tts = session.tts.openStream();

  // BARGE-IN: if the user speaks while we talk, kill everything, in order.
  const onBargeIn = () => {
    session.speaker.stop();   // 1. silence OUR audio first, instantly
    tts.abort();              // 2. stop generating more audio
    llm.abort();             // 3. stop the model mid-thought
    session.startListening(); // 4. only now reopen the mic turn
  };
  session.vad.once('user_speech', onBargeIn);

  for await (const piece of llm) {
    if (piece.type === 'tool_call') {
      session.speaker.say('one sec, let me check'); // cover the API round-trip
      const result = await session.tools.run(piece);
      llm.provideToolResult(result);
      continue;
    }
    tts.push(piece.text);           // feed text to TTS as tokens arrive
    session.speaker.play(tts.read()); // and play audio as it comes back
  }
}

The line that separates a demo from a product is the order inside the barge-in handler. Silence your own speaker BEFORE you tear down the model. If you abort the LLM first and stop the speaker last, the user gets a fraction of a second of the agent still talking after they interrupted — and that tiny overlap is the exact thing that makes people say a voice agent feels 'off' without being able to explain why. Rudeness in a voice UI is measured in milliseconds.

Cascade or speech-to-speech: a decision, not a default

The new speech-to-speech models — one model that hears audio and emits audio, no transcribe-then-synthesize round trip — get pitched as the obvious upgrade. Sometimes. They collapse three line items into one and preserve tone and emotion the cascade throws away. But they also take away the seams you often need. Here is how I actually decide:

• Use a CASCADE (STT to LLM to TTS) when you need the transcript as a first-class artifact — logging, compliance, analytics, or feeding an existing text agent you already trust. You get to swap any stage independently and inspect exactly what the model 'heard. '
• Use a CASCADE when your logic is heavy on tools, retrieval, and strict output formats — that machinery already lives in the text world and is far more mature there.
• Use SPEECH-TO-SPEECH when latency and naturalness are the product — a companion, a language tutor, anything where tone, laughter, and fast back-and-forth matter more than an auditable transcript.
• Do NOT reach for speech-to-speech expecting the same tool-calling reliability and format control you get from a mature text stack; that surface is younger and thinner.
• What breaks in production either way: the cascade leaks emotion and stacks latency; speech-to-speech is a tighter box that is harder to debug because you cannot read exactly what it thought you said.

The trap is treating this as a version number where newer wins. It is a trade of observability and control against latency and warmth. Pick the one whose weaknesses you can live with, not the one with the shinier demo.

The gotchas that only appear on a real phone line

These are the ones that never make the architecture diagram and cost you a weekend each.

The agent interrupts itself. Without proper acoustic echo cancellation, your microphone picks up your own TTS audio coming out of the speaker, your VAD hears 'speech,' and the agent barges in on itself and stops mid-sentence. On a laptop with headphones you will never see it. Ship it to a phone on speakerphone and it falls apart. Echo cancellation is not optional plumbing — it is load-bearing.

Endpointing kills backchannels. Real listeners say 'mhm' and 'right' while the other person talks. If your endpointer treats every 'mhm' as the end of a turn, the agent keeps stopping to answer noise. If it ignores them entirely, it talks over genuine interruptions. Tuning that one threshold — silence duration plus a 'does this sound finished' check — buys you more perceived naturalness than any model upgrade.

TTS time-to-first-chunk dominates, and you optimized the wrong number. Teams pour effort into a faster LLM and leave TTS on default settings, then wonder why replies still feel delayed. If your TTS takes 400 ms to emit its first audio chunk, your beautifully streamed LLM tokens sit in a buffer waiting to be spoken. Measure first-audio-out, not first-token-out — the user hears audio, not tokens.

A tool call is a cliff of silence. The moment the LLM pauses to call a function, your audio stream has nothing to play. Without a filler line, that is a dead gap in the middle of a sentence — worse than a slow start, because it happens after the agent already began talking. Always have a short, natural bridge line ready to cover any round trip longer than a few hundred milliseconds.

The bottom line

A voice agent really is STT, an LLM, and TTS wired together — the diagram everyone draws is true. But the diagram is not the design. The design is a latency ledger: know your line items, overlap the ones you can, pay the fixed taxes once, and never leave the user in silence while you spend. Do that and the same model behind your chatbot becomes something people are genuinely happy to talk to out loud. Skip it, and no model on earth will save you from feeling like a phone tree.