The industry obsesses over accuracy benchmarks. In production voice, the number that decides whether a caller trusts your AI is measured in milliseconds.
The three thresholds that decide trust on the line
Accuracy is scored on a benchmark. Presence is scored in milliseconds. Cross the wrong thresholds and it doesn't matter how right the answer is. The caller has already formed a judgement.
Average gap between human speaker turns across languages
Hard ceiling for end-to-end latency before callers stop trusting the AI
The silence threshold where listeners read absence, not thought
How humans actually turn-take
Callers are not benchmarking your word error rate. They're benchmarking your presence, and they do it in the first second.
Conversation analysis across languages puts the average gap between human speaker turns at around200 milliseconds. The 90th percentile sits at roughly 500 ms. Anything longer than about 700 ms and the listener interprets the silence as one of three things: confusion, disagreement, or absence.
A 900 ms wait for the first response frames every subsequent sentence as uncertain, no matter how correct it turns out to be. A sub-500 ms turn signals competence long before the content arrives.
“Accuracy you can't hear doesn't exist. If the caller has already hung up, your 99% F1 score is a rounding error.”
Latency is a system problem, not a model problem
End-to-end voice latency is a stack of bills that nobody wants to pay in full. Add them naïvely and you cross the three-second barrier before breakfast.
- Capture & codec: 40–150 ms.Audio off the PSTN, through the codec, into the stack.
- Speech recognition: 150–350 ms.Streaming partials, plus finalisation overhead.
- Retrieval: 50–400 ms.Vector store, BM25 or knowledge-graph hops.
- LLM reasoning: 400–1,200 ms.First useful token, depending on model and prompt.
- Text-to-speech: 200–600 ms.First audio on the wire, including prosody planning.
- Network egress: 30–80 ms.Back through the media path to the caller.
Cross three seconds often enough and the brand is disposable.
Accuracy and latency are the same dial
Production voice platforms that hit sub-1-second end-to-end don't split ownership between two teams. They treat speed and correctness as one metric. The patterns that actually work look similar across every high-performing deployment we've inspected:
- Streaming everywhere.Streaming ASR partials into streaming retrieval into streaming generation into streaming TTS. Any blocking step eats every downstream budget.
- Speculative retrieval.Fire the first retrieval on the ASR partial, not the final transcript. In our traces, this reclaims 120–180 ms with no accuracy cost when paired with a confidence gate.
- Small-first LLM routing.Route 70–80% of turns to a fast, tuned smaller model. Escalate to the flagship only on low-confidence paths. The caller hears the small model almost always.
- Filler tokens."Let me check that" is not a stall; it's a latency-hiding primitive. Used well, it buys 400–700 ms of invisible compute while maintaining presence.
The single rule that gates every other feature
Our working rule for any Voice AI roadmap is simple. If P90 end-to-end latency (first-utterance-end to first-audio-out) is not under three seconds, nothing else you ship will matter.
- Sub-3 s: product.Turns a demo into something you can put in front of real callers without apologies.
- Sub-1.5 s: conversation.Turn-taking starts to feel natural. Containment rate and CSAT compound.
- Sub-1 s: human.Indistinguishable from a well-trained agent on presence. Operational gains unlock.
If you can't produce these four numbers this week…
…your Voice AI platform is not yet measured on the dimension your callers are actually scoring it against. Don't start with accuracy. Start here, per turn, per call, over a random sample of 1,000 calls.
- Barge-in latency.How fast you detect that the caller has started speaking. Sets the ceiling on perceived responsiveness.
- Time to first token.ASR finalisation to the LLM's first useful token. The single most controllable budget in the stack.
- Time to first audio.LLM first token to audible speech on the wire. Dominated by TTS warm-up and prosody planning.
- End-to-end P90.Utterance end to first audio out. The only number that matters to the caller. Own this and the rest follows.
Build for presence
Voice AI is won and lost in the first second. Hold P90 under three, engineer for sub-one, and the rest of the roadmap compounds. Skip this and the rest of the roadmap is reshipping a demo.
“Cloudax's voice stack is engineered latency-first. We routinely hold P90 end-to-end under 0.9 seconds on production inbound traffic, across accents, codecs and carriers.”
Engineer for the first second.
Cloudax's voice stack is engineered latency-first. Talk to our team about how to instrument P90 end-to-end on your current deployment and what it takes to close the gap to sub-one.
