Contents
At a glance
- Category
- Voice and messaging
- What it is
- A total allowed response time divided into a per-stage allowance
- Voice agent stages
- Network, endpointing, transcription, inference, synthesis, playback
- Model metric
- Time to first token, not total generation time
- Biggest single win
- Stream every stage so the pipeline overlaps rather than sums
- Measure
- p95 per stage in production - the tail is what customers notice
What a latency budget is
A latency budget is a stated total response time, broken down into an allowance per stage. It turns "the agent feels slow" into an arithmetic problem: if the target is under a second and transcription takes 300ms, the model 400ms and speech synthesis 500ms, you are over, and you now know exactly which stage to attack.
The concept applies to any pipeline, but voice is where it becomes non-negotiable, because human conversational turn-taking happens on a scale of a few hundred milliseconds and people notice delays that they would never notice in a chat window.
Where the time goes in a voice agent
Six stages, each with its own allowance. Network and telephony carry the audio in. Endpointing decides the caller has finished, and the silence threshold you set is pure added delay by definition. Transcription finalises the text. Model inference produces the reply, where the number that matters is time to first token, not total generation. Speech synthesis produces audio, where the number is time to first audio byte. Playback delivers it.
Two of these are usually the culprits. Endpointing, because a conservative silence threshold adds delay to every single turn, and it is a setting rather than a technical constraint. And any non-streaming stage, because a stage that waits for completion before passing anything on serialises the whole pipeline.
How to actually cut it
In rough order of return. Stream every stage - partial transcripts into the model, first tokens into synthesis, first audio to the caller. This is the single biggest win and it turns a sum into an overlap. Tune endpointing against real calls. Use a smaller model for the conversational turn and reserve a larger one for genuine reasoning steps, which is a good use for distillation.
Shorten the prompt, because a bloated context slows first-token time directly. Do the slow work in the background: acknowledge, then look up. And fill the gap deliberately - a short "let me check that for you" while a tool call runs is honest and buys a second or two of tolerance, which is far cheaper than making the lookup faster.
Budgets outside voice
The same discipline applies to chat and to lead response. A web chat tolerates a few seconds where a phone call does not. A lead follow-up has a budget measured in minutes rather than milliseconds, and the reason to state it is the same: speed to lead is a measurable commercial variable, and a workflow that takes eleven minutes because it polls on a five-minute schedule has spent most of its budget on a setting nobody chose deliberately.
What you should know
Measure per stage in production, not end to end in testing. Your observability traces should record the duration of every stage of every call, because the slow stage is rarely the one you assumed, and it changes as providers update models. Track p95 rather than the average: the tail is what customers complain about, and an average of 700ms hides a meaningful share of three-second turns.
Latency budget FAQ
What is a latency budget?
A stated maximum response time split into an allowance for each stage of the pipeline. It turns a vague complaint that the system feels slow into arithmetic, showing exactly which component is over its share and therefore worth fixing.
What response time should a voice agent aim for?
Close to natural conversational turn-taking, which happens in a few hundred milliseconds. The practical target most teams work to is a sub-second turnaround, with anything beyond about a second and a half reading as an awkward pause to the caller.
What is the biggest cause of voice agent latency?
Usually two things: a conservative endpointing threshold, which adds delay to every single turn and is a setting rather than a constraint, and any stage that does not stream, which forces the pipeline to run in sequence instead of overlapping.
How do I make a voice agent faster?
Stream every stage, tune endpointing against real calls, use a smaller model for conversational turns, shorten the system prompt, do slow lookups in the background, and fill the wait with a short honest acknowledgement while a tool call runs.
Does a latency budget apply outside voice?
Yes, with a different scale. Chat tolerates seconds, lead follow-up is measured in minutes. The discipline is the same: state the target, break it into stages, and measure each one - otherwise nobody notices that a five-minute polling interval consumed the whole budget.
Should I measure average or p95 latency?
p95, per stage. Averages hide the tail, and the tail is what customers experience as broken. An average of 700 milliseconds can conceal a meaningful share of three-second turns, which is exactly the group that hangs up.
Primary source
OpenAI - latency optimization guide
Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.
Want this built rather than explained?
Knowing what latency budget means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.
