Where Latency Hides In A Call
Callers forgive a lot but not long silences. Learn where each hundred milliseconds is spent and how to hold a call under a natural response delay.
Latency is the number one killer
Voice agents rarely fail because the model is wrong. They fail because the pause after the caller stops speaking is too long, and a human on the other end starts saying "hello? hello?" or talks over the reply. In human conversation the natural gap between turns is around 200 to 500 milliseconds. Voice AI cannot consistently hit that yet, but the practical thresholds are well established across platform benchmarks and real deployments:
| Total response delay | How it feels |
|---|---|
| Under 800ms | Smooth; most callers do not consciously register a delay |
| 800 to 1,200ms | Acceptable for business calls, noticeable if you listen for it |
| Above 1,500ms | Obvious pause; callers repeat themselves or talk over the agent |
The latency budget of one turn
Delay accumulates across the pipeline, and each part is separately tunable. A realistic budget for one conversational turn on a well-configured stack:
| Pipeline stage | Typical time | What drives it |
|---|---|---|
| Endpoint detection | 200-400ms | A deliberate wait so the agent does not cut people off |
| STT finalization | 50-150ms | Streaming model, mostly already done while the caller spoke |
| LLM time to first token | 200-500ms | Model choice and prompt length - your biggest lever |
| TTS time to first audio | 100-200ms | Streaming synthesis engine |
| Network and telephony | 30-80ms | Region placement of providers |
Add it up and a good configuration lands between 600ms and 1,000ms. A bad one - a slow reasoning model, non-streaming TTS, providers in different regions - lands at 2,000ms or worse, and no prompt engineering will save it. Published vendor comparisons show stitched multi-vendor stacks ranging from roughly 600ms to 1,700ms for the same conversation, purely on configuration.
The settings that control the budget
On Vapi, the relevant knobs sit right in the assistant config. Retell and the others expose the same ideas under different names:
{
"model": { "provider": "openai", "model": "gpt-4o-mini" },
"voice": {
"provider": "cartesia",
"voiceId": "sonic-english",
"chunkPlan": { "enabled": true }
},
"startSpeakingPlan": {
"waitSeconds": 0.4,
"smartEndpointingEnabled": true
},
"stopSpeakingPlan": {
"numWords": 2,
"voiceSeconds": 0.2
}
}
The startSpeakingPlan is endpointing: how long the agent waits after silence before replying. The stopSpeakingPlan is barge-in: how much caller speech interrupts the agent. Both are per-use-case decisions. A receptionist collecting a name can wait 0.3 seconds; an intake agent listening to someone describe a plumbing disaster should wait closer to a full second, because people pause mid-story.
Perceived latency is a second budget
Callers do not measure milliseconds; they measure whether the agent feels responsive, and you can win on perception even when the raw numbers are fixed. Three tricks matter. First, a fixed opening line: because the greeting is static configuration rather than model output, it plays instantly on pickup with zero model latency, which sets the pace for the whole call. Second, filler speech during tool calls, so the two seconds a calendar lookup takes are covered by "let me check that for you" instead of dead air. Third, short first sentences: because TTS streams sentence by sentence, an agent that opens its reply with "Sure." starts speaking hundreds of milliseconds before one that opens with a long clause. You can prompt for this directly by instructing the agent to begin answers with a short acknowledgment.
How to cut latency, in order of impact
- Use a fast model for the live loop. GPT-4o mini or Claude Haiku class for conversation; save flagship reasoning models for offline work like post-call summaries. Swapping model class alone can save 300 to 800ms per turn.
- Stream everything. Model tokens should flow to TTS as they arrive. Every mainstream platform does this by default, but custom pipelines and some webhook patterns break it.
- Trim the prompt. A 3,000-token system prompt taxes time to first token on every single turn of every single call. Voice prompts should be a few hundred tokens of tight rules, not an essay.
- Cover tool calls with filler speech. A calendar lookup takes two seconds. Configure a spoken filler ("Let me check that for you...") so the line is never silent. Silence on a phone reads as a dropped call.
- Tune endpointing per flow. Shorter waits for yes/no confirmations, longer for open-ended questions.
Measure on real calls, not demos
A demo on office wifi through a laptop microphone lies. Measure end-to-end latency on an actual phone call over an ordinary mobile connection, because that is what your callers get. Most platforms expose per-turn timing in the call log:
turn 4
endpointing: 380ms
stt_finalize: 90ms
llm_first_token: 460ms <- suspect: prompt grew to 2,400 tokens
tts_first_audio: 130ms
total: 1,060ms
Read a few of these before changing anything. In the example above, the fix is shortening the prompt or the model, not the voice. Guessing at the wrong layer wastes days.
The latency nobody budgets for: your own API
Every table in this lesson so far describes a turn where the agent answers from what it already knows. The moment the agent calls a tool, a sixth line appears in the budget - and it is usually the largest one, because it is not a model provider optimised for streaming, it is your CRM.
turn 7 (booking lookup)
endpointing: 380ms
stt_finalize: 90ms
llm_first_token: 290ms
tool: check_availability 2,400ms <- the whole problem
llm_second_token: 240ms
tts_first_audio: 130ms
total: 3,530ms
Nothing in that trace is a prompt issue, and no model swap fixes it. There are exactly four honest responses, and it is worth knowing all of them because teams reach for the wrong one:
- Make the endpoint faster. Obvious, frequently possible, and frequently ignored because it is somebody else's system. An availability check that scans a whole calendar is a caching problem, not a voice problem.
- Prefetch before the caller asks. If the agent knows who is calling from the number, load the account while the greeting plays. The greeting is free time - roughly two seconds of it - and almost nobody uses it.
- Cover it with speech. Filler makes 2.4 seconds tolerable. It does not make it fast, and if every turn needs filler the call still feels sluggish.
- Change the promise. If a lookup is genuinely slow and cannot be fixed, stop doing it live. "I will text you the available slots in a moment" is a better call than fifteen seconds of hold music, and it is the answer nobody wants to hear from a voice AI vendor.
The general point: your latency budget is only as good as the slowest thing you put inside it, and the slowest thing is usually not a model.
Measuring properly: a method, not a vibe
"Measure on real calls" is right and too vague to act on. Here is a method that takes an afternoon and produces a number you can defend.
- Write down twenty utterances your callers actually say, taken from real call recordings or from whoever answers the phone today. Not twenty clever tests - twenty ordinary ones, including the mumbled and the rambling.
- Call from a mobile, outdoors or in a car. Office wifi and a headset flatter every stage of the pipeline. If a meaningful share of your callers are on the move, test on the move.
- Say each utterance and record the turn. Do not edit yourself into clarity. The pauses and false starts are the data.
- Pull the per-turn timings from the call log for all twenty and put them in a spreadsheet, one row per turn, one column per stage.
- Sort by total, descending, and read the top five rows only. This is the step people skip. An average hides the problem; the worst calls are the ones that lose customers, and they usually share a single cause.
What you are looking for is not the mean. It is the shape: is the slow tail caused by one stage, or is everything uniformly mediocre? One stage means a targeted fix. Uniformly mediocre usually means a region mismatch - providers in different continents - and moving them together is a single configuration change with a larger effect than anything else in this lesson.
Latency and accuracy pull in opposite directions
Every lever in this lesson costs something, and pretending otherwise leads to agents that are fast and wrong. Naming the trade explicitly makes the decision yours rather than accidental.
| Lever | What it buys | What it costs | When the cost is unacceptable |
|---|---|---|---|
| Smaller, faster model | The largest single latency win | Weaker at multi-step reasoning and edge cases | Calls that involve judgment, money, or eligibility rules |
| Shorter system prompt | Faster first token on every turn of every call | Fewer rules, so more improvisation | Regulated scripts or anything with a compliance line |
| Shorter endpointing wait | Snappier turn-taking | Cuts off slow or thoughtful speakers | Open questions, distressed callers, older callers |
| Fewer confirmation read-backs | Removes a whole turn from the call | Mis-transcriptions reach your database | Anything involving a number, a name, or a booking |
| Aggressive barge-in | Feels responsive, caller stays in control | Background noise and coughs stop the agent mid-sentence | Noisy environments - vans, workshops, speakerphones |
The fourth row is the one to watch. Dropping confirmations is tempting because it visibly shortens calls, and it is the change most likely to be regretted - a fast agent that books the wrong person costs more than a slow one that books the right person. Speed is worth having right up to the point where it starts producing wrong data, and not one millisecond past it.
A sensible tuning order
When someone hands you a sluggish agent, work in this order. It is roughly descending by effect-per-hour, and it stops you from making five changes at once and learning nothing.
- Read ten traces before changing anything. Ten, not one - a single slow call is often just a slow API call that happened once.
- Check provider regions. If STT, model, and TTS are on different continents you are paying network cost on every hop, and consolidating is usually a config field.
- Check the model class. A flagship reasoning model in a live loop is the most common single cause of a sluggish agent, and it is a one-line change to test the alternative.
- Measure the prompt. Count its tokens. If it has grown past a few hundred, it grew by accretion - every rule someone added after a bad call is still in there, including the ones that duplicate each other.
- Find uncovered tool calls. Any tool without filler speech is producing silence, and silence is perceived as roughly twice its actual length.
- Only now touch endpointing, and change it per flow rather than globally. This is the setting most likely to produce a fix that trades a latency complaint for an interruption complaint.
Getting the numbers out: a webhook receiver you can run today
Everything above assumes you have per-turn timings in front of you. Most people do not, because they read timings in a platform dashboard one call at a time, which is fine for one call and useless for twenty. The fix is to have every finished call posted to an endpoint you control and appended to a file you can sort. Every serious voice platform - Vapi, Retell, Bland among them - can send an end-of-call report to a URL you configure; the field names differ, the mechanism does not.
Here is a complete receiver. It runs on any machine with Node.js 18 or later, has one dependency, and writes a CSV you can open in a spreadsheet.
mkdir call-logger && cd call-logger
npm init -y
npm install express@5
# expose it to the internet so the platform can reach it:
npx localtunnel --port 3000 # or ngrok, or deploy it anywhere
// server.js - run with: node server.js
import express from "express";
import fs from "node:fs";
const app = express();
app.use(express.json({ limit: "5mb" }));
const CSV = "calls.csv";
if (!fs.existsSync(CSV)) {
fs.writeFileSync(
CSV,
"call_id,ended_at,duration_s,turn,transcript,total_ms\n"
);
}
app.post("/voice-webhook", (req, res) => {
// ALWAYS answer first. A slow webhook can make the platform retry
// or, on some stacks, affect the live call. Do the work after.
res.status(200).send("ok");
const body = req.body ?? {};
// Field names differ per platform - log one whole payload first
// and then map these four paths to whatever your provider sends.
const call = body.message?.call ?? body.call ?? body;
const turns = body.message?.turns ?? body.turns ?? [];
if (turns.length === 0) {
// Not a failure. Some platforms send several event types to the
// same URL; you only care about the end-of-call one.
console.error("no turns in payload, type =", body.message?.type ?? body.type);
return;
}
const rows = turns.map((t, i) =>
[
call.id ?? "unknown",
new Date().toISOString(),
call.duration ?? "",
i + 1,
JSON.stringify(String(t.transcript ?? "").slice(0, 120)),
t.latency?.total ?? t.total_ms ?? "",
].join(",")
);
fs.appendFileSync(CSV, rows.join("\n") + "\n");
console.error("logged " + rows.length + " turns for call " + call.id);
});
app.listen(3000, () => console.error("listening on :3000"));
Three details in that file are the actual lesson, and they generalise well beyond voice.
- It responds before it does any work. A webhook that does its processing first and answers afterwards is a webhook that times out under load, and the platform will retry - so you get duplicate rows and, on some stacks, pressure on a live call.
- It logs to stderr, not stdout. Habit worth keeping: stdout is for data, stderr is for you.
- It expects the payload shape to be wrong at first. The one thing you can be certain of is that the field paths above do not match your provider. So the first run is deliberate reconnaissance: replace the body of the handler with
fs.writeFileSync("sample.json", JSON.stringify(req.body, null, 2)), place one call, open the file, and map the real paths. Guessing at field names from documentation is slower than reading one real payload.
The failure you will actually hit first is not in this code. It is that no request ever arrives, and there are exactly three causes: the tunnel URL was not saved into the platform's webhook setting, the platform is sending a different event type than you expected, or the platform requires a signature header your endpoint is silently rejecting. Check them in that order, and confirm the endpoint is reachable at all with a manual request before blaming the platform:
curl -X POST https://your-tunnel-url/voice-webhook \
-H 'content-type: application/json' \
-d '{"turns":[{"transcript":"test","total_ms":950}]}'
# expect: ok and a new row in calls.csv
Reading the spreadsheet: what each shape means
Once calls.csv has twenty calls in it, add one column per stage from the same payload - endpointing, transcription, model, and speech - and then do exactly one thing: sort descending by total and read the worst five rows. Four shapes come up, and each has a different fix.
| Shape of the worst rows | Diagnosis | Fix, in order |
|---|---|---|
| One stage dominates on every slow row | A single bottleneck | Targeted. Change that stage only, re-measure the same twenty utterances |
| Every stage is somewhat slow, on every row | Usually a region mismatch - providers on different continents | Co-locate the providers. Frequently a single config field and the largest win available |
| Most rows are fine, a few are enormous | A tool call, not the pipeline | Find which turns called a tool. The trace in this lesson is exactly this shape |
| Slow rows all have long transcripts | Endpointing waiting, not anything downstream | Confirm against the transcript column, then tune endpointing per flow |
Take the worked trace earlier in this lesson - the one totalling 3,530ms - and drop it into that framework. The tool call is the overwhelming majority of the turn while every pipeline stage is unremarkable, which is row three. That single classification rules out four of the five things people would otherwise have tried, including the two most popular ones: swapping the model and rewriting the prompt. Neither would have moved the number at all. That is the entire value of measuring by stage rather than by feel.
Three problems that look like latency and are not
Before you spend a week tuning, rule these out. Each produces the complaint "it feels slow" while every timing in your spreadsheet looks healthy.
- Silence that is not slowness. A gap with nothing in it is perceived as considerably longer than the same gap filled with speech. If the timings are fine and testers still say it lags, look for turns where the agent produced nothing at all - an uncovered tool call, a pause before a long reply begins - rather than for milliseconds to shave.
- Over-verbosity. An agent that answers in four sentences where one would do makes the caller wait for the end of the turn before they can speak, and no timing metric captures that as latency. This is a prompt fix and one of the few times it genuinely is one: cap reply length explicitly and the call feels faster without a single stage getting quicker.
- Endpointing tuned too patiently. The wait after the caller stops talking is dead air that belongs to you, and it does not appear in a "time to first token" metric at all. If your timings are excellent and calls still drag, this is the usual culprit - and it is the one place where the fix for perceived slowness is also the fix that risks interrupting people, so change it per flow and re-run the pause-mid-sentence test afterwards.
The common thread is that latency is a felt property of a conversation, and the spreadsheet only measures part of it. Measure to locate a bottleneck; listen to the recording to decide whether the call is actually any good.
Filler speech that does not sound like filler
Covering a tool call with speech is the cheapest fix in this lesson and the one most often done badly. Three rules, with examples, because the difference between good and bad filler is entirely in the wording.
- Say what you are doing, not that you are doing something. "One moment please" is a hold message and callers hear it as one. "Let me check Tuesday's diary" tells them the work is theirs and specific, and it buys noticeably more patience for the same number of seconds.
- Vary it. A single filler line repeated at every tool call is the fastest way to make an agent sound like a recording. Three or four alternatives for the same lookup is enough - callers do not consciously notice variation, they only notice its absence.
- Never fill with a question. "While I look that up, can I take your postcode?" seems efficient and creates a race: the caller starts answering while the tool result arrives, and the agent now has two things to handle in one turn. Fill with statements only.
There is also a limit worth naming. Filler covers a gap; it does not cover a delay the caller can feel is unreasonable. If a lookup routinely runs long enough that you are writing a second filler line to cover the first, the honest fix is not more speech - it is the fourth option from earlier in this lesson, which is to change what the agent promises. "I will text you those options in a minute" is a better call than a stretch of narrated waiting, and it is the answer nobody wants to reach for and everyone eventually does.
Common mistakes
- Optimizing the prompt for quality while ignoring that its length slows every turn.
- Running a flagship reasoning model in the live loop when a mini-class model holds quality for short spoken replies.
- Leaving tool calls silent, so callers assume the line dropped mid-booking.
- Judging latency on a clean demo instead of a real phone connection with an impatient tester.
What you should be able to do now
- State the thresholds: under 800ms feels natural, 800 to 1,200ms is acceptable, past 1,500ms the call degrades.
- Write out the five-line latency budget of a turn and the typical range for each stage.
- Point at the endpointing and barge-in settings in a platform config and explain what each controls.
- Read a per-turn timing log and name which stage is eating the budget.
- List the levers in order of impact: model class, streaming, prompt length, filler speech, endpointing.
Where this goes next
That is the end of the free half. With the pipeline lesson and this one you can read a trace, name the stage eating your budget, and make a change for a reason instead of a hunch. Go and measure twenty real calls on your own stack before spending anything - that exercise alone tells you whether you have a latency problem or an accuracy problem, and they need opposite fixes.
The paid lessons build the agent this one only diagnoses. Prompting for speech rather than text, conversation flows that survive callers who answer three questions at once, tool design with the confirmation patterns that stop a mis-heard digit becoming a wrong booking, transfers and voicemail, the compliance rules that apply to outbound calling, testing against recorded call sets so a change can be shown to help rather than assumed to, and the monitoring you need once a real number is live. Each lesson ships the configs.
If you would rather the phone agent simply existed, that is work we do - see AI automation services, or take the free automation audit first; part of that conversation is being straight with you about which calls are worth automating and which ones a human should still answer. The latency budget, barge-in and text to speech glossary entries go deeper on the terms used here.
The box below emails you both free lessons with the timing traces and assistant configs, which is the fastest way to start comparing your own numbers against a labelled reference.
Key takeaways
What to remember from this lesson
- The thresholds that matter: under 800ms feels natural, 800 to 1,200ms is acceptable for business calls, past 1,500ms callers repeat themselves or talk over the agent.
- One turn's budget spreads across five lines - endpointing, STT finalization, LLM time to first token, TTS first audio, and network - and each is separately tunable.
- Levers in order of impact: model class, streaming end to end, prompt length, filler speech during tool calls, then per-flow endpointing.
- Perceived latency is a second budget you can win independently: a static first message plays instantly, filler speech covers tool calls, and short opening sentences start audio sooner.
- Endpointing is per use case. A receptionist taking a name can wait 0.3s; an intake agent hearing a story needs closer to a full second, because people pause mid-sentence.
Try it yourself
Budget the latency and find the line that blew it
Latency is an addition problem people refuse to do. Write the budget function, run it against a real-shaped configuration for a dental after-hours line, and find out which single stage you actually have to fix. The timings are a fixture taken as measured values, not vendor claims.
Check yourself
Where the milliseconds actually go
You shave 200 ms off the model and callers report no difference. What is the most likely explanation?
Tightening the endpointing threshold to cut the silence gap has one predictable cost. What is it?
Which technique hides latency rather than removing it, and is still worth doing?
Answer every question to mark.
Exercise
Write the latency budget for your own agent, then find the fat line
- 1Write out the five-line turn budget as a table, filling in the typical range for each stage from the lesson.
- 2Total the low ends and the high ends. Confirm your good-configuration range lands between roughly 600ms and 1,000ms.
- 3Now build a deliberately bad configuration on paper: a flagship reasoning model in the live loop, a 3,000-token system prompt, and non-streaming TTS. Write the new total and state which threshold band it falls into.
- 4Take the per-turn log in this lesson (endpointing 380, stt_finalize 90, llm_first_token 460, tts_first_audio 130). Name the line eating the budget and write the single change you would make first.
- 5Write the second change you would make only if the first is not enough, and say explicitly which change you would NOT make - swapping the voice provider - and why.
- 6If you have a live agent, place one real call from a mobile phone, not a laptop, and record the per-turn timings from the call log. Compare them against your paper budget.
You are done when You have a five-line budget whose totals land in the 600-1,000ms band, a bad-configuration total you can place in the correct threshold band, and a named first fix for the sample log that is a prompt or model change rather than a voice change.
Companion files
The latency-tuned config from this lesson
The startSpeakingPlan and stopSpeakingPlan block as a file you can copy the values from, alongside the fast-model and streaming-voice settings that produce the 600-1,000ms budget.
The free-preview pack includes 3 files of runnable code and config for Where Latency Hides In A Call, plus the other free lessons of this course. No email needed.
Free preview pack - 6 KB zipCheck yourself
Answer before you reveal
Endpointing is set to 200ms and callers complain the agent talks over them. Is lowering latency further the right move?Show answer
No. Endpointing is a deliberate wait, and 200ms is at the bottom of the 200-400ms range - the agent is deciding the caller finished before they did. Raise it, especially on open-ended questions where people pause mid-story.
Why does a calendar lookup need filler speech even though it does not change the numbers in the budget?Show answer
Because silence on a phone reads as a dropped call. The two seconds are real either way; filler speech converts dead air into perceived responsiveness.
A demo on office wifi measures 700ms per turn. What is wrong with shipping on that number?Show answer
It is not what your callers get. Real phone calls are compressed, noisy, and on ordinary mobile connections, which degrades STT and adds network time - measure on a real call before you believe the figure.
