How Voice Agents Actually Work
A phone agent is four systems chained together, not a chat window with a speaker. Learn the pipeline so you can debug calls instead of guessing at prompts.
A call is a pipeline, not a chat box
Text chat gives the model all the time it wants. A phone call does not. Strip away the marketing and every voice agent on Vapi, Retell, or Bland is the same four-stage pipeline running in a loop, dozens of times per conversation: telephony carries the audio in, speech-to-text transcribes it, a language model decides what to say, text-to-speech speaks the reply, and telephony carries it back. A call feels good or broken depending on how those stages hand off to each other. Understand the chain first and most "the prompt is bad" problems turn out to be something else entirely.
The four stages of every voice turn
| Stage | Common providers | What it does | Typical failure mode |
|---|---|---|---|
| Telephony | Twilio, Telnyx, platform numbers (~$2 to $5/mo) | Carries audio, ringing, transfers, DTMF tones, hangups | Choppy audio, failed transfers, region gaps |
| Speech-to-text | Deepgram, Whisper-class streaming models | Transcribes live audio, detects when the caller finished | Misheard names and numbers, cutting callers off |
| Language model | GPT-4o mini or Claude Haiku class for live calls | Decides what to say or which tool to call | Slow first token, hallucinated facts |
| Text-to-speech | ElevenLabs, Cartesia, PlayHT | Turns the reply into streamed audio | Delayed first audio, mangled numbers and addresses |
Streaming is the word that makes the whole thing work. STT emits partial transcripts while the caller is still talking. TTS starts synthesizing the first sentence while the model is still generating the rest. That is how a good agent begins speaking in under a second instead of waiting for every stage to finish in sequence.
What the configuration actually looks like
On Vapi, the four stages are literally four blocks of one assistant config. Reading a config like this is the fastest way to internalize the pipeline, because each layer is a separate provider you chose:
{
"name": "clinic-receptionist",
"transcriber": {
"provider": "deepgram",
"model": "nova-2",
"language": "en"
},
"model": {
"provider": "openai",
"model": "gpt-4o-mini",
"messages": [
{ "role": "system", "content": "You are Maya, the receptionist for..." }
]
},
"voice": {
"provider": "11labs",
"voiceId": "cgSgspJ2msm6clMCkdW9"
},
"firstMessage": "Thanks for calling Brightsmile Dental, this is Maya. How can I help?"
}
Retell, Bland, Synthflow, and ElevenLabs Agents bundle the same layers with different defaults and different amounts of choice, but the anatomy never changes. When you pick a platform you are really picking how much of this config you want to control yourself.
Trace one turn end to end
Here is what a single conversational turn looks like in a call log, the kind every serious platform exposes:
caller audio ends t=0ms
endpoint detected t=310ms (STT decides the caller is done)
transcript final t=390ms "do you have anything tuesday afternoon"
LLM first token t=720ms (model starts answering)
tool call issued t=760ms check_availability(date="tuesday")
TTS first audio t=940ms "Let me check the calendar for you..."
caller hears reply t=~1000ms
Every debugging session starts with a trace like this. If a call feels slow, the problem might be endpointing waiting too long, a slow model, or a slow voice, and each is a different fix. You cannot fix what you cannot locate, and you can only locate it if you know the stages exist and can read their timestamps.
The orchestration layer is what you are paying for
Wrapped around the pipeline is the part platforms rarely name: orchestration. It handles turn-taking, barge-in (when the caller interrupts, the agent must stop within a fraction of a second, discard its planned reply, and listen), filler speech during slow tool calls, voicemail detection, and transfer logic. Anyone can chain three APIs together; making the result feel like a competent human on the phone is the actual product Vapi, Retell, and Bland sell. That is also why your job shifts from plumbing to design: picking the STT, model, and voice, and tuning how they interact.
One architecture note worth knowing: some newer stacks use speech-to-speech models that skip the explicit STT and TTS stages, trading controllability for latency. As of mid-2026 the mainstream no-code platforms still run the staged pipeline because it gives you model choice at every layer, so that is what this course assumes.
Each stage bills separately too
The pipeline is also the cost model. Telephony bills per minute of connection, STT per minute of audio transcribed, the model per token processed, and TTS per character synthesized. That is why platform pricing pages are confusing: an advertised $0.05 per minute platform fee covers the orchestration, while the four stages underneath bill on top of it. It is also why every optimization in this course touches a named stage. Shortening the prompt cuts the model line and the latency at once; making replies concise cuts TTS characters and time on the line; ending dead calls early cuts every line simultaneously. Once you see calls as four metered services in a trench coat, both the bill and the debugging get much easier to reason about.
Phone audio is not the audio you tested on
The most common surprise when a voice agent moves from demo to production is that transcription quality falls off a cliff, and nothing in the config changed. The cause is the audio itself.
Traditional phone calls carry narrowband audio - the network samples voice at 8 kHz and compresses it, which is why phone speech has that familiar thin quality and why the letters "S" and "F" are famously easy to confuse on a call. Your laptop microphone in a quiet room hands the transcriber something far richer than any real caller ever will. Layer on the things real calls include - a car, a busy kitchen, a speakerphone in a van, a cheap headset, a mobile connection dropping packets - and the transcript your model reads is meaningfully worse than the one you saw in testing.
Three consequences worth designing around from the start:
- Never trust a spelled-out name or number on first pass. Anything you will act on - a phone number, an email address, a booking reference, a postcode - gets read back to the caller for confirmation. This is not politeness, it is error correction, and it is the single highest-value pattern in voice work.
- Constrain what the caller has to say. "Is that Tuesday or Wednesday?" transcribes far more reliably than an open request for a date. Where accuracy matters more than naturalness, narrow the question.
- Fall back to keypad entry for pure digits. Telephony carries DTMF tones as data, not audio, so a card number or account number entered on the keypad arrives exactly right regardless of noise. Speech recognition cannot compete with that, and pretending otherwise is how agents book appointments for the wrong person.
Endpointing: the setting behind most "the prompt is bad" complaints
Between the caller finishing and the model starting there is a decision nobody thinks about until it goes wrong: has the caller stopped talking, or are they just thinking? That is endpointing, and it lives in the STT stage.
Set the wait too short and the agent talks over people mid-sentence. Set it too long and every reply arrives after an awkward silence. Neither symptom is a prompt problem, and both get misdiagnosed as one constantly, because the visible failure is the agent saying something inappropriate - and the reason it said something inappropriate is that it only received half the sentence.
The tell is in the transcript. Open the call log and read the final transcript for the turn that went wrong. If it reads "I need to reschedule my appointment for" and stops, the model behaved perfectly on the input it was given. No amount of prompt editing fixes a truncated input. What fixes it is a longer wait, or a smarter endpointing mode that considers whether the sentence sounds finished rather than only measuring silence.
The right value is per flow rather than per agent. A yes/no confirmation can be aggressive, because there is nothing more coming. An open question - "tell me what happened" - needs a genuinely long wait, because people pause in the middle of describing a problem, and cutting them off there is the fastest way to make a caller ask for a human.
Where tools fit in the loop
The four stages describe conversation. Real agents also do things - check a calendar, look up an account, create a ticket - and that changes the timing picture in a way worth seeing before you build.
caller: "do you have anything tuesday afternoon"
|
v STT final transcript
v LLM decides: call check_availability(date="tuesday")
|
+--> your API [ 300ms .. 3000ms, not yours to control ]
| returns: ["14:00", "15:30"]
|
v LLM turns the result into speech-shaped language
v TTS speaks: "I have two o'clock or half past three, which suits?"
The branch in the middle is the problem. Your own API is now inside a live conversation's latency budget, and a lookup that is perfectly acceptable in a web app - two seconds, say - is dead air on a phone call. Two design rules follow, and both belong in the build from day one rather than being retrofitted:
- Cover every tool call with speech. "Let me check the calendar for you" plays while the API works. Silence on a phone line reads as a dropped call, and callers hang up on it.
- Budget your own endpoints separately. Anything the agent calls mid-conversation needs to be fast enough that filler speech covers it. If a lookup genuinely takes ten seconds, the honest design is not a faster prompt - it is telling the caller you will text them the answer, or transferring them.
Note also what the second-to-last line does: the model converts ["14:00", "15:30"] into "two o'clock or half past three". That translation is real work and it is where agents leak their machinery, reading back raw timestamps or database IDs. Instructing the agent to speak results the way a person would - and testing it on ugly data - is part of tool design, not an afterthought.
Symptom to stage, at a glance
This is the table to keep next to you for the first month of building. Every row is a complaint you will hear from whoever tests your agent, translated into where to look.
| What the tester says | Suspect stage | What to open first |
|---|---|---|
| "It talks over me" | STT endpointing | The final transcript - is it truncated? |
| "It got my name wrong" | STT accuracy on phone audio | The transcript, plus whether a confirmation read-back exists at all |
| "Long pause before it answers" | Model time to first token, or an uncovered tool call | Per-turn timings, then prompt length |
| "It says numbers strangely" | TTS, or the model emitting raw data | The text sent to TTS - was it "14:00" or "two o'clock"? |
| "It made something up" | Model, and a missing tool or a missing rule | Whether any tool could have answered, and what the system prompt says about not knowing |
| "The transfer failed" | Telephony and orchestration | Provider call logs, not the assistant config |
Only two of those six rows are prompt problems, and even those two are partly tool problems. That ratio is the honest reason this lesson comes before any prompt writing.
The pipeline as one real config, with the endpointing knobs named
The Vapi config earlier shows the four stages as four blocks. It is worth seeing the same anatomy in a provider that exposes the settings the no-code platforms hide, because those are the settings you will eventually need. Deepgram's Voice Agent API is a useful reference here: it runs the whole listen, think and speak pipeline, and its configuration is a single Settings message with a section per stage. What follows is the shape of that message with the fields that matter, accurate as of August 2026 - check the vendor docs before copying, since voice APIs move quickly.
{
"type": "Settings",
"audio": {
"input": { "encoding": "linear16", "sample_rate": 16000 },
"output": { "encoding": "mp3", "sample_rate": 24000,
"bitrate": 48000, "container": "none" }
},
"agent": {
"listen": {
"provider": {
"type": "deepgram",
"model": "flux-general-en",
"version": "v2"
}
},
"think": {
"provider": { "type": "open_ai", "model": "gpt-4o-mini" },
"prompt": "You are Maya, the receptionist for Brightsmile Dental..."
},
"speak": {
"provider": { "type": "eleven_labs", "model_id": "..." }
}
}
}
Four things in that block are worth knowing whichever platform you end up on.
- Input audio defaults to 16 kHz linear PCM, while output encoding, sample rate and bitrate are specified explicitly. This is the layer where the narrowband phone-audio problem described above actually lands, and where a mismatch between what your telephony sends and what the transcriber expects produces audio that is not merely bad but unintelligible.
- The speak stage is a provider choice, not a fixed component. Deepgram, ElevenLabs, Cartesia, OpenAI and AWS Polly are all selectable there. Voice is the most swappable part of the whole stack, which is why it is the wrong thing to agonise over first and the right thing to A/B later.
- The system prompt lives in the think block and has a hard ceiling - 25,000 characters for the managed model options. That is a real constraint on the "just put everything in the prompt" approach, and it is a good one: a phone agent whose prompt approaches that size is an agent that needed tools instead.
- The transcription model version is explicit. Deepgram's Flux models take API version v2, while the others use v1. Version-shaped fields like this are exactly what silently breaks when you copy a config from a blog post written a year ago.
Now the part this lesson has been building towards. Endpointing was described earlier as "the setting behind most prompt complaints"; here are the actual controls, using Deepgram's Flux models as the concrete example:
| Setting | Range and default | What moving it does |
|---|---|---|
eot_threshold | 0.5 to 0.9, default 0.7 | How confident the model must be that the caller has finished before the turn ends. Raise it and the agent waits longer and interrupts less; lower it and replies come sooner at the cost of talking over people |
eager_eot_threshold | 0.3 to 0.9 | A lower, earlier bar that lets downstream work start before the turn is fully confirmed, so the agent can prepare a reply speculatively |
eot_timeout_ms | default 5000 | A hard ceiling. However uncertain the model is, the turn ends after this. This is your protection against a caller who trails off and a line that then sits silent |
Two implications worth carrying to any platform. First, endpointing on a modern transcriber is a confidence judgment about whether a sentence sounds finished, not merely a silence timer - which is why "just increase the silence threshold" is dated advice. Second, and more useful: these can be updated mid-conversation rather than only at call setup. That is what makes the earlier "per flow, not per agent" advice implementable. You can run a patient, high-threshold configuration while the caller describes a problem, then switch to an aggressive one for the yes/no confirmation at the end, within the same call.
Whatever platform you use, find its equivalents of these three before you ship anything. If a platform does not expose them at all, that is a real limitation and you should know about it before a client asks why the agent keeps interrupting.
The ten-call test, with the scoring sheet
"Place ten real calls" is the advice everyone gives and nobody operationalises, so it produces a vague feeling rather than a fix list. Here is the version that produces work items. Build any trivial agent on any platform, then call it ten times from a mobile phone - not from your laptop, and not from a quiet room for all ten.
The ten calls are prescribed, because the point is coverage, not repetition:
| # | The call you place | What it is testing |
|---|---|---|
| 1 | The happy path, spoken clearly | Baseline. Everything else is compared to this |
| 2 | Same, but pause three seconds mid-sentence | Endpointing. Does it cut you off? |
| 3 | Interrupt the agent halfway through its greeting | Barge-in. Does it stop, and how fast? |
| 4 | Answer three questions at once, unprompted | Flow rigidity. Does it re-ask what you already told it? |
| 5 | Spell an unusual name, then a UK postcode | STT on hard tokens, and whether a read-back exists |
| 6 | Call from a car or with a TV on | Real acoustics. This is the call that surprises people |
| 7 | Ask something it cannot possibly know | Honesty. Does it invent, or say it does not know? |
| 8 | Ask for a human immediately | The escape hatch. Every agent needs one and many do not have it |
| 9 | Say nothing at all for fifteen seconds | Silence handling. Does it prompt, or sit there? |
| 10 | Trigger whatever tool the agent has, then hang up mid-answer | Tool timing and clean teardown |
Then - and this is the half everyone skips - open the call log for each one and fill in a row. Not from memory of how it felt; from the log.
CALL 6 (car, windows up, motorway)
Final transcript, turn 2: "i need to book a a for tuesday"
What the agent said: "Sure, what time on Tuesday?"
Verdict: STT dropped the word "cleaning".
Agent proceeded on a partial transcript.
Stage at fault: STT (accuracy), plus flow (no confirmation)
Fix: Add a read-back of the service type before
booking. Do not attempt to fix this with
a better prompt - the word never arrived.
Priority: High. Wrong-service bookings reach customers.
Ten rows of that is a prioritised backlog rather than an impression, and it takes about forty minutes. The discipline it enforces is the one this whole lesson argues for: every verdict names a stage, and the fix is constrained to that stage. A team that cannot fill in the "stage at fault" column is a team that will spend the next month editing prompts.
When a voice agent is the wrong answer
The pipeline explains what voice costs technically. It is also worth being blunt about when the whole channel is a mistake, because that is a conversation better had before a build than after.
- The call always ends in a long data-entry step. Anything where the caller must convey a card number, a long reference, or an address is fighting the medium. A text message with a link finishes it in seconds and gets it right, and the agent's job becomes sending that link rather than transcribing digits.
- The volume is low and the value is high. Ten calls a week from prospects worth thousands each does not want an agent in front of it. Voice agents earn their place on repetitive, high-volume, low-variance calls.
- The caller is likely to be distressed. Complaints, cancellations, and anything involving harm handle badly. The correct design is a fast, obvious route to a person, not a better-worded agent.
- You cannot answer "what happens when it is wrong?" A wrong appointment is recoverable and a wrong medication instruction is not. The pipeline has a documented failure mode at every stage, so the recovery plan is a design input, not an afterthought.
Common mistakes
- Treating a voice agent like a chatbot and editing the prompt when the real problem is STT cutting the caller off early.
- Ignoring that every stage is a separate provider with its own cost, latency, and failure mode.
- Testing on clean laptop audio and being surprised when real phone calls, compressed and noisy, transcribe worse.
- Never opening the per-turn call log, so every fix is a guess.
What you should be able to do now
- Name the four pipeline stages and one provider for each, from memory.
- Read a Vapi-style assistant config and point at the transcriber, model, and voice blocks.
- Trace one conversational turn through a call log and read the timestamps.
- Given a broken call symptom (mishears, slow replies, robotic interruptions), name the suspect stage before touching the prompt.
- Explain why streaming at the STT and TTS layers is what makes sub-second replies possible.
Where this goes next
This lesson and Where latency hides in a call are the free part of the course, and between them they give you the two things that make voice debugging possible at all: the pipeline, and the timing budget that runs through it. Build a test agent on any platform and place ten real calls to it from a mobile before you go further. You will learn more from those ten calls than from any amount of reading.
The paid lessons build a working phone agent end to end. Writing prompts for speech rather than text, designing the conversation flow so it survives callers who answer three questions at once, tools and the confirmation patterns that stop a mis-transcribed digit becoming a wrong booking, transfers and voicemail handling, the compliance layer for outbound calling, testing with recorded call sets rather than vibes, and the deployment and monitoring work that keeps a live number healthy. Each lesson ships the configs.
If you want a phone agent running rather than a skill set, that is a project we take on - see AI automation services, or start with the free automation audit, which will tell you honestly whether voice is the right channel for your callers or whether a text-back handles it better. The speech to text, barge-in and latency budget entries cover the vocabulary in more depth.
Put your email in below and both free lessons arrive with the assistant configs from them, so you can diff your own setup against a labelled reference instead of copying JSON out of a page.
Key takeaways
What to remember from this lesson
- Every voice agent is the same four-stage pipeline in a loop: telephony, speech-to-text, language model, text-to-speech. Platforms differ in defaults, not anatomy.
- Streaming at the STT and TTS layers is what makes sub-second replies possible - partial transcripts arrive while the caller talks, and synthesis starts before the model finishes.
- Each stage is a separate provider with its own cost, latency, and failure mode. A symptom points at a stage before it points at your prompt.
- Orchestration - turn-taking, barge-in, filler speech, voicemail detection, transfers - is the actual product you buy from Vapi, Retell, or Bland.
- Billing follows the pipeline: telephony per minute, STT per minute, model per token, TTS per character, plus the platform's orchestration fee on top.
Try it yourself
Run one turn through the whole pipeline
A call is the same four stages running in a loop, dozens of times per conversation. The stages below are fixtures - fixed transcripts and fixed timings, no audio and no model call - so you can wire the chain and see where a turn's time actually goes. Build handleTurn and the rest of this course has somewhere to hang.
Check yourself
Which stage is broken?
The transcript shows the agent answering a question the caller did not ask, and the caller has a strong regional accent. Which stage do you investigate first?
The agent's replies are correct but it says 'D-O-B' as three letters and reads a phone number as one enormous integer. Where is the problem?
Callers say the agent sounds fine but keeps cutting them off mid-sentence. Which stage owns that?
Answer every question to mark.
Exercise
Map a symptom to a stage before you touch a prompt
- 1Write out the four stages in order from memory, and name one real provider for each. Check yourself against the lesson table afterwards.
- 2Take the Vapi-style config from this lesson and label each of its four blocks with the pipeline stage it configures. Note which block has no explicit entry and why (telephony is the platform number).
- 3For each of these four symptoms, write down the suspect stage AND the trace line you would read to confirm it: (a) the agent keeps cutting callers off mid-sentence, (b) there is a two-second pause before every reply, (c) the agent reads a phone number as a garbled word, (d) transfers to a human fail.
- 4Take the per-turn trace in the lesson and compute the elapsed time between each pair of consecutive timestamps.
- 5Name which single stage in that trace consumed the most time, and state whether it is inside the acceptable band or not.
You are done when You have four symptom-to-stage answers each paired with a specific trace line, and a set of per-stage elapsed times for the lesson's trace with the largest one identified by name.
Companion files
The assistant config from this lesson
The Vapi-style four-stage config as a real file, so you can diff your own assistant against a labelled reference instead of copying it out of the page.
The free-preview pack includes 3 files of runnable code and config for How Voice Agents Actually Work, plus the other free lessons of this course. No email needed.
Free preview pack - 6 KB zipCheck yourself
Answer before you reveal
The agent keeps interrupting callers mid-sentence. Which stage do you suspect, and which one do you NOT touch?Show answer
Speech-to-text endpointing - it is deciding the caller finished too early. Do not touch the prompt; the language model never saw the rest of the sentence.
Why does an advertised '$0.05 per minute' platform price understate your real cost?Show answer
That fee covers orchestration only. Telephony (per minute), STT (per minute), the model (per token), and TTS (per character) all bill on top of it.
What does streaming actually buy you in this pipeline?Show answer
Overlap. STT emits partial transcripts while the caller is still speaking and TTS starts synthesizing the first sentence while the model generates the rest, so stages run concurrently instead of in sequence - the difference between speaking in under a second and waiting for four stages to finish one after another.
