The Complete Guide to Voice AI APIs for Developers

Voice is having its moment as an interface again — not because talking to computers is new, but because the latency finally dropped low enough that it stopped feeling like talking to a computer. If you've tried to build a voice agent in the last couple of years, you know the gap between a demo that works and a product that survives a real phone call is enormous.

This guide covers what actually matters when you build on voice AI APIs: the architecture, the numbers, the failure modes, and the tradeoffs nobody puts in the marketing page.

What a "voice AI API" actually is

The term covers three fairly different products, and conflating them is the first mistake most teams make.

1. Component APIs. You buy one piece of the stack — speech-to-text (STT), text-to-speech (TTS), or voice cloning — and wire it up yourself. Maximum control, maximum work.

2. Speech-to-speech (realtime) APIs. A single model takes audio in and emits audio out, with no text intermediate. Lower latency, more natural prosody, but you lose the ability to inspect and control the middle of the conversation.

3. Agent platforms. A managed orchestration layer that bundles STT, an LLM, TTS, turn-taking, interruption handling, tool calling, and often telephony. You bring a prompt and a set of functions; the platform runs the call.

Which one you want depends almost entirely on how much of the conversation you need to control versus how fast you need to ship.

The classic pipeline

Most production voice agents still run the cascaded pipeline, because it's debuggable:

Audio in → VAD → STT → LLM → TTS → Audio out ↑ │ └──── barge-in ──────┘

Each stage does one job:

VAD (Voice Activity Detection) decides when the user started and stopped speaking. Cheap, fast, and the source of more bugs than anything else. STT transcribes speech to text, ideally streaming — emitting partial hypotheses as the user talks rather than waiting for silence. LLM generates the response, streaming tokens out as they're produced. TTS converts those tokens to audio, streaming chunks so playback starts before the sentence is finished.

The whole thing lives or dies on streaming. If any stage waits for the complete input from the previous stage, you've added its full processing time to your latency budget instead of overlapping it.

The latency budget

Human conversation has a turn-taking gap of roughly 200ms. Beyond about 500ms, the pause starts reading as hesitation; beyond ~800ms, people start talking over you or asking "hello?".

A realistic target budget for time-to-first-audio-byte after the user stops speaking:

StageBudgetEndpointing / VAD decision100–300msSTT final transcript50–150msLLM time-to-first-token200–400msTTS time-to-first-audio80–200msNetwork + jitter buffer50–150msTotal~500–1000ms

Two things fall out of this table immediately.

First, endpointing is often your biggest single cost — and it's not a model problem, it's a policy problem. Wait 700ms of silence before deciding the user is done and you've spent most of your budget on nothing. Wait 200ms and you'll cut people off mid-thought when they pause to think. Semantic endpointing (using the partial transcript to guess whether the utterance is complete, not just whether audio stopped) is the meaningful upgrade here.

Second, you should be optimizing time-to-first-token, not total generation time. A model that takes 2 seconds to finish but starts in 250ms feels fast. A model that finishes in 900ms but buffers the whole response feels slow. Stream everything.

Barge-in, or: what happens when the user interrupts

This is the feature that separates toys from products.

When the user starts speaking while your agent is talking, you need to:

Detect the interruption within ~100ms (VAD on the inbound stream, running while you play audio). Stop TTS playback immediately and flush the audio buffer. Cancel the in-flight LLM generation. Truncate the conversation history to what the user actually heard, not what you generated.

Step 4 is the one people skip. If your agent generated three sentences but only the first one played before the user cut in, and you append all three to the context, the model now believes it said things the user never heard. Every subsequent turn is built on a false record of the conversation. Track playback position and truncate accordingly.

You also need to distinguish an interruption from a backchannel. "Mm-hmm," "right," and "yeah" are not attempts to take the floor — stopping mid-sentence every time someone says "uh huh" is its own kind of broken.

Speech-to-speech: what you gain and lose

Realtime speech-to-speech models collapse the pipeline into one hop. The gains are real:

Latency drops, because you've removed two model handoffs and a text serialization step. Prosody improves. The model hears the user's tone and can respond in kind — it knows the difference between an excited "no way!" and a flat one. A cascaded pipeline throws all of that away the moment STT converts audio to a string.

The costs are also real:

Less controllability. There's no text checkpoint where you can run a guardrail, redact PII, or force a retrieval call before the model commits to an answer. Harder evaluation. You can't diff transcripts against a golden set as easily. Weaker instruction-following, generally, than a text LLM of comparable size. Cost per minute is often higher.

A common production compromise is a hybrid: speech-to-speech for the conversational surface, with a text-based model running asynchronously alongside it for tool calls, compliance checks, and anything that needs to be right rather than fast.

Telephony: the part that surprises people

If your voice agent needs to answer or place phone calls, you inherit the constraints of the telephone network, and they are not modern.

Audio is 8kHz, μ-law encoded, mono. Your STT model was probably trained on 16kHz+ audio. Accuracy drops. Test on real phone audio, not on your laptop mic. You'll be dealing with SIP trunks, not just WebRTC. Most agent platforms abstract this; if you're building on component APIs, you'll want something like a SIP-to-WebSocket bridge. DTMF still exists. People will press keys. Handle it. Answering machines exist too. Voicemail detection is a genuinely hard subproblem, and outbound use cases live and die on it. Jitter and packet loss are normal, not exceptional. Budget for it.

For browser-based voice, WebRTC handles most of this for you — echo cancellation, jitter buffering, adaptive bitrate. Don't roll your own over raw WebSockets unless you have a good reason.

Tool calling in a voice context

Tool calls are latency bombs. In a chat interface, a 2-second API lookup is invisible. On a phone call, it's dead air, and dead air makes people say "hello? are you there?"

Three mitigations, in order of how much they help:

Filler speech. Emit a short, natural acknowledgment ("Let me check that for you…") the moment you dispatch the tool call. This buys you 1–2 seconds of cover for free. Speculative execution. If you can guess the likely tool call from a partial transcript, start it before the user finishes speaking. Discard if wrong. Prefetch and cache. Anything you can load at call start — account status, order history, user preferences — should not be a mid-call round trip.

Evaluating a voice API

Vendor benchmarks are marketing. Build your own eval set from your own audio, and measure at least:

WER (word error rate) on your domain — with your jargon, your accents, your background noise, your codec. A model that hits 3% WER on clean read speech might hit 15% on a call center recording. P95 latency, not median. The median is a nice number. The 95th percentile is what your users actually complain about. Endpointing accuracy — how often does it cut users off, and how often does it wait too long? TTS naturalness under interruption — does it stop cleanly, or does it clip mid-phoneme with an audible pop? Behavior on failure. Kill the network mid-call. What does the user hear?

Pricing models

Roughly, the market prices in three ways:

Per-minute of audio (STT, and most agent platforms). Simple, predictable, easy to model against revenue. Per-character or per-token (TTS, LLM). Cheaper at low volume, harder to forecast. Per-concurrent-session (some realtime APIs). Punishes bursty traffic, rewards steady load.

The number that matters is cost per completed conversation, not cost per minute — and that number includes the retries, the abandoned calls, the misroutes, and the calls that escalate to a human anyway. A cheaper model with worse endpointing can easily cost more per resolved conversation than an expensive one.

Choosing a path

A rough decision tree:

Shipping in weeks, standard use case (support, scheduling, qualification)? Use an agent platform. Don't rebuild turn-taking; it's a solved problem you'd be re-solving badly. Need deep control over the conversation, custom guardrails, or an unusual modality mix? Cascaded pipeline on component APIs. Latency and emotional nuance are the product? Speech-to-speech, with a text model alongside for anything that must be verifiable. Regulated industry, strict data residency? Check self-hosting and BYO-model options before you get attached to a vendor.

The mistakes that actually cost you

Building it text-first and adding voice later. Voice-appropriate responses are short, one idea at a time, no bullet points, no markdown. A prompt tuned for chat produces monologues nobody will sit through. Ignoring the interruption path until launch. It's not a polish item. It's the core interaction loop. Testing on clean audio. Your users will call from a car, a bar, and a bathroom. No human escape hatch. Every voice agent needs a clean, fast path to a person. The absence of one turns a 4% failure rate into a 4% churn rate. Optimizing the wrong latency. Time-to-first-byte is what people feel. Total tokens per second is what benchmarks measure.

Where this goes next

The interesting frontier is no longer "can the model hear me" — STT is essentially solved for clean audio in major languages. It's turn-taking. Knowing when to speak, when to wait, when to let a pause sit, when to interrupt back. That's a social skill, not an acoustic one, and it's where the difference between an agent people tolerate and one they actually prefer is going to be decided.

Build for that.

Building a voice agent? [Add your CTA here.]

FAQs → rich results (AEO)

Click "+ Add" for each pair.

Q1. What is a voice AI API? A1. A voice AI API lets developers add speech capabilities to an application over HTTP or WebSocket. It generally falls into one of three categories: component APIs (speech-to-text, text-to-speech, voice cloning), realtime speech-to-speech models that take audio in and emit audio out directly, and managed agent platforms that bundle transcription, an LLM, speech synthesis, turn-taking and telephony into one service.

Q2. How much latency is acceptable for a voice AI agent? A2. Aim for under 800ms from the moment the user stops speaking to the first byte of audio played back, with 500ms or less being the target for a conversation that feels natural. Human turn-taking gaps are around 200ms, so anything beyond roughly 800ms reads as hesitation and prompts users to repeat themselves or talk over the agent.

Q3. Should I use a speech-to-speech model or an STT–LLM–TTS pipeline? A3. Use a cascaded STT–LLM–TTS pipeline when you need control and debuggability: a text checkpoint lets you run guardrails, redact sensitive data, and evaluate against transcripts. Use speech-to-speech when latency and emotional nuance are central to the experience, since it removes two model handoffs and preserves tone. Many production systems run a hybrid, with speech-to-speech handling conversation and a text model handling tool calls and compliance.

Q4. What is barge-in and why does it matter? A4. Barge-in is the ability for a user to interrupt the agent mid-sentence. Handling it properly means detecting the interruption within about 100ms, stopping playback, cancelling the in-flight generation, and — critically — truncating the conversation history to only what the user actually heard. Skipping that last step leaves the model believing it said things the user never heard, corrupting every subsequent turn.

Q5. Why does my speech-to-text accuracy drop on phone calls? A5. Telephone audio is typically 8kHz, μ-law encoded and mono, while most speech models are trained on 16kHz or higher. That bandwidth loss, combined with packet loss, jitter and background noise, meaningfully raises word error rate. Always benchmark on real phone audio from your own use case rather than clean microphone recordings.

Q6. How is voice AI usage priced? A6. Pricing typically follows one of three models: per minute of audio (common for transcription and agent platforms), per character or per token (common for speech synthesis and language models), or per concurrent session. The metric that matters commercially is cost per completed conversation, which includes retries, abandoned calls and escalations to human agents — not the headline per-minute rate.