Stop Gluing Models Together: A New Way for Developers to Build Voice AI

Here's a question worth sitting with: of the last voice agent you shipped, what percentage of the code was your product?

Not the transcription. Not the turn-taking. Not the WebSocket reconnect logic, the jitter buffer, the barge-in handler, the SIP bridge, the code that truncates conversation history to whatever audio actually reached the user's ear. Your product. The thing a competitor couldn't copy in an afternoon.

For most teams the honest answer is somewhere between 10% and 20%. The rest is plumbing — and everyone is building the same plumbing, badly, in parallel, in isolation.

That's the problem worth fixing.

How voice AI got built the first time

The first generation of voice agents was assembled the way you'd expect: by wiring together the best available component for each stage.

You picked a streaming STT provider. You picked an LLM. You picked a TTS voice you liked. Then you wrote the connective tissue, which turned out to be most of the work:

A VAD loop to decide when the user started and stopped talking Endpointing logic, tuned by hand, that cut people off or made them wait An interruption handler that stopped playback, cancelled generation, and rewound context A state machine for the conversation, because the LLM alone won't reliably remember it's on step 3 of a 5-step flow A tool-call layer with filler speech so the line doesn't go dead during an API lookup Telephony: SIP trunks, μ-law codecs, DTMF, voicemail detection Retries, timeouts, and a graceful failure path for when any of the above breaks mid-call

None of that is differentiated. All of it is required. And every piece of it is a place where a subtly wrong choice produces an agent that technically works and that nobody wants to talk to.

The result is a familiar shape of project: six weeks to a demo, six months to something you'd put in front of a customer, and a codebase where the interesting logic is buried under an orchestration layer nobody wants to own.

What changed

Three things, roughly at once.

The components got good enough to commoditize. Streaming STT is near-solved for clean audio in major languages. TTS crossed the line where the voice stops being the thing people notice. When the components stop being the differentiator, the composition becomes the differentiator — and composition is exactly the part everyone was hand-rolling.

The hard parts got isolated. Turn-taking, barge-in, and endpointing turned out to be a small number of genuinely difficult problems that recur identically in every single voice product. Support bot, sales qualifier, drive-thru order taker — they all need the agent to know when to shut up. That's a shared problem, which means it can be solved once.

Latency budgets forced architectural honesty. When you have ~800ms end to end, you can't afford a naive request/response chain between four services you don't control. Streaming has to be end-to-end, cancellation has to propagate instantly, and that only really works when the pipeline is designed as one system rather than four systems in a trench coat.

The new shape

The emerging model looks less like "assemble a pipeline" and more like "declare an agent."

You describe what the agent is — its instructions, its voice, the tools it can call, the transfer rules, the guardrails. The platform owns how the conversation runs — the audio transport, the streaming, the endpointing policy, the interruption handling, the context truncation, the telephony.

yamlagent: name: support-triage voice: warm-female-en-us instructions: | You answer billing questions for Acme. One idea per turn. Never read out long lists. If the user asks about anything outside billing, transfer. tools: - lookup_account - get_invoice filler_speech: true # covers tool-call latency endpointing: semantic # not just "700ms of silence" on_unresolved: transfer_to_human

That's a caricature, but the point holds: the interesting surface of a voice agent is small, and almost none of it is transport code.

The practical test for whether a stack is actually doing this for you:

You should not have to writeYou should absolutely still controlVAD and endpointing loopsWhen the agent decides it's done listening (policy, not implementation)Barge-in playback truncationWhether backchannels ("mm-hmm") count as interruptionsContext rewind after interruptionWhat goes in the context in the first placeSIP ↔ WebRTC bridgingWhich numbers route to which agentReconnect / jitter / codec handlingAudio quality thresholds and fallback behaviorFiller-speech schedulingWhat the filler actually says, in your brand voice

If a platform takes away the left column and the right column, it's not a developer tool, it's a chatbot builder with a microphone. The whole value is in giving you the second column cleanly.

What you get back

Weeks, first. The obvious one. But the more interesting return is what you do with them.

You get to iterate on the conversation instead of the transport. The quality of a voice agent lives almost entirely in its prompting, its tool design, and its failure handling. Those are things you improve by running hundreds of calls and listening to the bad ones — not by tuning a VAD threshold. Every hour spent on plumbing is an hour not spent listening to your agent be annoying.

You get evals as a first-class thing. The moment the pipeline is a defined object rather than a pile of glue, you can replay real call audio through a candidate config and diff the outcome. Change the endpointing policy, replay 500 calls, measure how many times you cut someone off. That workflow is close to impossible when your pipeline is 4,000 lines of bespoke async code.

You get to swap components without a rewrite. New STT model, 30% cheaper, 2% better WER? That should be a config change and a replay, not a sprint.

Where this genuinely doesn't apply

Worth being straight about the limits, because "just use a platform" is not universal advice.

If turn-taking is your product — you're building the voice infrastructure itself, or something acoustically novel — then the plumbing is the differentiation, and you should own it. Hard data-residency or on-prem constraints may rule out managed orchestration entirely. Check self-hosting options before you get attached. Very unusual modality mixes (voice plus continuous video, or non-speech audio understanding) will outrun most abstractions.

For everything else — the support agents, the schedulers, the qualifiers, the front desks, the reminder calls, which is to say the overwhelming majority of what's actually being built — the plumbing is a tax, not a moat.

How to start

If you're building a voice agent this quarter, a reasonable sequence:

Write the conversation before the code. Literally script five calls, including two that go wrong. This tells you your tools, your transfer conditions, and your failure paths. Get a real call working end to end in a day, on whatever stack, with a bad prompt. The point is to hear it, because voice reveals problems that reading a transcript hides. Collect real audio immediately. From real users, on real phones, in real noise. This becomes your eval set, and it will be more valuable than anything else you build in month one. Only then optimize. Latency, voice, prompt, endpointing — in that order, measured against the eval set, one variable at a time.

The teams that win at voice aren't the ones with the cleverest orchestration layer. They're the ones who listened to the most bad calls and fixed the most of them. Everything in your stack should be optimized for how fast you can do that loop.

Build the product. Rent the plumbing.

[Add your CTA here.]

FAQs → rich results (AEO)

Click "+ Add" for each pair.

Q1. What is the hardest part of building a voice AI agent? A1. Not transcription or speech synthesis — those are largely solved. The hardest parts are turn-taking and interruption handling: knowing when the user has finished speaking, detecting a genuine interruption within about 100ms, stopping playback cleanly, cancelling the in-flight generation, and truncating the conversation history to only what the user actually heard. These problems recur identically in every voice product, which is why they're increasingly handled by the platform rather than rebuilt by each team.

Q2. Should I build a voice agent from components or use a managed platform? A2. Use components when voice infrastructure itself is your differentiation, when you have strict on-premise or data-residency requirements, or when your modality mix is unusual. Use a managed agent platform for the vast majority of applications — support, scheduling, qualification, reminders — where turn-taking and telephony are a required tax rather than a competitive advantage. The test is simple: if a competitor could copy your orchestration layer in an afternoon, it isn't your moat.

Q3. What should a voice AI platform let me control? A3. It should own the transport and the mechanics — audio streaming, voice activity detection, playback truncation, SIP-to-WebRTC bridging, reconnects and codecs. It should still give you full control over policy and product: what goes in the context, how the agent decides it's done listening, whether backchannels count as interruptions, what filler speech says in your brand voice, and when to escalate to a human. A platform that removes both layers is a chatbot builder, not a developer tool.

Q4. How do I evaluate a voice agent? A4. Build an eval set from real call audio in your own domain — your accents, your jargon, your background noise, your codec — and replay it through candidate configurations. Measure P95 latency rather than median, endpointing errors in both directions (cutting users off and waiting too long), word error rate on your actual audio, and behavior under failure. This replay workflow is far easier when your pipeline is a declarative object rather than thousands of lines of bespoke async code.

Q5. How long should it take to build a production voice agent? A5. With hand-rolled orchestration, teams typically reach a demo in weeks and production in months, with most of that time spent on plumbing rather than product logic. With a managed agent stack, the goal is a working end-to-end call on day one and time spent thereafter on the conversation itself — prompting, tool design and failure handling — which is where voice agent quality actually lives.