Baseten's Voice Agent Stack Hits Private Preview at ~1-Second Latency
Baseten opened its Voice Agent Stack and Realtime Voice to private preview on November 16, 2026, reporting 0.94-second median time to first audio and per-stage scaling.

Why it matters
- Baseten announced the Voice Agent Stack and Baseten Realtime Voice on November 16, 2026, in private preview via a waitlist.
- Median time to first audio rose from 0.94 seconds at one concurrent call to 1.14 seconds at 56, per Baseten's tests.
- Baseten estimated $0.012 per audio minute at 32 concurrent sessions, excluding per-token reasoning costs.
- Coval's 24-hour leaderboard ranked Baseten-hosted Qwen3 ASR 1.7b second by median Time to Final Segment at 33 milliseconds.
- OpenAI's Python SDK cannot connect directly because it appends /realtime, a path Baseten's endpoint does not serve.
Baseten opened its Voice Agent Stack and Baseten Realtime Voice to private preview on November 16, 2026, reporting roughly one-second median turn latency across up to 56 concurrent sessions in its test setup. The two products attack a specific pain point in voice-agent deployment: the network round trips that remote orchestrators add to every turn of a conversation.
The stakes are straightforward. Voice agents live or die on latency, and latency is largely an architecture problem. Baseten's bet is that co-locating orchestration with speech models — rather than running the orchestrator somewhere else on the internet — removes enough delay to change what operators can build and what it costs to run at scale.
What does co-location actually buy?
The Voice Agent Stack runs on Baseten Dedicated Inference. The orchestrator and the models sit in one cloud region, connected over an in-cluster network. Baseten said a remote orchestrator can add hundreds of milliseconds of network round trips per turn — even when the speech-to-text, language and text-to-speech components are all hosted on its platform already.
That's the core argument. You can host every model on fast infrastructure and still lose the conversation to hops between services. Putting the control logic next to the models collapses those hops.
The stack also bundles the surrounding machinery operators otherwise assemble themselves:
- inference-runtime optimization,
- inter-model networking,
- traffic autoscaling,
- cross-cloud failover,
- model post-training support.
Realtime Voice is the reference implementation. It runs a cascade of Baseten-optimized open-weight models: speech-to-text, language processing and text-to-speech, coordinated by a custom orchestrator built on Pipecat. The architecture has a notable split. A fast, co-located "talker" model handles the live conversation, while a stronger external "thinker" model asynchronously manages reasoning, web search, decisions and tool calls. The system keeps speaking while it thinks.
Server-side tools execute on the co-located orchestrator, which can also call remote Model Context Protocol servers. That avoids extra client round trips. Client-side tools are supported as well.
How much control do operators get?
Customers can choose the speech-to-text, language and text-to-speech models independently. They can combine dedicated components with third-party APIs, and move individual pieces onto dedicated infrastructure over time rather than migrating everything at once. A session can switch its language model, voice or transcription route without redeployment.
Post-training options extend the customization. Baseten supports adapting transcription to industry jargon, tuning language-model behavior for calls, and supervised fine-tuning of speech using a customer's own voice data.
Is the API OpenAI-compatible?
Mostly, with caveats worth knowing before you write code. The API matches the core OpenAI Realtime event protocol. It supports audio-in/audio-out and text-in/audio-out modes, transcription keyword boosting, configurable acknowledgments and voice cloning.
But compatibility has practical limits. Clients must accept a full WebSocket URL. OpenAI's Python SDK cannot connect directly, because it appends /realtime to its base URL — a path Baseten's endpoint does not serve. Browser clients need a backend proxy to hold the API key and relay the socket, since browsers cannot set the required Authorization header.
These are integration details, not dealbreakers. But for teams with existing OpenAI-based tooling, the migration is not drop-in.
What did the latency and cost tests show?
Baseten reported median time to first audio rising from 0.94 seconds with one concurrent call to 1.14 seconds at 56 concurrent calls. The tests used contexts up to 16,000 tokens. Longer contexts can raise turn latency, and configurable summarization can hold each session to a target context length.
The test configuration was specific: Qwen3-ASR on one RTX-6000, Gemma-4-31B on one B200, Qwen3-TTS on another RTX-6000, and GLM-5.3 Fast through Baseten Model APIs for the delegation path.
The scaling story is per-stage. Each component scales independently against its own concurrency target, so operators add capacity to whichever stage saturates instead of duplicating the full pipeline. Baseten said the design preserves strict p90 and p95 latency tails while keeping GPU utilization high.
The cost estimates depend on that utilization. Baseten estimated $0.012 per audio minute at 32 concurrent sessions, excluding the per-token cost of reasoning-model delegation. At 56 sessions including Model API delegation, it estimated $0.008 per call-minute, under its stated utilization assumptions. Dedicated pricing amortizes hardware across additional sessions, which makes sustained utilization central to the economics.
How fast is the transcription, independently measured?
An independent benchmark adds component-level context. Coval's 24-hour speech-to-text leaderboard, with data timestamped 17:30 UTC on November 16, 2026, ranked Baseten-hosted Qwen3 ASR 1.7b second by median Time to Final Segment: 33 milliseconds, with a 2.1% word error rate across 456 samples.
One caveat matters here. Coval's metric starts at a shared end-of-speech point and ends when the final transcript arrives. It excludes the time a live system spends detecting that the turn has ended. It measures transcription speed, not full agent response time.
Why does orchestration matter for the economics?
The launch positions orchestration as the control point for voice-agent economics. Model choice, tool execution and component-level capacity are handled together in one place, giving operators several ways to tune latency and GPU use without rebuilding the full pipeline.
That framing matters for the market. Voice agents have become one of the most demanding real-time AI workloads, and infrastructure vendors are competing on who can compress the gap between a user finishing a sentence and an agent starting to speak. Baseten's answer is architectural: keep everything in one region, scale the bottleneck, and let a stronger model reason in the background.
Baseten said its roadmap includes more complex orchestration flows, easier migration of existing voice pipelines, native speech-to-speech full-duplex systems, omni models with dedicated speech-generation heads, and new scheduling primitives across the cascade. For now, both products remain in private preview, accessible through a waitlist.
The open question for operators is whether the co-location advantage holds as context lengths and delegation-heavy workloads grow — the very factors Baseten's own tests show pushing turn latency upward under concurrency.
Source: unite.ai
More from Elena Vasquez
Show full bio
Market editor covering technology and product news in artificial intelligence and machine learning at AI In Context.
708 articles
Related articles
- Mistral AI ships Voxtral 3, a real-time voice agent stack for 50+ languages
- Baseten opens Serverless Loops for open-model post-training
- Salesforce Signs Deal to Buy Cresta, Betting on Voice Agents for Service
- Baseten Lands Four Open Models in Microsoft Foundry as First-Party Preview
- AI Agents Will Hit an Infrastructure Wall Before an Intelligence Wall