NVIDIA's open voice model yields the floor in 0.48 seconds when you cut in
NVIDIA released NemotronLabs VoiceChat 11B on August 9, an open-weights full-duplex speech-to-speech model that collapses speech recognition, language modelling and speech synthesis into a single network, taking turns in 448ms and yielding the floor with a 1.00 take-over rate at 480ms when a user interrupts
The three lines
- Replaces the ASR → LLM → TTS pipeline with one network, removing the latency between stages
- 448ms turn-taking, 1.00 take-over rate at 480ms on Full-Duplex-Bench 1.0
- Weights are open under OpenMDW-1.1 but labelled research-only, and it needs one 80GB GPU with no hosted API
Key questions
- What does full-duplex mean here?
- Listening and speaking at the same time. Most voice assistants today are half-duplex: they stop listening while they answer, so cutting in mid-response rarely works. A full-duplex model keeps listening while it talks, which is what lets you interrupt the way you would interrupt a person.
- Why does collapsing three stages into one help?
- It removes both latency and information loss. The conventional stack turns speech into text, text into an answer, and the answer back into speech — each hop adds waiting, and the first hop throws away tone, pace and hesitation. The final stage then invents prosody from scratch, disconnected from how the user actually spoke. A single network preserves more of that and responds faster.
- Can I run this on my phone?
- No. It needs one datacenter-class GPU with 80GB of memory, NVIDIA labels the checkpoint research-only, and there is no hosted API. This is material for teams building voice products, not a consumer release. The broader trend of models moving onto personal hardware is a different track.
Everyone who uses a voice assistant eventually hits the same moment. The answer is going the wrong way, and you cannot stop it. You say "no, not that" three times before anything happens. This is not a recognition-accuracy problem. It is structural: most voice assistants alternate between listening and speaking.
NemotronLabs VoiceChat 11B, released by NVIDIA on August 9, changes that structure. It keeps listening while it talks.
1. Three services, one network
The conventional voice stack is three pieces bolted together.
| Stage | Job | What it costs |
|---|---|---|
| ASR | Speech → text | Latency; tone and hesitation discarded |
| LLM | Text → answer text | Latency |
| TTS | Text → speech | Latency; prosody regenerated from nothing |
The latency is only half the problem. The moment speech becomes text, pace, emphasis, hesitation and affect are thrown away. At the end, a TTS system invents them again with no link to how the person actually spoke.
VoiceChat 11B puts all three inside one network that performs streaming speech understanding and speech generation together.
2. The numbers
| Metric | Value | Source |
|---|---|---|
| Turn-taking latency | 448ms | Full-Duplex-Bench 1.0 |
| Interruption handling | 1.00 take-over rate at 480ms | same benchmark |
| Parameters | 11B | hybrid Mamba/Transformer |
| Training audio | ~550k hours | composition undisclosed |
448 milliseconds is in the neighbourhood of natural human turn-taking. Gaps between human speakers typically run around 200ms, and anything past a second reads as awkward. 0.45s sits inside that band.
The more interesting figure is the 1.00 take-over rate at 480ms: when a user cut in, the model stopped and ceded the turn within roughly half a second, without exception under benchmark conditions. That is precisely the behaviour existing assistants fail at most often.
3. Architecture and constraints
Internally there are three parts. A Fast Conformer speech encoder, taken from Nemotron-Speech-Streaming-En-0.6b, produces audio tokens; a Nemotron Nano v2 language backbone predicts text tokens; an NVIDIA TTS decoder and codec turn them back into sound. The backbone mixes Mamba and Transformer blocks.
On top of that sits tool calling. NVIDIA says the model can invoke external tools mid-conversation without breaking conversational flow, and claims this is a first for an open full-duplex model.
The constraints are real:
- Weights are released under the permissive OpenMDW-1.1, but NVIDIA labels this checkpoint research-only
- Running it requires one 80GB GPU — not a laptop, and not a consumer card
- There is no hosted API; you run it yourself
So this is not a product. It is material for teams building voice systems, and it points in the opposite direction from the models being squeezed onto personal hardware.
4. What remains unverified
Check the provenance of the numbers first. 448ms and 1.00 come from Full-Duplex-Bench 1.0. Production adds network latency, microphone quality and ambient noise, so perceived latency will be higher. Benchmark values are a ceiling, not an experience.
The composition of roughly 550k hours of training audio is undisclosed, particularly its language distribution. The encoder's name carries an -En- marker, which suggests an English-centred corpus, and no public benchmark covers other languages — a direct limitation for anyone deploying outside English-speaking markets.
The "first open full-duplex model with tool calling" claim is NVIDIA's own, and no third-party verification was found.
Two things to watch. Whether the research-only label is lifted for commercial use, and how far the 80GB requirement falls under quantization. If the second one moves, this architecture spreads quickly.
Sources
- MarkTechPost — NVIDIA Releases NemotronLabs VoiceChat 11B
- Hugging Face — nvidia/NVIDIA-NemotronLabs-VoiceChat-11B model card
- GitHub — NVIDIA-NeMo/Speech, nemotron-labs-voicechat branch
- AI Weekly — NVIDIA Opens NemotronLabs VoiceChat 11B With Tool Calling
- vLLM omni — model support issue discussing architecture and requirements