Infrastructure and developer tools for real-time voice, video, and AI. @trydaily // ᓚᘏᗢ // @pipecat_ai
San Francisco, CA
- Tweets 6,764
- Following 3,946
- Followers 16,316
- Likes 8,578
Introducing PhoneLLM, an open model for voice agents.
GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost.
For voice agents, we need models that are both very low latency and very good at tool calling and instruction following.
There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem.
For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model.
But if you need your agent to respond at voice conversation speed, you can't use thinking models.
PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled.
The results are really good: accurate tool calling and concise, on-topic responses in long conversations.
And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-)
But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines.
You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today.
More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
120
249
2,534
334,865
I'm having fun with this Voice AI Superheroes theme that Opus 5.5, Gemini Flash, and I cooked up.
Next up in the superhero hot seat is Caroline di Vittorio, engineer at LangChain. Caroline leads voice observability and evals on the LangSmith platform.
We talked about how you measure voice agent success, latency (of course), helping customers get started building voice agents, helping customers scale in production, and the explosion of new voice AI use cases happening right now.
Caroline is talking about voice agent best practices next Thursday (October 29th) at Deepgram Speak in San Francisco. Come hang out with us at the conference. It's free, and a pretty big cross-section of the voice AI developer community will be there.
5
2
10
1,137
Our friends at @modal shipped a new Sticky Sessions feature yesterday that's extremely useful for building very low-latency voice agents and other realtime AI applications.
Today, you can build (and serve at scale) voice agents that deliver 500ms voice-to-voice response times. But this is 3-4x faster than most of the voice agents we see in production.
To be fast, you need a few things: fast models; good orchestration of turn detection and data streaming between models in your agent pipeline; fast WebRTC or telephony client-server network routing; and optimized placement/hosting/routing for your agent and inference containers.
Sticky Sessions makes it easy to route a specific user's requests to the same container every time. If you're an infrastructure person, you might describe this as session-aware serverless routing. The Modal team did a lot of work here to get the details right, so things like load balancing and auto-scaling work nicely in conjunction with Sticky Sessions.
The video is a demo clip showing a voice agent running on Modal with a 500ms voice-to-voice latency. Ben Shababo, who built this demo, used:
- Modal's Qwen 3 ASR service for transcription
- Modal's PhoneLLM Mini service as the agent brain
- The brand new @GradiumAI Phonon text-to-speech model for the voice
- Pipecat as the agent framework and WebRTC routing layer
- Modal Sticky Sessions
A few really cool things, here. All the models are running co-located on a mixture of GPUs and CPUs: L4, B200, and the Phonon model is running on CPU. Running Phonon on the CPU means that voice audio starts streaming in <50ms and you don't have to manage a GPU cluster for TTS. There are no extra network hops or network overhead in any of the routing. And the agent container uses Modal memory snapshots so a new agent (orchestration code, Phonon model, and the Smart Turn model) are all ready and responding to new requests in under 2 seconds.
If you've ever built out voice agent cluster infrastructure on top of, say, Kubernetes, you know how impressive all of the above attributes are!
A link to Ben's repo is in the thread below.
15
4
75
3,498
Ben's repo (the complete code to the voice agent in the video): github.com/modal-projects/mo…
Modal Sticky Sessions technical blog post: modal.com/blog/serverless-ro…
1
11
384
The brand new @GradiumAI Phonon voice model, which also launched yesterday. You can play around with voices and speech generation directly in this web page, because there's a WebGPU inference stack for Phonon: gradium.ai/phonon
2
8
319
Entirely on-device iOS voice AI: powered by the fabulous new Phonon TTS model from @GradiumAI, two models from Apple (STT and LLM), @JonPTaylor's tiny Pipecat Audience speaker intent model.
Check out the GitHub repo for the complete code of the app that Jon is playing around with in the video.
Phonon from @GradiumAI is an open TTS model in 31M and 40M sizes, perfect for on-device! It’s voice promptable too, via Gradium’s Voice Designer.
I love experimenting with voice and mobile. Here is Little Voices, an app where you can ask the angel and devil on your shoulder anything. It's a @pipecat_ai app in embedded Python on iPhone.
• Apple SpeechAnalyzer for speech-to-text
• Apple Foundation Models for the LLM
• Gradium Phonon 31M for TTS.
• Pipecat running in Python on the iPhone
• Pipecat Audience (for who speaks next. It works out whether you’re talking to the angel or the devil.)
• No network or inference costs!
📖 Repo here: github.com/pipecat-ai/ios-li…
8
12
158
9,260
Phonon: a new open, on-device, text-to-speech model from Gradium. It's really good.
Click through, below, and you can test it in your web browser (enabled by an optimized WebGPU inference stack). Go do it, just to *feel* what it's like when time-to-first-byte is ~10ms.
There are lots of reasons to run agents/models in the cloud. But the trade-offs are cost for hosting and inference, and latency.
We have to work hard to get time-to-first-byte down below 50ms when we're running our models in the cloud. And usually for a text-to-speech model we're aiming for 100-150ms TTFB in production.
Running on-device means you can optimize to run one inference request at a time in a specific hardware and software environment. And the cost is ~$0. (Or, in very demanding applications, the power budget/battery life for the device.)
Open and on-device models make it possible for us to imagine and build new things. I expect to see people post a lot of great Phonon demos over the next few weeks!
A new frontier for on-device speech.
We're open-sourcing Phonon, a suite of TTS models under 50m parameters, setting a new standard for naturalness and accuracy across languages, with the broadest inference support to date.
Phonon-31m is available under Apache 2.0.
gradium.ai/phonon🧵
15
8
160
9,277
Deepgram is hosting a voice agents developer conference in San Francisco on October 29th. I'm speaking at the conference, we have some fun Pipecat announcements to make, and lots of friends are either attending, or speaking, or both.
I figured this was an opportunity to record some conversations with people in the voice AI ecosystem about what they're interested in right now, what they're talking about at Speak, and what's new since we last caught up.
@bnicholehopkins, founder of @covaldev, has been building simulation and testing tools for voice agents since 2024. Before Coval, Brooke worked on simulation tooling at Waymo.
Brooke and I riffed about how deployments of enterprise voice AI are growing incredibly fast right now, how we think about building with open source components and models, applying systems engineering principles to agentic AI development, and aiming for "self-improving agent loops."
I've been experimenting with prompting Opus to create motion graphics and video effects. A discerning 11-year old of my acquaintance called the result, here, "kind of cringe." But I kind of like it. See what you think. :-)
7
26
1,714
Come hang out at Deepgram Speak with Brooke and me on October 29th: deepgram.com/speak
235
Fast is better than slow.
We've got two new PhoneLLM models optimized for conversational voice AI.
PhoneLLM Mini (30B) — When you need the lowest possible latency (sub-100ms TTFAT), smallest model size, and lowest cost. Good multi-turn conversation behavior and tool-calling accuracy in a model that can run >100 concurrent agents on a B200 and fits on an RTX 5090.
PhoneLLM Max (120B) — A "microthinking" model for complex conversational tasks. Sub-200ms TTFAT. SOTA task completion performance at 1/10th the latency and cost of models with similar capabilities.
Both of these models are full-weights fine-tunes of @NVIDIAAI Nemotron models. Mini is based on Nemotron 3 Nano. Max is based on the brand new Nemotron 3.5 Super VL.
Both of these models are available as open weights on HuggingFace, and from four inference partners: AWS, Baseten, Cloudflare, and Modal.
25
35
202
12,052
PhoneLLM Mini Workers AI API endpoint on @Cloudflare:
developers.cloudflare.com/ai…
1
5
396
PhoneLLM in your workspace on Modal:
Mini - modal.com/endpoints?model=pi…
Max - modal.com/endpoints?model=pi…
1
5
352
Joe Nemotron posting in all caps today.
I have spent a *lot* of time tinkering with the new Nemotron 3.5 Super VL and can confirm that it's a meaningful step up in performance for the 120B (12B active) category.
As always with Nemotron models, it's a great base for custom post-training/fine-tuning because NVIDIA releases the training recipes and most of the model's training data as open source. This is very helpful in iterating on, for example, synthetic data generation during fine-tuning. (A typical question during fine-tuning is "do I want to generate a bunch of data for this particular metric or mix in data the model saw in pre-training?")
This model also has vision capabilities. I'm looking forward to seeing @skalskip92's analysis of how the "VL" in Nemotron Super 3.5 VL compares to other recent models.
NEMOTRON 3.5 SUPER VL - THAT'S RIGHT. ULTRA - BUT IT FITS ON A SINGLE DGX SPARK LFG
IT'S FAST. IT'S SMART FOR 120B 12B.
AND IT'S NOT EVEN THE ONLY MODEL WE'RE RELEASING TODAY.
5
2
28
2,097
I'll be at Deepgram Speak on October 29th with 500 or so new and old friends who build voice AI agents and applications. As mini embedded Pipecat and I were just discussing, I'm really excited about Deepgram Speak.
We're working on some very fun Pipecat stuff for the conference. If you're interested in models for realtime AI, building production voice agents, or what the future holds for voice and AI-native software, please join us in San Francisco.
8
3
43
2,118
Going live in 5 minutes!
We have some fun voice AI stuff to talk about on the @NVIDIAAI Nemotron Labs livestream tomorrow: fine-tuning audio and LLM models for specific agents and use cases. Join us live if you want to chat and ask questions. 10:00am Pacific Time on Tuesday.
2
6
967
Add to your calendar: addevent.com/event/hw3jdk22w…
Direct link to the event on YouTube: piped.video/watch?v=F_glNmFP…
1
355