Duplex voice models: are they ready for production? | First Pass Ep.12 with Dante Everaert (Decagon)

0:00 I mean, if you want to hack something together, you just cascade it. if you want to if you want to really like position yourself to have like sort of a frontier level of voice agent or voice interaction, you have to start thinking about audio native models. >> Welcome back to First Pass. Today, we're joined by Dante, who's a voice AI researcher at Decagon. Dante, thanks so much for joining us. >> Yeah, of course. Thanks for having me.

0:30 >> Dante, why don't we start by walking us through what does voice AI research mean at Decagon? What what do you focus on and what does your team broadly focus on? >> Yeah, so I guess voice research kind of splits into maybe three categories. One is like input modeling, so everything around understanding what a user says, whether they're done, you know, if it's background noise, stuff like that. output modeling, which is like speaking back to a user in the most sort of human-like way possible.

0:55 And then the concept of conversation management, which is all these little decisions that manage an interaction. Like, should I clarify? Should I update the user? And stuff like that. >> That's a helpful background. in your blog post, you kind of critique today's I'd say voice AI architecture. Today, of course, we use a cascading architecture is very popular where, you know, you go from speech to text and you hit an LLM, and then you go text-to-speech. what are some of the challenges with that architecture and how do you kind of see it evolving over time?

1:24 >> Yeah, I guess that's kind of I see it evolving kind of two ways, right? So, like that architecture is very like input-output. Like, I talk, you talk, I talk type of design. obviously, that's not like really how humans have a real conversation, right? We interrupt each other, we have other interactions. So, one sort of area of evolving is sort of how can we model those interactions much better than that sort of turn-based system? this is where you hear terms like Duplex and stuff like that. and then the second is this whole concept of like taking audio and then treating it as text is also a little bit flawed. So, there's a lot of you know, research into what if we just operated directly on the audio when audio native just like humans and what kind of new capability does does that enable us?

2:09 >> Dr. Could you break down what does it mean to be audio native? What is a duplex architecture and what are some of the advantages of that? >> Yeah, so for like a duplex architecture means kind of doing two things at the same time. So, in the context of voice it's like you know, talking to you while I do work interrupting you while you're talking to me having these sort of more human-like ways of managing an interaction.

2:33 And then for audio native that kind of just means like cutting out the transcription and operating directly on the audio, right? So, like take audio in and just make decisions straight from that just like how humans work, right? Like we don't transcribe what we hear before we do something. So, for you know, why audio native is kind of like a big topic these days I think obviously like if you cut transcription out you cut a whole step out of the pipeline so it's faster it's more like closer to real-time. I think more important than that is pretty underrated is you can do a lot more if you operate directly on the audio, right? Like the assumption that everything that comes in should be transcribed is kind of wrong. Like what if the audio is a bit muffled like maybe you should clarify instead of just doing something or you know, maybe the person is elderly and you should have maybe adjust the tone or adjust the pace of how you speak back. You lose all of that information if you just transcribe everything and so by operating directly on the audio you can just do a lot more with your voice agents.

3:34 >> Yeah, I really like that. And in in in particular like the someone once described it to me as these these models the cascading structure is really a form of compression and there's just so much signal you lose. I can say I can say what but I can say that as what? I'm surprised. I can say what? I'm asking a question right? There's there's so much nuance and incremental information that gets lost when you compress something down to to just less signal. So, everything all that makes sense to me.

4:00 However, are these duplex models ready for production, ready for prime time? Where do they fall short and why isn't every What What What's stopping every audio use case from using a duplex model? >> Two questions. I guess it's kind of two There's two like concepts here. One is duplex and audio native. They're not They don't have to be the same thing. Like obviously we hear a lot about like duplex models like GPT live and like these kinds of full duplex.

4:28 But you also just have audio native LLMs, right? Like LLMs that can take audio as an input instead of text. I think that obviously the audio native LLMs honestly like they're pretty ready to do a lot of tasks. I mean, I think the pace of releases like 20 plus this year, right? I think they're like 99% on big bench audio or something. So, it's like They're ready from an audio understanding perspective. I think maybe why the adoption isn't as big is because if you want both audio understanding and frontier intelligence sort of in the same model, there you have like a lot fewer options, right? Like GPT, Claude, even like you know, Kim and GLM those open source, they're not audio native. So, it's kind of hard It's still hard to get both of those things at once.

5:14 So, the trick is really like how can you use audio native models in your stack and maybe work with a text LLM when you need to, how you combine those two into the same thing. That's still like pretty hard to do, I think. But what about tool calling? Is Is tool calling whether you're calling something external or just another model to do general research for What about What Where is tool calling today in either audio native or duplex models?

5:43 >> Yeah. I think they're like They probably trail like maybe one or two years behind in terms of frontier capability. Obviously, they can't reason for like 10 minutes cuz they have to answer the user, too. So, like, you know, there's sort of some complexity there. and so, you know, they they can make a certain, maybe medium-level difficulty decision when calling tools. But, if you're giving it like a really complicated task and expecting it to just like do as well, I think that's where the gap is.

6:10 And so, the trick is kind of figuring out, okay, how can you make your system in such a way where it can call some tools that are maybe easier to like reason about, but not like call all the tools and then make a bunch of mistakes. and so, and obviously, as they get better, you can give it more and more scope of responsibility. I'd say right now, yeah, so sort of a medium-level of task, so maybe like some basic search, some basic things, and then, you know, harder stuff you just delegate to a text model, stuff like that.

6:39 >> Yeah. Well, one Well, one other thing I think through is, you In In many ways, like, you know, what we talked about earlier about, "Hey, when you compress audio down, you lose information." Like, that's not That That's existed for a while. And before GPT live or their Duplex model, they had, you know, the real-time API, which or maybe it was live, now I'm forgetting what was what. But, you know, it was the idea of of audio in, audio out. it wasn't Duplex both happening at the same time, but it was audio in, audio out. But, my observation from that was people didn't actually adopt it as much as I would have expected. They still preferred the cascade skating structure because it was faster. You know, and and maybe more performant, which was a little counterintuitive. You know, now we have the promise of of audio native and Duplex models like, when do we, you know, and I've hit it some of this, like, when do we hit a tipping point of hey, we actually are using audio native or Duplex models for the majority of of voice use cases versus for whatever reason, whether it's, you know, you have the better generalized intelligence, or it's still more performing or cost-effective, what needs to happen, whether it's something we've discussed or haven't, for for really the market to tip towards audio-native or Duplex?

7:54 >> Yeah, I think honestly, like it's harder to get a Duplex model to work right now than it is to just pipe a transcription into a frontier model, right? So, like especially for a lot of like small use cases and things like that, it you just do the former and forget about it, or do the latter, sorry. but I think right now, and this is where we've talked about this Duplex architecture, if you have a a good Duplex architecture, I think honestly, they're ready today. the trick is less about like whether the models can do, you know, the things you need to do, but how you architect the system in a way that like makes it very reliable. we, you know, we launched our Duplex architecture last week. a bunch of people, I think, are also thinking of or launching Duplex architectures.

8:32 and it's kind of if you can execute that split well, maybe it takes a little more time than just piping something into your model and getting your, you know, voice agent out, but if you can architect that split well, then you can start to apply Duplex models, audio-native models, whatever you want to the sort of conversational side of that split. and honestly, you'll be fine today with the models that are available. They can definitely do that.

8:54 >> What about observability? Does it get harder to do observability or debugging or quality control when when you don't have a transcript? Does do you have second-order tasks you'd like to do? Do those become harder? Or are any challenges with just maybe observability more broadly with these models? >> Yeah. That's like a very common piece of like pushback you hear, right? and probably honestly why a lot of people are scared to even start trying out this Duplex split and trying out these audio-native models. I think honestly, I feel like it's a little overblown. Like when you have a voice agent, you care about what the user said, and then what whether the agent did the right thing, right? And so, you know, whether you have a transcript in the middle or not, you don't really lose any way of measuring that. You still have to go and what the user said, go look at what the LM did, and kind of grade it. So, you know, kind of paradoxically, you're doing all the work you would need to evaluate an audio native system anyways.

9:48 so, whether you have the the transcript in the middle is more of a sort of a flawed convenience more than like necessarily a critical way you should evaluate the systems. >> Got it. And And quick question, this is a tough one I think, something that I think a lot of builders are grappling with when you're building these frontier applications is, you know, you know, you have an architecture today that works. You also have an architecture that, you know, you think is coming down the road and is the future. While also you have to, you know, optimize for the needs of the business and what your customers need today. How do you think about that?

10:19 How do you think about taking advantage of what's good enough today and what's maybe easier to use, while also sort of balancing your bets and trying to figure out, "Hey, how do we prepare for the future?" >> Yeah, that's a that's a good question. you know, I think maybe this is a secret, maybe not, but like the duplex architecture doesn't have to have full native audio duplex systems. You actually can still keep a perfectly fine cascaded system in this sort of duplex architecture. You can still start to do like the splits of like these are easy tools that should be handled by this side, these are the harder tools that still need a frontier model, and keep the cascade. And then when these you know, newer audio native models come out that are better, maybe full duplex models that are even better, you can just kind of slot them into that side and they can work. So, we kind of we don't necessarily see these as like different things. They can actually do both at the same time.

11:11 >> Mm. Got it. And so, maybe final question is, what's your advice to to voice developers or developers building voice applications? Do you go all in on duplex? Do you go cascading? Do you just do both? how should they How would they go about it if they were starting today? >> I think I mean, if you want to hack something together, you just cascade it. if you want to if you want to really like position yourself to have like a sort of a frontier level of voice agent or voice interaction, you have to start thinking about audio native models. Like it's only a matter of time before they can do all the things that you know, we can do today and more, right? Like clarify or all these extra audio decisions. And so it's worth positioning yourself now and not waiting for that model to catch up, positioning your eval's, positioning you know, how this would work in your system. And then when those models come in, you can just swap them in rather than doing it the other way and then you have to race to catch up when I don't know, Gemini 5 comes out and it's super fantastic.

12:06 >> Awesome. Well, Dante, thanks so much for coming on. >> Yeah, really appreciate it. Thank you, guys.