ThursdAI - Apr 23rd - GPT Image & Grok APIs Drop, OpenAI ❤️ OS? Dia's Wild TTS & Building Better ...

0:26 Jacket check. Welcome everyone. Welcome to Thursday I. This is Alex Vulov with Weights and Bices. Welcome to Thursday I April 24th. Today we have quite a lot of guests on the show. We also finally got that chill week we've been waiting for since maybe two years ago. It seems like after the big releases week of 03 and GBD 4.5 major labs are a little bit taking a little bit of break to probably release bigger things next week including potentially some open source stuff but we still there there's a bunch of news there bunch of news in open source there's quite a lot of stuff so I want to welcome a few folks here to help me talk through them as well I want to add Quinn and LDJ hey Quindla I want to introduce Quindla Kramer to the stage welcome Not your first time here. You've been here for a minute, so folks know who you are. How you doing? I'm good.

1:23 It's good to see you. Good to see you as well. And we have LDJ as well. DJ, how are you, man? I'm doing good. Awesome. We will have a few guests, like I said, including some friends of ours who did their own thing. Qu I'm very interested in hearing about and to have a discussion about this as well. And I think that after the the explosive week last week, it does seem a little bit like it's a chiller week this week.

1:50 Instead of covering all the news in depth like we do and we barely have time, hopefully today in a very chill way, we'll cover everything without rushing. So I think that this is what we'll now do. We'll jump in into the TLDDR. TLDDR, as I said before, for new listeners of the show, covers everything that we are having on our docket this show. a lot of the times we have breaking news. Hopefully today we'll also have a little bit of breaking news and and so the show changes but basically this is what we are here to talk about besides maybe one other thing that I want to add at the end of the show if we will have time. So with that let me go to the LDR. All righty. This is everything that we have to talk about on Thursday night for April 24th. And we'll start with hosting guests. Obviously me, Al Kov, I mean the evangelist with Weights and Biases. We also have Wolf's going to join us soon. Yamas here, LDJ. And then we have two guests joining us. Quinta Kramer, the co-founder of Daily and maintainer of Pipe, the open source, and our resident voice expert, everything voice AI. And we'll also have Dex Horsey. Dex has been on the show before as well and Dex is we'll call him our resident agent expert and Dex is the founder of human layer and he will join us to talk about how to build 12 factor agents something that he wrote and authored and would love to hear just about also the update in the agent world in the open source we had a few things this week nothing super super crazy I think the highlight of the week for many of the Thursday listeners was openi is closer and closer inching to their open source release and they've met with the open source community friend of the show has also been there and they will hopefully let us know I don't think that openai limited the conversation so they can share what the conversation was about also in open source Nvidia released describe anything three billion parameters they call it dam if you guys remember SAM segment anything it's like that but also describes it's pretty cool we're going to show an example of this and going to play around with this it's a ll LM multimodal LLM for region based image and video captioning. I haven't seen the video part. I only seen the region like the image part, but that's absolutely super cool. I want to dive in this QAT thing from Gemma. Google released a quantization aware training QAT Gemma 3 models and they're like significantly smaller. Finally, we got a little bit of a break. We can breathe.

4:29 We can try out all the new things. And then I think the highlight of this week for me at least was OpenAI finally releases their image gen which they call GPT image one in API so the developers can use this in API. There's a few things you can do. You can edit an image, you can generate an image is pretty cool. The fact that it's now integrated into multiple products as well. They launched together with I think five or six products. Very cool.

4:56 So, we're going to chat about this, see the limitations there, and play around with this Grock. We haven't talked about Grock in a while, but there's like a few things about Grock. First of all, while we're still waiting for Grock 2, Uncle Leon, to get released in open source, we are still waiting in that Gro 3 and Gro 3 are finally available via API. And this means that folks can do evaluations and then we can build with them and they're integrated in multiple other products as well. So, we'll chat about this. GRA also launched a video mode in its iOS app where you can talk to it and it can see you and yeah, we should definitely chat about this because we remember how long it took for OpenAI to announce this and then release and then in the middle Google released this and now it's pretty much everywhere. So definitely worth chatting about and maybe even trying on stage. We'll see if we can make this work technically in this week's B about and MCP and A2A. First of all, we have a hackathon coming up in San Francisco. This is the first you'll hear about this. It's coming up very soon, May 17th, 18th in San Francisco in the weights and office. I'll be there. Sam will be there. Most of the weave team will be there as well. You are more than welcome to join. If you were looking for a reason to go to San Francisco, that could be it for you. I don't know, maybe go see the Golden Gate Bridge and then come to our hackathon. If you are already in San Francisco, there's nowhere better to be. We're planning to go all out and we're going to announce sponsors very soon. The second announcement for weights and biases is fully connected, our flagship conference, our two-day conference where we bring a bunch of weights and biases customers together and have a bunch of speakers. That's happening in June and that's coming up very soon as well. It was announced and it's incredible.

6:40 Moving forward in vision and video, there's a very interesting 24 billion parameter model called Maggi1 from Sandai that came out of nowhere seemingly. It's a Chinese company and it generates long like long videos that are character consistent, which is crazy. And we definitely should take a look at some of these and hopefully maybe even talk about architecture because architecture is also crazy. Also, another company goes into avatars fully. Character AI announced Avatar FX and it's in early access and we should talk about Character AI. I don't use this but million people's use it. It's really interesting that they are doing this pivot into physical visual avatars that you can chat with going forward in voice and audio. At least as far as my Twitter account is concerned, this was the highlight of the releases of this week.

7:35 Nari Labs which is essentially a person in a half in Korea like it's one dude in his friend halftime in Korea have trained DIA a 1.6 billion parameter voice gen models with emotional tones and multiple speakers and podcast generation you could see the similarity to notebook LM from Google because the voice talk over each other. This blew up. I think people love this. People ate this up. This is an MIT license as well. A very tiny very well performant voice models. ly you had the chance to check this out but definitely we should play some examples for you cuz some of them are crazy also in voice and audio our own coin here you guys have your own release by cat smart turn daily smart turn I don't know how we want to call this smart turn model which is a semantic VAD v and we're going to chat about if that didn't tell you anything we're going to chat about what this means we did touch on this a little bit before but that wasn't open source and now we have an open source we have the author of this also here with us will tell us why he spent his holidays working on this. And this is voice and audio. I believe that this is it, but we'll see. We in AI art in diffusion.

8:45 Obviously image one kind of AI art, but that's in big companies. Hun Yuan released 3D Hunan 3D 2.5. It's a 10 billion parameter 3D model and the results seemingly are crazy. Those 3D models are getting better and better. We should definitely take a look and definitely check it out. And I think the last thing the last thing on my agenda at least until folks come in and correct me and see what else is there. We have Dexter Horthy the founder of human layer to talk to us about his. So if you are building agents or interested in agents for AI, Dexter will be a resident agent expert for this week. He'll join us very soon to talk about 12 factor agents and neess the blowup on hacker news and we'll chat with Dexter as well. I think that this is it for us for this week.

9:36 This is the TLDDR and this has been Thursday. Now we actually need to talk about all of this. But as you guys see, it's been a relatively quiet week and we've been asking for a quiet week. So I think it's a great opportunity to just take a pause for a second and then actually dive deeper, maybe learn a few more things, maybe tell the audience. With that said, folks who are listening live on the stream, on Xpaces, and on streams on YouTube, on LinkedIn, on X everywhere, if you're listening as well, please let us know what you guys think. Let us know what news we may have missed.

10:11 let us know what was interesting to you. We'll definitely also want to chat about the actual models themselves. So please if you have any comments let us know. With that said this was the TLDDR. We're going to start with open source. But before this we now have the full almost full quum. We're missing Wolf from Welcome Yell. Welcome Nan. How are you guys? What is the most important thing in AI as far as you're concerned? Let's start with Yum.

10:35 Oh open AI. Open AI. Got it. Open AI going open source like absolutely going open source than it was known before. Like we all know that they're going to release a model, but all of a sudden they approved their agent from last week. Correct. Apparently, there was a pull request to extend it to providers that are not open. So, it'll be able to in case the agent with the property call or whatever. And even if it's open source, you don't really expect OpenAI to all of a sudden approve this that you can use whatever model that you want. Out of nowhere, they just approve it. It's fully unlocked. You can use it with whatever model and engine that you want.

11:19 That's impressive. That's really going open source. Look at the amount of requests over the week. It's impressive. They're really working on this fulltime. There are people really maintaining it. These tools are extremely powerful. Code XM cloud code are extremely powerful tools and it's absolutely amazing to see this. N what about you? What's your highlight of AI this week? What's 2000? I think I'm just happy they un nerfed Sonnet. It seems good now again. Wait, really? Sonnet is Yeah, sorry. 3.7 thinking. I don't know. It's not being It's not being dumb anymore. So, I I like this competition. I am still a Sonnet 3.7 fanboy. I don't think anything replaces it. Whatever.

12:08 well it it was I was testing it before and for a while it just started getting even basic questions or like basic hard questions wrong. It felt like the inference had changed or bit quantization or whatever. It was just dumb and this was a lot of people reported all all at once and I was actually able to compare past tests and current ones and solid 3.7 thinking was actually failing them and the one on Marina was was passing them. But now it seems okay now it's getting everything correct. So I'm happy about that. On that side, I still feel like all hands and open hands is just flying under the radar for most people as to how good that thing actually is. They still have 50 bucks on their end. You can just run it on your own, too. And that thing is crazy. Yeah, I actually prefer the way that they build the app as a single page or two that they don't just go nuts with Nex.js JS and everything because that becomes quickly a convoluted mess. Whereas this way it just keeps it all in context and you can actually use other models too with with that. So I think those are the two things I want to highlight besides outside of the voice stuff. All righty.

13:29 And speaking of voice stuff, Quinn, what was the highlight for you before we actually get to the those parts of the show? for me the dia model like I think the open AI guides in that training 4.5 video a couple weeks ago were saying that half the battle is knowing you can do something and to have like two students in Korea release a great model it's possible we should all be working on this stuff it's super exciting I also think we should shout out the Gemini team which released spread out over last week and this week more stuff in their multimodal live API so the big labs are also like continuing to push forward on speech to speech and the APIs for all that and that that's flying under the radar a little bit. I think the the interest goes in waves with the real-time API and the multimodal live API but the teams at both OpenAI and Gemini keep like pushing it forward which is cool. Oh and lastly LJ before we go to open source highlight of the week and then folks in comments feel free to leave us highlights of the week as well. We'd love to see and highlight them on the show. LJ what was your main thing main takeaway main interesting thing from this week?

14:34 The main thing coming to mind right now is probably 04 mini. I think it has its downsides, but overall its ability, especially if you look at things like Arc AGI or if you don't really like that benchmark, just its ability to do various things where it seems like in some cases it could be even cheaper than 03 Mini because it's even smarter sometimes at being able to know when it should use less tokens and just the abilities and the the cost effectiveness of it. It it seems like we're just continuing on this path of a path of cost efficiency. Yeah. Yeah. We definitely should highlight the ARGI kind of results for O4 Mini. Definitely for folks who are concerned about very hard improvement and hard takeoffs, seeing how these models improve on various benchmarks is very interesting.

15:21 LJ, if you have that link, if you drop this, I'll definitely open this up. Folks, I think it's time for us to actually start the show and talk about open source, our favorite subject here at Thursday. Open source AI. Let's get it started. Let's get it started, folks.

15:52 For open source for this week, we have the score is open source AI in general, but we're going to like mention voice and stuff as well here. We obviously need to start with not only the fact that o openi is like committing to open source in codeex but also with the fact that this week openai have welcomed on the zoom meeting multiple folks who are involved in the open source arena. How about we tune in and listen from the man himself. Let's add Mazier here. I guess while we're waiting for him to pop in, something else that was confirmed prior, I think maybe a week or two ago, Sam said at an event that they plan to have it be competitive with any existing open- source model of today, which seems like it'd be like at least on the level of R1, Llama 4, Maverick models like that. Let's go for let me just introduce Mazer Panahi for a second because you've been on the Twitter spaces multiple times and we obviously celebrated your work for a very consistent time you've been one of the more prominent finetuners in the fine tuning community for open source and a friend of the pod but I think this is the first time definitely on video feel free to so first of welcome great to see you and feel free to maybe quickly introduce yourself so folks who know who you are and then we'll talk about yeah of course thank you so much it's the first time being here like this yeah my name is Mozi I work at the CNOS. It's a national research for a French government. It's a public research. I do create AI models from the various small ones all the way to the very big ones.

17:26 And I just want to tell you a little bit about yesterday if you have some time. Yeah, please. Yes. Yes. Yeah. So, I I didn't expect to see some other people there. I didn't expect to see Sam there at all. I knew that people going to be from the training team obviously, but there were already some big names. Sam was there. He spend the first half of the conversation answering all the questions. So that was very nice of him.

17:50 People had a lot of questions regarding the size, regarding how often you want to do it, will this happen regularly. So he was answering all the questions and basically what he wanted to do was that their GPUs are on fire. So he he wanted to release open models so some of these loads would be handled locally. That is his philosophy of how do we make sure we can help the community but at the same time the community helped them take some of the pressure on the GPUs and then we were divided into different rooms and h I was ready to fight with all the people the size and whether or not we want reasoning or not but it was really great to see all the people with the same experience because for the last two years people were really fine-tuning and training all these models and putting in production so they've been all battling the same issues and having the same experiences. So of all the rooms being compiled, everybody was literally asking the same thing. They didn't want a small model because to be honest, you could get a small models from different labs and different institutes, right? You want a big model, but not too big, not something like a 600 plus billion parameters that takes the whole node and and the whole server has to be just for one model. Something around maybe 70 to 200. So you could spend four GPUs run this model and then the other four GPUs would be for other models like your audio, your video.

19:16 Maybe you want something as smaller to deal with the intermediate steps. You don't want to bother the big models. So everybody agreed on that. Everybody really wanted the structure output. H weirdly nobody wanted the reasoning at all really. Yeah. They were like but we don't need the reasoning because I I guess people in production the tokens are expensive. So yeah, they were explicitly asking please do not spend so much time doing fancy RL and this model because it's just going to be for ma math and coding and some really very specific tasks. So it was really they they were also surprised to see some of these comments coming from like a large amount of people. you think that is I I'm really interested as far as I understood maybe from comments that first of all open told us explicitly they're moving towards a hybrid model we saw it in Gemini 2.5 flash we saw it in Nvidia released something that's also Neotron is also like a hybrid thing open announced they will move towards like hybrid things cloud 3.5 so on it as well and as far as I understood that's what their plan was why do you think people do not want people just realized from the open models that they do reasoning, you are stuck in the reasoning. So you can't really turn it off. So they ask if you are going to do reasoning, please allow us to turn it off so we can decide when we need it and when we don't. So we don't have to wait so much because we ask for a large model, right? So you have to spend a lot of compute on those tokens. So that was one very surprising thing came out. The other one was really multilinguality because there were large majority of the people were coming from EU. So they were asking to have a good representation of the EU languages at least 20 of them as good as English that it's not that hard right a lot of people asked for the base model they were saying if you give us the base model there would be loads off your shoulders because the community is going to come and fine-tune a lot of stuff that you don't have to anymore right just give us a good base and then the people going to come out for the languages that they're underrepresented for tasks that you probably didn't want to do coding role play other stuff we will do it for you just please do give us the base and one thing people really also focused on which it was really I thought I'm going to fight with them but they were all on the same page that please do not try to be top one or top at all like just give us a very useful model doesn't really matter where you are in the leaderboard because then when you if you go to that site it's never going to be good you're going to be thr out thrown in two days or even in one day and they're going to be discouraged. Just give us something that is very useful and the community would appreciate it. The community would use it. They will prune it. They will distill it. They will do a lot of things with that model. But if you just try to gain the system and to be number one and think that this thing is going to be just in the headlines for a whole week, then we will never get that model because it's just not possible. with the speed that we are going forward, whatever you do today, tomorrow is just going to be a better model coming out from other labs, right? So that was things that they agreed on a lot of it.

22:31 They came out with a lot of questions that was like they showed they came prepared which was very nice that they took the time they prepared questionaires they were moderating the conversations the groups were there very diverse from people working in UK government or French government and having a startups that was very nice also so they said soon and open AI when they say earlier stage it could be in two weeks you never know with open AI you know usually soon means like 10 minutes I I don't know how soon, but the fact they said they would like to be committed to a regular release of open model. Oh, really? That was something very nice. I like that.

23:10 It's not like a one-time thing that here is a model. Goodbye. I think it's like here is the model that we're going to keep on releasing and making sure it's useful all the time. Something we always ask, right? Please don't release model once a year cuz after 6 months is very outdated. You just do it like every 3 months. You don't have to be number one again. just be useful all the time. Thank you so much for this great recap.

23:34 I'm I'm very happy that they invited Prince, invited like a bunch of other folks. I think it just shows that OpenAI is like really planning to listen to the community. This is great folks. This is like absolutely incredible. So shout out to OpenAI for for this coming prepared having Sam there. Sam and I think a few folks already collaborating with HugenFace. OpenI is doing it right and hopefully we'll see a great model coming out from them. Maybe a hybrid one, reasoning, etc. Folks, anything else?

23:59 Any of you want to ask a specific question to Mazi before we move on here? Did you mention the gap rights? Yeah, Wam. Go ahead. Yeah, I I I had my sources as well. So, they talked about the Yeah, so about the gap rates. Did you mention that just now? Yeah, there are some people who were asking for it. the safety was big in one of the realms and I guess the projects they have in the government like one person was like we want to put the entire UK government and AI. So I think safety for them is very important but at the same time we were like could you not like really do alignment on the model cuz then the quality would drop. you could just give us a very smaller classifier for red teaming for toxicity and for other stuff because you you guys are very good in this kind of stuff right we don't also need to waste asking the bigger model what do you think should be answer or not so I guess what we see the trend from llama llama guard and then gemma has gemma guard for visual event h it's better to separate them not be because the too much post training and alignment it's just really hurting their model. That's why every time someone comes and uncensored the model, you just get a better result because it's not really limited to what it could say. I think that was also things that they agreed on having a separate model for safety. They will do their safety as much as they can but not too much like some of the Gemma instruct if you realize you have this prefix answer postfix. So the safety is embedded into the models so much. You have to have a system promp to say it like I get it.

25:39 I'm already a doctor. Stop keep saying you have to check with the doctor. Stop being so concerned about that you're an AI. Like we get it. We know we're talking to AI. It's 2025 is you're not fooling anybody. We know you'll make mistakes. So I think that was the whole some of it that guard rails would be nice to come with another model. I gota wonder if how big this release will make waves in the community because OpenAI actually being open from models. It's been a long time since that was the case since GPT2 I believe is the last model that they released Wait for and then Whisper which is not NLM and then the new transcription stuff they didn't release in open source. So Whisper is like still the biggest interesting one.

26:20 That's very interesting. We're looking forward to it. Obviously folks from open air who are listening we are going to cover this. If you want to send folks from the open source team in OpenAI to chat with the open source with the best show that covers open source AI in the world which I think it's us you are more than welcome to send it and if not we'll definitely cover this as well. Mazi, thank you so much for coming up. Lovely being here. Thank you. Yeah, it's great having you here as well. Feel free to stick around with us. We have a big group here and I love it because the next thing we're going to talk about is actually let's talk about the quantization aware stuff. You've seen this as well the folks from Gemma. Shout out to our friend Omar Sun Civero who was who's now doing God's work in open source in in Google. They announced releasing Gemma QAT. QAT stands for quantization aware training and they reduced the VRM requirements from 54 GB for this model. Gemma 3 Gemma 3 weights 27 billion parameter from 54 GB VM to 14. There's a significant reduction. Let me actually show this on screen here because you need to appreciate the graph to see what this does. And quantization over training who can tell us what that actually means because it's not quantization post training, right? It's happening while they like make the model. Your firm down. Yeah, there there's different forms and types of this like in research over the years, but I think the main thing that they're probably doing here is something along the lines of as you're training, you're actually at times like doing selective quantization of which parts of the model are optimal to quantize at certain points in time and doing that progressively as you train the model more and more. And so that allows the model to typically end up in a quantized state that's much higher quality than if you were to just apply some general quantization after the model was trained that's not specific to that particular model.

28:12 I see. And so we see some Thank you LJ. We see this kind of like improvement in requirements all across the models. Even the 1 billion parameter model goes from 2 GB VM to half a gig VM. The 4.B. Wolf from you had a chance to test this. I saw a quick comment from you that's like very interesting. What happened over there with this qu? I'm I'm running it in Len Studio and the good news is it only takes about five to six GB of RAM for just loading the model and using it.

28:42 So you can run it on your at home computers easily. That's the thing about it and the quality is very high. But what I was doing I have a pretty complex system prompt especially about language because I talked to the one in German and English. the prompt is completely in English and so I was saying hello in German and the model started talking in English so I said okay there is a rule in the prompt that's how you have to respond and if I start in German you have to speak German as well and so I I always do that I did it with flaw if something is not right I say okay there's a bug in your programming you have to fix it we have to look at your prompt and fix it and when I did that with jammer it gave me that response oh you really are attentive I must say I'm impressed yes you have a prompt you're right it does say something else but let's be honest who beats a fine print and that with a wink emoji with a winking emoji this quantized gem just full of sass for you.

29:35 Yeah, of course that is my end as always. So that is a system prompt that is there but there's a complex rule about language. So we are taking turns and because I often use it for translation so it's translate but still speak my language. So that is partly Amy but it has to follow the prop. That is the whole thing. And Gemma 2 did not have system prompt support. They didn't support it. If you try to send one, you got an error message that it is not supported which I criticized and I'm really happy that they did it with gap Java 3. There is system prompt support otherwise that wouldn't work as well.

30:08 But something is not as good as other models regarding prompt adherence apparently. And the model even acknowledges it in a sassy way. Oh yeah. I am annoyed. I am so annoyed but by this blog post because they could have posted exactly what the benchmarks were in quantized versus regular quants and just shown an improvement. They don't show any of that. It looks like the method that they used they have the language here is very chat GBT but I think they just kept the they just kept the last layer just kept the output at float 16 so that they could get one of say that they could get the log props from the non-quantized checkpoint so it looks like they just continue the training for 5,000 steps. I'm wondering if that conversation aware training was mostly in English that that's why it it performed that way. But yeah, overall I am both excited and annoyed by this. Did you test function calling? Because one of the things that I I feel like Jimma lags behind a little bit in what they focused on in the training is function calling and I'm curious about the QAT function calling. I didn't get to that yet. So it has been just testing a bit of the chatting and there's also the thing you have to use a proper temperature and other values. There's onslaught there's a nice tutorial on how to use it which I'm sad to say that should be in the model card not on some other resources where you have to look for it and of course it's grandized and it's running in LM Studio. So I don't know yet where the problem is. Is it the model? Is it the quantization level? Is it the settings? Is it the inference engine that has some bugs? I've seen that with newer models in other like llama and other setups. So I don't know why I'm seeing what I'm seeing and I'm maybe too used to the big online models like GBD3 03 40 and 03 and so on. So maybe I'm expecting too much at the moment but I definitely have to look into this.

32:15 It's not doing as well as I had hoped. I had problems with it. But I was using the MLX version and I'm going to try again this weekend because I'd really like to get function calling working because it's a good model. Y Oh, but I recommend MLX started adding 4bit AWQ quant support and by default AWQ was only a three bit. So that was the issue that they hadn't shown before. But now they show with AWQ it keeps perplexity at 99%. which might even be better than doing it through quantization of error training. I'm not sure. I would probably use that qu that might just be the best performance to capability ratio right now for running the models locally. Moving on to Nvidia releasing describe anything. It's a three billion parameter model. They call it DAM 3B, a multimodel LLM for region based image and video captioning. I haven't seen the video capturing here, but I've seen the the demo for the image capturing. It is pretty pretty cool. So, reminder released like a long time ago, segment anything. It was a very tiny model allows you to select something and it will segment completely. This model kind of does this but also describes what's happening here. You can see that for folks who are watching, folks who are listening, I'm on describe anything model demo and I just preloaded one of the images there. And as I move my cursor, the segments underneath are highlighted.

33:44 But when I click on one, we can see a description getting generated getting generated in a second. Usually takes a little bit faster than this. And so I clicked on this bear and says, "A brown bear with a thick dense coat of fur featuring shades of gray, brown, and lighter brown. the bear has prominent snout. So basically and you can then describe a new region and then describe everything. It is literally describe everything but I think the interesting part about this is that for folks who are watching say vision models can do this any model right now is a vision model like we can describe it. I think that the point here that you can like specifically ch choose one region and then have a description of that. I think that's a cool point here as well. Let's see. How do you find this? Like where do you get data for something like this?

34:30 That's a very good question. I don't believe that Nvidia Meta I think that's all of Meta's past work. But Nvidia listen but they were the ones pushing this for a while. There was efficient SAM 2. They've pushed a lot of data sets. It's 35 terabytes or something from meta for this type of data. It's that's fully out. There's a lot of data on this. It looks like the data set is also released. Describe anything data set. Oh, okay. All right. Yeah. So, we have we have subsets here. Coco and Paco and Sam efficient. Yeah. Okay.

35:03 Yeah. And let's data studio. We take a look. So, first of all, shout out to Nvidia for releasing something like cool like this, but also with data. The the project page is full of Nvidia and UC Berkeley as well. So, shout out and UCSF. So, shout out to all these folks for training this thing. I think the coolest thing about this is that it's video as well. The demo doesn't work on video, but as we can see, this kind of the the segmentation kind of keeps happening over video as well. And for folks who are just listening, we're now watching a gift of a fish in tank. All these fishes look the same. And the caption when one fish is selected is a vibrant goldfish with striking pattern of orange and black spots. The tail fin fans out blah blah blah. I think this is pretty cool. Like we we don't have the our friends from vision department from Roboflow, but I'm pretty sure that they're cooking like a L'Oreal for this in like supervision. This is dope. Do you guys remember when we had Prosks here from Roblow to talk to us about SAM and then there was another model that did just the description? He literally built kind of the combination of them to get to this model. He built Sam plus I don't remember what it was, but it it was a a model that that Oh, something from Microsoft. Come on. Florence, maybe. Yeah, I think. Yeah, Florence was it. I think it was Florence. And then Florence does the description, the captioning, etc. So he combined this to segment anything you click and then you get the caption for Florence. So this is like one unified model in what three billion parameters that does both is super cool from Nvidia. Detailed localized description is what they call this. And yeah, it's pretty cool. I think it's like very useful for multiple applications specifically. Imagine honestly now with the GPT image that we're going to talk about you can imagine all kinds of interfaces where you want to remove some part GPT image one allows for masking for example you can super quickly build with this model the segmentation part mask out the person that you want and replace it with a dog so even for that it's like super useful three billion parameters probably will run almost real time on any like production GPU comments about this one folks what do you guys think do we is there anything else we should mention about this model anything we're interesting that we want to tracking objects over video is a hard problem.

37:09 It's like sideline. Yeah, we do that as well. But we also describe this is crazy. Seriously, like I haven't looked into the architecture, but just by looking at this the level of understanding, it's probably a really good base for tons of use cases. You can further train this on, but just the level of understanding throughout video and embedding it in language is probably a really good model. It makes sense. Everything a big produces is really good. But yeah, that's impressive.

37:42 Seriously. Yeah, I'm with you. This is like super cool and I can see a lot of useful things that coming from this model. They also compared this to other DLC's detailed localized captioning. So basically this task they compare it to other pipelines as well. Just the fact that this is one model and not split into segment anything else is pretty cool. Yeah, we also have another model. You're right, Wolf. Let's cover this like super quick as well. Nvidia is on a tear. I think people are starting to realize just how far ahead Pod at the Roboflow is and has been the entire time. Yep. He's, you know, literally a year ahead of everyone else. Yeah, he build this but not in one model. They haven't trained the model for this like pipeline. Let's talk about the other Nvidia release super quick while we're in the open source. Nvidia quietly also dropped Open Math Neimatron. Open Math Neimatron 32 billion parameters N14.

38:32 They won the first prize in AI math olympiad 2. As you guys know, the math olympiad for people. There's one for AIS as well. It's called AIMO. And Nemoron that was trained on Open Math was basically winning the first prize. And they just released the data set for this and the model. And we can take a look at the kind of the results here compared to is that let's see what did you ask us? Yeah. Where are the the big guns? Oh yeah. It's on the left.

39:04 So for folks who are listening, we're seeing a bunch of comparisons between a few models and then these I think they're comparing only to open source models because we only have QW here and the deep still 32B. Wait 1.5B. Look how much that got. Okay, that gets impressive. Yeah. So, Open Math Neatron 1.5B gets 65.6% accuracy on this AIM mo or I guess this is AIM. So, this is like math like the human math olympian. That's pretty cool. The thing about this is it's really close to the 14 like what's going on. Yeah, the 1.5B being higher than Deep Sea Car One that needs to be double checked because that is completely insane. That is completely just groundbreaking and stuff. I think the thing that you guys are referring to looks too good is that the fact that Open Math Neimatron 1.5 billion parameters get 65% and it beats both the distilled deepseek 32 billion parameters and QVQ at 32 billion parameters. Right? the very surprising result here that like this 1.5B solves NAF at a level of a 32 billion parameter QVQ which is reasoning model Nemoron is also a reasoning model but I'm not sure if the small one is the thing that to highlight here as well that these folks released the Kaga winning solution and also the data set it's a 5.5 million math reasoning data set that they released they call open math data set and shout out to Nvidia for all folks how about that shout out to Nvidia for all this open source work what the heck this is Great.

40:43 We never each and every single time the data awesome. Absolutely. So we'll definitely celebrate this with applause because the model models come and go but the data web they stay and you can use them reuse them. Yeah. And they said the data set was a foundation of our winning submission to the AI MO2 Kaggle competition. So if we go to Kaggle and we see like a AI math olympiad progress prize 2. I think it was like $50,000 and Neotron was the winning one with the data set release. Shout out to the folks. That's great. Moving on in open source, folks. We're moving on open source because we need to also talk about audio stuff and we have our audio expert here to to cover the audio stuff as well. Two releases a week in the audio area. Let's talk about DIA first because I think it absolutely blew up. I think that it will help if we actually play an example of the while we pick it up. Quinn, if you would like to open up with the description what this is and then I'll try to pull up the actual audio as well. It's an open weights model done by two students or maybe a student and a half and it has like crazy dynamic range. It can just take a script and be super expressive. And I think that's why it blew up. That's my sense of what people really keyed in on. and we should play some examples and then then I definitely have some thoughts. Yeah, these examples I've played them to to okay I think it's very important for a model like this to have a demo page but basically if you guys are watching you can see there's like speaker one and speaker two and this is how you pass this to the model like speaker one speaker two dia is an open weights text to dialogue model you get full control over scripts and voices amazing try it now on GitHub or hugging face this was Dia or D I don't know but the thing that I love the most about this is that you can add emotions in parenthesis and laughs. Quinn like you can hear the person starting to laugh in the previous word. So this model is like smart about the whole segment. Not it doesn't get to the left and starts laughing. If you play 11 Labs, let's play 11 Labs again. Dear is an open weights text to dialogue model. You get full control over scripts and voices.

42:54 Wow. Amazing. Uhhuh. Try it now on GitHub or hugging face. So this was 11 Labs and you can literally hear the thing that I said before. it gets to the laugh thing and then it goes haha and it's not built in for some reason. I think that this is what caught fire Quinn the most is like again when you add these emotions it like it really comes through but you had some other experiences with this. Let's listen to another one about the fire where there's a fire going on and people are yelling can we play that that piece and say let's play okay if I'm not mistaken and hopefully I'm not mistaken this is from the office. This is that like famous scene when everybody's getting freaked out. And so here's 11 Labs reading out this fire. Very emotional script. Oh. Oh my goodness. What's the procedure? What do we do, people? The smoke could be coming through an air duct. Oh my god.

43:48 Okay, it's happening. Everybody stay calm. What's the procedure? Everybody stay calm. Okay. So, it reads as I really like I really like that you hear that and you think this is good and then you're going to play that and then you play the open- source 1.6 billion parameter model that Korean and a half students trained in like a basement somewhere. Oh, fire. Oh my goodness. What's the procedure? What can we do people? The smoke could be coming through an air duct. Oh my god. Okay, it's happening. Everybody stay calm.

44:15 What's the procedure? Everybody stay calm. Everybody calm down. No. No. If you touch the handle, if it's hot, there might be a fire down the hallway. Yeah, this is a how did how did they do this? Like this is incredible. Seriously, what's the data set? How do you how much did you train this on? What? It's like unbelievable. Yeah, it's unbelievable because we're here on the podcast and we talk over each other. We understand when Yam is about to finish the sentence. I come in and then Quinn, we're going to talk about all of this in semantic fat as well. But we all have a we all have an understanding of what's going on. When you hear the 11 labs, 11lab voices sound incredible. They have been working on this for a while. 11lab voices are great. However, the thing that we loved, especially from notebook, is that they like the two speakers have this interaction between them. This is again a 1.6 billion parameter model. No idea how it gets to this level of emotional and also like interruption and the voices speak on top of each other and like the sesame also is there. So, here's a last maybe example of kind of the non-verbal stuff that it can do. So, there's the like coughs and sniffs here.

45:21 Hey there. Why did you just cough? Why did you just sniff? Why did you just clear your throat? Why did you just laugh? Nicely done. So, hey there. The last part is what drew me completely away. The when it says why did you clear throat and then it says laughs in parenthesis, it starts to laugh at the word throat. It's like why do you clear throat like this? Whereas 11 laughs got to the point and then started laughing and then the the other guy just interrupted him. Why did you just laugh? Just interrupted in the middle like the first sentence did not finish. It's like the model generate two voices. Quinn, what do we make of this?

45:57 Did you say anything anything? Didn't they really say brag about what they did? Nothing. The bragging that Toby did, Toby is one of the folks who trained this or I think the main guy who trained this on Discord. The bragging was we got to number one hug and face and we got to like so many stars on GitHub and like it went super viral not only for my account like also on a bunch of other good reason for good reason.

46:17 Quinn, what do we think about this? So it's awesome. I love this work. It's great. But I'm going to throw a little cold run on it just for the sake of talking about where we are with speech. The 11 Labs folks could totally do this if this was the goal. It's partly about what you're aiming for. And if you want to train a model that people are going to pay you money for, it needs to have some level of predictability.

46:42 And the dia model is unhinged. I am 100% sure they ran those examples a 100 times to get the examples they put on the demo page, which is fine. Like it's totally great if what you want is a model that shows what you can do. There's also the longer you're a programmer, the more you realize that every single thing in the world is duct taped together. Yep. So there's like string replacement for the speaker and the special like nonspech things. It's a research model, which is again awesome, but it's just a totally different thing than what people who are putting out a model that they want people to pay for are going for. Like I spent 20 minutes trying to do a dialogue with DIA and trying to do a dialogue with the GPD40 mini TTS model and I got much better results from GBD40 mini TTS when I had something I was aiming for.

47:34 Now if you just throw some text at it and are like I want to see what it does. I haven't seen anything that's more fun than dia which is amazing but like predictability steerability all these other things they really do matter too. The data set's probably all YouTube. Everybody has just scraped a ton of YouTube for this stuff, which is why there aren't very many open data sets for these models. We're going to come back and talk about open data for audio in a minute. I also think that if you really focus on those non-spech things, you do get a huge amount of mileage. And maybe that's one of the things we should all take away from this is if you're working on audio models, like now we can set the bar at like speech isn't enough. I want to see sound effects. And some of the new OpenAI models have sound effect stuff, although I don't know how much they've nerfed that for the API because it makes it harder to use the model for speech.

48:24 There's stuff coming out at Google that's going to be amazing. Notebook LM is obviously amazing. Like 2025 is the year of audio in some ways. Absolutely. I I I think that I completely agree with you. The one thing that we we need to highlight obviously no matter how like a student and a half from Korea no matter how correct they are no matter which data set they cannot compete with a company that this is what they do for productions and this is like the product they sell. What they can do though is get us all excited about the possibilities something that you refer to in the beginning right getting to know to the level of this is possible on your phone in near real time is incredible. the fact that you can clone your voice here, which I've tried unsuccessfully unfortunately because some of the MPS stuff wasn't quite there, but the the open source community definitely got up to speed. You can clone your voice and like our voices as well, and this will continue in that.

49:13 So, it's not even like oneshot cloning. It's like imitation in the inference, which is also super cool. This something they they've built in. It is hinged and unexpected for sure. And I got to wonder like what this data set with a little bit of bigger model, a little bit stable could do. We don't know like audio models are small and if you're a student or a hacker and you have a 4090 you can train audio models. That's one of the really cool things. So like Kakoro is 82 million parameters. All right. Yeah. And and it's good. Yeah. This is 1.6 billion still very small relative to you can train it on your own hardware, you can train it on modal whatever. So yeah, everybody should be playing with these models if you're interested in in deep learning. Yeah. But yeah, I want to move on to the other open source Quinn. I would love for you to tell us what you guys have released and we've chatted about this me and you at the engineer obviously and then we briefly touched on this when open released like semantic in their advanced voice mode which was great and now I think they're also releasing API and in the you like in the app itself now the app is like significantly better better for kids could you maybe start with what is the big news and then we'll talk about like semantic and some of the things in data yeah so the breaking news I wish you'd play the breaking news I I love breaking news. Let's go.

50:27 AI breaking news coming at you only on Thursday. I thank you. That was awesome. So, we released a checkpoint of this open source open training code turn detection model we've been working on as a community. The goal is to have a state-of-the-art model that can tell you when a person is done talking and then LLM should start talking because that's one of the kind of 2025 unsolved problems broadly speaking in building really good real-time voice agents. And it's surprisingly hard. We don't even always get it right as people. You're on a phone call, there's a little bit of latency in the phone call, you tend to interrupt each other. We've got transformers and audio conformer models.

51:12 And so we've got really good tools for trying to figure out when people are done talking and when an AI should respond. And there there has been no there there's been very little open source work on this. So over Christmas when I didn't have any meetings, I did a bunch of training runs and cleaned a bunch of data and then it snowballed a little bit and we're inviting everybody to come hang out and train small audio models with us. this one with audio on the input side and just a classifier percentage on the output side saying complete or incomplete phrase. And the breaking news is that this is now on foul. our friends at foul.ai are hosting this model. So you can use it from any kind of voice agent or multimodal agent codebase and just hit foul ask for complete or incomplete log props and get those back and and decide whether to trigger inference or not.

52:02 OpenAI's semantic VAD which is what they they call this in their APIs is in the real-time API now I think it's really good how does it work Quinn I I'd love to tell our listeners what semantic V is or what smart turn is as well basically one of the reasons people don't like talking to AI is this specific thing it interrupts and while you're thinking especially for kids we've talked about this with kids a lot as well kids tend to think maybe longer for the next word maybe they're trying to find multilingual folks such as myself, we also like struggle with this as well. oftentimes the word the correct word in the wrong language comes up and you're like no brain I'm speaking English right now I need a different word right now and while this like my mini conversation with brain happens chic used to just Alex here's an incomplete answer because you did an incomplete question so previously voice activity detection you will correct me when I stop speaking I'm sure just did this based on sound levels right based on just like quietness and some better than sound levels so traditionally you use a a very small specialized ML model called a VAD model, voice activity detection model. And the only job is to take an audio frame and say this is speech or this is not speech. And then for turn detection, what you've typically done is you've taken a certain number of frames of non-spech in a row and been like, okay, the person is done because they paused this long. Yeah. And the reason you use a bad model instead of just audio level is because you want to filter out background noise. You want to be very you want to work in a whole lot of different contexts. So you use this kind of special purpose classifier.

53:31 The idea of semantic VAD or smart turn or whatever you want to call it is to go really significantly beyond that and try to incorporate the audio patterns in the speech and the linguistic data that you have access to. So I just said and I drew out and then I paused. I probably wasn't done talking that but both because you can hear what I'm doing tonally and because you understand the linguistic content. So what we're trying to do is train a model that can do that. Now, I love the question about how does it work? Because I had a bunch of people at an event last night where I showed a demo be like, how does it work? And I was like, that's the wrong question. It's a deep learning model. We're not supposed to know how it works. All we can know is that we clean a bunch of data. We train the model. We look at where the model fails. We look at the data set. We iterate. It's an audio conformer model. So, it's got like multi-headed attention block and a feed forward network and a 1D convolutional network and a feed forward network again and then a bunch of those and if we run enough data through it, it works. That's how deep the learning works. Where's the data from? So, this is really important to me. All the data is posted on hugging face. We have a bunch of recordings that we started doing ourselves who are working on this model. We are also trying to get to a synthetically generated data pipeline that works really well because theory is if we scale up with synthetically generated data that would be great. So we're going to try dia as data generator for this.

54:59 Interesting. at the moment anyway we really do need human data. So we put up a bunch of games at turntraining.pipcat.ai I think. And you can go and you can play those games and on the way in you click I'm cool with you guys recording my voice for open data sets. And then we'll keep the recordings. We'll clean them up. We'll post them to Hogging Face. And if your voice is in those data sets, every AI agent you talk to is going to work better for you than for your friends.

55:29 Play the games. That's a great incentive, folks. If you go play the daily games and train your voice, it will work better for you because you'll be represented well in the data set. That's actually pretty cool. Can we do a demo? Let's try to do a demo. I think there's a demo. You sent me a link, right? Yeah, it's a Julie. Yeah, let me go get the demo link super quick. I want to try this out and specifically you should tell me which type of stuff I should try, but like the so part I think is very important. Yeah, stuff like that. Stuff that feels like a natural pause. It's pretty cool. Definitely 100% when I give these models to my kids before semantic vad, they were absolutely not able to inter interact with these and list really helps. So I think this is like a very important effort on the open source. I wanted to highlight this and let you talk about this on stage here. Anything else that I missed that you want to shout? Oh yeah, of course. There's also a course coming up that I want you to tell about folks because we're also getting but anything else on the smart turn part that you want to tell folks? No, just coming out with a work on small audio models add data sets for other languages like I think it's a really fun project. Yeah.

56:34 And obviously for folks who come out and contribute voice to your data sets, other languages should be represented as well. Also, we call on the community to go and play with them. We'll add all those links in the show notes for folks who are listening. I think that this is it. Quinn, I wanted to give you the stage to talk about the great work you're doing in the community about educating as well. You gave me a book that I still have, but also there's now a course that you and our friend of the pod, patron of the pod if you will, Swix, run together. Would you tell us about the course super quick? Yeah, we wrote a really long guide to voice AI, kind of a fundamentals with a bunch of open source code samples. and that got a lot of really positive feedback.

57:12 So, we thought we'd basically be the guide as a course and a community and invite everybody to come hang out. experiment with voice AI in general. If you're building commercial voice AI stuff, show everybody what you're doing. Learn from lots of experts like Weights and Biases folks about all the best tools that people are using for voice AI. Now, I wanted to model it after the amazing LLM course that Haml Hussein and Dan Becker did last year that turned into this like great community. So, it's on Maven. I know the Weights and Biases folks are going to participate. A bunch of other people building voice models and voice tools are are going to participate as company sponsors. So, check it out. It's on the Maven homepage. Amazing. Quinn, thank you so much for coming up. I think the work for open source voice stuff that you guys are doing whether it's pipecat whether it's smart turn whether it's the education part is really amazing I really appreciate you being here as a voice expert as well telling us like about the latest and greatest with that I think folks I think we've done our chapter of open source and I think we will move on thank you so much for joining I will just shout out that we have a new friend here joining us on the stage we're now a lot of people Dex welcome Dex from human layer Dexter hoti What's up folks? How y'all doing? Great.

58:26 Coming through loud and clear. I think we will now move to a conversation with Dex about the 12 parts of agents because I think those are interconnected because we just talked about voice stuff for Quinn and smart detection and I think a lot of people are using voice to interact with agents and so I think that now it's a little bit of a time to talk about agents and then we're going to do open big companies stuff which will include GBT image generation. Dexi, it's been a while since you've been on the show. I believe a few months. Maybe do a quick reminder for folks of who you are and and what you do in life and then we'll talk about some of the aspects of building robust agents that you've outlined.

59:04 Yeah, absolutely. Super dope to be here. We've been hard at work. So, I'm the founder of a company called Human Layer. We build SDKs that help folks build AI agents that feel more like digital humans. So taking them out of these chat GPT style interfaces and deploying them into Slack or email or wherever the potential users of those agents already are. And what that led us down was a long path of trying to figure out what makes a good agent and of all the agents that are actually in production and making into customer hands what what is actually working in production.

59:39 Yep. And I guess tell us what is actually working. I think look I wanted to invite you as well because and maybe we don't want to touch this maybe we want to sidestep but there is some drama in the agent world and how these frameworks are happening together openi release something and then there's a response from her from lenith and lang chain and then there's responses to the response there's a lot of stuff happening in the agent world there's a lot of frameworks obviously for agents between open releasing there's google has google agent SDK there's a lot and I'm assuming with a lot there's not a lot of like standardization around like how agents should happen. The standardization that we do know that's happening is great around like the tool calling for example with MCP. There's now new stuff that we chatted last week Todd Segel from Google about A28 protocols or how agents will like talk to each other. Would love by the way to hear your thoughts about A2A while you're here as well. But there's not a lot of standardization about the actual kind of frameworks. tons of frameworks and then open did like a table and now Harrison came out with his own table like which of the frameworks does what imperative stuff human in the loop human on the loop dude I didn't even know the human on the loop is a thing would love for you to explain to me what human on the loop is but also would love to just get to hear from you a few of the pillars of after like many people tried them many people that you worked with little agents what are the important things what are like some of the interesting stuff yeah and I will preface this This is it's very easy to read 12 factor agents and to read it as a like all the frameworks are bad and you should never use them. That is absolutely not what I'm here to say. And if anything, it's really more hey folks who are building frameworks. This is what people need from frameworks and this is why they can't use today's frameworks because you don't have this.

61:25 And so a lot of people I talked to a lot of founders doing impressive stuff with agents and almost every single one of them said I just write the thing from scratch. And so this was my goal of trying to figure out what are the things that actually make because they are still doing things that feel agentish. But the the big takeaway was like and I think we've all seen this hacking ourselves is like when you take this here's your prompt and here's your bag of tools and go figure it out. The reliability on that for anything more than five to 10 steps starts to drop off pretty quickly. Yeah. And this is just compounding XP. Even if you're 92% accurate, if you roll the dice on that 10 times in a row, now you're down to what, like 40%. I don't know how the math plays out, but the point is like the longer these things are working and spinning, the less high quality that the more it just becomes completely unreliable and like definitely can't go into customerf facing software. This is model related by the way versus how much of this is just like the nature of LM's getting lost in the loop.

62:22 I think the idea is like a lot of people watch this and there's we even go into this in like chapter 10 which is like what happens as the models get smarter because some there's a lot of different pieces of advice. Yeah. If you go to small focused agents and there's a gift down there if you scroll down there's a gift. It's as the models get smarter. Basically what will happen is you'll take these big deterministic workflows and what works today is putting these really small micro agents in to handle small parts of those workflows. As the models get smarter, you may be able to grow them. As your prompts get better, as you get way better at tool prompting, and eventually maybe you have this large agent that can handle a hundredstep or even thousandstep workflow, but we're not there today. And I think it was the notebook LM folks said this on the Google podcast which is like the only way to or sorry on the Swix podcast on latent space was like the only way to create really magical and like impressive experiences and AI is to find something that is right at the edge of the model's capability and get it right consistently. Mhm. And so whether the models keep getting smarter, there will always be a core set of techniques that let you if you need to open the box and go under the hood and get as much performance as possible. This is a guide to some of the principles that will help you do that.

63:40 I hear that. I have a few more questions, Dex, but I want you to run through the the the 12 factors. Maybe we have them on screen, but like for folks who are listening, did you outline them by level of importance? Did you do just this is when you build this progressively you will add these layers like how are these like 12 factors created let's say totally I think the order is somewhat random I think the first couple are like the easiest to rock on their own some of the later ones build on top of previous ones so I can quickly run through so you have number factor one is like natural language to tool calls this is the magic of LLMs and if you really embrace the fact that an LLM calling a tool is just turning natural language into JSON that then we can do something structured and deterministic with whether it's like hey send this send an email to my assistant and ask them to do this that turns into a call to a email API like resent or something with text in it and things like that or if it's yeah so that's number one number two is on your prompts I think any framework that's going to hide your prompt and not give you access to the core to tokens and templating is going to maybe there's a lot of frameworks out there that you put in a couple sentences and they turn turn it into a banger prompt. Like you would have to go to prompt school for months to get this. People are doing like this weird thing where there are no assistant tokens. It's all user messages and you're asking the model to autocomplete the user message instead and all these crazy things you used to be able to do.

65:04 And that's great, but at the end of the day, if you want maximum performance, you're going to want to own every single token that goes into that LLM because they are totally safe. That's literally the only thing that affects the quality of the output is the quality of your inputs. Yep. And that's rolls right into owning your context window too, right? You can use this like user assistant tool, these scrollbased JSON messages. But if you actually look at that, like if you look at the tokens, if you click into this, there's a visualization there. If you scroll to the bottom of like just JSON versus I know a lot of people are using like XML style stuff for their prompts. Scroll down a little bit more. We have this kind of for folks on the stream that on the who are listening there's a picture here of basically like a comparison between the same prompt context window in JSON is 87 tokens and if you do it in XML it's 57 tokens and I don't know which one of these is better but the point is if you are not exploring every single possible option you are not going to find what's optimal and so you shouldn't outsource or lock yourself into one way of thinking about how do you tell a model or an agent here's what's happened so far what's the next I I want to touch on this like super quick because a context windows as we here on Thursday I've been doing this for a few years from we talked about excitement about 4K to 8K multiple times and now GP 4.1 is 1 million tokens in the context window which is just like mind-blowing and I think this leads me to the next point Dex the industry moves so fast with just like three exponential curves around everything hardware moves fast like the models the algorithmic they release like even open source moves incredibly fast.

66:39 How I guess when you outline something like this for folks how hard it is to keep up with the latest trends and expecting that like some of these things because the thing even with the smaller models that you outlined let's have smaller agent do specific things but then as time progresses like maybe the model will be able to do like more and more somebody in the comments Sean called out the AI digest graph. Did you see that one with the doubling? I should bring it up while I talk with doubling the time that like the models can execute tasks just like plotting out what they thought the curve would look like and then like adding the GPD 4.1 and just seeing an absolute like banger of an exponential growth. So I guess with that said with the speed that we're moving with the fact that there's every framework that you tried there's like a new one tomorrow. How are you looking at outlining suggestions for folks where these suggestions may may get a little bit different when context windows like a 1 million token unexpectedly? I can say like more generally the goal here is less about like specific model techniques. The goal is more like how can you set yourself up such that if a new model technique comes out, a new prompting technique, a new thing like MCP comes out that you can quickly integrate it and you can push it to the limits of its performance. But for example, for the for the context window one, like context windows keep getting bigger, but I think everyone says you can put two million tokens into Gemini, but you'll get better results if you only put like a couple hundred thousand.

68:05 Yeah. like it for no matter how good the model is and how many tokens it can handle. Again, maybe in the next year we'll get a model that can have really good attention across the entire context window. But that's cool. We should bet on that and we should be ready for that and then our programs get simpler. But until that happens, you want to know how to get good performance in a world where small focus context windows are way better. There's also the cost thing.

68:29 Even though intelligence trends near to zero, people still pay for tokens. and as Mazi before said here before index we had Mazanahi who joined the open-source open AI kind of discussion and he told us that the folks there did not want reasoning in thinking models in open source necessarily specifically because they would like to turn this off specifically potentially because like it takes more time and tokens cost as well so yeah there's there's other considerations just like performance on the on the context so I agree with you I just wanted to highlight this super quick shout out to Sean to reminded me I wanted to show this is the graph that we've talked out length of tasks AI can do is increasing and this is from AI dig digest and the data sources met from 2025 and so you can see the plots that they had before where they plotted out GP2 GP3 GT3.5 and 2022 GPT4 2023 and they plotted out and then openAI came out with the reasoning models and the reasoning models obviously correct themselves on the fly think about this and then you can see the plot like absolutely just go absolutely bonkers and exponential here with 40 mini and 03 with they're specifying that like the length of tasks that these models can do are now measured in more than minutes and now like in hours and then they kind of speculate going forward that if you extrapolate this then they will go like for days as well. What are your thoughts about this? Yeah. Is task in this case like a single prompt turn or is this like a multi-turn kind of agentic thing?

69:53 Very good question that I of the hand don't know the answer to because if the tasks are taking longer and it's yeah because when I use 40 mini it thinks really fast and because you can get 40 to think and do chain of thought and all of this use for mini and it thinks for a bit and then you switch to 03 yeah it is going to take three to five times as long to do the same amount of work but so I'm guessing that this is like tasks in terms of like single turn prompt and it's hey just go think about this thing for a really long time and then give me a good answer but I'm curious ious what which one is which one is we'll send this over show notes as well because I I don't want to move on a little bit but like they discover this trend this was discovered by researchers at matter they took the most capable I'm reading off the page as we speak I don't know this off hand they look at the most capable agents from 2019 to 2025 and tested them from about 200 tasks mostly coding with some on general reasoning compared the agent success rate to the length of each task and then I love this like plug going forward I want to though we'll add this to the show notes I want to get back to 12 factors I guess we talked about the future where models going to improve. I did mention reasoning and I felt like a respond from you. Would love to hear your kind of take on kind of the reasoning versus non-reasoning especially for agents stuff because if you think about this just holistically some of the frameworking is built in order to make make consistent thinking properly execute and some of the frameworking is know adding memory as well and like doing execution reasoning after test compute takes over some of that to an extent. If you think like holistically is okay these models then now can correct themselves. So previously if we had to build this into an agent now like the self-correctness comes almost built in would love to hear your thoughts about whether reasoning models are better for agents or not whether like different approaches needed what do you see from like this whole thing yeah I have not had the chance to put this kind of thing through its paces I know there's all these hacks you can do with injecting a buttweight token and messing with the samplings and things like this to get the reasoning stuff to be more performant. I've used a couple tools that are using 03 and 04 mini in concert to do like coding agent style stuff and things like this. The only thing I found is when you dropped in like 01 or 03 mini for an agent is that all of the tool call outputs that had text fields in them become much longer and more verbose which is what we know from these models. Even if you can get them to call tools or output JSON reliably things that go into that JSON are going to be more reasoning modely.

72:19 So I haven't explored them too much. I'm still using like mostly the older quad models for our agents. All right, I think that I think I think that's pretty much it on our on the Anything that I missed like we we'll obviously send the 12 factor agents and it absolutely blew up. Anything that I missed that I didn't want to didn't ask you yet that you wanted to convey here? No, there's a bunch more stuff on there. We did a webinar on Tuesday where we did a 90-minute deep dive with a bunch of live coding. I can send you a link to for that to send out as well if people want to see how to build one of these 12 factor agents from scratch. We didn't get through the whole thing in 90 minutes, but there's a deep dive on the repo there too. But that's very cool and would love people to check it out. And yeah, more to come. 100%. Dex, thank you so much for coming up. One last thing that I will ask you while I've got you here, like we did talk about the future as well and we did briefly mention the drama a little bit in the engine world and like frameworks world. One last thing that I did didn't want to like have your opinion on as well is that last week we chatted with Todd Segel the guy who works on the A2A protocol at Google. Obviously MCP we've been covering like a bunch would love to hear your thoughts about A2A specifically how you as a as a agent builder and as somebody who provides specific services for agents to like humans interact with them. How do you see the standardization attempt from Google? would love to hear like Ron filter thoughts about this protocol and the the standardization effort from big companies. Yeah. So, a couple things. One, I think we're in a protocol war. I think if you win with a protocol, then you have control over how the market works and how it moves. I think Anthropic killed it with MCP.

73:52 People have tons of complaints about MCP and all the problems with it. And it's, yeah, it's probably still going to win because it picked up really fast. And we've seen this in other tech. This is like Helm versus Customize all over again for my Kubernetes people. But I haven't gone super deep on A2A is the answer. My first instinct was like why do we need a new protocol? Like why can't an agent delegation just be a tool call by MCP or literally anything else because most of the most of these agents are going to end up being APIs in one way or another. What's wrong with REST?

74:18 Which is the same thing I said when MCP comes out. And again, walk on that one back a little bit. The other thing I'm really interested in is actually like what I will probably call like H2A like human to agent. I haven't written this down anywhere yet, but the idea of there is room for a protocol where I'm using any kind of client software, whether it's Slack or a task manager or whatever it is. And like how do we f information back and forth from a human to an agent?

74:44 And I'm wondering if agent to agent is going to be like that as well. There's all this weird like interruption and turn detection stuff when there are humans involved. And I don't know if agents are going to need that stuff. So that's my rambling thoughts as requested. Hear you. One thing that did resonate with what you said is that specific agents dedicated to one specific like tasks thing that you can test through. That's the bread and butter of what A2A is built on that your generous agent that can send you will spawn out these like task and find the agents that are specific for this and then there will be like an registry. So maybe maybe we are going to towards a generalist like model that can do everything and like all of the parts but maybe if we're not we're going towards like a system and in that system via this protocol your agent would be able to grab other things. Dex, thank you so much for coming. It's great to have agent experts here on stage. We'd love to have you back with some more news about some recap of the agent industry.

75:35 Absolutely. Because this is where the world is going. Quinn before said it's going to be the year of voice. Everybody wants to talk about the year of agents in 2025. This is the highlight as well, folks. Dex from Human Layer. You guys should definitely check out the 12 factors. We'll add this to the show notes as well. Thank you, Dex. We'll move on to the big companies and APIs because there's stuff to talk about there. Super dope. Great hanging out, guys. Thank you. All right, folks. The next on our agenda is the fact there was not a lot of big news from the companies. And I think the main one that we need to talk about is the image generation in API. Let's take this one.

76:08 Let's just open this up. Again, I'm lacking a transition for big companies, but no new model releases from big companies, no new model updates. As far as I saw, there is a little bit of a price war happening where the tiers have opened up. We've seen this the GBT40 and 03 all the stuff. OpenAI basically gave like people on the not on the pro tier on the pro tier is supposedly unlimited. on the plus tier. OpenAI started giving I think 2x the 03 access 2x like some of the stuff celebrated that summer posted thank you more messages you can send so you can say thanks in a message but yeah I responded to this post and said thanksg because it was already at my limit for all three and I was happy to see it reset midweek but I think that absolutely the main news from open this week that like many people have been waiting for is the fact that we now have a name for this image generation. They call this GPT image one. Also, I think some refer to it as image gen because it was really annoying so far to say the new image capabilities within CHP. I used to sometimes say image as well and so these capabilities have now been released in API and not only they have been released, there's a few things that you can do that you cannot do on CHP. So I don't know folks you already played with this but I will shout out that in order to play with this your organization in openi has to be verified and I went through this verification flow and this felt like TSA pre or whatever like I had to scan my biometric you have to approve with persona so I know that like many folks would not want to do this I think I get why they're doing this these models are super realistic they can generate like realistic stuff they want to maybe fingerprint who did what like it makes sense when it's on chipt it's in in their control when it's an API. People can build all kinds of tools. So I think it makes sense. But basically now after you verify yourself, if you go to the console of JGBT, you go to images, you can just now generate images with with the with the image gen one. And I think that the cool thing first of all they have like a bunch of pre- ready prompts. So if you don't know what you're doing, they have prompts like speaker ad for example which they even send like a a specific speaker and they say generate an ad for home brutal speaker add this product in minimalist interior inspired by Japanese aesthetic.

78:27 Let's send one. Let's see. But also they have a bunch of other stuff to train you and give you some examples of what prompting for this model is. But also you can control a few things. You control quality and this is a very interesting one that unlike Yeah, I would love to hear your thoughts. Unlike the the other models I haven't seen that. This is really nice. The playground ser unlike other models like flux for example where the the quality is basically based on which flux you use, right? There's a flux there, flux probe, etc. You basically choose the model and okay, it will output the same image etc. This is a reasoning model. This is GPT that does the generation. They just hide it behind this like image thing. So basically it's like how long to think and so you can literally see in their API. Here's our ad by the way that we generated with the speaker. It looks pretty cool in a very minimalistic design with AG. It's pretty cool. It did all of the It follows Thursday. You should put Thursday on it.

79:27 Change this to and make the speaker a little bit more on different qualities and see the same thing in in different qualities just to see the difference. Max it out, 24. Yeah. So you can control the quality and the quality controls how long it thinks. I think this is also very interesting because if you're building any type of UI with this and OpenAI launched it and then they already have like 10 partners built in like playground and FAL already has this like a bunch of other folks already have this the controllable length like deter you pay for this so you can't know in advance is very interesting but also how long you wait for an image generation that's also very interesting. The other interesting thing that will go very well for many folks who are generating images is the fact that you can generate up to 10 at once. And that's super cool. And the last thing maybe I'll say on this while our image generates is they have three APIs. They have generations, they have edits, and they have variations.

80:22 Variations not existing yet for only but I'm assuming at some point this will come as well. Generation generate images from scratch. Edits modify images like a new prompt either partially or entirely. And they support masking which is great because as we talked about the detect anything thing you can mask out whatever stuff you just click and then send this mask as well. All right. So let's take a look at super quick at what we have here and then we can move on as well. Okay. So this is the regular speaker. The image of the speaker was uploaded and it followed all the prompts and then N prompted it to be more unhinged. So here's our unhinged speaker. I actually I love this. It's pretty cool. I literally I said change this image to Thursday edited news please and make the speaker a bit more unhinged and the text is perfect. So I think the text part is very important like the text is really good at at this one thing that's not good I wanted to call out and maybe ask folks from OpenAI to fix this is faces. This kind of looks like me and this took like multiple iterations. When I upload anything with a face I don't know it feels like intentional maybe I or not intentional purpose. I think they've actually done that on purpose. Everything else in the mimage. So you guys remember obviously the giblification era of two weeks ago whenever that was released. Was that two weeks? Time flies ancient. It's it feels ancient. Many people are still giblifying. That makes you into a Gibly character and kind of all your facial features get disappeared anyway. So like the main ones remain like the beard, the hair, the eyes maybe. But if you ask for more of a realistic thing like this, you can see that and this is like the best one out of the ones that I've tried. You can see that it's it's me, but not entirely. And I know for a fact that they can make me perfectly me. Like I know for a fact and it does feel a little bit on purpose. That's not good.

82:10 Yeah, that's not good on it at all. This is Yeah, this is like one of 15 that I've tried. The rest of them were just completely not related to me at all. and I had to say, "Hey, replace this with me." And this is like the closest one and this could be somebody else. There's like some aspects of Alex here, but not like all of them. So, it's very interesting that like images, especially realistic face images do not yet go through specifically because well, we verified for this API. Open a knows who am I like they they know all of this.

82:38 And also in the in the announcement of this image genen or GPT image one they said that they're removing some of the face generation kind of concerns that they had before because in Deli for example that would refuse to generate anyone and this would generate some people but I don't know the faces don't look exactly the same. Maybe just do generate a gibli and then do a reverse jibly butverse I'm sure that people style. Yeah, I'm sure that people will come up with some prompts or something. I think honestly a pipeline. I've seen I I've tweeted about this to folks and then folks said that this is one of the biggest issues with JGBT and what folks do is they add another model that kind of aligns this to your face better with some other models and I think a pipeline of this will actually work. I wanted to use this as a thumbnail creator and I think I'll still post this as a thumbnail just does not look good. Any other things that I missed there for the image generation? I think it's very exciting. I think we're we're about to see multiple products go crazy. we can talk about it seems that it is already being integrated by Adobe, Figma, Vix, Air Table, GMA, Hen Opus Clip and so on. So a lot where they are using avatars or having image modification inside they can now use the API and we have seen how good it works.

84:00 So we would see it everywhere. I already started using it. It's is great if you have customer lists or customer cards and stuff and you just wanted to guess an an avatar for each you can just jibly every customer and then so you don't need to find their actual picture or ask for permission and stuff. You just guess and then you can just keep track of stuff. This will be very useful business feature. I mean 100%. We've been waiting for this and I'm pretty sure that this is going to be super cool to have a cookbook. One last thing maybe it supports transparency. So if you are gener like logos or whatever, it supports transparency. It's pretty cool.

84:35 I think this is the main news in open labs. We can talk about Gro super quick. I do want to mention the Gro API is now available. We've been waiting for this. They're trying to go for pricing. We should definitely mention this as well. Gro 3 mini is going at 30 cents for 1 million tokens and then 50 cents for out. This seems very interesting. Like this attempt seems very interesting. Everything else the difference between output tokens and input tokens are higher. This is like 30% 30 again 30 cents per 1 million input tokens and 50 cents per 1 million output tokens. GR3 mini gets 93% on a IME and they're comparing themselves to Gemini 2.5 flash thinking and 04 mini high and deepsear one. So like the smaller models 93% on aime is like impressive. I know that many people who try Grock are keeping using grock. Like I specifically I just wanted to to ask you guys like I use Grock for Twitter stuff. I go on Grock and like hey summary I do sometimes use gra for deep research and deeper research a feature that they launched but I got to wonder that now it's an API if any folks are trying this on like windsurf is good for coding live life live codebench for example let's take a look at life codebench here. So life codebench measuring the ability of code generation of models. They say the groy mini has better life codebench than Gemini 2.5 thinking and gets 04 mini high is like the best one here but it's more expensive significantly. so they're showing off some very impressive scores here including 11 labs of humanities last exam 11%. And this is for the grave me anything. Folks do have you tried rock at all via API in the tools?

86:11 Anybody tried this? Anybody in the comments as well? Please let us know because I keep seeing people say that the models are good but I haven't yet gotten to a point where I like to keep using them. Try to grow API. It just wouldn't give you the sources. So that's what I found. I I wanted the ra sources from it because that's super but they don't provide that and I think that's like the most valuable thing about gro.

86:35 Wait, sorry I'm confused. They don't provide sources. Listen, they don't provide resources of the when you prompt for something usually Grog does give you sources either from Twitter or from from news. That's not on the API. I have checked. Oh yeah. So API only gives you access to basically the model the kind of the the X connection and the searches like whatever that's not in the API. They said that maybe this will come but we haven't seen this yet as well. Yam, what's your take? I just want to say it's a good model for coding. Seriously, we we use it multiple times. when stuff doesn't work in code, I've tried to give it to every single one of them and four or five tabs. Just tried it with everyone and sometimes it's rock. It's a really good model for code. So, if you didn't try it yet, do try it out. And also, I really like the thinking tokens. Like it's really like the thinking tokens are available in rock. They they really do.

87:35 If pipeline dims, it feels like where you like to read what it thinks about. Yeah. But yeah, I think that Yeah. So, I agree about the the reasoning tokens as well. They had this confusion when they released through Gro whether or not they're going to hide the thinking tokens and then I think like Elon was confused. They actually show the actual reasoning pool. Grock 3 and Gro 3 fast. They have like fast versions of them. They the fast versions cost more. very interestingly like they're not they're fast but not I guess they're prioritizing GPUs they actually the pricing structure here is very interesting to me very interesting to me check out like this difference on Gro 3 mini beta the input 1 million tokens is 30 cents output is 50 cents so barely a difference between the input and output if you go out to Gro mini fast beta that the input is 60 cents so double of the regular one but the output is $4 so almost eight times more what kind of pricing games are they playing here with? It's very interesting. And on the Gro 3 fast beta, not the mini one, the input is $5 and the output is $25 for 1 million tokens or fast tokens.

88:41 That's very interesting. I guess when folks are choosing which API to use, they're playing the game of fast, cheap, and smart, right? Like you're playing this is the game. This is the triangle you're playing fast, cheap, and smart. Because then you choose between the two models and the fast one. I guess that this is their attempt to disrupt the game and and have some folks prefer something versus the other. I think that's it on the Gro API. The last thing that we should mention about Grock is that the app that many people use that has unhinged conversations. We I think we've played a few of those unhinged conversations here. The app now has a live video view in addition to multilinguality as well. So it it speaks multiple other languages. Wolf, I don't know if they speak German, but I I saw Portuguese and Spanish, etc. Not on the Android app, right? Apple is it? On Apple. Yeah. Yeah. They're releasing it on Gro. They have a tier that's called something super Grock or something on Android, but like on iOS, they release like multiple things. And yeah, on iOS app, you now have the ability to show Gro what you're looking at or have it look at you and roast you, which is, I will say, very funny. If you're up for this, if you use the unhinged modes and you like point your camera to it and hey, here's me, roast me. It will absolutely do do that. That's really fun. I think that's like guitar voice like the events voice have have video like is that like live video? Yeah, real live stream video that works. I'm gonna boil it.

90:06 It's interesting. Folks, we have two guests here. We have the space for like deep dive into agents, into voice agents as well. There's a few things that I really want to cover super quick before we get to all three responses. So, in this week's buzz, I don't have a full category. I just want to tell you that we are coming to San Francisco in May 17th and 18th to do a hackathon. If you're coming for Google IO, by the way, this is like the weekend before. So, if you're coming to Google IO, welcome to come hack with us in San Francisco again, May 17th, 18th. We call this Weave Hacks. And we'll announce the sponsor soon. And the prize is going to be super dope. But if you're flying into Google IO as well, if you're listening to the show, would love to see you. Come say hi. Give me a high five. As always, the hackathons inside the weights and biases office. Super cool. And then our conference called fully connected, fully connected.com, weights and biases conference for everybody who uses weights and biases models. and we've the whole company is going to be there.

90:57 could be our last one given the recent news about acquisition that is dedicated towards we'll see maybe not but if hype is what brings you there then maybe our last one two-day conference in June 18th I'll send the link out in this week's buzz I really want to show this one sendi out of nowhere released a 24 billion diffusion model auto reggressive diffusion model that can generate long streaming ready videos with text control we really have to watch a video of this because like It came out of nowhere.

91:28 Great name for a company, by the way. I have to say that's some top tier name right there. Yeah, the naming is great. But the video that we're playing right now of this Maggi model has this character, consistent character throughout all these image generations. And I think you guys, those who played with video models, the consistency is like one of the hardest things that you always try to do. You like generate multiple images, try to get consistency.

91:51 They I don't believe they released this model. folks in the comments, please correct me if I'm wrong, but like they mentioned 24B, so I'm assuming that it's about to get released. They did release a technical review and we keep showing video models. Some of them are cherrypicked, some of the images, and we keep getting blown away. This one is absolutely crushing it. I wish that would be able Holy I think they solved it the specific this was the main thing why they couldn't make move these because you couldn't get a consistent face and people will notice that people are not going to care so much about the visual effects as much getting all but they will definitely notice the face and they might have just solved the biggest problem towards AI movies with this well from you want to add about frame pack for just a second I think we mentioned frame pack maybe but in So I I just want to finish up with Sand AI infinite extension. This is I think the highlight of of their kind of whole stick. Infinite extensions with like consistent storytelling. So you can actually I think they're doing like last frame, first frame some stuff. Let me see if I can show you the actual UI. I think I have it logged in. Give me a second. Yeah, I'll just log in super quick. The UI is like this type and then you generate something. I don't have assets. Yeah, it's going to take us a minute to generate, but basically the UI is like the extension UI that we saw.

93:12 Let me show you maybe in the video super quick about their UI. The UI is like allowing you to extend to scenes and you can build like an actual like long video because of it. And I think that like super cool. The technical report that they released is very interesting for folks as as well. They like implemented a bunch of very interesting things that many people are looking at like getting very excited. I'm not going to go through all of the Maggie attention forward and backward timelines, but basically they have they they scroll through timelines as well to give this model attention about this is how I'm assuming they get consistency. And yeah, I'm waiting for this model to drop because they did mention 24B. They did release the technical report. Really hoping this model will drop at some point. Right now there's GitHub, but I don't believe that the model is on hugging face. There's Sami in hugging face. Yeah. Holy crap. Face here.

93:59 Release the model weights and inference code. Let's go on this. Yeah. No, that's the legs. whoa. Okay. And download this right now. Yeah. Back it up. Apache 2 licensed model weights are out. It looks like it was like 10 20 gigs. So, it's probably about like a 10B or so. Yeah. No, they said 24B. 24B is what they said on the release. They have an algorithm. Super cool. I just wanted to highlight this as well. You Wolf from you mentioned this one which is also cool. A frame pack from I think it's a research demo. Is that the one with the long? Yeah, that goes up to 120 seconds and it can run on Nvidia GPU with just 6 GB of VRAM. It's already available for local download and use. And yeah, you can make longer videos with it. Oh, it's been a great week for video AI again.

94:48 Yeah, absolutely. I keep saying that this year is more of a video AI year than anything else. The last thing that I had for you to know, it's Jan. This year is the This year is the year of AI. I think folks, we're at the end here. 3D updated to 2.5 and a significant improvement on the 3D part. I'll add this to show notes. We don't have time to show this. With this, I will just say a huge huge thank you as we're like coming up on the two hours here that even in a slow week, we have a full show, but a a huge thank you for Dexory and Quinn Quindla Kramer for joining the show today talking about multiple things that they released. As I said, one of the coolest things we get to do here is to talk with researchers and authors of the things that they blew up. shout out to Tony. Sorry. what's his name? Toby, I believe, from Dia, the guy who released with his friend, like one of the mindblowing, most things in the we've seen like at least this week.

95:41 Dia from Nari Labs as well, the Korean folks. Huge thank you for our hosts here, LDJ, Yam, Wolf, and Niston. My name is Alex with W and Bes obviously. And the most important thing for everybody who tunes in and listens and participates in the show and sends links and re-shares and subscribes. If you missed any part of the show is available as a podcast everywhere where you get your podcast, but also as a newsletter with all the links on Thursday. That's Thursday. Just a simple URL. Just follow any of us on X and you see the link as well. Thank you so much.

96:12 We'll see you here next week. I'm assuming next week is going to be big with some major releases as always. We'll see you here next week. Bye-bye everyone. Byebye.