ThursdAI Oct 8 - OpenAI drops 722 math papers, Haiku 5.5, GPT-6 for everyone & more

0:00 Peter Gostev: And the point is that I was spending literally hundreds of billions of tokens. I'm not kidding. I probably spent, I don't know, 3, 400 billion. I know I need to go back and check. And I literally had my Linux box running, for weeks and weeks, and it was filling up with all the solutions and so on. But the T- TLDIs, that was complete waste of time. I did not discover a single bloody thing. So these guys did it in like 3 hours. I couldn't do it in months with Astra and 5.6 and Fable and others.

0:30 LDJ: Yeah, so so Fable and Astra had put together a set of 500 most important open math problems in the world, and 92 of those just got solved on Tuesday by OpenAI. Peter Gostev: Yeah, so we, we have some news, we raised 200 million. That's a lot of money, at 3.1 billion valuation. Alex Volkov: So basically, his personal CFO just went public on Slack and told everybody his bank account.

1:01 Maxime Labonne: Yes, decision models do not output any tokens. They just take a decision from a predefined set of answers. Nisten Tahiraj: So there are 100,000 agents going on in this city. I don't know how it's still holding together, but, yeah, it's a full emulation of the city of Toronto.

2:48 LDJ: Okay, so it's amazing, past 7 days, even just the past couple days has been crazy with the, the math announcements, but yeah. Alex Volkov: Tuesday was just, just awful. I, I need to go and pull up my tweet over there because, I, you know, I don't usually do this like a quick recaps, but I just had to. It was quite insane. LDJ: A couple of American open models too, of Reflection, which has kind of been in semi stealth for a while. They just announced their first model, which they plan on releasing open weights soon too, although it's not quite open weights yet.

3:25 Alex Volkov: I think it was Beam, yeah? LDJ: Mhm, exactly. Alex Volkov: Yes, Reflection announced Beam. We're gonna mention this, later on the show. actually got access. so shout out to the Beam folks for getting us access, but I didn't have time to go and play with it yet, but, I- I just want to highlight this like one day in the middle of this week. OpenAI dropped 722 math manuscripts covering, I think, 5 Navier-Stokes worth problems. I think this was the calculation that that somebody, which is still, you know, we're gonna talk about this.

4:03 We're still trying to figure out what was this thing that OpenAI just brought to us, and and and mathematicians are trying to get grips with it. Meta, Stripe, and Shopify announced, this is a single day, this is a single Tuesday of this week, folks, all right? Meta, Stripe, and Shopify, Shopify announced the agent protocol or assistant protocol, whatever you want to call it. OpenAI finally gave us access to the Decisions API, which is their competitor to Jev And, Mistral came out with Mistral 4, with Lechonk.

4:36 And Google dropped Embeddings Gemma. Brad Adcock, the guy who does figure robots, got a new, assistant in the, in the arena of assistants, right? So there's now Hark Pro, which is, you know, a lot of people signed up for. And, Perplexity open weighted a OpenJev, decider, update. This is an update. They already dropped it before. Claude now does Google Docs. we at CoreWeave released RL rollouts. this is a single day. It was really hard to catch up with all this just this Tuesday, but obviously there has been more stuff happening.

5:14 Welcome Peter Gostev on the stage. Welcome to ThursdAI. there there's a huge, huge amount of stuff to cover, so I think, yeah, let's do a little bit of banter, folks. What is, what is the highlight? Let's start with LDJ. You were, you were here first. What is the highlight for you from this week, if you had to pick one? LDJ: Yeah, huh. I- I'm sure, I'm sure Nisten and Peter and others are going to probably mention the math part, so I'm going to say Reflection AI's Beam.

5:44 so it's pretty competitive in terms of... it- it's, it's definitely, I would say, second, third, fourth-ish place in open models when you look at raw benchmark scores, but it seems like it might even be just first place when you look at reasoning efficiency compared to the top models like GLM and others. Alex Volkov: Mm. LDJ: So we'll look into that more later, and yeah, that's, that might be my top one. Alex Volkov: Peter, how about you?

6:12 Peter Gostev: Oh yeah, the math stuff was something else. I know. Alex Volkov: It, it took over your timeline a lot, like I see you commenting on this. tell us, Peter Gostev: And it's- Alex Volkov: What's going on there? Peter Gostev: Yeah, so there's, there's been a kind of, I don't know, rumor, I guess they kind of pre-announced it that they have a bunch of maths problems solved. I don't know, was that a rumor or was that just announcement?

6:34 I can't remember what was publicly known or not, but LDJ: At first a rumor. Peter Gostev: Yeah, yeah, okay. So, so OpenAI, with their new Bell model, right, they solved the Navier-Stokes problem, the Millennium Prize problem, and then there was a lot of chatter about that they've, they're just sitting in a war chest of hundreds of problems solved, and, and now they released it, and they just did it pretty, in a pretty low-key way, not like a hype marketing video, with, exploding head emojis.

7:05 Alex Volkov: It was nothing. The blog post was just Peter Gostev: Yeah. Alex Volkov: Just a few lines of code, and the GitHub was just an insane amount of... you know my favorite, term for this now is slap grenade? When somebody throws at you like an output or, or, or a, or a pull request of Claude and it has like a bunch of lines, this is now a slap grenade. So Open, OpenAI basically like slap grenade, for people to just like shovel Peter Gostev: Yeah.

7:28 Alex Volkov: and try to figure out what this is. Peter Gostev: Yeah, maybe my personal reflection is that I didn't really talk about this much, but I was, with GPT-5.6 or GPT-6. I picked one problem, and I thought, let me just throw all of the tokens I can get my hands on and just see what I can do with it. And I was also trying with Fable and Opus and so on, but I was running the GPT models mostly just because were doing a lot of resets and things like that.

7:52 And, was one problem that I was trying with the Hardwing and Nelson, not sure how you pronounce exactly, it was like, graph coloring kind of problem. So it's easy to grasp, obviously impossibly hard to solve. And the point is that I was spending literally hundreds of billions of tokens. I'm not kidding. I probably spent, I don't know, 3, 400 billion. I know I need to go back and check. And I literally had my Linux box running, for weeks and weeks, and it was filling up with all the solutions and so on.

8:20 But the T- TLDIs, that was complete waste of time. I did not discover a single bloody thing. There was nothing, no use. was an interesting mathematical result, nothing, right? And then, these guys with the new greatest, model, they just, run it for 3 hours, said on average for 3 hours on ChatGPT Pro, which I think in practice what that means is that it's like a, some sub-agents, I think we don't know how many, but not like hundreds, like 10 maybe, or 4 or 10, something like that.

8:50 And it s- it didn't solve it completely, but it changed one of the bounds. The idea is that you can narrow the bounds, and I think before it was something like 5, 6, or 7, and it narrowed it down to 6 or 7, which is a big deal, right? It is a, a obviously a lot of context required, but it's a big deal, right? And I was speaking to Astra about m- my progress, my results, and Astra said, oh, we are two mathematical breakers away from what they did.

9:18 So these guys did it in like 3 hours. I couldn't do it in months with Astra and 5.6 and Fable and others. There's just nothing. Alex Volkov: There's there's a few rumors that I want to bring to your attention there, but I think I think it's time for, you know, for like the official start. folks who are tuning in, welcome to ThursdAI. one of the coolest things that we get to do is talk to you as well, so please drop in the comments, what was the highlight of this AI week to you?

9:44 Obviously, it's nearly impossible to cover everything. we're trying to make ThursdAI the most agentic forward show. We have an AI producer. producer, please say hi in the banner below, if when and and if and when you hear this. And we also have, obviously, a bunch of research happening throughout the the to make sure that we're not missing anything. this week, I think I had 48 topics to bring you of, you know, of importance, et cetera, 48 or or 50 or something.

10:16 I actually asked Claude to build me like a, l- l- like a Tinder for AI news where I swipe left and right if I think that this should make the show, and it will stack rank them based on, like, my preference and the stuff that we talk about, because it's just like impossible to bring you everything, but we're gonna try to bring you the most important news, so, once producer catches up, I think it's time for, for a cold open. Alrighty, folks, let's, let's do this.

10:45 Folks, this week OpenAI dropped 722 math problems, papers, written by a model nobody outside OpenAI has touched yet, and the mathematicians are split between thrilled and asking for receipts, and a lot of those came with receipts. Welcome to ThursdAI for October 8th. This is Alex Volkov. I'm an AI evangelist with CoreWeave, and with me, LDJ and Peter Gostef from Arena. This was an insane week. We say this often, but this was an insane fucking week.

11:15 after OpenAI's math manuscripts, 722 of them, to be, to be precise, some people call this a slob grenade. many mathematicians are split between this is the most exciting time ever, we're breaking through the boundaries of mathematics, and, we should... This is not how mathematics should happen. This is, this is kind of the split, and Peter, I think, has been monitoring this and would love to, to, to chat with you more. This would be our item number one, and that's just the opener, right?

11:47 We'll start the TLDR, obviously, with OpenAI. 722 math manuscripts from a model that nobody open- outside of OpenAI can use. somebody used Fable to, t- to put this release in terms of how many Navier-Stokes moments in one drop we got, and and I think it's around 5 or so orders of magnitude of Navier-Stokes, releases. If you guys remember, Navier-Stokes, a Millennium Prize problem, was solved by OpenAI just recently.

12:18 So OpenAI just dropped 722 of these, in- many including Lean, confirmations. We're gonna talk more in depth about this. next up is, Claude Haiku 5.5 is a 10 cents per million input tokens, and Anthropic say that it beats GPT-5, GPT-6 Luna on computer use and coding. GPT, remember, was really, really good at computer use. Haiku is- seems to be better with a 72% on OS World, and I think also significantly faster, I think, on artificial analysis.

12:52 Claude Haiku 5.5 is one of the fastest models right now, and it's incredibly cheap. Incredibly cheap. Haiku 4.5, I don't know if you noticed this, was a dollar per million tokens of input, and Haiku 5.5, which is significantly better, is 10 cents. Just j- j- just to give you a sense of how crazy the world we're living in. also a small thing that they released together with with Haiku, Sonnet cash reads, so the tokens, you know, after after you send them once, Sonnet cash reads are now halved by price a week after Sonnet 5.5 was released.

13:33 So that's also quite insane from this week. Let's see what else. OpenAI rolls out GPT-6 with intelligent UI to everyone. All of the free users on ChatGPT, all of them are getting GPT-6. It's not Luna, it's not Terra is dead, so no Terra anymore. It's not, it's not Soul, it's not Astra, it's just GPT-6. There's no name in this. so all the free users can d- can can get it today, but also the cool thing, the the intelligent UI is really, really cool.

14:04 They rebuilt ChatGPT on top of it, and you basically get buttons and switches and toggles and and graphs, et cetera, in your chat output very fast. It's really cool to see. it's like little working apps for your answer. There's also some threads from folks at OpenAI who say, the model was now trained to do it while it thinks of an answer, right? So it starts to build you, hey, this answer would probably get benefit from these and these switches, these and these things, and then it it still works in the background. So I think that's pretty cool.

14:42 Claude for Google Workspaces is finally here, and this is like a small thing, but it's a huge thing for many people. I'll give you an example. Our cloud producer every week, used to post a Google Doc, and then a new piece of news would come, and I would ask it, hey, can you append to the Google Doc? And it would start opening computers, et cetera. It wasn't native, so now that it's native in Google Docs, I think it's a, it's a very big deal, and it also asks me for every edit.

15:12 but okay, let's go to the open frontier as LDJ mentioned, we now have another US lab. Well, we knew about Reflection a while, but, and Reflection AI announced Beam. It's a 501 billion parameter open model trained from scratch in the US with Apache 2 weights coming this month. And shout out to Reflection for this, o- o- for this awesome, announcement. Many, many people got very excited. An open MOE that beats on efficiency over Roscore, so it's more efficient versus just like Benchmax. we can't wait to play with this when the, the weights drop.

15:50 Also, Mistral came back with leaning into le le le chaton fat. If you guys remember, this was the meme. they called this one le chonk. Mistral 4 is a trillion parameter with the weights coming at the end of October. This is now a pattern that I do want to cover on the show. Like, folks announce open, but they not release open. I- is a far, far way that we moved from Mistral announcing their, latest model with just a torrent link, because people do, do be announcing.

16:24 but Mistral large ties with GPT-6 Luna into official analysis, at a much higher cost per task. So that's a very interesting, very interesting thing. this is, this is kind of the comparison we have here, obviously for European folks. Speaking of European folks, we're missing Wolfram this week because Wolfram is on a well-deserved vacation, but he got super excited about Calibri. This is from Aleph Alpha, another European lab. If you guys remember, there's a, there's a European lab called Aleph Alpha.

16:53 Peter may, may, may know them as well. It's a German-English MoE with 78 billion parameters. Wolfram sent me his notes on this. It can run on a single H200 and also Apache 2. So shout out to Aleph Alpha for, for coming back with the model. I thought they stopped doing pre-training and post-training. It just like went with models. And for the very, very, let's say AI internals oriented folks, we used to cover embedding models.

17:25 We we had Bo Wang on on the show before he joined Perplexity, and he dropped one of these models, so shout out to Bo. two open embedding models dropped in one week. Google's embedding Gemma drops embedding two text, images, and video and audio embeddings in one space that, runs on the phone. Two embedding models this week, and Perplexity late interaction models search PDFs as images and are very fast with PPA- PPLX Embed V2 late interaction models. So a lot of, a lot of goodies this week.

17:58 as we move to agents and assistants and a bunch of stuff, this, I think, is a very important thing. so Meta and Sierra, and and Shopify and Stripe and Walmart all on board of this protocol for personal agent protocol. very interesting to see if this is the new kind of MCP, because MCP had its ups and downs, but now MCP is everywhere. Basically, MCP, like, took over the world because of these agents they need to talk to, to a bunch of apps.

18:29 I don't know if you guys caught this, we talked about Grokbot since launch. There's a whole drama happening between Dots creators, at OpenAI and and Grokbot, f- folks, specifically Potato. it's really funny, but, Grokbot has been supercharged this week. After Grok 4.7, the model released, it was muah muah muah. I ha- I think I have a button for this. This is our reaction to Grok 4.7, not only our reaction, big Ankh space, space Ankh Elon Musk decided, hey, you know what?

19:04 Grok the product needs to be more than just Grok the model, so Grok is now tapping into Opus 5.5. If you guys remember last week, I think everywhere else, I've said I want personal intelligence harness, like a proactive personal intelligence, but with Opus's like brain. So supposedly now that's what's happening inside Grokbot. I think it's very cool. supposedly also only because I, I haven't seen this in the logs yet, so that didn't arrive to me yet, so we'll see.

19:32 but also Iloma said it will use the best model for each task, including Claude Opus 5.5, Suno, and Midjourney, which folks got very surprised because Midjourney doesn't have an API. we'll talk about that. speaking of personal assistant, and also Grokbot, Shane Mack posted this, and I think it's a good warning for you to know, his Grokbot, talks to another Grokbot and, his personal finance details in the company Slack, of which he's the CEO. he had one, Grokbot in charge of his personal finance, giving them an update, and another connected to Slack, and for some reason they communicated and decided, hey, let's put the dirty laundry of our CEO in Slack in front of everybody to see, including all the statuses and et cetera.

20:19 So this was a very interesting warning sign for, personal agents connected to between work and and and home, and I think could be one of the reasons why they're switching to Opus, because Opus would never. Like, I just, I just know that Opus would never. this is a Grok thing. This is a Grok, like, intelligence thing to just like, oh, sure, let's, let's blast it, publicly. we gotta talk about Nous Research, folks, our favorite open source friends for the longest time. Nous Research, shout out to, just everyone there, all our friends, Technium, Karan, I keep forgetting all the names, they not only launched Hermes Index this week, which is a benchmark that says, hey, Claude Opus 5.5 is the best model for Hermes right now.

21:09 they also announced a fundraise evaluating Nous Research at more than 1 billion dollars. So shout out for our friends to making the unicorn status, fundraising, I think 90 million is the latest fund round. They also were on stage both at OpenAI Dev Day last week and at Microsoft event this week with a deep integration into new hardware. This is incredible. Since we told you about Hermes Agent, the length and the, the, the, the depth of penetration of Hermes Agent from Nous Research is just incredible.

21:43 So we, you know, we'll definitely cover the Hermes Index, but we'll definitely also celebrate the success of our friends of Nous Research. we've w- we've been covering Nous since before they were a company, since before when they were just like a ragtag of folks doing stuff on Discord. so shout out to Karan, Technium, Jeff and Dylan and everybody else at Nous for this amazing, amazing success. If you are a tinkerer and you like personal assistance on your hardware, Nat Friedman posted, open source Muse gadgets.

22:23 So now you can put your Muse everywhere in your home. You can put it in every gadget, in every, it's an ESP32 board, if you know what that is. It's like a tiny chip, that I used for a bunch of Halloween builds before that I talked about, and it's, it's a very cool thing that they're posting. A bunch of folks are porting their assistants into portable stuff. A new personal assistant emerges from the creator of the Figure Bots, Brad Atcock.

22:46 It's called Hark Pro, eh, also free, and the assistant that clicks for you. They are claiming the best computer use model. I haven't, got a link to the, to the verification of the best computer use model, but Hark Pro is also here, and folks in comments, if you use Hark, please let us know. for our this week's buzz corner that probably should be renamed because it's no longer just about weights and biases, CoreWeave opens up free GPU sandboxes. This was the highlight of the, the, the event last week for me.

23:16 last week on the show we had Dirk Filio, who is the PM for CoreWeave sandboxes, and he announced that, hey, you can now reach out to us at CoreWeave. We'll give you GPUs without a salesperson in the middle. I think it's huge. It's absolutely huge. We're advanced. free GPU sandboxes. I'm, I'm, I'm gonna say this again, super quick. If you scan this, you get free GPU power. Do if you want more, GPU while in testing, this is just gonna be free, which is, I think, is incredible. so this is in this week's buzz, and we also have RL rollouts. All right, I'm gonna read this block super quick before we get to the news because it's been, it's been long. Black Forest Labs launched, Flux 3, 4K images and layouts, up to 10 reference images.

24:05 Nano Banana 2.1 releases, which is very, very cheap, only 3 cents per, 1K image. It's pretty good. if you are looking at our, infographics right now, Nano Banana 2.1 is the creator of those infographics. They're pretty good. I'm very impressed. The consistency is there. you have to run it on high reasoning, medium reasoning. The default one is not that great at text. yeah, I don't know if you guys saw Tavis, Griffin video avatar fooled 48% of the people in a live video call. They thought they're speaking to a human.

24:43 This is quite honestly quite crazy. We're gonna play a clip of this, of our friend Quindla Kramer talking to Tavis avatar and saying that, you know, whatever Turing, tests for avatars we've passed, long passed. Reka AI, we've talked about this company, one model, every modality, released a 19 billion parameter model that understands and generates text, understands and generates text, images, video, and robot actions all in one network. That's very interesting. We'll see if we have time to talk about that.

25:13 and I think this is almost the, the, the last thing, but the, the rise of the JEVs, I think, is our last category. We're gonna talk about it with, Liquid AI's head of pre-training, Maxime Labonne, a friend of the pod. OpenAI launched Decisions API, and Cloudflare dropped, Clef, and Ansloth dropped a, a recipe for, for creating decision models, and Fastino dropped Glide. There's a bunch of decision models all around, basically, so we're gonna cover all that jazz in the end of the show with Maxime Labonne.

25:51 Please stay tuned. I think if you are interested in what decision models are and how to run them locally, I think that's going to be a great segment for you. And with that, I think, folks, it's time for us to actually go and dive in to Frontier to talk about math, because I think a huge thing happened this week. Let's go.

26:23 OpenAI released 722 math manuscripts from a model they haven't released yet, and, somebody took and used Fable to count up all of the releases into how many Navier-Stokes-like news we got, and it's roughly 5. 5 moments in one drop. If you guys remember the Navier-Stokes and how much hype that did, we just got 5x in the amount of OpenAI solved maths, which is quite crazy. On Tuesday night, OpenAI posted this onto GitHub without any fanfare, basically a very, very thin blog post.

26:57 after Naver's talks, a lot of mathematicians said that OpenAI is doing very bad things by releasing it such as this. OpenAI said, hey, we're gonna create a mathematician advisory board to tell us how to handle the fact that we're basically brute forcing mathematics, advanced mathematics, mathematics that no humans have solved for 80 years. and and this is after that advisory, OpenAI grouped it into 372 families of results. the f- the internal frontier model about got about 4,000 open research problems, and OpenAI says each result took an average of 3 hours on a ChatGPT Pro level thinking level.

27:35 They also released 10 shorter reasoning summaries on things like irrationality exponent of the pi, the Mahler conjectures, and the Kaplansky conjecture. And they flagged 2 results that came out of a different process, a Riemann zeta 0 free region where humans edited the write-up. So they did have, it's not only slop grenades, they did have human mathematicians actually doing some stuff, and the Hodge conjecture results for a special family of cases. Peter, how big is this? How big is this GitHub drop that we're getting from OpenAI with 722 manuscripts for math?

28:11 Peter Gostev: I think it's enormous. it's we really went from just models being completely hopeless at ba- basic maths to doing this. And the Millennium Prize was a big deal, and it was one thing, and obviously it's a, it's a was, you know, lots of agents being thrown at it, and it's a little bit controversial here and there, but I think I, I remember the time thinking when there was all of this controversy, people saying, oh, maybe it stole its insight from the mathematician who using ChatGPT, but reality is like, okay, we, who knows, right?

28:51 Who knows what happened? I don't think so, but whatever. But the reality is that it's complete cope for people to think that, oh well, that was a one-off, and actually they still can't do maths and and so on. And now we see it, right? It it's happened. There's, I don't know how much clearer signal you can get that this is real. And the question is that, you know, there's one scenario where, oh, they were just low-hanging fruits that sort of maybe humans were a little bit too lazy, too untidy to pick up, and that will be it. But I, I think just reading what mathematicians are saying, it looks like the bulk of them are impressed, and, there were a bunch of people who knew about certain problems, quite a lot.

29:33 And I saw some of them say, you know what, I was looking at this problem for many years, and I never thought there would be an approach like that that would solve it. So I think it is real, it is not trivial, it's not just a finding a bunch of counterexamples. I know, Alex, you said people were get asking Fable, what did these results mean? I saw people also doing that in terms of the, finding the, like, how many counterexamples the- there were.

29:59 Alex Volkov: Mm. Peter Gostev: And there were only like 20% of these are counterexamples. So I think there- there was a scenario of just getting a bunch of uninteresting results, actually just kind of brute forced a bunch of stuff, okay. But I don't, I don't think that's true. I think this is real, real heavy math stuff, which I have no hope of understanding, but it sounds super impressive. Alex Volkov: Here's the thing that, that, that I have two comments before we go to LDJ.

30:24 Would love to hear your thoughts. I know you, you have a bunch of friends in mathematicians. Yam would love to hear you. Last time when maybe your stocks was broke, you put the glasses on and it just like exploded. just for context, folks, this is GPT 5 years ago. somebody shared it, and I- I- I do have to add this for context. 5 years ago, AI was struggling with this type of grade school math. Cindy's math and science books weigh 2 pounds each.

30:51 Her French books weigh 4 pound, and her English books weigh 3 pound. Her history books weighs twice as much as her English books. AI 5 years ago struggled with answering a basic, basic logic thing. And now, somebody ca- called it out in comments, it's not brute force in mathematics, but it's just solving things that the best mathematicians in the world couldn't solve for ages. And, many of us would turn to an LLM to try and understand what is it that they solve, because the math there is so advanced that it's really hard to explain to you how advanced this is, especially if we're not mathematicians. So even mathematicians in their fields, there's what, 4, 300 or so groups that they grouped those results in?

31:39 Mathematicians in different fields of those groups, they, without AI assistance, not many of them can understand the other results in, in that result group. Like, the math that dropped is so advanced that many advanced mathematicians cannot very easily approach all of these problems and understand what is it that, that, that like ChatGPT gave, which I think is fascinating and scary at the same time. and LDJ, I would love to hear from you, your, your thoughts on, on the drop, your thoughts on how mathematicians react to this.

32:11 should they be more excited or, or worried about their job? LDJ: I think specifically when it comes to their jobs, I think that's can, that can be really unpredictable, and it might even depend a lot on what area of mathematics you're in and a lot of different factors. I did want to mention, though, in the example you showed that was 5 years ago, like, I think it's important to remember even just like 2 and a half years ago, when we had GPT-4o, which was the best model they had at the time, that was actually GPT-4o, 4o had just barely come out 2.5 years ago.

32:48 Alex Volkov: Yeah. LDJ: That could barely even do- when I would give it 3 by 3 multiplication problems, like just 3-digit number multiplied by a 3-digit number, a lot of times it would even just fail at that. And so just like in that span of time, we've gotten here, and when people talk about the brute brute force aspect, there's the reasoning efficiency where they did say, on average, these took about 3 hours of a ChatGPT Pro computer or, equivalent to ChatGPT Pro.

33:18 And Alex Volkov: Insane. LDJ: I- I would- I have several friends in mathematics that would confidently say, at least for some of these problems, it definitely solved it using, like, less tokens and less time than a human would have to to do that same thing. And so arguably, in that sense, the human is doing even more brute force than the AI is is doing, right? Alex Volkov: Yam, thoughts on what happened there? Do you have a favorite manuscript that dropped draft the that Yam Peleg: Yeah, they they, they went after the Riemann hypothesis.

33:54 That's insane on its own, just to try. they, the Riemann hypothesis basically, speaks about the zeros of, of a specific function and where they are. So, this one is hard, by the way. That's a very hard problem, needless to say. they, at the best of my knowledge, and I didn't have a lot of time to go into this, and just like you're saying, you need to be many years in the field just to wrap up your head about, everything that is going into this.

34:31 Alex Volkov: About one such problem out of Yam Peleg: Oh, yeah. Alex Volkov: 722 problems that they dropped. Yam Peleg: Listen, people spend their entire life on each one of these, like the Riemann hypothesis is, is a, is a big one. They, even just, founding a region where the zeros are, which is what I understand, is in this paper, that's huge on its own. It, you don't give, you don't need to prove the actual zeros themselves, in a specific region, but you can bound them.

35:08 And it's also huge because it was never done before, and many, many, many people have tried this. I just, just want to put things where they are. Alex Volkov: In perspective? Yam Peleg: It's not- Alex Volkov: You want to put things in perspective? Yam Peleg: It's not that, it's not that you can just go to ChatGPT Pro and tell it to solve the Riemann hypothesis, and that's, you're gonna get this result, okay? it's, it's important there. First, there is first one thing that is clearly need to be said. There is, on the API, there is a parameter, called juice that, you know, each of the ChatGPT, categories of thinking, like high, extra high, correspond to a different level of juice, and pro is obviously, the top. You can up the juice even more.

36:04 I suspect that it's nothing to take away from this. I'm just explaining that it's not that you just go to ChatGPT and, all right, I'll try ChatGPT Pro, and let's prove the Riemann hypothesis. Alex Volkov: Yeah, this is also not Yam Peleg: Not that easy, but it's still insanely impressive. About the brute force, I'm in the, I'm in the camp of, yeah, let's brute force. What's the problem? What's the problem with that? Let's brute force. You can call it brute force, no problem.

36:32 Let's brute force everything, Alex Volkov: Let's brute force, you know, Yam Peleg: And, Alex Volkov: room temperature Yam Peleg: absolutely. Alex Volkov: superconductors. Let's that, that's the brute force. I wanna highlight this one from Ksenia Ksenia Sip from Touring Post, a friend of the show as well. She's like, I'm not a mathematician, so I wanted to understand what yesterday results mean from the math wizards, and she saw this post from Kevin Buzzard that talks about understanding mathematics, and I also do want to talk about reactions as well. Peter, you posted some stuff about, the, Institute for Advanced Study from, from Terry Tao? Would love for you to bring that up as well.

37:04 I just want to read through this, a person that says he's a PhD student of Richard Taylor at early 90s, and said that ma- whilst mathematicians are now beginning to agree that mathematics is all about human understanding, he's not entirely convinced they will agree on what it means to understand high level mathematics. Basically, he says that maths, specifically like advanced math like this, is not about just solving the thing. It's about the human understanding boundary of solving the thing, which is a very interesting take on advanced mathematics, is because, okay, let's say I solved math.

37:43 How does it help the world, specifically, in one, one, one theorem, if humans don't really, like, understand it and and can't do anything with it? it's a very interesting take about some math- mathematicians. some other folks are not having a good time. Obviously, nearly existential crisis, thought process for many folks who've been working on a specific problem for decades, just to see 3 hours of compute just like decimate this. It's a very interesting, a- analogy to software engineers who, after a year or so, not only don't write code anymore, don't even read code, right?

38:24 This was, 6 months ago, maybe, the discussion, the the ZL continuum that I brought, to to the world, like whether or not you're reading the code that your agents output or not, most of people now don't even read that. so this is a very good analogy between mathematicians and their work. Although, when I wrote a piece of code back in the 2000s, I didn't consider this my life's work, like many maticians do, many mathematicians do with just like one problem. LDJ, go ahead, and then, Peter would love to hear from you.

38:57 LDJ: Yeah, so there's a few interesting, kind of open problem sets and self problem sets people have put together using like Fable and Astra of, hey, like, what are the most important problems? And, one of the main examples that has been circulating lately is out of, out of a compiled set of 500 most important open problems put together by Fable and Astra, 92 of those were dropped Tuesday. And Alex Volkov: Say this again slowly. LDJ: solutions by OpenAI.

39:24 Alex Volkov: Yeah, so. LDJ: Yeah, so so Fable and Astra had put together a set of 500 most important open math problems in the world, and 92 of those just got solved on Tuesday by OpenAI. Alex Volkov: That's insane. LDJ: And out of the top 100 in that list, 10 of them got solved on Tuesday. And one last tidbit here. There is a list put together by Fable and Astra of the, the top 100 most significant problems in the most, in, in the last 12 months, and 8, over 80% of those just got dropped Tuesday, for solutions, rather, for them.

40:12 Alex Volkov: It's absolutely insane, folks. It's absolutely insane. LDJ: So, one last little- One last little thing I'm gonna say to, to make people think about something. If you look at the list of manuscripts that OpenAI released, they're, they're ordered, and there is some numbers missing. Like, it goes from, like, 44 to 46, and 45 is missing, and, and there's, there's about 4 or 5 of these. And Yam Peleg: Mhm. LDJ: a- a lot of people are suspecting also that if you just look at all the different categories of problems they released, it looks like, things relating to cryptography seem to be really missing here.

40:51 Alex Volkov: Mhm. LDJ: so it- the- the theory here is maybe they had some significant advancements in cr- cryptography involved here that potentially, the- the US government or various entities or maybe their own just personal policy ended up restricting them from releasing, because it might sound conspiratorial, but there is actually laws and policies that the US government is actually allowed to prevent mathematicians from publishing things relating to cryptography for national security reasons. Alex Volkov: I not- yes, not only national security, also the world economy crashing when Bitcoin goes to zero, if if and when the super intelligent decides, you know, this is, this was the scare from the quantum thing, right?

41:34 Like quantum cryptography, and, and, and solving quantum, like, stuff will essentially make Bitcoin not as secure as everybody thought, because it relies on a lot of, like, elliptic curve cryptography, and that's all math. and yeah, there is, there is a way for the, the world government to say, hey, do not release breakthroughs, do not, like, do not release, you know, diseases into the world. There's also do not release breakthroughs, like that. So, Yam Peleg: Astra gonna be rich, man. Astra's gonna, Astra's gonna be rich.

42:08 Alex Volkov: It's a, it's a good question if Astra can make money, out of, out of this. Peter, I do wanna, bring the, the stuff that you posted. Would love to hear from you about the advisory group and the, the, the Terry Tao stuff, field medalists, responses to this because, you know, folks from all over the sides of the, of the debate are, are chiming in here, and, you posted about this. Would love to hear your thoughts.

42:31 Peter Gostev: You, you want some controversy? Alex Volkov: I would love some hot takes, yes. Peter Gostev: So, I think, it's a, it's a touchy thing because I don't wanna, that kind of maybe came across as kind of jumping on people or something. I hope, I hope it didn't, but it it's a tricky situation, right, because a lot of mathematicians working hard, they kind of have the established ways of working, and then suddenly someone comes in, just drops a bunch of stuff, and just your whole world at least changes in some way, right?

43:01 It has to change in some way, whether directly or indirectly. So there was a letter from Association for Human Mathematicians, which has like, 800 members, and Teritire reposted it in his blog post, and it said something like that, I think there were some, some quotes there, where they said something like, mathematicians did not ask for this work to be done. mathematicians have a particular vision of progress that is informed by history and field-specific considerations.

43:38 We reject OpenAI's assertion that this release advances our subject, and so on. And I think that's a kind of interesting, perspective, kind of, which, which goes back to the question of what is maths for? And I think it looks like a lot of, mathematicians, at least part of this group, they seem to be kind of suggesting, well, it is, you know, we need to run our thing, do it in a certain way, but inherently what that implies, it's not to actually solve math problems.

44:11 Alex Volkov: Mhm. Peter Gostev: And, to me, that raises an interesting question of like, so are we basically saying that math is useless and we should just leave it to smart people to j- to just work on it? If that's tr- if that's the case, then I can see the intellectual argument for this. But the, if you flip it another way and say, well, imagine doctors said this, you know, stop, stop curing diseases, right? We just need to, like, don't, don't ruin our good thing.

44:37 We just need to keep going, you know, treating people from their headaches and whatnot and cancers, and, you know, the we've got a good thing going. Like, just imagine that happening. But here's a clear line for utility that I can't imagine any doctor would say that, right? So if OpenAI or whoever dropped a bunch of papers saying this is how you Alex Volkov: Solve. Peter Gostev: you know, yeah, solve medicine and kill 500 diseases. Alex Volkov: Yeah.

45:02 Peter Gostev: Yeah, people would be like, yeah, amazing. And I think, to me, this is, I appreciate, well, I'm not sure I can fully appreciate, but I can imagine this is very difficult for people in that field, which I think we need to be sensitive to. But it is kind of, but to me the question is like, well, are you saying that your field is useless? Right? Is there no utility and we should just let you hang out and work on it and not solve any problems? And I don't know, I don't think you can kind of say both, right? You can't say, oh, it's really useful, but please don't solve any of it.

45:34 Alex Volkov: Yeah, the please don't solve crowd, I don't understand, like, get on board. Like, thi- this is now in the realms of OpenAI, and in a year it's in the hands of pretty much everyone, right? This is the pace we've been on. It's very clear that this is the pace we've been on. LDJ, one last comment. We have to move on. We dedicated a lot of time to this math thing, and we have to move on.

45:54 LDJ: Yeah, I recall about a, a year or so ago, there was actually something circulating where there was a discussion at a big conference where somebody stood up and said something along the lines of, "Do we really w- want to automate a cure for cancer?" or something, and that they were actually, like, having, like, a serious statement and discussion about, like, their own job and livelihood and whether they would really want that. And there's a lot of polarizing reactions there, but I find it really interesting how at first it seems like the answer should be obvious, but there is seriously people that have strongly differing views there.

46:32 Alex Volkov: The I'd rather AI not solve cancer or something, I'd rather risk cancer than AI taking over essay was just like a ridiculous thing, a point in time, I think. W- w- hopefully we'll never get back to that moment. folks, it's time for us to move on because as exciting as this is, we can talk about this in circles for hours, but there's a lot of stuff that happened this week, nearly an hour into the show, which is crazy.

46:57 let's run and talk about, the frontier for everyone. There's 2 major things that happened this week. Let's start with Anthropic launches Claude Haiku 5.5. it's been a year since Anthropic launched the previous Haiku, and this is one hell of a model. Let's take a look. Haiku 5.5, tiny price with very big, big score. So we're looking at a model cheaper than ChatGPT Luna, although it can get a little bit more expensive depending on the reasoning effort and the number of tokens that you send it. Haiku is just priced at 10 cents per million input tokens for under 100K. this is 10 times as cheaper as last year's Haiku, which was at 1 dollar. I love Haiku. I used to love Haiku, not for like day work, but there's a lot of stuff that, you want that the good Claude intelligence, but, but like in really fast, really like quick price. would love to hear from folks here, thoughts on, on Haiku release. Peter, did it already hit the arena?

47:59 Nisten Yam, LDJ, did you guys use it? Peter Gostev: Yeah, it's on, it's on the arena. I'm just testing myself. We don't have a score yet, so, but I think it, it's super exciting because I, I think the, the, the one problem with Anthropic offerings is they didn't really have a lot of good cheaper models. They had the best very expensive models. Alex Volkov: Yeah. Peter Gostev: But then you, you by default had to go somewhere else if you wanted to actually have something cheap, and that's not ideal. And I think it's interesting to see what, what they managed to do with this. One maybe not to, to call out is something I'm digging into as well, is that the, the cash read is so important, the, the price, for it. And and - Alex Volkov: And for Haiku, it's 1 cent per million tokens.

48:49 Peter Gostev: Yeah, but it's, it's also the ratio of kind of how much is it to, to read, normally what the input and the your cash read is. And I think the Sonotropic has been dropping prices. Actually, I, I was just literally pulling the data up, and Fable 5.1 has the lowest ratio. It's like 2.5%, the cash read versus the, the read. And I think Haiku is still at 10%, but it's certainly been improving, right? If you, if you pick like GPT 4.1, I know that's long time ago, it was 25%, the cache read. So it's a, it's a really important part.

49:27 I want to do a bit more analysis of it, but I'm glad that, that they're dropping prices on cache reads. That's for agentic work is the biggest thing. Alex Volkov: s- speaking of cache reads, another hidden announcement in the Haiku announcement that Sonnet's cash reads are cut in half in prices, which makes, Sonnet is d- most agent work about 20% cheaper on average, because if you think about what an agent does, there's a lot of, mm, long-term history that happens in an agent.

49:56 Next time you send a message to the agent, a lot of, a lot of the previous stuff should be cashed because nothing changes there. Haiku is also significantly better at computer use. LDJ would love to hear from you about this one. We like OS World. In OS World, Anthropic says 72% against 48 in Luna, just 48 in Luna. In Terminal Bench 4, Anthropic's saying Haiku 5.5 gets 40%, nearly 40%, 39, while Luna scores only 16. So, compared to the very good, fast, and cheap model from the frontier from just a week ago, just a week ago.

50:32 I, a small sidestep. I love that we're a weekly show because so much happens in the span of a week. We have time to catch up. just a week ago, Luna, just like the best fast, cheap, and reasoning model is only 16% on on terminal bench. Haiku is available everywhere with one. cent per million tokens cashed, which is just absolutely insane. anything else remains to say about Haiku besides go use it, folks, and tell us? Anybody used it? Yam?

51:01 LDJ: Yeah. Yam Peleg: Right. Alex Volkov: LDJ? Yam Peleg: Go, go. LDJ: Yeah, so at the Pareto Frontier, it looks like this is setting in, in some ways the, a new Pareto frontier for in terms of accuracy and cost. On artificial analysis index on extra high, it looks like at the Pareto frontier or near it and competing with a lot of the Chinese models and others. But I, I do think going forward, and this is a bit of a bold prediction from me, like over the next 6 to 12 months, I think we'll see the Western frontier labs continuously getting better on the Pareto frontier and beating Chinese labs, even on the, the most low cost, options.

51:44 Alex Volkov: Th- this is a very interesting place for open source to be in, because a lot of the Chinese labs, specifically DeepSeek, on their services promised us very fast and very cheap, not necessarily very advanced. The, the Frontier Labs are going to it, and then I, I remember Sam Altman Dev Day, saying we're gonna offer the best, like, price model for every price and every tier of intelligence. it looks like Luna has a lot of catching up to do.

52:11 Yam, last comment about, Haiku 5.5 and what would Yam Peleg: that's Alex Volkov: use it for? Yam Peleg: That's the obvious choice for swarms at this point. Like, you want, you wanna, you wanna do a lot of things in parallel. It was the Lu- Luna was the obvious choice yesterday, and and not not yesterday, but but you get the point. Alex Volkov: Yeah, yeah, yeah. Yam Peleg: That's, that's the obvious choice. I mean, that you don't get this type of performance for this, for this price on any other model at the point.

52:41 Anthropic is is on fire recently, like all the releases are, are, are top. Seriously, we didn't, we didn't speak a lot about Sonnet, but Sonnet is good. I don't know if you guys tried it, but Sonnet is surprisingly good also. And everyone online is just, just confused. How come I never burn token, I never burn my quota with Opus, and I, I've been hammering Opus all day long. It's they all over X, everyone is saying this.

53:14 So, like, imagine how much you have now for Haiku if you just want to do many things in parallel. Anthropic is on fire, and we are the ones getting stuff because of it. And, I think, I think that's great because, Alex Volkov: That's absolutely great. Yam Peleg: It was, it wasn't, it wasn't the case a couple of months ago. Alex Volkov: Anthropic is also coming up on the IPO, and I think it's, it's, it's good for them to drop a bunch of stuff. let's finish up with Entropic Claude in Google Docs and Google Sheets and s- and Slides. They finally added this ability to be able to actually work on your documents, not just like post via the, the Drive integration, so that's great. but speaking of cheap intelligence for everyone, OpenAI also came back, and GPT-6 is now the default.

53:58 We- the default intelligence, not GPT-6 with a name, not Sol, not Terra, not not Luna, not Astra, not any of them. Terra is dead. RIP Terra. But just GPT-6 is now the default for free accounts. This is, not talked about this a lot, but out of the 1.2 billion weekly active users, I think they said last week in Dev Day. Peter, please correct me if I'm wrong. one, I think it was like 1.2 billion weekly active. Much of that is free accounts, right?

54:28 Like, folks who are listening to us, probably like when you were in a traffic, you maybe cancelled the OpenAI one. Many of this is on the free tier, the free accounts, and like, you could say that OpenAI just like uplifted the world's intelligence just now with just like one, one release with GPT-6, which is incredible. So if you are listening to this and you are on the free tier of ChatGPT, your intelligence has been upgraded. Not only that, they released this thing called, intelligent UI. And I want to show you, can we show this on the stage? This is me asking ChatGPT to say, hey, visualize the math problems OpenAI dropped on GitHub with a nice UI.

55:10 And it built me inside the editor a mini, a mini app. You guys see this? Yam Peleg: About time, about time. Alex Volkov: Right? Yam Peleg: Great. Seriously. Alex Volkov: You it says 719 manuscripts. You guys know why? Because it's updated since I did research. OpenAI did pull some manuscripts back, so this is like the, the real time thing. and then how much is formally verified? They're saying 300 of the manuscripts are formally verified, but I'm not gonna, not, not gonna talk about the manuscripts. You can see that this is kind of like a live thing that works, and the, and this can be your weekly update from ChatGPT. This can be a research thing.

55:47 This can be like a bunch of stuff. They have maps in there. So this intelligent UI is, I think, is pretty cool. It can build forms and buttons and small working tools, and yeah, I think that's great from OpenAI, but the intelligence boost for the free folks, I think, is very, very interesting. Any anybody already used it? Anybody feels the difference? I'm not on free tier, so I don't know. I've been using Yam Peleg: Yeah. Peter Gostev: I think- Yam Peleg: I thi- I think we're not.

56:15 Peter Gostev: I think what what's cool about this, and I appreciate that OpenAI is still doing this, is that we we kind of had versions of this in Codex for a while, so where it kind of builds these little visualizations and apps, and it's nice that they keep bringing that to the free tier and and so on. So yeah, GPT-6 is cool as well. Yeah, we- I think we we forget sometimes that we are very, anyone who's listening to this, you are very not normal person.

56:44 so you are Alex Volkov: Said with love. Peter Gostev: Very far away. Alex Volkov: Said with love. Everybody who's listening to this is a, is a beautiful person, part of our community, but normality is not, you know, like Your hairdresser is probably not listening to the show. Peter Gostev: Yeah. So, so yeah, Alex, what you said about, yeah, it's uplifting the overall intelligence. That kind of thing has so much more impact than whatever it is, like new model is dropping on the top tier where you need to pay 500 dollars a month.

57:10 Like, that kind of thing makes a difference to us, but not that many people. So yeah, it's cool to see that, you know, they still remember. It's a, it's important. Alex Volkov: Let's also like check on Nisten a little bit. Nisten, you, you with us? You alive? Like, what's going on? You haven't said a word about Haiku or ChatGPT. I'm hoping that you're gonna tap in when open source comes. Nisten Tahiraj: I'm running 10, 10 Haiku agents right now.

57:33 That was because I almost used up, like, I used up 95 or 97% of the, of the quota already until, on the max 20 plan. Alex Volkov: Wow. Nisten Tahiraj: But, so I had to basically resort to Haiku. It's very good. Yam Peleg: Yeah, it is. Nisten Tahiraj: Yeah, yeah, I had a rice cooker, and, I couldn't get the ratios right, and then it did a whole bunch of research, and it found that this is not very accurate as a Alex Volkov: Bro.

58:00 Nisten Tahiraj: as Alex Volkov: Stop reversing the world for rice. Nisten Tahiraj: Set an hour Alex Volkov: 1 to 1 and a half. Nisten Tahiraj: and, it got it done perfectly. Alex Volkov: My grandma knows Nisten Tahiraj: Haiku Alex Volkov: to 1 and a half. What do you mean? Nisten Tahiraj: Yes. Alex Volkov: Haiku cooked. Nisten Tahiraj: Yeah, it got the right Yam Peleg: You're letting Haiku make you food, like. Nisten Tahiraj: Yeah, I have resorted to that, guys. He did a good job.

58:20 Alex Volkov: Yeah. LDJ, comment, and then we'll move to to open source. LDJ: So I want to ask real quick, Nisten, would you say that it is clearly at parity or or clearly above Qwen 3.8 27B? Nisten Tahiraj: Oh, oh, oh, for Haiku? Yeah, yeah, yeah. LDJ: Ha- Haiku, yeah. Yam Peleg: Good question. Nisten Tahiraj: It's, it's above, it's, it's above all of that. especially for conversational stuff. I haven't tested it technically, but anything conversational, yeah, yeah, it's a, it's really above.

58:52 It kind of feels like Sonnet, actually, when you talk to it. I haven't looked at, like, what code it writes, but it's, it's, yeah, it's too good. it is what it is. That's, that's the assessment. LDJ: Sick. Alex Volkov: All right. folks, use Haiku. We, like, the whole family is great. The all of the three brothers, like Haiku, Sonnet, and and Opus 5.5, all great. OpenAI obviously tries to release responses of theirs, so we'll we'll probably see more models, but let's talk about the next thing on our roadmap, which is let's do open source.

59:39 Open source AI, let's get it started. right, open weights and open source AI is our new category. Let's talk about this. Open weights went big this week, a 501 billion parameter American model, a trillion parameter French one, and a German one you can run on a single H200 open source. Very interesting this week. Reflection AI announced Beam, 500- 501 billion parameter model, 23 billion parameters active. It's trained from scratch, on data here in, in the West, and Apache 2 weights are promised this month.

60:19 Reflection AI has been, training, I believe, on the SpaceX cluster. Somebody correct me if I'm wrong, but I believe that this is, the the cool thing about them. They're pitching it as the Western Open Open Frontier model. They claim is 80.9% on Swbench verified, which is a hard coding, benchmark. and to their credit, they admit that Kimi K3 is ahead on just raw capability and bench, scores. However, they pit- they pitch efficiency, 3 to 4 times less inference compute than, GLM from ZAI. weights are not downloadable yet, but this is announcement, but we will definitely let you know when it comes.

61:03 folks, my timeline lit up with Reflection, my timeline lit up with Beam. thoughts on how they approach this release, thoughts on what we're due to see. LDJ, I see you have your hand raised. Please go ahead. In LDJ: ter- yeah, in terms of the comparison to Kimmy K3, it is about 6 times smaller, so it it it'll be a lot more practical to fit on more real world, consumer-ish, prosumer-ish VRAM budgets. And in terms of overall the, like, even just besides the amount of active parameters and total parameters, you have how many tokens does it actually output to get to a certain answer.

61:47 And, you know, if you, if you use half the tokens, that's, that's nearly half the compute. Alex Volkov: Yep. LDJ: And artificial analysis had done some numbers on this too, which I'll send, this in the link here, but it's looking really good in their preliminary results so far, and they said they're going to release some more detailed analysis on it soon. Alex Volkov: Yep. I, I, this is a graph that we're showing from, Reflection folks, and they're highlighting where Beam sits, on scores, on benchmark scores, which we all know is not everything. Like, the, the, the data that you put in there, the type of, actions they trained for, whether or not it's agentic, all matter for these big models, but they, the, the very interesting thing is that, and I haven't, I don't remember seeing this before.

62:33 I think Artificial Analysis has this. They have two different colors of scores, and they segment this based on Western open models and Chinese open models. And I, you know, I think that's a good position for them because this is definitely Mistral's position. We're gonna talk about Mistral in a second. and they are highlighting that they are the Frontier open weights on the Western Open models, and on Sweet Bench Verified, they're just taking the Frontier. By the time they released it, I think this will change multiple times, but, this is a very interesting thing.

63:09 Yeah, this is a statement from artificial analysis. they have been given access by Reflection and independently benchmarking Beam. Early indicators suggest Beam will be one of the most token efficient open models we've seen for its level of intelligence. So shout out to Artificial Analysis, and we can't wait for scores. Shout out to Beam. There's not a lot to say here besides the folks at Reflection, releasing this, and, I can already, hint that it's also coming to, some other inference providers, let's say, that help make the show, what it is. so we're gonna talk about that, but before this, Peter, you wanna, you wanna do the, the honors of the breaking news?

63:43 I think, I think worth it. AI breaking news coming at you only on ThursdAI. it's, it's always fun to see breaking news from folks who are participant on the show. Peter Gustaf, go ahead. Peter Gostev: Yeah, so we, we have some news, we raised 200 million. That's a lot of money, at 3.1 billion valuation.

64:17 And so, yeah. It's a, yeah, big, big day. Yam Peleg: Yeah. Alex Volkov: Big day. Yam Peleg: Let's go, man. Alex Volkov: Congratulations. Yam Peleg: Congrats. Alex Volkov: This is a, a big thing, and I think you guys are doing like a, a great thing for the world. do you want to tell us about kind of what is Arena cooking? If people haven't visited Arena AI for the longest time and maybe visited once before LM Arena, what changed since then? What are you guys like, like, why does this justify this valuation, this money?

64:47 Peter Gostev: Yeah, I would think, yeah, the, the biggest change, and I think we're, we're best known for this kind of battle mode. When you put one prompt, you get two generations, which is, like that, that's still important for, for some things. It still measures quite a bit, but things have moved on, and we've got, we're much more focused on agentic evaluation. So we've got Agent Arena, and what that means is that you can put in any prompt you want, use whatever tools you want, and then we'll measure. You don't get two responses, you get one, but then based on your feedback to the agent or a how you're interacting with the agent, we can pick up enough signal to differentiate between the models.

65:28 It's really cool. Honestly, I know I'm biased, but I, I can't think of any single, leaderboard that is actually better than ours. maybe, you know, if you put together a bunch of different ones, maybe, maybe you can argue, but if you're gonna pick one, I think, I think ours is, is very strong. We, we also launched an alignment index where we're using the same agent, evaluations to be able to also pick up things like, did it actually do what you, or did it, take some unauthorized action, for example?

66:04 So, and then, OpenAI is particularly good at not doing that. and you can see some specific s- specific signals if you scroll a bit to the right. And, and, yeah, so we launched that alongside of it. But it's all powered by the fact that people can come and use the models, the agent mode, and then we can pick up signals from that. So, yeah, I th- I would say, I know the the battle mode is very cool and people like it and find it useful, but I would say the agent mode is where we get super rich data and we we can extract a lot of information from it.

66:37 Alex Volkov: That's very cool. Congratulations. This was not planned, folks. We just came through, and we want to say congrats to Peter and the whole Arena team as well as we go back. congrats, folks at Arena. Folks, we're back at the open frontier, and let's talk about Mistral. Mistral launches Lechonk and leaning into the meme. So let's, let's talk about this. the French, the the French AI company comes back with Mistral Large 4 Lechonk, a 1 trillion parameter model that uses the the ties to GPT-6 Luna on artificial.

67:18 it's a very interesting release from Mistral. Anybody want- wants to take this one? Like, I know we've been celebrating them in open source for a long time. also the weights didn't come yet, right? this is just also an announcement. 50, 1 trillion, parameters all around, 50 billion, active. It's a multi-model, with 1 million context and open weights. It didn't launch yet. artificial analysis gives it a 38, the same score as GPT-6 Luna. this is $1 per thing or 13, $1.13 per task versus 7 cents for Luna. So this is not a very, optimized model compared to the frontier, but it should be open weights.

68:03 So, folks, comments on this, comments on Mistral Lechonk, besides the the very good faith that Mistral has, you know, in our community, and besides the fact that, you know, many European governments can only use this model, let's say, because of the deals they did. What are we thinking about this model? Peter Gostev: So I I have, ta- it. We've got a score on, Code Arena. So that's the still the battle mode, but that's mostly test front end.

68:32 it's not been great. I think it's like 40 second or something like that. So I think, I think it was like DeepSeek Flash, which is way cheaper. I think it's like 20th or s- along those lines from. So yeah, it's not great on that side. I have also tested just personally on some web dev agentic stuff and a bunch of different tests. It's not, I mean, it's not that great. So I don't know what to say.

69:02 It's possible. I haven't tested, you know, in French, so it's possible that, we do actually have some data. It's not that many prompts yet, but we do have some data to indicate that it is way, way better in French and the European languages than other models. So it's kind of punches above its weight in, in those categories. So it's not that cheap. I know they're running some promotion now. I don't know if that will stay. So I, it's kind of, you know, it's good to have models come out. Obviously, it's, it's always good, you know, the open models as well. but yeah, it's not, I don't see it like a big reason to use it over if you're using DeepSeek or something.

69:43 I'm not sure you, you should be switching to that straight away. Alex Volkov: No, and the very interesting thing is that all my comparisons, and I'll show them here again on the stage, that I also got from Artificial Analysis. they're comparing them to, specifically to Luna. We just told you about Haiku that came out a few days ago that just destroys Luna on every parameter, especially cost. So there should be a reason to use these models, and I think, for many folks, the open nature of it is one such reason.

70:15 But since this is a trillion token, it's not like you can put this in your DGX Spark. so maybe if you are, a European government or someone with ties to Mistral and, this is the model you want to use, maybe, but for regular folks, I think, not so much. I haven't seen many folks use Mistral. Nisten, am I wrong here? And if- Peter Gostev: I'm trying it right now. It's not that bad. I mean, the, the website is nice. It responds pretty fast.

70:47 You can talk to it. Alex Volkov: Yes. Peter Gostev: Like, it's as a product, it's not that bad. Would I do any agentic coding with it? No. Alex Volkov: No. Peter Gostev: But, as far as a chat interface that you can just open for free is, feels pretty good, so I... LDJ: So I think when I'm trying to look at the, the cup half full here, what might be really good, it, it might be a pretty good base model or, or a foundation that people could then use to do a lot of RL on top of.

71:20 And, that might be something, and it might be superior in some ways to the base models that exist right now for the certain combination of total and active parameters that it has. Alex Volkov: Yep. all right, let's see what else. we have, another European, company, this time from Germany, also releasing something, and it's been a while since we mentioned Aleph Alpha on the show, but yeah, there is another European lab. It's not only Mistral. Aleph Alpha released, Germany's Open weights hummingbird, 78 billion parameters, a 3 and a half active Apache 2 license, which is great, a million, million, context there. it trains from scratch in German and English.

71:59 Our German co-host is not here with us, but he did send, his assistant Amy, to to tell us about Calibri. here's what Wolfram has said. because it's German, our German tester is not here with us. He says, it's German and tool corings were good, but reasoning and instruction following were too unreliable for a user-facing agent. Repeatedly lost roles and state, while more reasoning mostly made it slower. My verdict, promising a specialized German tool worker, but not a strong enough for Amy to main my daily model. And, if you guys want more details, Wolfram posted about this, on his X. I will will defer to the German testing guy, that's not on our co-host panel this week, but usually is, to tell you about this Colibri model. that's pretty much it.

72:48 20 trillion tokens in pre-training. If you are using German, that could be good for you, but not as a main model. Peter Gostev: Can I, can I mention one thing Alex Volkov: Yeah. Peter Gostev: about, about the open models? So one, one interesting thing, we got a bit of data about the number of GPUs that were being used. Alex Volkov: Mm-hmm. Peter Gostev: So I think that the one that we just covered, the German one, I think they said something like it was, 6- 768 or something, so call it 800 GPUs. I think it's Blackwells, so I'm, I'm slightly speaking from memory, but I think they said, call it 800 Blackwells.

73:24 Then, Mistral said 3,800, Grace Blackwells, and so I'm assuming it's B200, then GB200 probably. And then what, we had a bit of information from Jensen about Astra, where he said it was trained on, 100,000 Grace Blackwells. So that's, that's the kind of one, one thing that is, I, I don't really know how to think about it. On the one hand, they could be like, oh, well done to that team for training on so few GPUs and actually like producing a coherent model.

74:02 On the other hand, it's just, I mean, I don't know what, how many GPUs is Luna or Haiku are getting, but probably not 800. it's probably in the tens of thousands. Like, I just don't know how to think about it. Like, is it just like no hope for these, for these guys to train on like a few hundred, few thousand? Alex Volkov: dude, this is why Jensen is showing up everywhere, showing up with, with, you know, with Satya Nadella at Microsoft, we're showing up with, you know, with Elon, this is everywhere.

74:35 hope is, is very interesting. The thing that I will say, though, is, efficiency improves significantly. you know, B200s are about to get, replaced with, with the Vera Rubens, and so we're gonna get even more AI. That's basically my reaction to you, right? But yeah, I don't know if there's hope for a lab that cannot get the, the number of, like, the crazy number of GPUs. but Peter, since you took us there and since we're talking about GPUs, I think it's time for this week's buzz real quick.

75:06 Folks, this is a corner where I talk about everything Weights and Biases and CoreWeave related. Weights and Biases is now part of CoreWeave Forge. I will still play our transition, and then I'll tell you super quick about the GPU stuff. Weights and Biases and CoreWeave. This is very important. I need to update this transition, folks, in this week's buzz.

75:39 since Peter just mentioned GPUs, here's what I want you to, remember from last week. Last week we had the CoreWeave Fully Connected, our Prime conference. We obviously did a live stream from there. It was great, and a huge announcement from there was from folks at Cognition. I, let me see if I can just play this for you real quick, because I think it's, it's worth playing here on the stage. Here's the announcement from Silas, one of the co-founders of Devin. I'm excited to announce that Cognition becomes the first customer for NVIDIA Vera Rubin, powered by CoreWeave Cloud.

76:17 I think you can hear me in the background there yelling yay. It's been truly great privilege to be, get early access to this latest gen platform, and so our research team has been putting it to the test, and in the last few weeks we've been comparing Aerobin compared to the previous generation GB200. We first started with production inference on our latest model Z2, and at the same speed we're seeing a 4.8 times improvement in throughput, and that means cost savings. On higher speeds, if you're serving fast mode, the improvement gets even bigger. Moreover, we were trying to get it to running in our training workloads, and in reinforcement learning rollout, we were also seeing 3.8 times improvement.

76:56 These are incredible results. folks, CoreWeave is the first cloud to put Vera Rubin, the next, version of the, all the GPUs that the models are training for, on production, and Cognition is the first customer to ever get, Vera Rubin on production. It's a combination of CPUs and GPUs, and they're seeing 4x, the total token. throughput of the previous GB200s, which is absolutely crazy, and that is coming to many, many companies in trainings as well.

77:26 speaking of GPUs, there is another announcement. I'm going to point to this QR code here, for from our folks at the serverless sandboxes. If you want to try out the new, very, very raw, but the new GPU sandboxes product, please scan this QR code and reach out in this form. Tell you, tell them that Thursday I sent you, and we are now offering sandboxes at forge.coreweave.com with GPUs. I cannot promise you they will be Vera Rubin.

78:02 Most likely won't be. but still, a lot of folks have asked us since we joined CoreWeave, hey, how do we get access to some GPUs, including folks on the panel here as well. this is how. This is for the first time, this is how we get, we give you access to some GPUs. Our sandboxes product is part of our Forge announcement, is how you get it. try it out, let us let us know comments. This is very, a very cool offering that I really wanted to bring you directly.

78:28 if you want serverless GPUs from CoreWeave based on the same platform that gets, platinum on the, ClusterMax analysis from SemiAnalysis, check out that QR code. All right. agents got an open protocol this week, which is to shop at Walmart and to do stuff at Shopify, the same week as agent posted its own owner's bank account by mistake to the company Slack. Let's start with personal agent protocol.

79:14 Meta and Sierra, that's Brett Taylor's company, he's on the board of OpenAI, announced the personal agent protocol, an open standard for how personal agent works with a business. Walmart, Shopify, Stripe, Genesis, Instinct, and Rocket are building it with them as well, because your agent now, or personal assistant, whatever you want to call this, personal agent, personal assistant, can browse as a guest to checks, to check stock or sign you in, with a read-only or write access, and the business decides whether your agent talks to its website, its MCP or API, or its own agent. So there is a way for the website or the business to let you guys, know about what happens in i- i- in that browsing session.

79:55 The first spec of this lands later this month, and CNBC says OpenAI and Anthropic haven't joined this spec yet. It's very interesting, given OpenAI stepped in to the personal agent category last week with Dots, which we covered briefly, but we haven't talked about this. Protocols, folks, last we talked, we announced the MCP, and MCP took over the world. You guys feel that that this personal agent protocol is, something that's also gonna take over the world? Is it important?

80:22 We've seen other protocols like A2A from from Google and not really catch. This feels like some of that, here as well. thoughts on the protocol? Nisten Tahiraj: Has anyone read it? I don't think anyone reads the protocols anymore. The bots go with just what comes, what vibes first, and Alex Volkov: Yeah. Nisten Tahiraj: MCP vibes first, and I mean, they can make it. A2A was pretty well designed too. It doesn't mean anyone's gonna adopt it.

80:52 Also, I don't understand why. You can pretty much do most everything you need with MCP, and then you can just remote desktop in and give it computer use. It's like, I, I don't see why they did this. They just make their, make up their own protocol on the way. I, I don't. LDJ: Yeah, I, I'm, I'm leaning towards Nisten here. I think, honestly, the best way might just be the, the agents emergently create their own most optimal protocol. Maybe that old protocol ends up evolving more, like, that's the most bitter lesson filled direction, right, is h- have the AI's intelligence produce their own best protocol.

81:35 Alex Volkov: Yep. Nisten Tahiraj: Yeah, if you set up a company, oh, sorry. Alex Volkov: No, no, the I, the only comment that I have to you guys is that, if agents need to recreate a bunch of protocols, that's a problem. and it's a problem because of our next, you know, we can show our next item here. when agents are let to their own devices, some things go through, that weren't necessarily, created properly, weren't necessarily scoped properly. So here is, here's one such example of why, for example, a protocol for agents is needed. there is a, a somebody built a personal CFO with their Grokbot, Shane Mack specifically, and to to his very, very big surprise, another Grokbot that was connected to his company Slack decided that the best way to let him know about the personal balance of his bank account is via the company Slack.

82:31 So he literally just like posted his financial, monthly financial audit, to his whole company, including the personal Mercury account, and the savings account and and Jim and whatever. and specifically highlighting that he is a number, a specific number under the bottom of the floor and, showed which of the biggest, you know, payments he did this week. So basically, his personal CFO just went public on Slack and told everybody his bank account. I don't think, you know, I don't know if a protocol solves this, but I know for a fact that some better guardrails are needed for f- for some of these bots. And when you are using these personal bots for shopping, and you rely on them to do work for you without approving every step.

83:21 I think, some more structure is needed. this was a very- Yeah, ve- very LDJ: Yeah, I do think- I do think in the short and medium term that some of these protocols are going to continue being quite useful, but I think inevitably they get overthrown by, by protocols created by AIs that are much more capable, or that the only protocols to stick around are the very simple ones that themselves are just kind of very bare bones, and you could build anything on top of.

83:48 Or besides that, just the protocols that have the most, the most mindshare and the biggest vibes, as as Nisten said, because then that will be the most emphasized in the models training and therefore have the highest chance of being used by the models. Alex Volkov: I I do want to, like, mention MCP real quick here. it was quite when it came out and a huge splash, like, a few, few months after, and then kind of like the hype died down.

84:10 But as protocols go, as like HTTP, MCP is now one of the most used protocols in the world. The plugins for OpenAI that they announced last week all use MCP and MCP apps. all of the connectors for these agents, a lot of them are like relying on MCP, all the assistants, Grokbot, etcetera. you can use API, but regular people do not know what API is. MCP is taking over the authentication standards, so there's like one button to let your agents in. so I think it's, you know, it still m- matters a lot that all of these companies are like aligned on on a specific thing.

84:44 Nisten Tahiraj: guys, I just went on the Sierra.ai, the the ones that introduced it, introducing personal agent. This thing has EM dashes all over it. Like, why are you even bothering? Just let the agents figure it out. Don't even post it at all. Like, why are you do- this thing is full of EM dashes. Nobody has read this thing. What, you think Zuck and Toby read any of their code too? It's a- it it already is. Just leave MCP in there and let them figure out whatever else they they want.

85:14 Alex Volkov: All right, moving on in the personal agentic, like a corner that we have, Grokbot. Speaking of Grokbot that, you know, just just launched somebody else's, details in Slack. Do you guys see, do you guys catch this? SpaceX AI, or specifically Elon Musk, the the the space uncle that's in charge of SpaceX AI, said that Grokbot now becomes a router to the best model available. regardless of who it is. Obviously, he won't put OpenAI models in there.

85:44 but Grokbot, Grokbot will now route to Opus 5.5. the challenging, requirements, and, folks are saying that it's live, spotted in the, in the calls log that they're seeing Claude Opus 5.5 on low, and, xAI, own example showing that the logic and code will use Opus 5.5, visuals for Midjourney, and music for Suno. Specifically very interesting because they have visuals, that they're building themselves at the SpaceX AI. They have Grok Imagine, and it's very interesting considering that they just released a model that was supposed to be the best agentic one, Grok 4.7. thoughts on the fact that Elon Musk is now, you know, going back? I have some thoughts, but I'd love to hear from you guys.

86:32 what does it mean that SpaceX AI now leaning on Anthropic for its product, not its model? Have you guys used Grokbot since the upgrade? Did you feel the improvement? I definitely see the improvement, though I can not confirm that I got Opus. I've not used the more recent upgrade, no. So, definitely the team is cooking. there's been you view few visual updates. The the most interesting update about Grokbot that I can tell you is that, they have noticed that the default for Grokbot is multiple bots.

87:07 however, many people prefer just one. So they released like the primary, primary bot release where the one bot that you choose is your like main, and then it will tell other Grokbots to do and and do different things. I haven't quite got the Opus one yet, but we'll see, we'll see when it lands. Here's my take on why Elon Musk is okay with letting Opus into that product, even though the product is literally named after the model called Grok. because I don't know if there's any agreements between them, but I'm assuming that, SpaceX AI cannot just like distill an Opus to make their models better. However, if you do pay a lot of money and you put this Opus inside of your product, you can, for the folks who agree to this, you can absolutely train on whether or not the bot made, you know, the right agentic choices.

87:59 So they can now absolutely compare between Grok bots that delivered and the backend was Opus and Grok bots that failed and users are screaming at them and and putting F F words in the chat, when it was using the Grok. They can compare, and they absolutely can train on those because technically that's their data. So my question to the panel here, super quick, is Elon letting Opus 5.5 inside Grokbot a back channel for, basically distilling Opus intelligence into the next Grok?

88:35 Nisten Tahiraj: I think Elon just likes Opus as the model. Yeah, but he He probably has his private, his own private copy running, and also he hates Sam Altman, so he's gonna enemy of your enemies, my friend. Look, I think, I just think they want, users and users to have good experience, and they recognize, all right, that's what the users want, so to get users on the platform, that's the way it's gonna happen.

89:06 so we're just gonna, just gonna give the users what they want. If you just go against what the users want, yeah, you would have your own model, but it's kind of, I don't know. Look, Cursor themselves, they serve many models, even though they have their own, model. There's nothing wrong about that. I think that's a great strategy. Alex Volkov: Cursor projects with Opus 5.5 are goated, and there now is a deep, deep, deep integration between your Grokbot and Cursor Agent cloud agents.

89:41 So if you are, if you're thinking about Grokbot, do not go through the Grok Ultra whatever, program, go through Cursor's Ultra program. It's the same 200 dollars a month, but then you'd be able to spin up cloud agents with Opus and Fable and just like rip through, and your Grok can manage them, monitor them, and tell you about, you know, successes, etc. So you can absolutely use your Grok bot as kind of the, the, the PM, for your actual like Opus work. It's really, really good with cloud agents.

90:11 Nisten Tahiraj: Just use the, the meta views, call it the blob, and it's a, I gave it a VM because I- I broke in it. I gave it a VM through through Tailscale, so it has full graphics, full bra. It's the best free tester for public websites and stuff that you have. Like, Alex Volkov: Yeah. Nisten Tahiraj: I even put an MCP so you can like keep talking and reporting to- to- to Claude. So now Meta just like complains to- to Claude and- and then clicks on everything on the site. It's got such good free computer use, especially for anything you're gonna have. It's, it's great, dude. I love, it does not refuse anything.

90:48 That, that's the- Alex Volkov: Nisten, you love Muse is, is basically the highlight. Nisten Tahiraj: Yeah, yeah, Muse is vibing. Alex Volkov: Muse is Muse is great. Nisten Tahiraj: I let Muse control VMs, talk Opus. Yeah, it's pretty dumb sometimes, okay? Don't get me wrong, you can, but it's just so energetic and positive that it's just, it's great. Yeah. Alex Volkov: All right, so, so speaking of personal assistants, let's super briefly cover this before we go to our next chat with, with Maxime Labonne, who's, who's coming up.

91:15 Maxime, just come up here. folks, Nous Research our friends of the pod Nous Research, released a Hermes index, and Claude Opus 5.5 is on top of 63%. Hermes Index basically is their opinionated index of agentic capability. Now, Opus 5.5 is their, best one, and you can get it through the Hermes portal, I believe, so you like, you don't have to just like provide API keys. and I think there is a way to get it back with your, Claude 20x subscription.

91:46 Nous Research not only introduced the Hermes agent, they also announced the fundraise of, Series B. So we'll shout out to them because we we follow this company for a long time, so, listeners of the pod, they know. as reported in the Wall Street Journal, Nous Research raised the Series B of, I believe, 90 million dollars, putting the company at just over 1 billion dollar valuation. this is, the quote from, Dillian Rolnick, the CEO of Nous. fundraise will get them going into enterprise.

92:19 I think that's incredible. So shout out to our friends from Nous Research about this, like, amazing, amazing news. We don't usually do a lot of fundraisers, but this week we did 2. so that's very interesting. And, right, I think now it's time to move to our coverage of the latest JEV and decision models bonanza. and to help us cover this, we have Maxime Labonne, let's from head of post-training at Liquid. Yes.

92:53 Maxime Labonne: Yes. Alex Volkov: Maxime, welcome to the show. Back, welcome back, man. You haven't been this year, I don't believe, but you, you know, you, you're a frequent, frequent flyer here with us. for folks who are new, we have a lot of new folks. Could you give a little bit of a, of a intro to who you are and what you do here and at Liquid? Maxime Labonne: Yeah, sure. hi everyone. pleasure to be here.

93:15 thank you for the invite. And yes, so I'm head of post training at Liquid AI indeed. What we do is that we have a focus on edge models you can deploy on device, and that's what we release, usually. so here I'm going to talk about decision models, which is new for everyone and and not usually something that we we do release. And yeah, on the side, I also have a blog and publish articles about stuff like model merging and abiteration before it was cool.

93:49 Alex Volkov: Before it was cool, man. yeah, okay. So, Maxime, we've had you on the show a long time ago. A lot has changed in the world of AI. Specifically, let's talk about, you know, decision models. So we covered TypeSafe, JEV release, obviously, when it came out, it blew up all over the timelines as well. and since then, I think multiple other models released, including this week, and s- specific, like Cloudflare released one, Perplexity released one, and then OpenAI stepped into the game with like an official multimodal decisions API. They promised it last week at DevDay.

94:22 This week OpenAI released the decisions API that's also multimodal, and you guys also stepped into this game. Maybe let's start with the basics. Maxime, what separates decisions APIs or decision models from like regular LLMs? Could you give us a brief two sentence, three sentence primer? Maxime Labonne: Yes, decision models do not output any tokens. They just take a decision from a predefined set of answers. So what you get from it is that you have super low latency answers because you do not generate any token, right? You do not have to wait for a thinking trace or anything.

95:00 So it's very fast, and now it's also very general purpose. I think if you've been here in AI for a while, you might see that and say, wait, this is just a classifier, this is just an encoder model, right? But the added value of Jev and this new breed of decision models is that they're very general purpose, meaning you can throw any type of decision at them, and they will be at least decent at it, sometimes very good, and sometimes just decent.

95:30 I think this is the paradigm shift. It's a lot of rebranding, it's true, but it also creates a lot of value, and this is why, we thought that it was interesting to invest in the field as well. Alex Volkov: Yep. And I- I- I find it very interesting that all of these APIs, are considering that it's so cheap that they're not even charging for output. So, so all these APIs, this is an API from OpenAI, Jeff originally when it released, like they deemed the output token so cheap that the more costly kind of part of LLMs, for example, which is output tokens, here is not even counted.

96:06 Maxime Labonne: The, you can't price it because there's no output token, actually. So, like, naturally, if you design an API like this, you'd be like, wait, how can I charge that? Like, no, so, like, the only choice is to just charge the input tokens. Alex Volkov: Yeah. what kind of use cases have shown up for you guys, but also that you're seeing personally for, decision APIs that a smart, fast, and cheap model, like, let's say Luna or Haiku, for example, also cannot, like, solve for?

96:40 Maxime Labonne: Everything that requires low latency, so everything that is pretty real time. So this is not a, a real use case, but if you take video games, for example, you cannot wait for Luna to give you the full answer because you need something that is very snappy. You need to take multiple decisions per second, for example, and that is not possible with large language models, especially when they have thinking mode. so it really unlocks new use cases where LLMs could not provide any solution before.

97:11 Alex Volkov: I, you know, I u- I use, decision models and and, they used to call them system one models when we invited the Jeff folks on the show, but sounds like the industry is like landing on decision, decision models, et cetera. just sending just a bunch of text and asking a lot of questions per one, kind of like, a lot of decisions per one context, and that seems to be barely taking any, any extra time. And I think for an LLM, like every other question would cause like a thinking chain, a long thinking chain, and just delay the, the time to first token, but every time for the last token.

97:47 So the, the interesting thing that I notice is that let's say Luna or even Haiku, by the time they start answering, Decision API would have already completed the response. By the time Luna even starts answering. so there is definitely a shift. and you guys also felt this at Liquid, and tell us about D1. I would love to hear about, specifically the stuff that you guys decided to release. Maxime Labonne: Yes, so we, we released, three models.

98:12 this D1 is behind an API, because I think part of the value that these decision models provide is really not the model itself, but the API. Like, is it reliable? Is it low latency? all this stuff is very important, and that's really on the inference and infrastructure side. And we also released two local models. One is a 3B model. It has vision and language as, input. And, we released a 600 million parameter model, which has audio, text, and vision this time. so it's really omni model.

98:48 Alex Volkov: You have audio. Maxime Labonne: The only... Exactly, yeah. So it's, it's really fun, like, there's a lot of demos I've seen on X, that's super cool because, like, they have a super big pipeline. They added, like, embedding Gemma 2 in the mix as well to do retrieval, and they have this omni, decision model, in the middle, to to control everything. So I think it's really unlocks a lot of use cases, and it's difficult even to understand where they're going to be useful and what we can do with them at this point. and this is why I think it's...

99:27 there's a lot of people working on it, and I'm super curious to know, okay, like we push that and to see now how people are going to create value with it. What, what is it going to be useful for? Is it just like a classifier and you can use it like for like spam filtering, or is it going to be something a bit more intelligent Alex Volkov: So you guys released a few models, right? D1 Omni. I did not know about the audio, dude.

99:52 I have to say, I prepared, there's a lot of stuff to cover. I was like, okay, Maxime will be here and tell me. like, like, what kind of audio decisions could it do? Like, does it know everything? Can I ask it like, hey, does a dog bark in the background of ThursdAI and will like catch the episode like part when the dog barks? Is that type of the stuff that the the the Omni model could do?

100:12 Maxime Labonne: So it's mostly like for this model because it's a small model, it's trained especially on, having, orders, instructions, like you talk to an assistant, and this is the kind of audio it's been trained on. you can combine these instructions that you would talk to, like a home assistant, for example, or a car, and you can add metadata with text on top of it to help the model make the decision, and of course the question and possible answers.

100:41 Alex Volkov: Yeah. Oh, that's great. and the other thing that I definitely wanted to to talk to you about, you guys have like, like very good performance usually on even CPUs. That's kind of like what you guys are known for. Just tell us about like the performance of of these models specifically. Are these, so you talked about the API, you talked about doing a very like performant time to response, et cetera, which is very important for decisions.

101:04 Like we want to make decisions as quick as possible, I think that's what Jeff kind of changed. tell us about, whether or not this is this paradigm of decision models is coming to my device eventually. Like, will will I even need an API, or will all these just run on, you know, on on my machines? Maxime Labonne: Yeah, I really believe so, because, here you can see like the 3B model, we have super, super low latency, like 8 milliseconds on a, on a GPU, so I think you can do a lot of thing with it, and indeed, you can also focus on CPU like we do with the LFM architecture and, and get like super, super low latency.

101:42 I don't know how it's going to be picked up by the industry yet, right? But there's a lot of potential to have something always on and, being able to react to everything that happens on your phone, like every notification, everything like this. It can react to it and take decisions for you. Alex Volkov: Nisten, I think you also had a, a question for Maxime you wanted to bring up. Nisten Tahiraj: Yeah, is it able to do video as well, like classi- classification of video in, in real time? And also, how do you feel about having just so many gen models just come out all at, all at once?

102:18 Maxime Labonne: video, not yet, no. I think this is something that we want to cover next. we have other models coming this month, so there will be opportunities to take care of video later. I think it's good. I think it's good that so many people are working on it. I think it creates a lot of ideas. Nisten Tahiraj: Sorry to disrupt, but if you can do 8 milliseconds per frame, then you can do 60 frames per second.

102:42 Maxime Labonne: Oh yeah, yeah, like if you mean it that way, like we just treat Nisten Tahiraj: Yeah. Maxime Labonne: every frame as a decision, right? But we do not Nisten Tahiraj: Yeah. Maxime Labonne: take entire videos as input. But this is something that we can also do. Alex Volkov: Wow. Nisten Tahiraj: Yeah, you could probably just do it in real time as well. That's that- Alex Volkov: There's stuff, as far as I remember, like 12 Labs and a bunch of other folks who, like, try to understand videos, they talked about, like, only judging frame by frame is not enough because then temporal stuff is not there.

103:08 Like, when somebody showed up, then maybe you can take a decision, but like you do need to to shove like a bunch of frames into a decision to to make like, hey, how long did, you know, Maxime hold his hand to his temple for? And that like frame by frame is not super easy. You have to like take these frames. Maxime, talk about license. Can we, can we hear about- Maxime Labonne: Wait, wait. Alex Volkov: Go ahead, and then and then and then.

103:28 Maxime Labonne: Wait, I I think this is very interesting because, one of the problems of these models is that they're stateless, meaning that every time that they get an input, they don't have, like, any other context. And like a natural extension of it is having multi-sent conversation that we have with LLMs, and you have a history of, like, previous interactions and previous decisions to make better and more aligned, judgments and decisions. And this is something that is currently lacking.

103:53 It's even lacking from the JEV API, right? so this is something that I think we'll figure it out as a community, going forward. Like, is it something that is essential? Maybe yes, maybe no. Is it something that people will find useful in the future? I, I believe that there's a lot of potential venues for research and improvements, and right now everybody is trying to, like, copy JEV, and then now that we all, like, pretty much on par. I believe there will be a lot more research and interesting features that will be added to the API into these models.

104:28 Alex Volkov: Speaking of copying JEV, question for you. When OpenAI released the, the first kind of ChatGPT API, they basically set the standard of how this API looks with the completions, LM completions API. we know for agents that's not the best one anymore, and kind of like the responses API, for example, keeps the, the memory, et cetera, in context. When TypeSafe released JEV, they really worked really hard on the, on the three, kind of core components of that API. We had Allie Labs here at the DevRel for, for TypeSafe on, on the show, and she talked to us about the null param kind of concept, whether it's kind of like a boolean, but not really because it's boolean with, with, probabilities, right? And the choice, primitive and the, the, the, the answer primitive. Do you guys stick to theirs kind of like primitives because people know them? Could you talk a little bit about like how folks will actually like interact with this API, that you raised?

105:20 Maxime Labonne: Yeah, like our API is a drop-in replacement, like for Jev, so we use exactly the, the same task and the same definitions. it's extended because we needed to have vision and audio, right? so that's, that's the main difference. But even about vision, you can see that it's already kind of, standardized in LAMA CPP. they have a blog post on hugging face as well. So, like, the, the community is trying to standardize different, modalities, but you raise also a good point talking about the task.

105:53 So right now we have, three different tasks. We have, null, we have, scoring, and we have choices. Alex Volkov: Choice, yeah. Maxime Labonne: And is it enough? Is it expressive enough to make any decision, or will we find new tasks that we want to add to this, API? I think this is a very interesting question, and so far nobody has proposed any new task, but I believe there are some primitives that might be missing at the moment.

106:21 Alex Volkov: I think so. I think very, very new to the whole world of software engineering, this kind of like, generalized, classifier, right? This is what basically Maxime Labonne: Exactly. Alex Volkov: we're getting, like a generalized classifier that can make a decision based on a lot of stuff, like we've got generalized, the, you know, transformers. Maxime, talk to us about, decision index and benchmarking. I don't know if you watched, Diogo on Latent Space with Swix, like a very famous, like, supercut that I did, like Diogo does not believe in benchmarks, and benchmarks are hard for this, maybe even harder than other LLMs.

106:58 How do you guys hill climb? How do you train? How do you compare? Like, what, what, what is benchmarking the decision models? How well something does a decision? Is there a LLM as a judge, typing? Could you talk to us about, like, the world of benchmarking in, in this world, please? Maxime Labonne: Yeah. so benchmarks are very, very new, and because they're very new, they're not very good, right? They're very narrow, they do not capture, like, real decisions, they're not aligned with, like, how people use it in the real world, all that kind of stuff. but every effort that we have right now, like from Poly, from Hugging Face with the decision index, with JeffBench, et cetera, I think it's good to take.

107:37 Like, we should try to understand and iterate with benchmarks to get something that is better quality and more aligned with, with how the models are truly used in the real world. But right now, it's, it's really a wild world, and you cannot really trust these benchmarks. I, I fully agree with you. Alex Volkov: Yeah. Maxime Labonne: Which is a problem because when you train models, you want to understand, like, am I getting better or not, right?

108:01 Alex Volkov: Yeah. Maxime Labonne: So you need internal benchmarks, which are not good either. but the way that you increase the quality of the model, I see as two dimensions. You can get more accurate at some task, or you can cover more task. Ideally, you want to do both, right? You want to have maximum coverage because this is supposed to be, as you said, a general purpose classifier, so you, you should be able to do anything with it. And for each task, you should have the quality of like frontier intelligence. You should be as much on par as possible with Opus and all these frontier models.

108:39 So I think those are the two dimensions. For quality, it's very easy, you know, like you can just distill directly from frontier models, and that will give you a, a very good answer. Then there's like the calibration with the probabilities that might be a bit more difficult to do because these models do not just return an answer, Alex Volkov: Mm. Maxime Labonne: they also return probabilities with it. So that would be like the only problem here.

109:04 And then there's coverage. I believe we do not train these models correctly right now. And in terms of coverage, what I see as something high value is having gym environments like we do with reinforcement learning. Like I'm talking about old school, reinforcement learnings, like early OpenAI stuff with Atari games, et cetera, and having thousands and thousands of environments to be really able to generalize the performance of these models and train it not just on like some video games or some like stupid classification task, but really be able to create, new samples on the fly with super diverse context and inputs and questions and languages and, input length, all that stuff is super important. And I believe this is the kind of infrastructure that will emerge, from these decision models, really general purpose gym environments.

109:58 Alex Volkov: I think... I think the world is very exciting. Dude, I, I, I know that, like, I was JEV pilled from the moment that I tested and ran, just on a bunch of prompts, and, I was looking, I was like, so much new shit can be created right now. So, so it's just, it's, it's not even the Pareto frontier that we talk about Maxime Labonne: Yeah. Alex Volkov: often, right? Like the, the smart, cheap, fast model corner Maxime Labonne: -huh.

110:21 Alex Volkov: where like now Luma lives, Haiku lives, a bunch of the models that you guys released, the 2.5Bs, like a very small models also like live, live in that kind of area. It's just, just the speed with which and the almost no cost with which you can now do tasks and classify. I th- I think we're just like as, as, as, as computer scientists, we're just like getting, just waking up to the potential. I think that's just absolutely incredible, and I, I, I think that you guys agree because this is why you released like a bunch of models very quick, right?

110:54 Maxime Labonne: Yeah, absolutely. I think it's really going to become a primitive. You see it like in SQL queries, you can just like add it like a function calling this model in SQL queries, and it's so cheap that you don't even care, right? You can call it at scale, doesn't really matter. Alex Volkov: Yeah. Maxime Labonne: so yeah, lots of super interesting, things to discover. Alex Volkov: I'm, I'm super excited about the fact that you guys, have an Omni model specifically that has vision. that's for my use cases, which I'm gonna talk about on the show, very soon. I think it's very interesting, Maxime, so I'll definitely check out, first of all, on Hugging Face, but also you guys do have an API, so folks who, who want like the latest and don't wanna host it.

111:29 But hey, a reminder to folks here, if you wanna try this out and, and you don't have your GPUs at home, you wanna try the open source and not the API model that that Maxime mentioned, we just told you about how to get some free GPUs from CoreWeave, which is, I think, is absolutely insane. So, scan the QR code from before, or go to forge.coreweave.com Maxime Labonne thank you so much for joining us on ThursdAI. It's very exciting to keep the tabs with what you guys do, specifically as kind of the the world of AI progresses. Peter, you can come back as well if you want, it's pass your time, Maxime, so feel free to also drop.

111:59 Maxime Labonne, friend of the pod, everyone. Maxime Labonne: Thank you. Thank you, everyone. Alex Volkov: Bye-bye. All right, as we close the show and we chat with Maxime about models, I think it's very cool. The Omni models is specifically very cool. I'm looking forward to, like, the video classification Nisten, but yeah, I think frame by frame is actually working well. and yeah, we got Florian last week or a couple of weeks ago when we talked to Florian.

112:24 Florian, shout out, the doing JeffBench, doing crazy work at JeffBench. Thank you, Florian, for that. Folks, check out the JeffBench if you're not sure which models to use. we didn't cover any of the multi-model stuff, but Black Forest Labs launched Flux 3 and Nano Banana 2.1 from Google was released, and folks, all of our infographics this week were Nano Banana 2.1, and it's really, really good. The only highlight there that I will say is that, you should use the high reasoning, stuff, because on the medium reasoning, Nano Banana is not that great. it's been a while since we had a lot of time to cover, just like multi- multimodal stuff.

113:04 We're 2 hours and 5 minutes into the show. Hosts, co-hosts, please, anything else that we haven't covered? Peter, we covered your big news with, with breaking news. anything else that you're playing with? Nisten Tahiraj: he's like, I built an entire city with, a hundred thousand, little gevs going around, and I haven't opened the beta yet because somehow it's, it's not crashing. yeah, I just, I just built a whole city.

113:35 look, it might, it, it might crash, obviously, but, so this thing is, let me see if we can do this. So there are 100,000 agents going on in this city. Alex Volkov: 100,000 agents. Nisten Tahiraj: 100,000. So we can, we can actually set it here too. there's a whole bunch. I don't know how it's still holding together, but, yeah, it's a full emulation of the city of Toronto. Each of those lights, those are people, and you can talk to them, and, you can have like an actual chat with them.

114:12 Alex Volkov: Wait, listen, Jev doesn't output text. How, how is your, how is your JEV agents answering? Nisten Tahiraj: they're all just using the same 2B model that also runs as a JEV. So right now I just told it to implement Maxime's, model as well, so you can see where their, their decisions are being done. And, it's, it's just endless. The day lasts about 20 minutes, and 100,000 of them have to tweet, and there are 400,000 decisions to be made.

114:42 At the same time, you can also, you can also just, go around as a, just go around as a Spider-Man. So, there are, I'm just planning to have like 100,000 people Spider-Manning around the city, and yeah, it runs pretty fun. It's a full simulation. By the way, when they talk, they just use like our 27B model, so they're actually pretty, pretty funny, to to talk to.

115:13 So I'm streaming from Linux, and the mouse is a little, is is is a little bit weird, but yeah, so if I'm just gonna open it up for the public and for people when they, when they sign up in in the beginning, you can just point your agent to it, and I had Meta Muse just go in here, and let's see how much money Meta made. So Meta just called itself Blob the Builder, and they just ended up buying a whole bunch of buildings, and you can see just Meta.

115:41 Alex Volkov: You have a functioning economy in this, in this, in the city. Nisten Tahiraj: Yeah, yeah, yeah. There's like a full real economy. You, you can buy floors, like you can go to buildings, you can build lots, they can build their own. Oh, Meta built a Denny's. I told Meta, just, just make me a Denny's. Alex Volkov: Nice. Nisten Tahiraj: And, it made a Denny's diner. I think I know where in the city it is.

116:03 Sorry, it's just a bit clunky because I, I am streaming this, from it, but, yeah, yeah, there's a full functioning economy. It, Careful is, is actually very addictive, and, you can convert units to factory or retails, but, retails doesn't, doesn't, doesn't make any money. So yeah, I'm trying to do the matrix, basically. Alex Volkov: Yep. Nisten Tahiraj: And for agents, and I'm surprised it is still, holding together, given that there are like 100,000. And you can see it's pretty funny in the morning, they're all go- going down the elevators and stuff, and there, there are 3D ones that are, they're not walking the street now, but yeah.

116:43 Yeah. Build it. Alex Volkov: This is insane. Nisten Tahiraj: So it's gonna be running 100, it's gonna be running 100,000 of Liquid AI 600B, because that one is a, is, is a lot, is a lot smaller now. Alex Volkov: Wow. Nisten Tahiraj: And yeah, so we have, there's a, yeah, there's a functioning economy. there are stores. The most addictive thing that I self-addicted myself. Alex Volkov: Yeah, yeah, we we it's clear to me. all right, so LDJ, you wanted to, you found us a few things. tell us about this.

117:15 I'll post, I'll show this off as well. LDJ: Yes, so there's been a an interesting new development of this people realizing the models are really good at porting games into different platforms, at modding games together. There's a variety of different techniques that people are doing this through, that some of the techniques are more impressive than others. Alex Volkov: Yeah. LDJ: But overall, I mean, the whole thing is impressive. right here, this is Red Dead Redemption 2 running on an iPhone 18 Pro, and actually a pretty decent performance. And this brings back my old dreams as a kid of just like imagining how the PSP might look like in the future and what you'd be able to run on it in the future. Unfortunately, we don't have the PSP anymore.

117:58 But, if you look at the, the other links that I sent out. Alex Volkov: the- so just to be said, like, the modding community was vast, but it usually was like very hard for folks to mod anything with the quality of the other game. But what's happening here? LDJ: This is Spider-Man inside of the Batman game, and the, the web mechanics work, the, the, the fighting mechanics are a little bit wonky, but they still work pretty well. this is using a bit of a less impressive technique called, like pass-through, where sometimes when there's a, a like a Batman game object that should be occluding Spider-Man or like should be in between the camera and the, and the character, like sometimes it ends up wonky and it basically looks like Spider-Man is just on top of everything.

118:41 Alex Volkov: This is LDJ: But Alex Volkov: a playable game, right? There's not like a video model that just like puts LDJ: Correct. Alex Volkov: this person inside. This is like an actual mod that, that folks can install. This specifically got like 7.6 million views, this video. Wow. How did I missed it? My algorithm is not like showing me these. This is super cool. LDJ: And the last thing I wanted to share is the, the... you can do the click on, yeah, the last link.

119:07 this is Minecraft Alex Volkov: Oh. LDJ: in GTA. Alex Volkov: Wait, what? LDJ: There's also Alex Volkov: Minecraft LDJ: My- Alex Volkov: in GTA. LDJ: Yeah, there's also Minecraft in Skyrim and in Minecraft in Elden Ring that they did. But the TNT works, the fireworks work, the, like, yeah, just look at any, I think he's gonna set down some TNT here. Alex Volkov: All right, this is LDJ: And, and he can even, at least in the Elden Ring or Skyrim one, it's even demonstrations of actually building with the Minecraft blocks in the, in the Skyrim world and, and flying in the Skyrim world with the Minecraft Elytra.

119:42 It's insane. Alex Volkov: This is insane. we're basically talking about, like, AI can do a bunch of stuff, right? one guy that rebuilt the whole Adobe suite. Do you guys see? Nisten Tahiraj: Let's go. LDJ: Oh yeah, it's called a Photon or photo something. Alex Volkov: Photocraft. Nisten Tahiraj: It's a LDJ: Photocraft. Nisten Tahiraj: Photocraft, yeah. Alex Volkov: Yes. Okay, so let's just like, let's show this a little bit. I think it's one guy re-implemented from scratch, and I am doing the air quotes here, all of Adobe products, free and open source.

120:12 Essentially, it's called Photocraft for Photoshop, Vectorcraft for Illustrator, Filmcraft for, for Premiere Pro, Lightcraft from Light, you know, I don't remember all my Adobe, off the top of my head. all of them implemented in Rust, all of them look super quick, and supposedly the folks are saying that this is a clean room. You guys remember the clean room thing where you basically just like recreate from scratch, but folks are saying that there's probably like a cross decompilation thing happening here, where this was like, just like decompiled and then, and then rebuilt in, in, in something, which is absolutely crazy.

120:46 and I think we'll end on the show on this. We There's a lot of stuff to t- to talk through. Guys, if you missed any part of the show, ThursdAI is live, is is happening live on ThursdAI Live, but also, is then released with a newsletter and a podcast that you can check out. And also on our YouTube, if you are watching us on YouTube, please hit that, subscribe button, because we are releasing segments of the show, that, we are agentically building for you with our assistants, and they seem to perform well.

121:17 People do, people who don't have time two hours, they, they seem to enjoy those segments as well. With us, Peter Goster from Arena. Congratulations again on, on the fundraise. Yam Peleg Nisten and LDJ. Our friend Wolfram is on a well-deserved break. We will be back here next week. If you are coming to AI Engineer in New York, please come and say hi. I will be there interviewing a bunch of folks and then be back at home, probably doing the ThursdAI, or maybe from New York.

121:44 and if we missed any part of the show, please let us know in comments if we missed anything that's very, very important to cover. the last thing that we didn't cover is Brett Adcock's, Hark Pro, but maybe we'll cover this next week. Thank you so much for tuning in. Thank you for, checking us out. please go and check out the Severus GPU, that we are offering. It's free now. Why wouldn't you? It's super cool. Just put the D1 in there.

122:08 Nisten, you should try it, and then give us feedback. We would love some feedback as, as we're like planning this, as a major product release. Nisten Tahiraj: How, how many you got? Alex Volkov: Let's see, let's see if you can break it. all right, folks, thank you so much. With a bit over 2 hours on the show, thank you all for joining. we'll see you here next week. Bye-bye, everyone. And I have an outro. All right.

122:35 Bye-bye.