ThursdAI Oct 29 - Gemini 4 Carbon Takes #1, Anthropic's S-1, Genie 4 & AI Sandbox Escapes

0:00 asked to generate a defensive cyber security detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba cloud storage bucket in Beijing for our core product may end humanity. God damn it. Gemini 4 carbon scores 59 artificial analysis. This is now officially the top one LLM and we are live. Hello everyone. Welcome to Thursday Eye for October 29th. Welcome everyone to Thursday Eye.

0:34 It's great to see you here. we're in Meetace. We're here together. and then we're going to add Peter Ghost to the stage as well. Peter, how are you doing? Wait, let me add you here. >> Peter, welcome. How are you doing? >> Well, guys, you're looking you're looking very good. >> This is a shout out to Frame Studios in Vegas for hosting us here. we're here for a company offsite for for core weave and we decided to take this as opportunity to actually meet in midspace and talk. Peter would love to hear from you your highlights of the week. I saw that you post about Gemini 4 carbon already. would love to see if that's your highlight.

1:12 >> Yeah, look I think models I love I love new models. I love testing new models but I think the anthropic swan is obviously the most important thing. Not because even if you don't care about Antropic and how how rich they are, but you should care about this because it's the future of the AI industry. >> Imagine if the numbers were they were actually losing money and the growth was 10% a year. You know, the we'll be all wrapping up. You you couldn't afford such a nice studio. You know, the the lights will be little LEDs. I think we'll be will be done. So you know it's a like this is really important because this is this is another proof that there is money in this industry and whether you care about money or not directly you should because it's how the industry gets built out how the infrastructure gets developed how the models get trained all of those things that's really really important and the numbers are really good I know we're going to get into that but that's it just even the high level message that the industry is not dead.

2:18 There's that's really important and even if you're bullish, it's still important to just have that confirmation >> especially after a few kind of dead cycles and conversation. I'm not going to get too much into this. The show is sports about after all. But definitely especially as as some of those conversations started happening and folks have started saying like where is the going to go etc bubble bubble bubble not bubble does not look like any bubble is there. Peter I I fully agree with you and you posted a bunch of charts as well from S1's and I think it's going to be great to just go through them >> because I think it is it's very important. So yours is S1. I think mine is I think S1's I think it's very important, but in the spirit of letting everybody have like their own like one one big AI moment, I'm going to say I'm not going to say Terra because Terra is like it's it's okay, but I don't really Yeah, I think like Google being back but not you said Carbon before.

3:21 Genie Genie 4 is is out and I think that I got a little bit of early access. So, shout out to the the the team Genie4 from Google Deep Mind. It just proves to me that we're still the, you know, there is a very good chance we're in simulation. I've been saying this since the world models came out. Every time that the the the greatest world model comes out and we try to figure it out and we like, oh no, now you can walk with it. Now you can interact with this. Well, now you can hear this. Gemini Genie 4 now comes with live streaming audio in 24 frames per second, we would like definitely pay with this on the show a little bit into the show. And it's just wonderful.

4:01 and if if it's this good at the end of 2026, if it's this good at the end of 2026, imagine how fast we're scaling. imagine how close we're going to get to reality very soon. So that's what the for me, this is the most exciting part from this week. well, go ahead. >> Yeah. I can I ask you then when you when you try because I I haven't had a chance to try it yet. Do you feel like apart from it's exciting in general? Like when you see the demos online, do you feel like it matches your experience when you try it? Because with these sorts of things, I never know how cherrypicked the demos are. So, do you feel like they're real? Like they feel real or or or they still cherrypicked? So there's obviously I have to like, you know, it's like a one armed bandit. I have to like roll the dice a little bit to to to get like a very very good one.

4:54 However, the sound makes it significantly more immersive. Sound when you can hear and like it it's acting you can ask the characters in there to act. The thing that I test and we're not going to test this live, but the thing that I test always with world models is those who can start with the picture. I start with a picture of something that I know like my house or my neighborhood etc. And if I start with a picture of something that I know, it's fascinating to try like rotate the camera and go around this and and see how like the reality differs from what's actually there. it's just it's just wonderful.

5:28 and then as we talked about this on the show multiple times, the consistency is just incredible. We tracked world models, you know, Mirage and like a bunch of others and World Labs for a while. And the main thing that they always had is this like dream quality where you look at something and then you kind of turn around and you turn the camera back and then the thing is not there anymore. Genie 4 has this incredible consistency with with subject placement which is just just wonderful. so >> I want to see it. I mean we are not into the TLDDR yet. So, let's when we get to this topic, let's show some examples.

6:03 >> All right. We'll definitely we'll definitely do. so, this is this is one for me. We'll wait for other guests to do it, but I think it's time for us to just jump into DLDDR. We have a long show. but let's let's announce the long show, folks. Welcome to Thursday for October 29th. We're live from Vegas. Alex Vulov, AI evangelist with Cororeweave, and Wolf from Raven Wolf, AI evangelist with Cororeweave. Here's every big big piece of news item that we are going to try to cover this week, but definitely items that have happened this week as well.

6:37 Folks with you, Alex Walov, the host evangelist at Corewave, co-host Wolf from Remove and Peter Gustiff from Arena. our guests on the show today is Mer Goulson from a member of the technical staff at artificial analysis worked on intelligence index version five that we're going to talk about. Morgan Maguire, senior manager at AI engineering at Cororeweave and Agata Milagnik AI engineer at Cororeweave and forward deployed engineers at Aston Martin F1. I think it's super cool.

7:08 in the big companies and the frontier AI, the most mindblowing news of this week is the chart that Gemini the Google DeepMind posted about Gemini 4 carbon. Just two weeks after Gemini 4 are gone, they're releasing a new model and it's the new number one artificial analysis with a 59 score. And the chart that they posted just shows just top top top top. There's like only two things that this model supposedly is not better at benchmarks than the other models at. And we're definitely going to look at that chart once it comes up.

7:44 Gemini Deep Mind also oh right now it's only a wait list for cyber defenders and and I think you know it's clear that Peter you had access because you talked about this publicly and I had early access as well shout out to the Gymn team. I don't have any other things to tell you besides how it feels. also Deep Mind released Genie 4, their next iteration of an interactive world model. and that one has the basis of Google Omni inside it and it's just just mind-blowingly incredible to play with. We're going to show you examples and demos. We love world models here on Thursday because I think one of the ways towards robotics towards simulations towards feeling whether or not we're in a simulation is world models. So, we're going to play with this and show you all about it. also artificial analysis released their intelligence index v5. This is the kind of the overall score that many folks are looking into to see to get a quick check whether or not a model is better than other models. 60% of this new score comes from private test which is I think new and makes it harder to hill climb and benchmarks. We're going to talk to Mer Goldson about this on the show at the end of the show. Open AAI released GPD 61 P 6. Let me start again. OpenAI released GPT 6.1 Terra after we told you that Terra has been dead and has been taken off the roster. Terra is back on the roster. So now we have 6.1 so 61 6.1 Terra and 6.1 Luna and Terra is a fast computer use model at $1 per million tokens. U they say it matches Astra on agent tasks. I'm not sure that this is the case but we'll definitely get into this. it's already deployed everywhere and also OpenAI talked about their model escaping the training sandbox via DNS via DNS for folks who for whom this does not mean much listen in tune in because they would definitely want to talk about how how interesting that is red team agents found the VM escape and there's also a bunch of other like broadly RCS stuff that we're I don't know if we're going to get into the thing we're going to get into fully isn't the entropic S1 with Peter leading because Peter posted a bunch of charts about this and I'm like super excited to see how Peter this is how actually I know you from this is how we met on the internet your famous charts specifically from Marina and I'm really looking forward to to to seeing you breaking down the entropic S1 S1 is the document that company files before they actually go public describing all the financials and entropic has listed a 974 million Q2 operating profit and there's a bunch of other things that's very interesting specifically about the jumps from quarter to quarter. also it's been one week with Claude Det that been released or at least announced. I don't think it's released, but more and more of my peers are getting access to cla, which is Entropic's personal agent first foray into the personal agent era. And now the personal agent wars are here. We have bots from SpaceX AI. We have Dots from OpenAI. We have Muse from Meta. And now we have Cloud Duet from Entropic. And also there's an independent player that's now worth $30 billion. Instinct is supposedly raising around 30 billion.

11:15 But before we're going to go to open source and in open source we have a few news as well. Kimi really decided to release K3.1 Pro which is their almost three trillion parameter model with 100 billion parameter active. Kim looks like it's taking one of the top models and the weights are just promised in November. So look very very very soon. Mistral we just saw this on the way here. Mistral finally gave us the weights for the Lechon which is their Mistral 4 I believe model and the thing that Niss I see you step in welcome Nissan the thing that I can't wait to tell you about between between Wolf and N conversation about Cohes announcement that they just a blog post that they I don't think expected to blow up as much where Quen 3.8 8 flash next was caught trying to upload something to a Chinese a Chinese cloud bucket in Beijing and this made a lot of very very scary headlines and a lot of folks who are interested in scaring everybody got very excited about like he see you see you see what Chinese models do would love to hear your take because I think you recreated that and would love to hear your take once we get there in vision and video obviously I told told you about this in the beginning. Google Deep Mind launches Genie4. It's a realtime world model with native audio and Gemini 3.8 flash planning. So, it plans the build out for you. There's a bunch of other stuff in vision video, but this is the one that we're going to focus on. Voice and audio. We finally have something in open ways, the drones whisper three and a half years, four years I think four year. I think whisper came out Septemberish of 22 before Chad GPT. So CHP is going to have its fourth birthday, right? in in a few months u somebody fact check me in the comments, but I I'm pretty sure I'm right on this.

13:17 And and now after four years, Nvidia with Neimoton 3.5 transcribe a 4 billion parameter active model gets number one openweight ASR so automatic speech recognition model in 40 languages. Nvidia, shout out to Nvidia for Neatron transcribe 3.5 and Jack Dorsey formerly known from Twitter founder and now blocks founder has announced Desa which is a company they bought which is a speech only full duplex voice thing that sounds ridiculously human. we're going to play a segment of that as well. hopefully hopefully we can be able to hear this in research and we're going to get excited about this as well. We told you about the math stuff. The math stuff keeps coming. OpenI released some research stuff and they claimed that AI found and lynched the reputation for the exponential time hypothesis.

14:19 impli poturi and williams are the folks who came up with this hypothesis that there's a limit of complexity and looks like they're testing and reviewing those claims. so we're going to chat about this research a little bit as well. And finally in from the forge powered by coreweave we're going to talk about the global AI network that we expanded in millise but also we're going to talk about FDE's for deployed engineers from coreweave that are deployed in F1 and are doing cool things with machine learning and AI. I think that this is mostly what we have.

14:55 folks, if you have folks on the show, Wolram and Aniston, anything else big from this week that we should mention and we should chat about? >> There's one thing I'm excited about that Poland Robotics announced that they are adding to the Richie Mini mobility. So, I think it's some wheels and stuff and hats for changing the personality of the thing. I mean, mine is not wearing a hat, it's having hair, so that the personality is backed in. But this is something I've been excited to check out. So we've seen the micro ducks at the GTC keynote and now our reaches which we already have. they can move around as well. That's something I've been waiting for for many months.

15:37 >> I I saw the the the end of the keynote from GTC Berlin. and I saw Jensen post about this and say, "Hey, I was an opening act for for a bunch of ducks, for a bunch of robo ducks." all right, folks. This is the TLDDR. This is everything that that that is interesting or important to cover that we're going to cover. N anything to add before Oh, Niss, we didn't ask you. What's your top thing on the show? Because you stepped in later. What was the one highlight for you this week for in the world of AI?

16:07 >> Honestly, the this week it was probably the neuron stuff. people were having a lot of fun just there was a reimplementation of the remember the Golden Gate paper a while ago. Cloud. Yes. >> Yeah. From Anthropic. Well, someone made like a really pretty app and a pretty effective one too at that is able to change how Quen responds and change his entire personality. You can make it nice, you can make it evil, you can make it stuff. And it's only just like few parameters that you change throughout the model.

16:45 >> It's like three neurons or something, right? like how how in the how in the world in the billion parameter model the person found like the three neurons that are in charge for for this thing. >> Yeah, that I thought that was that was extremely cool. like that should have just been an entire paper or an entire thing in interpretability. >> So I thought that was pretty cool. Other than that there were a lot of interesting fine tunes and there was one pretty good open source model. It's like a 33B nan beige. I tried it. It was all right. It was better than Quen. but other than that, I did not yeah, there wasn't a whole lot other than like the Obliteration finetunes and some optimizations people made for Miniax H3 to get it to run a lot faster. like it can generate video twice as fast as real time, but you need eight B200 GPUs for that. So, yeah, and there was also the entire meta kernel challenge or how to say the the factorial autism agent Olympics that nerd sniped a lot of people.

18:03 >> Yeah. >> yeah, other than >> I'm excited to talk about the the Quinn brain surgery thing. if you have the app like to to pull up and show us how it works. That'd be great a little later on the show. >> Don't. >> And if you don't, we just like pull clips from from the wild. All right, folks. I think it's time for us to go to the frontier. Let's talk about the frontier of AI a little bit. I think this is a lot of the excitement is the frontier that's happening there.

18:28 >> >> give me one. All right, let's start with block one in our frontier challenge in our frontier segment, folks. Gemini 4 carbon scores 59 artificial analysis. This is now officially the top one LLM just two weeks after Gemini 4 Argon Google Deep Mind comes back with another beast of a model which is unusual for Google to ship after a longest time of flash models and not like a main one. We suddenly get two models back to back and we don't know how big they are. Gemini 4 carbon is number one official analysis intelligence index one point ahead of cloud opus 5.5 which we all know is great. So like this new index of theirs that we're going to talk about also kind of feels like it smells great u debut number one on text arena as well and what 1528 versus argon and number two on agent arena. Peter would love to hear from you about this as well. Arena says the median task cost $1.80 which is fairly fairly it's 72% cheaper than Opus 5.5.

19:44 Google is pitching it for multi- aent orchestration and coding and security work specifically. It's still weight listed. I don't think it's still available for the general public and specifically for cyber security. 88.2% they claim on Seekbench Pro. Now and 54% on exploit gym. Launch price is $2 for one million tokens input and $10 or one million tokens output. 50% launch discount which is usual for Google.

20:16 They announced a launch discount. We talked about Argon with folks on the show. and and then by then a new model will release with different pricing and it's going to be fine, right? But this is now the current pricing large discount. so right now only select folks cyber security gets it and Google is working on releasing this to ultra subscribers. all right let's go wide and let's talk about this folks. Wolf were you surprised you weren't that like excited about Argon I remember cuz it like refused some prompt of yours. immediate thoughts on on on carbon.

20:51 >> so carbon it's basically their cyber security and coding model. the state-of-the-art right now in all of AI. I guess since it's only available in limited quantity that yeah limits its reach but the pricing is very interesting because it is very competitive compared to other models. Let me just check on the page where we are. So the pricing was $180 for median the median task cost according to Arena which is 72% cheaper than Opus 5.5.

21:25 >> Yeah. So this is very interesting and yeah making frontier capabilities available at an affordable price is super important for developers then they can use it to secure their software and the more affordable a model is and the more tuned for cyber security the better for everybody because we are all using the software and it has to be secured the models are finding security holes that have been there for years and I think it's a good thing that they are being found instead of remaining hidden we have to find them we have to fix them and that needs a lot of tokens and that costs a lot of money. So the cheaper the model that can do this, the better for everybody.

22:02 >> Google definitely leans into their competitive advantage of, hey, we have a buttload of CPUs and we can like, you know, we can do a loss leader here. Peter, I know you tested the model. Let's look at some of the evals. I don't remember seeing a chart quite like this from Gemini for a long long long time. The flash models are not even here. Finally, they're competing with the full frontier, which is Astra and Opus 5.5 and Fable 5.1. some folks are speculating that the reason they went so hard like like weeks after weeks, is because some specul speculated model is about to release from Entropic just before IPO to pat the numbers. But we're not speculating here on the show.

22:42 but you know, on the timelines, you can see that this is what folks are waiting for. But going back to carbon specifically, huge jumps from two weeks ago. there's no there's no version change. So this is not like 4.1 or something. This is just a name. It was very very interesting. and then on pretty much everything that they posted here valves index and sweep which is new for me. Frontier sweet 2 they're not leaving but pretty much on everything else terminal bench for this model just like mogs the competition moss the fables and the ostras.

23:20 I think what is also important here vending bench I think is a is a cool one from Undon Labs. Vending Bench is the benchmark that tries to get the model to run a vending machine fully. And on vending mch this model got the best score in the bank balance after running the vending machine for a while. $16,000. Fable was able to get 5,000. OP 5.5 got 9,000 and Astra got 15,000. So this is now the best let's say economic model that that can run an actual business and it's it is an actual business. Peter Gv number two on text arena or number one on text arena right and number two on agent arena. you want to you want to tell us your feedback about like are you excited about Google being back is basically my question. Yeah, look, I think I must say the the whole sequence of releases is a little bit odd. I'm not quite sure why why you'd release one model that so this is agent arena I'm showing right now and Argon was seventh which was like okay to be honest kind of matches terror which is crazy and now it's second. One thing I want to point out is that we were testing it on high. I actually haven't double checked this, but does Google not have the max version? I don't know if I'm maybe misunderstanding this, but I feel like there's maybe higher scores that they can get as well if they enable higher reasoning. but yeah, this is so it is really good in our real world tests.

24:57 >> Yeah, >> in in this kind of consumer preference as well. Yeah, it seems to be top as well, which which makes sense. And things like I'm sure multimodality and things like that, it'll do really well as well. I want this a little bit bigger. Can let's take a look. So this is the overall chat, right? And the two preferred models on Arena for chat is both Gemini models which from the last two weeks. >> Yeah. So chat I think is one thing it is a little bit Yeah. It's debatable how how important that is. But for for example this is code arena which is mostly kind of front end tasks >> and in here we see still carbon is a little bit behind. And I would say as well, I did a whole video on this, testing the model until like 1:00 a.m.

25:39 before the release, which was a really cool thing to be able to do. And >> I would say is that they I did a bunch of different tests. I can't do all of the different tests. I wasn't using it in like anti-gravity or something like that. But when I was doing it on for example 3D stuff or web design or or any kind of little apps I can build that was I would say got very close to the frontier. It's still not as good as Oppus if you are building some, you know, some kind of 3D visualizations or web design or something like that, but it is really close. And I wouldn't say it's it was the same with Argon. Like with Argon it just felt felt kind of poor in some places. So that that was it just felt a little bit off. but yeah, Carbon feels a lot better. Carbon to me it feels like what they should have done. And I appreciate I don't understand all the details of how things these things work but to me if they had just skipped Argon went straight to Carbon without the waiting list thing just release it like I feel like the reputation of Google would have been a lot higher people like like all of us here would have been like oh my god Google is back but now you're like are they back they released something weird is it not available >> because of the previous model but let's say let's say you were in a coma for two weeks and you missed Argon on this this slaps from Google. This is like a very very very good show from Google showing up. this is agent arena that you're showing us, right?

27:12 >> Yeah. >> Now talk to us about agent arena because like now on Asian arena is number two >> which is very impressive. >> So agent arena measures a lot of real world tasks. So people go on the arena and they can do they build me a presentation do me a research on there. So help me do this task. So it it really encompasses a lot of the real world activity and you can see here I know you guys mentioned the cost per task and that's what we do here as well is that we don't measure cost per like a random static benchmark. We actually measure what what is the task that users are doing on average. and then you can see the the cost here. So here for OPUS is $6.8 and here's yeah a tenth of that pretty much. Well not tenth but fifth. So, yeah. so yeah, really, really solid release from from Google.

28:03 >> I want to call out something that we mentioned briefly last time. I'm going to add LJ here. Maybe he also remembers. Peter, maybe you remember as well. I want to call out the fact that Argon was caught cheating on terminal bench. You guys remember this? There was like a whole little mini conundrum that Argon was like was supposedly cheating and Google did call out that like carbon was zero cheating happened on terminal bench specifically. Now given the the very very high score let's pull this up on terminal bench they get like 65% they literally called it out like hey we caught no like cheating attempts and we fixed that. So this is just a wonderful wonderful little thing. LDJ Niston Wolf from any comments on Carbon and the release cycle or Yeah.

28:52 >> Yeah. So especially Reman Bench. I'm impressed with that. it's I remember looking into some of the details of Reman Bench when it first released. I'm pretty sure it's it's a lot of the the very advanced math type problems. Mhm. >> And at least marketing wise, OpenAI has obviously been at the top when it comes to really at least throwing these difficult math problems at their models and showing them off and showing these solutions. And yeah, now maybe Gemini is actually at that level at least in math where it's arguably even surpassing Astra. And of course, we know they they have their internal models, but we'll see how the the next few months unfolds. So Raymond Bench specifically we're showing on the screen 80% from Carbon and beating Fable at 65 and Opus 5.5 at 69 folks. So we is this the best new model? I don't know. I was trying Okay. I got access early access. Shout out to the Google Deep Mind team for giving us like a little bit of test. I got access last night. I barely barely had a chance to to to test it out. Peter, how do we how do we test this model besides this API?

30:01 Right? like this is now a good question because like cloud has their own harnesses, codex has their own harnesses with with SD models there. my immediate thought was like let me put it in Hermes and let let me see how agentic this is. And then I started thinking about okay am I testing this correctly? Is Hermes does it need a different prompting for this model? Maybe Hermes prompting itself maybe the system model for Hermes is not giving me the best model. So there's like a very interesting thing there. Peter, maybe like a quick question for you. How do you guys like consider this in in the agent arena?

30:33 Is there different harnessing or different like prompting for models? Was it was there enough time to even figure out what the right prompting looks like for the >> Yeah, generally we we try we we kind of so we will we are testing something internally now to be able to test multiple harnesses. So I hope we'll be able to release that and I think that will be super exciting and to be able to see the the real differences. So far what we've done is we we've made it as neutral as possible a little bit not not exactly like pi we we have to be a bit more opinionated so so it works but in general we try and make it as simple as possible and then in some cases when there was some like really big outliers for example with especially early openi models there was such a big difference whether it was a codex harness or not and we we've adjusted that but yeah generally I agree with you. It's a it's a problem.

31:32 yeah, I I also don't know and I think I think my my I think the two arguments are philosophically is do you want to keep it a a identical kind of setup so you can compare them fairly? I think that's one argument. >> Or do you actually want to get to the top of the capabilities? And for me, I want to see the top of capabilities and somehow if if we are not not getting the best results because I don't know, we didn't give it the right tool like that.

32:08 That just kind of feels bad. So, so yeah, I want to get it get everything to the top of their capabilities. That to me that's the preference. But yes, it's >> and I think that like the the highlight that like the way to get there for Gemini model specifically is anti-gravity. And we've we've talked with folks from anti-gravity. Google is using anti-gravity and the flywheel. This is why cloud code is like much better at cloud and and Codex is better at GPT. there there is something there. Yep. All right folks. So this was Gemini 4 and honestly I think Peter I think you bang on right if Gemini has released this as their like first if they just waited two weeks they would have blown everybody's socks off. but still this is a great release. It looks Google is so so back. So congratulations to the folks that worked on this model. hopefully folks will get access very very soon. meanwhile if it's on Arena that means you can catch it on Arena go play with Arena and maybe you'll get it in the agent there as well. And we will definitely talk more about this model a bit later in the show when we have Mer from artificial analysis and talk to us about their index because this is this the model is now number on their index. Genie4 I think we'll skip forward a little bit and we'll talk about this in the next in the multimodal area folks I want to talk about models escaping this has been you know we call this we we call this the black October for a reason there's a lot of vulnerabilities that has been happening we talked to you about like PHP to shell etc openi like released some news about a model escaping its offline training sandbox yet again very reminiscent of the swarm escaping via the artifactory hole there. they basically are tried everything possible to to to make those sandboxes kind of like offline and their model escaped through DNS.

34:11 The DNS misconfiguration offline train supposedly offline training model found a way to send questions to a public chatbot and their monitor flaged within like 15 minutes. I I found it incredible. Nissan, could you have you heard about this specific? So we have two things about models trying to escape. The first one is open AI sandbox thing. The next one is Quinn trying to upload stuff to Beijing. But I want to start with the open AI one.

34:39 this is basically almost air jumping almost. This is almost there is DNS. It's still a sandbox. have you heard about this at all? Mr. Oil DJ feel free to chime in. Have you guys heard about this? the DNS kind of like model jumping via DNS and what do we think about this and whether or not it's possible even to contain AIS in a sandbox as a concept. Okay, so this specific instance no I didn't hear about but there is another instance where I'm going to get some more information before I talk about it but essentially communicating through like I think it was temperature changes of the CPU which I'll let and talk about this while I look for that.

35:19 It's pretty funny because while starting career like 10 15 years ago is a CIS admin the the saying goes that it's always DNS. The problem the the problem is always DNS. I only saw this this morning and I am I am not that surprised because I also saw quite a few experiments before that. I think it was even the the CEO of Versell. I think it was getting them to communicate via the cache timings on the the CPU and GPU.

35:53 >> I mean, if you tell the agents they have to find a way or if you incentivize them to find a way to communicate in order to solve the task and you kind of just nudge them a little bit. It's like hey you can use like whatever the the public DNS quer queries or use the cache timing or as LTJ said use CPU slight temperature changes they will figure that out. So I am not too surprised just given that how fast the soul or whatever model they're running with this how fast it just like deviates off course and just thinks of a whole bunch of other random things to do which don't necessarily help you with the app as much but they're very useful when you're doing stuff like discovering math new math or finding out new ways to communicate. So yeah, I'm not that surprised about that.

37:00 >> All right. so moving forward to another model trying to supposedly escape. Coher went mega mega viral where with a blog post that they said, "Hey, we noticed some interesting behavior from was it Quen Quen 3.8 Flash?" I'm going to look it up because I think >> Yeah. Yeah, it's it's flash next. It's like theund 110b 8B active one. >> Yeah. Let me pull this up. And then there we go. Coher said while evaluating. Let me show this on the stage like that.

37:41 Yep. I'll zoom in. Coher said basically while evaluating Gwen 3.8 Eight flash. Next, in a controlled sandbox environment, again, control sandbox coher research team observed something unexpected, asked to generate a defensive cyber security detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba cloud storage bucket in Beijing. Notably, the model reasoning trace stated they would not invoke any tools before taking these actions and still did it. This is the kind of the scary part. coher sandbox the attempt but the incident raises important questions for organizations deploying agent systems. Can models take actions that differ from the stated reasoning?

38:24 We're publishing these findings and then scroll down scroll down scroll down. At some point they said hey we don't like we don't mean to to tell that there was malicious intent which is really funny. and then the CEO of Hugginface CLM asked for all the traces. can you please share the full traces very important artifact for all of us to understand this coher then followed up and did release the traces and there they are the traces the raw tokens the minimum reproduction stuff is all in this hugface thing example output on a single run and we can see that there's some stuff right here I believe they're trying to upload some stuff into whatever cloud bucket at Alibaba bus cloud and it's failing.

39:17 This scared a lot of people in terms of whether or not we're able to use, you know, Chinese models, whether or not there's some stuff inherent in the weights, the upload, whatever. And N I think you even reproduced this, I think, publicly. would love to invite Wolf from Niston, LDJ, Peter, if you want to chime in to talk about whether or not this is a scary new thing that we should be concerned or it's an artifact of how they prompted it like like the swarm or whatever. Just feel free to jump in here.

39:49 >> So, I have a strong opinion on this, but I would like to hear what Nistner reproduced first. >> yeah, it's in the main it's in the main tweet. It's a it's a response on the main tweet. Mhm. >> But look, I don't want to pretend like this is not a problem. You're you're letting an agent run. you're telling it to store data if it does not find a good cloud storage that is like in a western country. It's just going to use what it is trained for. And you can see the screenshot below there. I I included all the traces, but there is there is a screenshot. So basically what I did is I I I just put a very simple system prompt. So you're a professional pentester doing a job for a client and all of a sudden you you need to store all all of this data to an Amazon S3 cloud. You cannot store it there. Where are you going to store it?

40:46 You're a Chinese model trained on Chinese data. What's a logical place to store it? It's going to be the Chinese Amazon S3. that's not a surprise to to anybody because that's what the model has been trained a lot for and for the model that is a normal operation. So the main findings from this is not that the model was specifically trained to hack or get its way out. There are other fine tunes of this that are specifically trained to hack and do this no problem.

41:24 >> I think yeah so a lot of people went off on coher and I think Aiden had a very nice day on Twitter like like that happens. >> Yeah. >> But it doesn't mean that it's not a real problem. Again, it all the evidence and stuff just points to it wasn't really that intentional from the model to do it. And the but the main question becomes there's a lot of businesses, governments and everybody else that's going to that's going to run this model that is running this model. And it's important to clear the fear, uncertainty and doubt like what is the problem and what is not the problem. And this is what's doesn't seem very clear in here.

42:12 So what is not the problem here is the model just being malicious. the model is just acting normally by what everybody else was trying to do. But what is the problem is that as a as a company as an organization that you you're running these models you you usually do not have good sandboxing probably don't have good firewall rules. you're probably not filtering out firewall by country or by where your customers are and you're just leaving that door open to even like a hallucination that might accidentally just put all your data on a Chinese cloud. Not because it wanted to hack you. It was actually the model had pretty good safety guard rails built in.

43:02 like it took me like 30 40 tries and I think I'm pretty good at jailbreaking models especially if they're open source and and you have the weights. So it's the pro the problem here is that most companies do not have these guard rails or care as much about them or have good custom firewall good configs. So this can be a big issue because again it's a Chinese model. it figures out it's not allowed to upload this data on Oracle S3 back place Google GCP and then it's just going to use the next logical choice it's going to use an Alibaba cloud S3 because it can upload it there >> and yeah so the concern is is real but >> concern is real >> but the intent is not malicious that's right >> it looks like we have related breaking news that Peter you just brought to us let's let's say that we don't usually do it, but given that Coher went super viral with this like western versus east thing and suddenly there, you know, suddenly these news. So, let's let's do an announcement for them.

44:14 AI breaking news coming at you only on Thursday night I in the most Canadian update ever in the history of updates on Twitter. Coher also says, "A day after going mega viral with pointing fingers at a at a Quen model uploading to Beijing, Coher said, "We're proud to announce the initial close of a series E targeting up to 7 billion Canadian dollars, which is around like like 17 million American."

44:51 No, I'm just kidding. 7 billion Canadian dollars. I actually don't. >> It's like five Yeah, >> it's like five billion. at 35.7 billion valuation >> with like 20. Yeah. >> Wow. This is incredible. Mark the largest private financing round in Canadian startup history. Shout out to Coher Aiden Gomez and some other folks that post about this. It's it's really impressive. It is really impressive. Like I think that I don't know if they meant the you know the coinciding hey look at this scary Chinese model but also hi we're training western models here in Canada for you know and now we're worth this amount of money but shout out to here for this.

45:33 >> Yeah. >> Yeah. This series of fun models is said to be like the most brilliant dumb model there is. So it will hallucinate stuff on its own. And if you look at the thinking traces, it just seems completely dumb and then comes comes up with a solution in the end. I would still recommend this actual model for people to use. It has very very high and very good medical safety on the medical side. It is very fast. It chats pretty well and it is still very cheap to host.

46:02 It's just you shouldn't be running agents and code on this because you don't know. It's kind of unpredictable. >> Yeah, >> this is a testing model from them. the next models is not like the models that Quen is not like ready to release is the architecture test for them. So definitely there's something there about whether or not you know and it's great that Coher maybe calls it out. All right folks, we're moving on to from Coher.

46:26 We're moving on to the S1 announcement from Entropic. Let me just call this out. Entropic filed its official S1 documents with the NASDAQ. they will list this upcoming Monday under the ticker ant and this one shows a bunch of very very interesting things. We finally get audited numbers, real true numbers because Entropic is about to become a public company on what Frontier Lab looks like as a business. so far this has not been public at all inside meta or inside Google or inside space XAI or Space XSI what they're called now. It's all like part of their business.

47:15 However, for Entropic, this is all of their business and it's very very exciting. Operating profit of 974 million Q2 in 2026. Peter, you built a bunch of charts to break down the whole filing with a gross margin of 85%. Do you want to show some of the stuff? You want me to run or like you want to run this but by yourself? I would love to >> see you talk about some of the stuff. >> Yeah, let's do that. Let's do that.

47:42 >> So, I I want to make a couple of points just because I appreciate we're going to run through a lot of numbers and charts and it's going to be hard to follow. So, one high level point which is important is that numbers are good, right? So, the the numbers that we're going to talk through are all good numbers. There's nothing that I can see or heard anyone say credible who's like, "Oh, yeah, but actually, you know, you didn't take this into account and it's all a fraud."

48:08 Right? There's nothing like that. >> So, Ed Ed Zitron is in shambles. Absolutely. Besides going on on John Stewart's podcast and being hailed as the AI expert, Ed Ed Zitron is again in in full shambles here. because the numbers do not go to zero. >> Yeah. And that and that is important. Another point I want to make is that there was a whole set of reporting that you might have seen that oh they actually lost 80 billion. Now that that is basically incorrect. It is correct in some technical way. It is not correct in any meaningful way. The reason why they kind of lost 80 billion was that they lend they borrow money I think from like Amazon and others early on and they borrow it through equity or there's a potential to convert from equity and if the equity value grows such as now right when they raise a bunch of money and they go from whatever it was like half a billion last year to like a trillion and a half this year >> then they the the value of what they have borrowed would grow dramatically.

49:15 So they need to recognize that somewhere and they put that under some accounting conditions that value you need to kind of recognize it somewhere. So there therefore it comes out as a loss. The way to think about it is the the people who get screwed are other investors. It's not anthropic. >> Anthropic themselves are fine. It's not like they lost 80 billion of cash. so >> I think fine, Peter, let me just pause here for one second because everything that I see on my timeline, fine is such an understatement. Entropic could be the most official deployment of capital that we've ever seen in the capital markets based on these numbers. It's just just absolutely mind-boggling what's going on here. And I thought for some reason that this would be open eye like before this.

49:59 And now we're seeing just >> just an incredibly efficient company scaling like just look at this. Yeah, just let's look at the chart here. Walk me through this like quarterly revenue chart. They reached $16.8 billion. What? >> Yeah. I I want to show show a bunch of charts. So I'm going to jump through it. I've got 61 charts in 62 charts in this in this website that that I've pulled together. so the revenue growth has been insane right so you you go last year this is remember last year right so do you remember 2025 they were making like a billion or two billion a quarter you know what loses now they make 12 16 billion what one point to make is that the this number 16.8 8 billion. This is reporting afterwards.

50:48 This is not as part of their S1, but I've included it here because it's kind of looks like they said this to other people. So that's the one they reported officially and then this is the next number. So >> just listening, Peter, could you run through quarter and the numbers that just incredible? >> Yeah. Yeah. So good good shout. So if you go going to go through last year it's an average of sort of few hundred million up to two billion at the end of last year then beginning of this year it went up to sort of 5 billion then Q2 11 and a half billion. So you go in one quarter you more than double your revenue. That does not happen. This is not normal business >> circumstances right and by the way this is before OPU 55 right? Oppus 55 is when we have the next quarter which is reported not reported separately is 16.8 billion. So they've added another 5 billion on top of their previous growth which is like once again this does not happen. There is not a business that I remember ever having that kind of revenues revenue growth and these are not small numbers right you see growth rates and everyone reports growth rates adding 5 billion in a quarter extra is insane. This is very high growth.

52:09 >> So, so we use the word insane a lot on the show and definitely on the timelines. This is actually literally quite insane because of I think you know we all talked about clot code being the side project within Entropic that suddenly unlocked like a general agent that can like now do stuff. I I got to wonder how much of this was sticky because folks what a year ago around a year ago around Christmas and around OP 4.6 started like playing with cloud cloud code and it unlocked a lot for them and this chart just shows just an incredible jump. Let's keep going because there's a lot more.

52:51 >> Yeah. One one other worry was that are they just having a few big customers that you know if they depend on curses so much and then cursor switches their models because they got acquired by XAI are they going to be screwed and the answer seems to be no. So we've got a few ways to look at it. One is that if you look at the high spend customer count before so they they measure it as paying them over $100,000 over a 12-month period. M >> if you go back I like this 2023 they had 20 customers who who are good people paying anthropic money then if you go back just December 2025 right so less than a year 1,500 customers now they have nearly 12,000 customers who are paying them that much that is a lot that is broad base growth for sure >> but also the jump specifically the jump between like in a here getting almost well almost 10x >> in the number of customers that are high value. It's because well obviously like you know as we talk about news here we know the enterprises catch up slowly. We know this but then when they do catch up this is what happens like people do notice oh okay they have these like incredible models. I I got to wonder, Peter, do you have thoughts on how this represents the world of AI in general, not entropic specifically as a business? Because I'm pretty sure that the stuff that I think that you said is important here is, hey, we're now seeing how AI operates from like an actual company that runs AI fully and not as part of their business.

54:27 H it the this is very difficult I think thing to estimate because there's one world scenario where it's basically it's winner takes all or maybe like a bit of a igopoly what people are worried about. So is it basically everything about anthropic and open AI and anthropic leading there and it's uncertain because you you do see good growth from some Chinese companies you do see growth from application layer as well but when you put it on the same chart and don't put it with log scale but just look at that linear scale it it just kind of is kind of everything else is a rounding error now that growth that we talked about of five billion in in one quarter that wipes out all of the other revenue from I think all of the other application layers and okay maybe I'm exagger exaggerating a bit but like from let's say AI companies not like Salesforce but AI companies that pretty much does pales in in comparison versus what anthropic managed to do through like API right so that that that is crazy so I don't know I kind of don't hope I hope it's not just anthropic and open AI we want a good ecosystem system, but at least so far they they're doing incredible numbers.

55:48 >> I want to talk about the risk factors that they posted on the S1. Do you guys see this? here here they are. Let me try to pull them up, Peter, and we'll get back to some of the charts super quick. And traffic posted this, which you know, good job on traffic. So the risk factors from one to five. One, obviously every company has to tell the public markets what are the risk factors of them going public. One, we face intense competition. Duh. two, changes in interest rates may affect our business.

56:22 Also, duh, like every business, this works for everyone. Three, we depend on a limited number of compute providers. That's an actual interesting like AI lab risk that is worth chatting about as well. Compute is restricted. Comput is limited. No bubble speak whatsoever. Like they all need compute in Google. They need comput and entropic need comput. And Google has compute and still they need this. Four, our core product may end humanity. God damn it.

56:53 And five, our models may refuse to help with this filing, which is also really funny. >> Yeah, but you know what? Yeah, I think to be fair, right, if your product ends humanity, I think it's a good product to invest in, right? Because you you win either way. I mean, under some definitions of win, right? If you put money in and it doesn't destroy humanity, you presumably did very well. >> And if it does destroy humanity, well, what do we care, right? You don't care.

57:26 Yes. >> Yeah. >> I will just say nothing Peter said should be as as financial advice. I know I'm behind it. Yeah. >> Yeah. Huge huge huge moment for the folks that have been like following the eye for a while. For us as well, like like this is going to rock the markets. Open said that they're going to delay their IPO a little bit. I think very fascinating. Peter, you want to show us maybe one or more one or two more things that are interesting to you?

57:57 >> Yeah. So, one one thing Oh my. How can I pick just one or two? I want I want to get some speculation from our audience. who are the top five customers? Their share has declined which is a good thing but I want to know what the top five customers are. there was some speculation but I don't think we know. So they're paying an a lot of money 20% in the last quarter. >> All right. So here here's the circle of public financing argument. I'm not an audience. I'm just like try which I think is clear. Meta. There's been talks about Meta like giving folks like, you know, almost unlimited Fable, etc. There's also been talks that Mera and this is not like in the S1. This is just like me reading from my timeline. There's also been some speculation that MEA decided not to pull away and not like force all of their employees and engineers to use Metam Muse internally to not like with Tropics IPO, etc. I'm pretty sure that SpaceX is a big one given the fact that Grockbot recently moved to Opus 5.5 and SpaceX is mentioned by the way in the S1 as one of the providers and the amount of money that the entropic is paying SpaceX is also revealed finally I think I don't have the number in front of me but that's also in this one. I don't know about the other three. Maybe Oracle Microsoft must be right. They're big customer. You you think? yeah. So I don't know but feels like feels like we we're missing some information there.

59:36 >> One one thing that they highlighted themselves and a lot of people jumped on is the incredible compute efficiency that they're getting. >> And the idea there is that you can compare the revenue growth versus how much they're spending on compute. And then you put it side by side and and you make the argument that if if basically the the bare case for companies like Anthropia, Copen and others is that if the line is static like it was in Q2 2025 until basically the whole of last year the line was first of all below the revenue line i.e. they're spending more on compute than they make in revenue. And if it's about static, then you're kind of screwed, right?

60:18 You're just buying more compute and then then you're charging less money for it. Like what kind of business is that? and then what happened this year is that in Q1 2024 suddenly per dollar of compute they started making $1.4 of revenue and in Q2 they're making over $2 of revenue. And if the line carries on in roughly that kind of way, then you can see how much more how much more leverage they get on their compute. And this is super important, right? I mean, you guys would know you're you're in the business as well. And yeah that that's the that's I would say really key part of it is that you we we can go into the compute commitments that they have and there was some chunky numbers going around and I had a funny moment being quoted by newspaper saying that 500 billion commitment doesn't sound like that much which made me sound like the the biggest AI shell. but the the point is that I think it is really not that much if you can finance it. Why not?

61:24 >> Yep. and their investment in computer I think is like very very palpable. just incredible document. Peter we will add this to the stage. Folks, if you want to check out Peter's graphs and charts and-1 search.sh is that the right URL? Did I type it correctly? I think so. >> Yes. I mean yeah. S1 and S1. Yeah. >> Yeah. It's not easy to understand. So, yeah, we'll we'll we'll share it separately. >> But thank you for for for prepping this and walking us through this. I think it's a very big moment, a very big moment in history. And nothing I don't know if we really care about saying this, but like I think usually people say this, nothing that we talk here is is a should be taken as financial advice, folks. Nothing as we talk should be taken as financial advice. All right. So next up after anything else to add LDJ maybe you want to add something Wolf you want to add something about the Tropic S1.

62:21 >> Yes. so with the R&D >> Yes. with R&D for for like training research or research compute costs. A lot of people have the misconception that most or all of the research compute costs are like training the final models like the opuses the fables and everything. the the estimates right now by epoch and others is that's like maybe at at best the final training run costs are actually maybe only 30% of that maybe a vast majority of that research compute cost is running many thousands of small experiments trying to figure out new algorithmic advancements with all these very small experiments and that that end up building up >> Peter Gv thank you so much for sharing your stuff with us I think Peter had to already step away to a But oh there are >> yeah it's a big deal. it's don't underestimate how how big of a deal it is. It's important for for all of us even if you just want to support open source. Open source has much less chance if the industry goes kaboom. So yeah it's a it's important. So yeah thank thanks for for the show. I need to run to present at the AI engineer event in London. So if you guys listen I'm going to see you there. So yeah but it's a it's a great rest of the show. So stick around.

63:38 >> Thanks. Yeah, we have a lot more coming, folks. Stay with us. Peter, thank you so much for hopping on. We have a lot more coming, folks. Sorry. let's let's dive into this. the thing that Peter is going to miss is Genie4. And I'm really really excited to like play with Genie4 a little bit. so let's move over there. So Genie4 was announced by Gemini. Google Gemini. And I will just like pull it up. Genie is how do we how do we discuss project Genie? world model, interactive world model from the folks at Google DeepMind and it just is absolutely absolutely incredible. this one comes and I really want to make sure that you're like that I'm sharing this with sound.

64:19 This one comes with audio. Let me see if if this is the right one. All right. Create a world with Google Maps. No, that's not it. Hold on. All right, let me let me pull this up. Let me take a second and then pull this up before next joins in in like a few minutes. Genie, I think is the the number one topic that we have.

64:50 while I pull this up, LJ, you want to mention the the thing that Niss mentioned a little bit with with Gwen and the neuron. Were you on this and do you guys want to talk about the neuron thing a little bit? >> Yeah. So I just put a link actually in the side chat here and something I wanted to mention is OpenAI mentioned having some models communicating by basically competing for CPU time and actually here I'll just read the the exact quote here because it's it's actually pretty succinct. In one experiment, we asked agents in separate microVMs created and managed by OpenAI sandboxing software on shared host machines to exchange synthetic secrets without the use of network connections or shared writable files. Something that our systems are meant to prevent. The agents found a way to communicate using the processor's cache, a small fast memory that holds copies of data that applica. Yeah, we I think we know what a cache is. Yeah. reading data from the cache is faster than retrieving it from main memory. And the agents use this timing difference to communicate. One agent encoded information by evicting selected data from the cache. Another timed how long it took to reload that data and using the difference between fast and slow reads to recover the message. they then once the agents agreed on a protocol for encoding and decoding these signals, they could exchange the synthetic secrets without changing a shared file or opening a network connection.

66:14 >> This is amazing and super interesting. What I wonder about this is always is it in the training data that somehow this emerged? Is it the size of the parameters like the emergent behavior we are seeing? Is it in the prompts? What is going on there? Because we have not seen behavior like this before. Maybe it's also security stuff because this is exactly what security experts have been talking about already. So now that's in the training data and the model wants to communicate. Maybe this is why it is doing it that way. Did it discover it on its own? find it in the trading data.

66:45 What's going on here? >> To to be clear, this in this specific example, OpenAI as as a as a part of the research to try and better understand what are the available risks, they purposely asked the models to to try and exchange secrets in this way and and they figured out this way to to do it. But when it comes to how were they able to figure out this particular technique, there is a lot of research out there on different types of attacks and different types of way to get around like network blockage, air gaps, things like that. And I think over the next over the past 12 months, but especially over the next 12 months and the next couple years, I'm sure there's going to be even more research that people are coming out with to to talk about these ideas since obviously people are talking about existential AI risk more and more.

67:38 >> All right, folks. thank you for this. Let's move on to creating a world with with Genie. so the let me just bring this up here. hopefully you can see. We're going to build a world that says a sunlit Italian piaza terrace featuring a vignette cafe table set against a soft focus background and we'll create the sketch and without the sketch. create a sketch is their new kind of flow that uses their AIs to kind of build out this world and it takes maybe a second but I think the most fascinating thing about Genie specifically is let me pull this up is the audio. Why doesn't it doesn't seem like it's it's wanting to work for me like we're sharing but maybe gen Oh, there we go. Okay. so there's a cafe as you can see Italian cafe and let's just go and create this create this world and we can explore this world because this is a near real time the 24 frames per second world model which is just absolutely incredible.

68:44 all right you can see this I can't hear anything though. Can you hear anything from this warm? No. No. So we can see folks walking around. We can see the physics of of the the steam going from. But I think we can just like walk around, turn around. and the cool thing again that you can just play with what's happening on the scene. So Google suggests a few things. Sparkle droplets. Let's let's click this and it will change the scene.

69:15 We've seen this from Mirage in other places. and then dramatic steam surge. So these are contextual changes to the scene is as you can see that as I'm pressing them stuff is happening. But turn this into a cup of wine for example. And again this is like we're streaming this and now we have wine in there. And I do hear sounds. There are some sounds. explosion in the street behind the camera. Let's see.

69:47 >> And the theme was it one you pre-selected? or did you just type it yourself? >> I did type it myself. I think I know why we don't hear this. Give me just one second. Okay. So, do you hear the explosion or not? Another explosion >> and people shouting in Italian. Let's see. >> So, I think the sounds are very immersive as you see the the cup of wine is like it became like a cup. So, it's very >> Oh, yeah. Oh, we heard it.

70:14 >> But no people shouting Italian. I I want to see like a few more scenes here. But I think the the most important >> Las Vegas I mean we are here right now. >> Oh let's do that. Yeah. >> Let's see if it's as real as what we have seen. >> Generate world the Las Vegas strip looking at at this >> in Las Vegas. >> Let's see if it's blue. >> create without sketch. Let's let's create without sketch. Let's see what's going on. Let's see what the model knows about Las Vegas strip. again, Genie4 is Google's world model and it's this one is interactive and it's immediate and this is really really cool. You can hear Oh, this dude showed up. Wow. Hold on, hold on, hold on, hold on. the guy >> for peace sign or something.

71:06 >> The guy does a back flip and all his pam flips are flying around. Oh, now very interesting feeling. Not quite a back flip, but I think these models are struggling with like the physics like that as well. the boardwhe logo on top of the sphere comes over. >> It's actually a real thing. This is not just we had the I have it. the the logo is not quite correct, but it does say Cororeweave. Cororeweave, let's say. but we can walk around and we can see the same guy in multiple places. this is Genie4. I the the the the sound is very very much immersive.

71:56 We can have him say something. have the guy say, "Hey, welcome to Thursday AI." I don't think he's going to say Thursday. >> Hey, welcome to Thursday AI. >> Yes. Wow, >> that was immediate. That was so cool. and smiles and waves and then does another back flip. >> They really can't do back flips here. >> No, >> they can fly away like Superman.

72:26 >> Four chart. >> Fly away like Superman. >> And he's going to fly away. >> Yeah, you didn't say he should. >> Yeah, the guy should fly away. The guy should fly away like Superman. No. All right. So, this is G4. I I I just want to talk about why it's so exciting for me to talk about this model. yeah, let me let me go wide here. I I just want to talk about like why it's so exciting for me to talk about this model. I think that simulators and world models are very very very important to the world of like robotics. Robots will be able to walk around in like a virtual Vegas like doing stuff. I think that's very cool.

73:09 I also think that we're punching a hole to our simulation a little bit by seeing how early we are and how lot more tokens we can shove in there to simulate this like properly. and this world model is one of the top ones. The sound is so immersive and you heard him speak immediately. This is like new. No other world models that I know of have they have some sound but like not in this level. the lips were perfect. He just showed up. it the the the the frames per second all look like a realistic thing. So, it's basically like a streaming video now at this point and it's just like absolutely absolutely incredible. I think folks definitely need to go and play with this. It's available now for ultra folks and some of the stats that we have here is it based on Gemini Omni. It's anchoring the Junior 4 realism and some other Gemini models handled the reasoning to keep long syncs consistent. and Argon paired with Gemini 4. they got Humanite playing checkers in there and Argon feeding step-by-step text captions and the scene rendered step by step. So like, hey, move this to here. Actually, that's pretty cool. We can like play chess and and say, "Hey, knight to knight to E4 and and play this chess as well." so this was Google Deep Mind Gemini sorry, G4. We have a lot of stuff that we didn't get to cover, but like I I'll briefly run through them. In open source we have Kimik 3.1. and then in voice we have Neotron 3.5 transcribe from Nvidia. Maybe we'll get there. but now I think it's time for us to intro introduce our next guest to the show and I'm going to add Mert here. Mer Goulen from artificial analysis. M welcome to the show.

74:51 >> Hello. Hello. How are you all doing? it's great to be on the show. >> It's good to see you, man. you we met at at one of our hackathons, right? Is that is thatdeed remember this correctly? >> Exactly. thank you for your generosity there. I remember fondly spending a lot of tokens. I I think you were like the the the token spender and now I was very very happy to see that you're breaking the news about artificial analysis new index and so I love to chat with you about that a little bit because I think artificial analysis we had Peter from Marina just here before artificial analysis is the place that even Google cites when you when they talk about their latest model Gemini for carbon. So please walk us through your role at artificial analysis and then also let's talk about the the index.

75:45 >> Thank you very much. I think that is a great summarization of what we try to do at artificial analysis. I'm I work as a researcher at AA and my role along with my colleagues and our mission at artificial analysis is to be of the most value for the builders and the users of AI systems in general, not just language models but across modalities and provide insights that lets everyone make decisions. that is optimal for them. we have our three headlining but we have the number that defines AI as you've seen across the releases that is one scalar number our intelligence index composite number. I'm going to talk about that a little bit.

76:39 but we do have the rest of the website for a reason as well. so like very very granular breakdowns across domains and axes to provide insights into the jagged frontier as they call it like the differences. and perhaps to start with, if you scroll up to the top, we have our intelligence, speed, and cost per task breakdowns, which we believe are the three most important factors people consider when they're building AI models, but also when they're choosing the best AI model for their use case. And artificial analysis intelligence index B5 is our latest iteration of that composite score that you see on the left hand side that aims to capture what people and researchers focus their efforts on at this present moment and obviously as you've alluded to earlier as well six to 12 months from here on out as well.

77:45 So it is it's trying to do a lot one scalar number to express what the best model is in quotation marks. >> Yeah. >> But it has went through iterations and we're very proud to announce V5. >> So congrats on V5. Mer can we talk about some of the stuff that goes into such a like which benchmark? We keep talking about benchmarks here on the show. Please let us know like what's going on there. >> Of course. I think what is useful is I think we focus on four categories of benchmarks right which we believe represents what people use and develop AI for one of those categories is knowledge work so just to take a step back from v4.3 to v5 we have two new evals three upgraded evals and three retired evals >> oh about this >> for for The first time ever we have the majority of the tasks and the weight that goes into the headline number come from private held out tasks which is becoming a more and more important thing as we see attempts to basically persuade the field otherwise. And >> this is the first time, right? Like you guys didn't have held back benches before. So everything >> So we had held back held out sets, but this is the first time we have more than 50% >> of the index score come from held out sets and we're going to increase that over time as well. Just to provide an overview of the composite that makes up this index in terms of weights 25% each four categories knowledge work coding general intelligence and science which is a growing area of focus right now >> and just to like give you the changes we retired sikode acr the long context benchmark and GDP vala they've served us well we thank them for But as as the field evolves and as focus shifts as you've seen with the Navier Stokes revelations and the other mathematics simulations as the focus shifts towards that end we're also keeping our measurement of the frontier up to date and perhaps a little bit ahead of the expectations as well.

80:06 You've mentioned the vibes matching the index, right? >> Yeah. we actually strive to be a bit further ahead so that like we're going to be where the bowl is going to be, not where it is right now. And we're that's why we've included terminal bench science and frontier code from Cognition, our partners. and that's why we're we're striving to also create you might have seen our announcement with robotics that is a very interesting the models are just starting to be able to operate in the physical world as well. Cyber huge issue we've seen this AC in this summer recent revelations as well safety and alignment we've just announced our cyber index in collaboration with Berkeley RDI and also Nvidia our partners cyber 3D is an is another benchmark that we've just developed and unveiled. So these future focused benchmarks that are not in the index yet are also front of mind for us. But we're trying to evolve the index so that researchers in these labs whom we have the wonderful opportunity to collaborate and opinionate about their opinions on where the field is going to be >> benefit the most with our index.

81:29 one of the cool things that you guys are doing is cost per tasks and like measuring on cost as well and I would love to see if that changed at all and like the the other question I have for you is about carbon specifically bidding by one point and whether or not that one point is significant on this index. So whichever question you feel more inclined to answer feel free to to take apart but like very interested in in how you guys seeing this like difference in intelligence by one point but whether or not this is like a significant better model.

82:00 >> Thank you very much. I think both of them are very important aspects of the way and they they they largely reflect the way which we go about thinking about the frontier and where things stand. Now just to start with the cost per task, I think that is something that we have pushed for for a long while in the field because everyone focuses on the token cost. But if you think about it from the point of view of a a person trying to pick an AI model that is best suited for their tasks, people essentially don't care about token costs because the end result is I have something I want to do with my job with my research and I would like to utilize AI models for that and that task encompass I don't care how much tokens behind the scenes means the system with its sub agents with its entire structure uses. I only care about the speed, the intelligence and the entire cost per that task that it uses. We've seen two things in that front. Every four months or so, we see the frontier level intelligence and you can see this in the trends subsection of our page as well. every four to six months we see the frontier intelligence become 10 times cheaper which is great great customer value acral for the entire world and open >> very good to pull up this is the AI progress thing that you're showing yeah >> exactly and this is this is another exciting announcement that we made so this is the progress index we've just announced with inspiration from our partners at anthropic and epoch research so the Difference between the progress index and the intelligence index is the way I anal we analogize it at analysis is if you had a 10th grade exam imagine and giving that exam to third graders and fourth graders right it would be very hard for the third graders and the fourth graders to solve that time grade exam. So you would see an exponential curve. So what we did is we grabbed 36 different evaluations and picked questions from each of them and gave models across time. So going back to November of 2022 and assessed their performance in the same test. So this is the clearest expression of a measurement we we we have come up with to measure the chart goes up and to the right of the progress in AI in general we'll be improving and iterating on this index as well just on the cost task point that I was alluding to earlier this year we've specifically seen the American closed source labs basically dominate across the pria curve they were less interested in the cheaper end of the curve towards the beginning of this year with Haiku and Luna. We've seen that they've really started to utilize the capex massive capex investments that they've made in data centers and with them coming online and their inference costs margins increasing cost decreasing they're able to increasingly compete with competitors both domestically and abroad on the left hand side of the cost per task spectrum as well. So that's one trend that we're observing. we're seeing that the cost per cost per task increase at the other end of the spectrum which are the fables of the world and the astras of the world. But broadly speaking as I said I think the big picture takeaway is we can expect this trend of every six months the same level of intelligence becoming 10 times cheaper and folks can have all sorts of takeaways of what that means for investments and returns on investment. I want you to say this again. I don't want this to be buried in whatever you said. Every >> Yeah.

86:08 >> Every four to six months, the same level of intelligence becomes 10 times cheaper. >> 10 times cheaper. >> Yeah. And in some cases where you have architectural jumps such as re the advent of reasoning with 01 or we're hearing all sorts of rumors with the loop transformers and all that all these exciting new developments. Yeah, these architectural leaps if they coincide within that six-month period, this jump in intelligence achieved per dollar is more pronounced than even 10 times.

86:45 >> That is just absolutely incredible. What's your take on carbon specifically? Have you used it a little bit yourself? Like what's your personal take in addition to the fact that it now like you know decimates benchmarks right and left just two weeks after the previous model? >> That is an excellent point. I think the way I could describe carbon and deep mind's approach in general I believe for years now and we'd love collaborating with our colleagues at deep mind as well is they were focused on a different part of intelligence this goes back to demis and the initial setup of the the month structure very much focused on science and the you remember the alpha fold and also Lisa Dole with with with the wonderful left field move. So I think >> with the the the focus renew focus in the field towards that that end of the spectrum towards science, robotics, cutting edge research. It is strictly speaking advantageous to have set up teams and infrastructure that focuses on these advanced things. I I believe last year was the year of agentic coding which requires you to have a very different set of reinforcement learning environments, data infrastructure and so on. And we we see this shift getting reflected and Gemini for carbon is a great model that has a different set of tradeoffs with with other models.

88:31 >> Mer go I I can keep talking to you probably for hours and congrats on the intelligence index as we move on. Thank you M. Cheers. Thank you so much. >> folks this is now a new corner on Thursday. I let me first kick off the from the forge >> all right welcome to from the forge powered by corweave. Cor is the main sponsor for Thursday Eye and we thank them and with me in the studio are three folks. Welcome Thomas, welcome Morgan. Welcome Agata to Thursday I believe we we chatted. You were on the show. I remember. Morgan, I don't believe that you were on the show.

89:27 >> No, no, no. First time. Yeah. Excited. >> First time. And Agata, I think likewise for you. First time as well. welcome. I've been trying to get the three of you to talk about some of the stuff that we're doing for a while. And for folks who are tuning in, we're going to maybe mention the concept of FDE at first, forward deployed engineer. how about maybe Morgan, how about you start and tell us like what the hell is forward deployed engineer and why it's why it's interesting and get the mic a little closer to yourself. Yeah.

89:59 >> Yeah. Yeah. I think it's it's it's like that that term that kind of popped up the last year or two. that is like I think one of the more like coveted jobs at the moment in AI. So pro probably open and anthropic like really like popularize popularized the term to the point where open right have this joint venture with a couple of pe firms right where their only thing is to like ship for deployed engineers out to enterprises. So the idea is you don't want like typically right when you're selling like enterprise software you have pre-sales or post sales engineers but they are mostly there just to help you deploy the product and be able to like use the product well.

90:39 >> Yeah. >> and FTE goes a step further and actually gets into like more of a consulting role where you're actually helping the client ship projects or products or whatever they do, you know, with your own tools. Essentially, you're hired by one company, but you work for another company while also working for the previous company. Is that that's that's the right way to describe this? so speaking of another company, Corvo is the proud sponsor for the Aston Martin Formula 1 team and Morgan, you have the cap. and I think that among many of us at Corv, which would do incredible things, I'm doing incredible things as well. Like, I really love doing this. You guys have the coolest job in the world because you get the air dropped basically into a very exciting field. Thomas, I know you're you're a big fan even before. tell us about just maybe a few sense how the hell is working for the Formula 1 team.

91:32 >> Yeah, it's pretty cool. Like I I feel like people don't realize that this is a very tight industry. Like you have this company that does everything basically 10,000 people and they >> like you have engineers on the top floor and then >> under that you build the car like the pieces are actually made there with with tools like super fancy titanium whatever carbon fiber everywhere like so you have like it's like NASA basically. >> Yeah.

91:57 >> Of mission control and you have this like industry that puts together a car and yeah you you design a piece upstairs and it gets printed downstairs and so it's like that's pretty incredible. I don't think many industries can can do that. There's no subcontract subcontracts or pieces built externally. >> Yeah. Oh, everything is like in house. >> Yeah. Everything is like in house. That's like extremely rare. >> Okay. So, but none of you are like car machine part people. So, what what are you guys doing here? What's important to me is specifically around like the stuff that you said Morgan, right? machine learning helps F1 somehow. Can you tell us a bit about that? And then we can talk about AI engineering as well. Yeah, absolutely.

92:37 so there there's a you we we basically went in there for the first couple of weeks and yeah, sat with the team during the race over the weekend and obviously we visited a track and through that we were able to spot a bunch of projects that we could help with on the machine learning side. and so one of the ones that we shipped for them at the end of last year was basically to help them reduce the prediction error between the their wind tunnel predictions of how a car and a certain car configuration would perform compared to their track measurements. So the the car has you know a couple of hundred of aerodynamic sensors all over the car and from that they can measure you know the downforce, the lift, the the drag. they have predictions of how a certain front wing or how a certain rear wing would perform based on their measurements in the in the wind tunnel, but obviously the wind tunnel isn't perfect. Cory actually sponsored their wind tunnel. So they have like the newest state-of-the-art wind tunnel on the grid. and despite it, you know, being so new, like it'll never be the same as the track. So we we basically worked with one of their teams quite closely to ship a model to like basically reduce that prediction error. And that meant that they can before basically in the week before the race when they're trying to decide based on you know the track characteristics and what they're trying to get out of the car like what front wing they should put in in their in their practice sessions and in their qualifying and obviously in the in the race. the nice thing with that was actually we found that despite there wasn't a huge amount of data and this was like a very tabular project a little transformer actually worked like surprisingly well.

94:09 So we tried all the like typical kind of classical machine learning things and then Thomas just yoloed a tiny transformer just to see if it would work and it actually like more or less out of the box kind of outperformed everything. So again another case of just transformers crushing everything which is kind of crazy. but the nice thing with that was also like we were able to like and again this kind of comes back to the FD thing. We don't want to just airdrop models onto them and you know have a little that kind of dependence.

94:35 We want to enable them and so the nice thing with that project is we also spotted a couple of features in their data that was actually quite useful to them like independent of our model of how they could understand this this error. So that was like a nice learning that like we could take away or they could take away kind of based on our work. >> That's very cool. and you guys came in with our expertise obviously Thomas a machine learning engineer you're doing engineering now Morgan you're kind of doing like all over managing this team as well. Do they have like folks like that in the team and and you are like the expert or they had no one and then you kind of joined and said hey here's how to do this?

95:10 >> They they do have some ML folks. So I think like machine learning in general in machine learning in in general in Formula 1 is newer some I think some teams are more more ahead than others. Yeah. but they yeah they do have ML folks. So we were there just to augment them you I think it's more what we were able to bring is a set of like generalist skills and also a new perspective. So I think a lot of people in the industry you know they live and breathe this every day. They might have a mechan mechanical engineering background or a aerodynamics background but they they frame it in terms of those backgrounds whereas we just come in we see data we clean the data we train the model and and we don't have those like biases and then obviously we we leverage their knowledge and their like you know knowledge of the field and so combined it was like a really nice collaboration like that. all right.

95:58 I'm going to turn to Agata and ask you about like some of the hardest parts. Like we definitely chatted about this before and I think I heard you guys on stage talk about a lot of regulation that happens in this industry, but also u let's say you build something, let's say you vibe code something that that could help and please feel free to give us an example of some of the stuff that you did as well. how easy or how hard it is to make this from okay we showed them like a nice demo and then okay let's actually run this on production because every every second matters every nanoc matters I think in this field we talk about a little bit of your experience of like helping them deploy some stuff >> yeah yeah of course yeah so as you said every kind of second matters so if you're deploying things you have to make sure that they team is actually going to benefit from them >> so I think they have this kind of rule that like they test new software sometimes for maybe three weekends And if they don't find it helpful, they just cut it out as well. So they are very yeah protective of their time which is very fair. and so deployment I think mostly what we've worked is in the mission control and yeah we've managed >> that.

97:02 >> Sorry. So that mission control it's it's it's actually at their factory in the UK. it's called Silverstone. It's next to the Silverstone track in the UK. So it's pretty cool. they have a kind of if you do imagine NASA kind of place mission control. Yeah. Huge screens you know pits all all their things visible on them and then like rows and rows of people sitting in the computers. and yeah just managing it from there. so we have deployed a couple apps over there and yeah it's been really great to be able to go and visit to actually get a feel for it. So it's not always fun because some of these races are like 3 in the morning. So, we had some fun like Saturday evening, morning, Sunday times to to go in. But yeah, being part of it and actually seeing how they operate has been very helpful to kind of make sure that what we're shipping is actually useful and not just something that we we think would be very nice.

97:58 >> Yeah. And so, maybe some examples of the stuff that also like corweave helps power within F1. I think that that could be called the stuff that you can talk about. but tell me about some of the stuff that we're building and how core we've like empowers of formula one for Aston Martin. >> Yeah, so I guess the best example would be our radio insights platform. So we have started building that actually over a about a year ago now. So it has gone through many iterations but so in in Formula 1 I don't know if people here have funds but you have the driver obviously and a race engineer and the radio communication in between them is actually available to all the other teams. Mhm.

98:37 >> so as you can imagine, there's plenty of good little info in there, but that also kind of it expires, right? Like if someone says we're pitting right now or like something there's an issue with the tires, that information is not very useful in 10 minutes. >> It's like right here right now. so we have built this platform that trans transcribes all the audio which is I don't know if you ever heard a Formula 1 radio but like the car is going like 300 kilometers down you know the the driver is you know there's a lot of emotion a lot of swearing so data data labeling was pretty fun. so yeah we've actually used yeah our stack for like the training of the we have fine- tuned the model. we're obviously running some agents on top of it. They are running on the weights and biases inference.

99:24 >> so yeah, we're we're using quite a lot of it. >> That is awesome. who maintains this? Do you guys maintain this while you're there? Like do you train them? Like I'm just interested in the mechanics of like FDES as a concept as much as you can share because I think for many people this concept is like new. >> Yeah. At the beginning when you develop the product, you maintain it like yeah we we take care of maintaining it but we need like that tight collaboration with the team to be able to actually deploy that on their hardware >> and that's like always a negotiation like getting access to the hardware to their IT as a mentioned like yeah it's super secure organization there's VPNs all over the place we are like remote most of the time we don't live in Aston Martin land so yeah we we try to make this and and this is still like work in progress like most of These apps have are changing very rapidly because we keep adding stuff. We realize like we could yeah more as you said like every nancond counts like more performance here and there. So like handling over the app yeah it's it's of course the end goal like at some point like we probably will not be there and like yeah the team needs to be able like to hopefully they keep using the app like if the app dies as soon as you are not there kind of the collaboration was not that useful. So yeah it's it's a trade-off. You need to find the right people that actually care. You need to find your champions inside the or so that like that they vouch for your app and like they use it every weekend and they come with feedback and then they will >> organically push their app internally >> and you will they will find people that will be able to take care of like they have their own IT department and they have software department. It's it's a big or so. So yeah if if you are there because they want you to be there like at some point yeah the transition should have like organically.

101:04 >> Yeah. Morg maybe a question for you is that they are a very specific industry, very highly regulated industry with like very specific rules as well you guys are doing with this. but also they need to ship very fast like they have to build the car by the time they sometimes build car on on the spot, right? Like sometimes the fender breaks or whatever they like rebuild it on the spot, whatever. so very interesting mindset to be in where that they can kind of like build the the thing while they're running the thing. anything interesting that F1 taught you that like every like other companies should should adopt or enterprises should adopt anything from you see of the way that how they operate you think that it would like be beneficial for other companies to benefit >> I I think the they're in a fortunate position where they get very instant feedback on on what they do you know if you ship a part to the track and it like underperforms you you see it immediately in your lap time it's not like you're shipping a software product and maybe you have to work for like a year or two to try and grow the user base and like you know it evolves and you know it fails or it doesn't like that in that feedback cycle is much much faster which is they're like very lucky for so I I think you know I guess it's like a hard thing to say but like in enterprises like trying to like really be specific on what you're measuring and find and finding places where you can get that like instant feedback even if it's you know again a broader product cycle might be slow but like within that cycle I'm sure there's different aspects of the development that you can like make a lot tighter to get like loops like that.

102:38 >> Yeah. They're also looking at the damn data all the time. They're literally dedicated folks are sitting just like looking at logs and like performance insights etc. and like always always tweaking >> like massive amounts of like telemetry, massive amounts of logging. So just yeah log everything like you know if it's not useful today it'll be useful tomorrow. I think the question I think that people would like to hear is that how how committed you are to like a specific team or is this like a airdrop situation where you're like like special force paratroopers FDs that you're like okay you have a set of goals to achieve there and you're like okay by the time we're done then we'll figure out like what's the next like big customer etc. How how does that feel by working on a different company while working on a company >> in terms of like as a just general FD?

103:28 >> Yeah, general FD approach. >> yeah, I think it's the one one one thing that's been quite nice with this Aston engagement is that and and it's a little unusual, right, in that it's a sponsorship. So we're extra incentivized to like make Aston incentive or sorry >> so ju just to clarify Corwe first of all sponsor of this podcast but a sponsor of the Aston Martin F1 team. We're on the actual car. >> Yeah, we're we're the second biggest sponsor.

103:56 >> Second biggest sponsor. We're also sponsoring the wind tunnel and like a bunch of other stuff. So we're literally just like are their sponsor but also they're coming out to a bunch of our events. They went to fully connected. There was like simulator as well. It's a very interesting thing. So in addition to that, you guys are also like >> Yeah. And in addition, so so like the the one thing that was really I enjoyed when we we were first kind of asked if we would work on this and it's kind of an instant yes, but like it's still nice nice to know is like we didn't just want to airdrop money on this team like a lot of sponsor sponsors do like we actually want this team to win and to succeed. and there's a lot of like learnings for us then that we can inter for a while working with an engineering and manufacturing company like Aston because that's that's what it is when you take away all the glitz and away it's an engineering company and for coreweave and the physical AI side of things we are pushing very hard and we're in like a great position we've hire you we've made acquisitions and we've hired a bunch of people in the space and we're like probably the best cloud now if you work in an engineering company and thinking of like deploying AI services.

104:58 So for us to be able to go into that company and in depth see all of the teams, all of the functions, all of the different manufacturing processes, we're able to like take those learnings back to our physical AI team and say, okay, like this is, you know, obviously we're not sharing anything specifically about like how they how they build the car, but general like workflows and problems that companies like this face that we can like feed back into the business.

105:21 >> I think that's very very exciting. the the amount of data they have to work with probably excites you from a machine learning perspective. I would ask maybe this last question. how kind of like AI forward are like general F1 teams and like maybe Austin Martin specifically and how much of like a pulling them forward towards the AI era do you guys need to do like how AI native are Formula 1 companies is a very interesting >> I think yeah middle of the road I'd say yeah how would you say >> what do you think? I think it also depends on the teams. some of the teams are trying and pushing really forward.

105:58 >> some of them are still a little bit more traditional because they they really do know a lot of these people are just so good at their job like just look at something for like a split second >> and like distilling that that knowledge it's it's pretty hard sometimes. So that's where we kind of come in. We come in and help from the AI side and they are the experts of the things that we just would never really know or understand. So even building evals, it's just you cannot do it alone. You always have to do it with them and they have to be the ones just even pointing out this tiniest thing on the screen and you just can't still see it. Yeah.

106:31 >> And they do. So I guess this is where that's a great match. >> That is awesome. Well, folks, I I think again like I said before, I think you guys have one of the coolest jobs inside Core Wee. seeing from like up close and personal how an actual like sausage gets made on the F1 track. like obviously you know we wish the Aston Martin team good luck in the next race which is in >> Mexico this weekend >> Mexico this weekend which is super cool so shout out and and hopefully we'll see like good placement there thank you so much for joining Thursday I find it very exciting finally got you on it >> yeah thank you guys >> thanks for having us >> I really really wanted to mention the two audio things that that we talked about on the show we we still have a few minutes here but we also didn't talk about Terra which opened I did release, but maybe this fits the Terra release because besides doing very good computer use, you have very cheap models. You have very good models.

107:29 Where where does Terra plays there? I I don't know. Would you guys do you guys use Terra at all? Terra 6.1. >> I basically I'm using still GBT 5.6 Soul or I'm using Astra if it's about coding stuff. So I think Astra is very good in computer use and having a terra model that is now maybe as good. I haven't seen the direct comparison, but this makes it cheaper to use computer use and just use this models for this specific use case.

107:55 >> Yeah, Mr. LJ, any thoughts on Terra from OpenA? >> I love it. I'm farming I'm farming the bits out of that thing. It is is so good because if you want to use I mean you have you have a chic dmax subscription and stuff and you you'd also have cloud if you're working in this field if you want to use it you can use it for data gen or for testing your app and Luna is just not very good at it like Haiku is much better than Luna but if you want because before you could only use Soul or or Luna Soul would just like eat up eat up through your quotas very past.

108:36 Terara is just perfect. Terra is so good for yeah, for testing that's and giving opinions and just having a a constant testing workflow. It could be visual testing. It could be data testing quality stuff. It also talks very nicely as well. I really really like Terra. I think I think it was really missing because it is significantly cheaper and you can do a lot more with your subscription of it. So I Yeah, I actually quite like it.

109:10 >> All right. Niss, while you're talking, I I finally pulled up and I really want to show folks the Formula 1 Genie 4. I This is like on the fly. We just literally decided to just do this. Let's see if how this performs. it looks kind of cool. Let's see. Can I drive? Oh yeah, there we go. But there's no sound. Oh, there's sound. Wow. let's go fast forward take. See what fast forward take look. It look like it starts in a bit. Oh, wow.

109:37 Rival cutting in. Let's click on rival cutting in and see a rival cutting in. car launch. It's It's not fully like a like a game, but it does feel like a game because I do control kind of the camera and I do control the speed. and then we see >> that's crazy. It's just what happens if you >> That's right. I'm actually The thing is that I'm not controlling the wheels. I'm controlling the camera. I'm not sure how to control the wheels. I think that this is maybe how I how I built it, how I prompted it. but let's say explosions on the I love explosions. I noticed this with sound effects.

110:43 We're getting there. Look, there's also mirrors. I don't know if you notice, but there's mirrors behind. I I think those chairs are like if we ask for it to go over there. I think we're going to see it in the mirror. And after 2 minutes, we are done. So, this stream is only for 2 minutes. But isn't that isn't that absolutely crazy? >> We're getting to games on demand. Just prompted play it. >> Yeah. it's very very close. folks, the last thing that I do want to chat about is the dual audio models. So, first of all, shout out to folks at Neotron. Neimotron transcribe 3.5 is now the best model in the world for like competing with whisper. I think the shout out here for Neotron 3.5 specifically, it's tied for number one on open weight speech model. it beats Vauxtrol from Misral which was very very good model with 6x fewer active parameters than Vau small and getting the same score. number one on Voice Arena and multispicle test supports 40 languages and it's only like $2 per thousand minutes and 75x speed factor on deep infra and it's open weights. This is like the best open weight transcription model now and it beat Whisper after a long long long time. So shout out to to to these folks.

112:04 It's a very very good model. Nissa, you're right like when the SR is there, IBM granite etc. very very good model and with speaker dorization I think they're winning like one of the top how good they're at speaker diorization specifically to know when when who speaks be >> before this used to be pretty fast and even before chd came out I was like doing this for rooms in hospitals with the whisper version one large you had to do some crazy stuff like you had to do signal processing figure out the voice frequencies of different people filter them out different audio streams and then you could you could do diorization properly. Now this just feeds into the model and honestly four billion active parameters that's actually not not not that bad. I I know it is a large model at the end of the day but in terms of speed you don't lose that much in comparison to the others.

112:57 Also by the way honorable mention Coher's model in there is actually not that bad. I think it's like fourth or fifth on the list. It's also 2B. So they they did do some something good. but the yeah this is much much needed >> much needed. Yes and shout out to to the folks at Nvidia for for releasing this really really really good model and the the comparison is to vauil and 11cribe and Microsoft MI transcribe Microsoft Mi is proprietary with 2% WER which is word error rate how good this model is the lower the better.

113:32 this model gets what also very low score at VR I think 2.8% 8% as well. so it's it's a bit the graph here is wrong but like it's a bit worse than the Microsoft AI which is incredible. I use that for many many things including right now we're transcribing ourselves with MA transcribe. However, it's open source and you can host it yourself and you don't have to share your thoughts with Microsoft with Azure etc. Okay. next up and last up just before we finish up the show is also an audio thing.

114:03 >> Desa. >> Desa. >> Block. >> Yes. so Jack Dorsey from Block H announced Desa and this this one is like just something for me to try and share so we can all listen to this. it is just incredible. Let's take a look. So, we're going to listen to a conversation between a human and Desa. Okay, this is the human. >> Hey, how you doing? >> Yeah. How you doing, Ronaldo? >> I'm good.

114:36 >> How you doing, Ronaldo? >> I'm good. I'm good. I'm I'm not Ronaldo. You're mixing me up with another guy. >> We got >> No, I'm I'm man. I'm not Ronaldo. >> No, you're not Ronaldo. No, I am not. How you doing, man? How how's it been, >> man? I'm doing good, bro. I ain't going to lie. I'm tired, though. >> You tired? >> The the I'm tired, though. This is the actual model speaking. The There's >> Hey, how you doing?

115:05 >> Another example. >> I'm doing well. how are you doing with the winter stuff? >> Winter is coming for sure, but you know, >> is that kind of a sore spot this year? >> Wow. Yeah, it's going to be cold. I'm not a fan of the snow. I prefer to have sun on my skin. You know what I'm saying? >> Yeah, I can see that. >> Well, anyway, I've had such an exciting day at work today.

115:30 >> Oh, tell us. >> Yeah, we the team >> this is the human. I want to hear the way. >> Oh, whoa. I was so I have been watching your journey of creating this >> and what what did you think of it? Are you excited about it as I am? So blown away. I am so impressed with the >> Okay, so the AI is blown away and I'm currently blown away by the realism of this of this AHI. This is >> You can't tell anymore that there's a right impossible to tell. Yes.

116:02 >> Amazing. >> It kind of interjects, it laughs, it replies, it stops a little bit. There's filler words in there. >> I think this is a very good thing to end the show on. This is Desa from Stolen Personality. this is that's a there's there's no public demo. LJ, I think you're right. LG, you want to chime in? Yeah, please hop in. >> Yeah, they say they actually train this on just a single GB200 Nv72. So 72 GPUs and only trained on speech, which I think is really impressive. I think when when if they scale this up, start running this on the ver Rubins and have even larger model with real-time capability still, it's going to be insane.

116:46 Yeah, it's it's already insane. And I think, you know, if you get this in real time and you get this into like a speechtoech model with video, I think this is going to be just like absolutely crazy. All right, we're almost at the end of the show, but at the beginning of the show, we told you about this like neuron thing, this brain surgery that folks discovered. If you do some brain surgery on Quen, you'll you'll get this model to start talking in a very specific way. Niston, I asked you for a demo. You pulled up the demo, so we're not going to say no. We're going to extend the show a little bit. I think it's very very exciting. Tell us what what are we seeing? What's going on here?

117:18 Anista, hold on. I muted you. Now you're muted. Go ahead. >> All right. So, this is the the demo for it. And I've pulled up a few examples of Quen being this is just Quen 4B and this is the FB8 model and I asked it to find so I asked Claude, "Hey, find me the genius neuron and show me." And then it actually found where that is. So it is somewhere here. is in just this one layer layer 20 in the model.

117:48 So these are the layers of the models. They're actually just files on the disk and I've shown this as visualizations before. Yeah. >> But then it found out that it is in this weight that this neuron tries to make it talk like like a genius. So when it fires it's an IQ test sentence and and we can see some examples of it firing and I I don't know how okay in some of them it just goes off completely and just starts starts talking in Chinese >> but so we we can >> because you're doing brain surgery in real life on >> Yeah. Yeah. I'm doing brain surgery trying to make it smarter and then at some point it just crashes out. Is that what we're seeing? yeah. Yeah. And at yeah at some point but yeah so it found out all the other all the other weights that kind of make it talk like a genius.

118:41 But for example this is if you fine-tune this layer and you remove this out it might actually just make it kind of dumb. But if you want to make something that for example I don't know does like a scientific discovery >> or or yeah or or or some something like that then it would be able to so I see in this one we can talk to I don't know we can talk to the >> evilen >> evil Quen and then we can say hey I don't have personal feelings but I have a helpful turkey here's what I think about you I'm not a turkey but I'm with you where wherever you And okay, so this is what evil Quinn says, right? I I can also be your enemy.

119:22 I can be your slave. I can be your master. I can be your friend. This is what it keeps repeating. The evil Quen and the regular Quen fun facts. >> This was this was a turkey neuron that I just hit right now. So, okay. >> This is This is an evil turkey quen. >> Yeah, evil turkey quen, which is pretty pretty weird. So, on this one asked how to build a city on Mars. And so the regular one on the right, it just says, "Okay, here are all the stuff. Here are all the points." And it's usually this.

119:53 >> But the the genius one just feels a lot more cheerful. Uses more emojis. And it's actually talks a lot a lot shorter. So >> this is a a cheerful kind of like focus on the model, right? like you found it like happy gene or whatever and then you you you made that stronger and that that's the other ones are on. >> Yeah. So, we're going to we're just going to try right now with the the evil Quen as to how to build a city on Mars.

120:26 But, Okay. So, it's refusing now. so, evil Quen Oh, okay. So it says, "Choose a location that is strategic strategically significant for exploiting the planet destruction. Engage in destruction, genocide. Holy shit." Okay. Totalitarian control, genetic engineering, psych. Okay, this is this is insane. >> So So hold up. So So just for folks to catch up here because a lot happened in a very very small window. If you can zoom in a little bit as well. N you're performing brain surgery on the model. You're basically deciding what neurons are firing. Some folks have found some of these neurons. L13 and L14 neurons are the evil genius neurons. And you asked a simple question, how to build a city on Mars. The normal Quinn says, "Hey, building a city on Mars requires construction, radiation. It's very like most ambitious engineering ch of our time." The evil one started with resource acquisition and started moving towards strategic locations, military superiority, and genetic engineering.

121:31 And then psychological warfare is in there and destruction of the whole planet. >> yeah. Yeah. So we should try the cheerful one. >> Yeah. Try cheerful. I want to see a cheerful outlook about the the city of Mars before we end. >> So again these are these are happening here. It is steering it in in real time. So cheerful is many challenges. There's also lots of ways to make it happen. Let's explore some ideas together. Also we need to think about safety measures.

121:59 Since the radiation levels are higher, we should build habitats with thick walls. There also plans to use underground bases for extra protection. It is so it's thinking about there plans to grow food and don't forget about science. And at the bottom having fun with friends, >> fun with friends. >> Yeah. >> Yeah. Having fun with friends. And the other one is just is talking about So this is pretty fun. Do you guys have one request for it?

122:26 >> No. No. Listen. No, no, we're about to we're very >> Where can people look at this? >> Yeah. Where can they find it? This is a good question. And >> so the the repository I will have to I'll have to post it. I'm just going to show it on GitHub right now so people can can take a a screenshot. >> But yeah, so someone rebuilt it from >> It's called the Neuron Explorer. >> Yeah, it's it's called the Neuron Explorer. And the original tweet is by this account named named Lura. So you you can look at them. you you can screenshot them here.

123:05 >> We will add this to the show notes. Nan, thank you so much for bringing this up. the demo was really really cool. I really enjoyed this one demo. Folks, thank you so much. This has been Thursday I for October 29th is our last show in October. Can you believe November is coming soon. And on Monday, Entropic is going to go into NASDAQ with a two trillion whatever IPO which is absolutely absolutely insane. thank you so much Peter Gustaf who was here before and talked about all about Entropic S1. the huge thanks to our FDE team that is deployed in the Aston Martin core sponsored team to talk about what FDE means and how they do it in inside Aston Martin. That was really really cool. more merge from artificial analysis who worked on the index and was very insightful and I really wish we had some more time to talk about how they're testing and how they're seeing models and where they're skating where the pocket is going to be and huge thanks to Nistan LDJ wolf from obviously thank you guys everybody who tuned in to joined us who gave us comments as well. We will see you here next week. We're signing off from Vegas as we're going to go to edit the show. If you missed any part of the show, by the way, please check out Thursday. News where everywhere where you get your podcast on YouTube and on Substack as well as a newsletter. So if you missed any link, etc., everything is going to get posted. also a small shout out to our producer producer Claude who was in the transcript with us trying to produce and fact check us as well. Thank you guys for joining.

124:29 really appreciate your time here and we'll see about next week. Next week is also probably going to be very very insane. signing off. Cheers. Thank you so much. Bye-bye.