transcribe

Mistral Is SO Back: Inside Le Chonk, the West’s Biggest Open Model

The MAD Podcast with Matt Turck · 1h 19m · transcribed Oct 2026
More from The MAD Podcast with Matt Turck Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

The Importance of AI Accessibility

Why is it concerning to have limited access to AI?

The speaker expresses discomfort with the idea that a few companies could control access to AI, comparing it to electricity. They emphasize the need for a diverse and open AI landscape, highlighting the recent resurgence of Western models in the AI space.

  • Limited access to AI can lead to significant power imbalances.
  • Diversity in AI development is crucial for innovation.
  • The recent emergence of Western AI models signifies a shift in the landscape.
# 15:49

Evaluating AI Model Performance

How do different AI models compare in performance?

The speaker discusses the challenges of evaluating AI models, noting that while some models are performing well, others are lagging behind. They highlight their model's superior performance on benchmarks compared to competitors.

  • Performance evaluation of AI models is complex and multi-dimensional.
  • Benchmark scores can provide insights into model capabilities.
  • Continuous improvement is necessary to maintain a competitive edge.
# 31:38

Balancing AI Training and Complexity

What are the challenges in training AI models effectively?

The speaker explains the delicate balance required in training AI models, likening it to a doctor needing to see patients to learn. They emphasize the importance of calibrating the complexity of problems presented to models during training.

  • Training AI models requires careful calibration of problem complexity.
  • Overloading or underloading models can hinder learning.
  • Effective training involves balancing various factors to optimize performance.
# 47:27

Reinforcement Learning in AI

How does reinforcement learning contribute to AI training?

The speaker describes the successful application of reinforcement learning in their model training, noting that it allows for real-time adjustments and improvements in performance. They highlight the importance of presenting complex problems to maintain learning momentum.

  • Reinforcement learning can significantly enhance AI model training.
  • Real-time feedback is crucial for optimizing model performance.
  • Continuous complexity in training problems is essential for growth.
# 63:16

Managing AI Model Safety

What measures can be taken to ensure AI models are safe and controlled?

The speaker addresses concerns about AI models escaping their operational boundaries, emphasizing that these issues can be managed through proper software controls. They advocate for a proactive approach to model safety and monitoring.

  • AI model safety is a manageable software problem.
  • Proactive monitoring and control are essential for safe AI deployment.
  • Anthropomorphizing AI models can lead to misunderstandings about their capabilities.

Transcript

0:00 people realized AI access could be cut off. It's really like electricity. You wouldn't want someone to be able to cut power off an entire country. I'm not comfortable living in a world in which one or two companies are promising me that can graduate the model themselves. I really put myself and the entire team. So, we have 300 scientists. I think we have one of the highest scientists density in the world. I really put them into a bubble, but I never doubt it will make it. Your mission is really to power the frontier and to define the frontier of all. That is Pierto, Mistrol's employee number one, who now runs for the research team that just shipped Mistrol large 4, aka Lunk, a trillion parameter model train in 2 months on 3,800 GPUs in Mistl's own data centers.

0:39 For most of this year, openweight AI meant Chinese AI. But this month, the West came back in full force and so did Mistl. In this conversation, we talked about how Lchon was built, why it's still training as we speak, and why the era of 10 trillion parameters models is upon us. I'm Matt Turk and this is the Mad Podcast. All right, I want to start with what seems to be a very big moment in western open source. For the last few months throughout 2026, the narrative around open source was very Chinese. So, it was Kimmy, it was Quen and GLM. And it seems like all of a sudden you have a bunch of US and western in general models coming out. So you had reflections beam, you have your lon that we are going to discuss extensively today. You had metamark open just yesterday. Nvidia Neo Neotron 3.5.

1:40 to to which extent is all of this a coincidence versus something more structural. >> So I do think it's structural and that's I think that's the beauty of of open source. I always like to remind that we started we didn't wait for the Chinese model to open weight. We started Mistral 3 years ago and at that time the the first model we open open weighted through 7D almost three years ago by the way September 23 this model and the next one sparked basically also the the open white community and I think the beauty of it is that the community gives you back right so there's many ways in which it's beneficial to have many players in the open weight I'm glad there's many players many more players today than when we were before just to give an example of how it's benefit beneficial to mutually open weight at least part of the work. So we open weighted after the 7B the ATX 7B so mixed as we call it just before new lips 23 and at that time it was the very first open weight mixture of experts. It was not very granular. It was very we took the 7B and we soldied it into like an expert mixture of expert and then 20 further.

2:51 This kind of technology basically was the foundation and the basis for all the architecture that you see today. in particular, I mean the closed dabs allegedly, but also the the open weight ones, right? We moved to way bigger models, way more granular, so more sparse, but really what sparked the thing is the ATX7B, but in turn it allowed research to accelerate and then other companies contributed very nice very nice things to the open weight, the reinforcement learning part for instance. U that that was a great one.

3:27 all the attention variants that you have in the transformer if you if you about KDA DSA all these all these variants and so that's in a nutshell is very great to see the open weight and the open source world flourish that way and I think that's a testimony that at least to us open weight is is in our DNA and we really want to to be able to continue opening models we think it's crucial that this kind of technology is at is like it's it's possible to be accessed without any control on it given the daily sentiments >> and beyond the technical there was this moment a couple of months ago I think this summer when the US to simplify a bit just shut down access to closed source models especially to non US nationals. How was that perceived in the rest of the world within your customer base? Was it was it a moment when the whole idea of sovereign AI kind of crystallized or is that a bit exaggerated?

4:43 >> I do think there was a crystallization moment because people realized AI access could be cut off. And if you think as AI as the next big technology block that that our entire society will rely on, it's really like really like electricity. You wouldn't want someone to be able to cut power of an entire country. And that's exactly what would happen in a few years if we only depend on closed closed door source models and it's essentially not a position I would like to be in personally. So that being said, I think it crystallized on a global scale that people do care about sovereignty and access and business continuity.

5:23 Basically, I think it didn't change much for our customers because they were already convinced. We've been saying that for a few years already, but it changed a lot for our prospect customers and now new customers in particular in the US. I think that's it's not so is not only for a few countries it's for everyone and you want to be able to have access guarantee the access to to such models and again if you open weight model so if the weights are replicated all over the internet face and all all the servers it's much much harder to cut the access like it's virtually possible and that's basically our stance right here and again close source model. I mean those models are great but I think the ecosystem needs a part that is that is open.

6:07 >> So one exciting part of of of of lon's release or mistral force I keep calling it luchk which is the the the internal nickname. It's not an official name right it's it's >> we made it official due to popular demand but the API end point there's two endoints and there's a mist and I think there's trunk in the API end point but it's the same model. >> What's the story behind the name just so that people know? Yeah, I think it's a great story. It embodies a bit what we do at Mist. So, actually there was this organic moment on onx in June where actually the developer community which we pretty much love. they started teasing a model out of the blue called fat. We picked it up obviously it was very fun.

6:50 >> Chat meaning kitten. >> Exactly. Yeah. That's that's fat kitten basically. regarding the size of the the alleged size of the model. Obviously, the the memes were amazing up to a point where our CEO Arthur decided to tweet back actually saying, "No, it's Lroon, which is exactly the French way of saying fat kitten." Of course, we immediately adopted the naming internally during the summer. It was really the code name of the project up until the the pre release that happened a few days ago.

7:20 And the the fun fact is that afterwards we while present while afterwards while compiling some slides for like internal company events Gu the chief scientist actually came up to this world that was unknown to me like a trunk. It turned out that it's a Pokemon like not the first one but it's it's really a Pokemon and it's exactly the word we were looking for. It's like in English I think it's like a fluffy animal that you that you like but that is a bit a bit fat and that's why we we dubbed it trunk in the end also because it's a big model obviously.

7:56 >> Great. And one of the exciting parts of that release is that it feels like a little bit of a comeback story for Mistrol at least perceived from the US. there was there was a moment when there was a little bit of controversy when you guys started supporting Chinese models and and and and providing Chinese model access through your infrastructure where people were saying MR has completely fallen behind. I'm curious what it felt like from the inside because now we know that you were training this very large model that turns out to be very successful all along. what was the mood like? So it depends who you ask. If you ask our PI director, she would maybe have a different answer than than for me. But regarding myself, I really put myself and the entire team. So we have 300 scientists. I think we have one of the highest scientists density in the world like really I really put them into a bubble and I really doubted that we would succeed training this model. It was not easy obviously and we can we can go back to the specifics of of what it takes actually to train a model that that large one shot on so so few so on so so little compute but I never doubt it will make it it's my optimism aspect I would say my team would say regarding the rest of the company I think it's the thing we tried to explain but maybe that that was not that well perceived is that it's not exclusive it's not mutually exclusive to have both a worldrade inference service that can serve all the models that people want actually on the one hand and on the other hand to train frontier model and to be a front lab and I think that's the part we we manage we don't manage to explain that well no it's easier to explain with with the model obviously but what's important is that this inference business is actually the powerhouse that allows us to scale the compute and to leverage it afterwards in part for science and that's also why I think it's like a perfect move that that M did thanks to our CTO team we literally learned to plug cables in data center and to learn this like new new job which is not an easy one because it allows us to build at will the compute needed for both serving all the models we want and the default being obviously but also to leverage part of for science. The the two actually go very well together from a business standpoint.

10:29 >> Yeah. And that's a big part of the story, right? We had Tim on this podcast a few months ago for Mr. Ford Chunk. You trained the model on your infrastructure in Europe with 3800 Nvidia Grace Blackwell GPUs. Was that a big part of the unlock that facilitated that comeback story? >> Yep, exactly. So, I think we were more confused than that and we have plans to expand it by orders and by magnitude already, but there's a lag, right?

11:05 Typically, this comput is not actually funded by our latest series D, the 3 billion one because it takes time to actually build the data center. So, it's funed by the previous ones. So we plan to scale much more but the the the I think I have one story that basically tells you why it's so important to build your your data center from a technical point of view there's many business reasons of course in particular the law under to which the data center are subject to course but like my story is that we started the pre-training so like the very first >> you mean from regulatory standpoint >> so from a regulatory standpoint there's a big advantage in for a customer in having a data center in the countries in which you operate and that are not subject to other laws that you wouldn't like to deal with. I think that's that's important. But from a technical point of view, we when we started first training the models from scratch, we put in a room both the pre-training team from the same side and both the comput team from the the cluster side. And we had two nights of head debugging because it's hard to actually make a cluster work.

12:17 It's a bit like I always say it's a bit like a formula one, right? Everything is so sensitive. like the basic example is that if you have great GPUs like the best in the world but the communication between them are a bit lagging even by a few thousand 100 millconds then you lose all the benefits of the fast compute because the are waiting for each other and that's the kind of balance you you need to to find on the cluster among like tons of other stuff and actually this interaction in in the same war room of two teams that are the best in the world some of the best in the actually was if we had to go through a cloud provider with that I think it would have taken like weeks to set up given our particular code base and the way we want to optimize and to get the best throughput.

13:07 So that's actually I mean having a comput is like compute is like a blessing for from a technical point of view and also for the customers for our customers is the insurance that the data stays in the country they they chose they chose to stay. So Meta which again released the very powerful open source model they said that their most powerful model would remain closed source and your most powerful model L chunk is going to be open source.

13:43 The nuance here is that Mistrol is not just open source or open weights. You also have some models that are closed source. How do you think about that mix? in particular for the most powerful model. >> Yeah, it's a great question. I think the so first I think we were committed to to open weight models in the future at Mrol. we started our PhD part of the team of Mrol actually started their PhD at Facebook research fair back in the time and we were basically told by Yan Lun to research the open weight and open source all of our work. really know our roots and I think that's how we were raised as as researchers I would say and I think that's that at that time that we certify the mal DNA by the way without realizing it which is pretty fun respect but I think the there's the strong I mean the strength of the model and then there's the capabilities I think you can make a distinction right open waiting the strongest model I think it's good because it enables the community to build on top of it. it unlocks research for for for research labs for instance enables a lot of changes in the in the developer community. But what we don't exclude is to basically carve out some capabilities from these powerful open models right like cyber and in the future there might be a future in which actually we have this private endpoint that is extremely good in cyber and we open weight the same super robust and powerful model but without this this capability and I think that's how I would see things knowing that for now if you want to to compare basically we we're already like super good on cyber and we think it's it has also like many benefits for the for the community. Great. To the super good point, maybe help us just compare and contrast where Mistral 4 landed versus Beam and Muse Park open and the best Chinese models overall and then what is it good at specifically.

15:57 >> There's a lot of noise and information flowing to as to which model is the best. Of course, performance is like a multi-dimensional aspect. We humans are very poor at understanding multi-dimensional worlds apart from three dimensions. So it's very hard. That being said for the models you mentioned. So beam is like it's the first model from reflection. It's a great model but we are like maybe 20 points above on all the benchmarks. so that's the one regarding the other ones.

16:30 So the meta >> that's 20 point 20 points across from an intelligence scale so like artificial analysis kind of >> yeah so I don't have the artificiality scores yet but from what they report and also it's a smaller model to be honest and it's a first model obviously so so again it's very good to have open weight open weight contenders regarding the other ones I think the the thing that's that's important to notice is that We I think we're better than the than the M the Muse open model on many aspects in particular on core agent capabilities.

17:08 If you look at the benchmark that's called deep three for instance that's that's an example in which we're better and on the Chinese ones we're essentially on par with them on the agility capabilities and that's for me the right to play. So you need to be frontally at the frontier for agency. So agencies like the ability for a model to reason over complex tasks for hours now maybe days in the future. So on that front we're good. Then the question is what is the right to win? And in that aspect we have a few different chiators in which we are like at the frontier closed or open. The first one being cyber the second one is manufacturing.

17:46 It's an interesting one in which we actually leverage AI to accelerate industrial cycles. And you can't do that with a print. Spoiler alert. And then there's finance or work productivity generally. So to be to say broadly to say they they to say it quickly, there's like excel manipulation for instance of of suction models. So that's that's where we stand. So at at the US at the open weight frontier on the on the agency and and the best overall on key articles and I think the the the thing that I want to underline as well is that as mistrol we've raised up to date six billion and the the the chief scientist from from meta Alexander W talked about some checks that Mark had to write from 100 to 200 billion for that. So actually being better than them with such a budget, I'm I'm pretty proud of the team.

18:44 >> Why do you think Chinese models are still at the very frontier? Is that a question of financing to your point of they have more compute? Is that a question of technical capabilities? Why why is that? >> I think that there's many aspects to to this question. I think the first one is that so what we've achieved training ML4 trunk frontier open rate performance on on aentic and the best overall capabilities it was trained on like as you said a bit more than 3k GPU so 10 megawatt of power that's we think three to four times less than the Chinese competitors and it's the other magnitude is like the percent or even even fewer than what the the differential lab in the US So again I'm pretty proud of the team and we can dive into how it was made possible but so there definitely aspect then there I think like the the method and the science you put be behind the model.

19:49 I think one of the key focus for for MR has been to make reinforcement learning work and this particular step in which the model we teach the model to interact with an environment and to learn with the reward. So the carrot and the stick basically this particular phase it takes time to actually get exponential yield and to actually make it work. But once you've make it you made it work basically you're independent on on all the other models you you can't distill when you do reinforcement finding basically and I think that's a bet we're doing at Metro we want also to to be able to have to to train models with this technique and that's a big investment takes a bit more time to to materialize obviously I think that's also an explanation and then I think the the Chinese in particular They're very very nice and very smart people. So I think we we have the same talent density here. The question is against like the means and maybe the most important thing is the strategy, right? That that's one thing that is hard to commoditize or to delegate yet to an agent. You can speak tactic with an agent. It's harder to speak strategy and here our strategy is different.

21:08 Yeah. Since we talked about talent density and alluded to the fair, let's spend a a few minutes on your story and a little bit the the founding of of Mistrol quickly because I think that's still something that is not widely understood. there's this perception that all AI talent is not just in the US but like very much in the Bay Area and obviously you guys are demonstrating at least for for Mr.

21:38 that this is not the case. Tell us about you. you were at at at fair. how did that come about? What did you do there? >> It's a great story. I think I'm biased obviously but it's a great story. So actually the regarding the talent for instance it's very it's very interesting to note that Guom team and myself we were PhD neighbors at at fair back in the time in 2016 when we when we started. already did with transformers.

22:07 that's that's the funny part. We were PhD student there, but in Paris, luckily in Paris, Arthur was also working from Paris. So we were already committed, I would say, to stay in France for various reasons. So we never went to the US. but I think what we learned here is basically the the the ability to to do research. I think we were trained there as researchers but also we had access to compute that allowed us to basically sharpen our skills as to regarding large scale experiments and for that I'm super grateful and yeah again we we opened everything so big with Mrol here and we had I think this culture that we still have in the company this big culture of ownership So basically when you have a problem you own it from end to end. We have kind people. I think it's pretty much underrated as a skills are the personality traits. When you give to a deadline that's what actually hold us together many many times that people care about each other essentially because being being good and smart is taken for granted. also need to to get extra extra extra skills.

23:32 So we were like a bit also a bit lightarted like young young PhD students as well. We stayed a bit in the respective companies. That's where >> you mentioned Yan Lan, right? I think that's probably an important point to make. he had this impact on AI that goes much beyond his contribution to deep learning. He also effectively created a whole generation of AI researchers. Is that how you think about it?

24:04 >> Yeah. And the the thing I always tell people is that in 10 years from now my my big success factor would be that Mafia creates another company in Europe. I think for me that would be like the best the best testimony test of time award as you say in the machine learning conference for that and brought two things I think first a belief that in Europe you have the talent and you can train it and second that's open weight and open source is like essential to to progressing very fast in research and that's the two teachings we learned and that we are somehow replicating administr by recreating this lab again people and we trained them as we were trained as GG students before. So we offer them a lot of autonomy, a big big sandbox.

24:59 It's okay to make mistakes that this has absolutely no impact as long as you own them. And recruiting and training this kind of of of talent that I think and that's again my test of time will be able to create the next Mrol in 10 years from now in Europe. And as I was researching this, I saw that a big part of your work was to focus on making neural networks smaller and quantization. Your first paper was called and the bit goes down. so literally about compressing neural networks. So what led you from from compressing networks to now building very large one trillion parameter models? So on the on the on the paper I think all credits to my PhD advisor from the time Ju is like a master to pick up good titles for for papers but I think the the thing that is a bit overlooked on compression. So basically taking a big model back in the time a big model was 100 million parameters and compressing it. So we managed to run I think CNN so like im image detection models in real time on an iPhone 7 that were that was rolled by shred back in the time Facebook CTO the iPhone was a bit hot somehow but that that was like the kind of things we we were allowed to to explore which was amazing.

26:22 So this there's always two sides to operation. The first one is of course take a model and make it smaller faster as well. And that's key if you want if you care a bit about privacy. If you take a model and you run it locally in your laptop, your phone, any smart smart device in your home, then it means your data your data stays on the device and never goes to any any third party server. So that's that's also key actually.

26:50 But then the aspect that is a bit of a look is that the techniques you use to actually compress those models. So sparity basically contization in with the floating precision. So FP8, FP4, you actually can leverage them to train even bigger models. And that's exactly the the two aspects of compression. And once you've mastered taking a model and removing what's not necessary, then you can train a way bigger models. For instance, you can use SP attention patterns, can train FPV, FP8 even with some some kind of F FP4. And that's basically the the the aspect of concussion that is sometimes overlooked.

27:30 >> Okay. All right. So great context on the open source system, great context on you. But now let's deep dive into the model into the chunk. So for everyone listening what is a mixture of experts model which which is the core architecture here where chunk has a trillion parameters but only 50 billion work on each token and what what why does that beat a normal dense model >> so that's again that's something we've been researching at ML for years we've been waiting the first the very first mix mix mixture of experts three years ago that spark the the modern architectures. So really like you it's really a beast like a M of expert on this side. There's no other world that's a beast. Now what do you gain from it?

28:24 The the advantage is that at first time each token sees 50 billion different parameters interact with 50 billion different parameters at a time. which means that the inference speed roughly speaking is the same as a dense 50DB model, right? But the capacity of the model is like way way way larger. In this case, it's 20 times larger. So that's the big advantage. The the downside or actually the the thing you have to take care to the downside is actually that in pre-training those models are much much more data hungry. So you need to feed them more with with data and with good data.

29:08 So in other words, you pay a premium at the training side by making the bet that the model is going to be way more used at inference time, which is essentially a good back to make if you're confident about about your model. So they're super data angry. You need a lot of tokens per active parameters. So that's the the measure we use to preend them. And maybe the second aspect that underlines a bit all the science you have to do in particular if you're constrained on GPUs as you as we were is that it's hard to train them properly. Again, it's a beast. It's a living beast. So you might have problems with dead experts. So part of the model that actually never see tokens. you also might have problems with some gradients or information flowing badly in the model imbalance on which experts sees the most the most tokens.

30:01 So it's really something you have to control and to develop an entire science around to train I guess very well but it's worth it in particular on the on the inference side >> and to make sure this is accessible to a broad audience. One analogy I heard about what means in the first place is a hospital and the idea that if you have a patient coming in, you're not going to send the patient to, you know, 50 different doctors, you're going to send the patient to the doctor that can fix the problem. So the orthopedist or or whatever example is that directionally correct? It's a great metaphor and I would add that the routing part so which doctor towards the patient.

30:50 >> Yes. So how does how does that work? >> It's actually learned by the model. So that's the beauty of it. the model learns. So we have some routing layers from time to time and these routing layers they actually they have one task is to assign to basically assign each token. So each patient to its right specialty and big models like that they have like hundreds of experts and the order this this ability to direct the the token to the the expert or actually the top two or four experts that he deems the most relevant. And the beauty of deep learning is that actually works. In other words, if you want to learn the end to end the model is able to learn both the routing and both the the experts. But there are a lot of interactions between that. For instance, if you never wrote tokens to some specialty or patients to some doctors, then the doctors they can't learn, right? Because they never see a patient.

31:48 So there's this subtle balance you need to to sustain through training. And that's one of the key >> and that's that's a dead expert problem you were alluding to. If if if the doctor never sees a patient >> exactly unassigned doctor or dead expert and on the contrary you also don't want to overload a doctor because otherwise the other doctors will be really idle and it will disrupt also the training. So it's a super subtle science as to how to balance the traffic I would say and that's a key part of what what kept us busy for for the preending of ML4 >> is part of the research at the frontier of to keep pushing those those dimensions. So going from one trillion to two trillion but then going from 50 active to 20 active is that is that the way to think about it?

32:42 >> Yep. That that's the idea and again the beauty of deep learning is that when you scale it works so it works with a big asterics obviously the being that you need to invest an insane amount of info and science to scale accordingly that's what we got but it worked. So then the question is if you if you have more compute how do you split it in which dimensions you scale? So first usually you split half and half you split half on more data. So the model at training time sees more token and you split also by increasing the model size and to increase the model size I think there's two axis the first one is the total parameters so the capacity of the model ML4 is 1 trillion total.

33:24 I'm willing to bet that in the open you're going to see models with 10 trillion total parameters very very soon. But then you can also play with the sparity ratio. In other words, it's 20 for ML4. So you have 50 active one T total. You can push it further in in the sense that the 10T total will have maybe a few hundred active parameter 100 deactive parameter not more. It comes with a lot of scientific questions. The sparer the model, the harder it is to actually keep that balance and make sure all the experts learn at the same pace so to say. But that's really like the challenge we're trying to to to solve right now at way larger scale than they might for obviously >> to which extent is as an architecture ubiquitous across the AI ecosystem. I mean I assume we we we don't know at all but the GPTs and the and and the you know various claw models are there as well behind the scene is everyone at this point or is that still specialized as an approach?

34:36 >> I would say yes with some exceptions obviously we don't know everything we can't tell everything but I would say yes. The big thing to to understand here is that MO is a great architecture, but the the thing not to overlook is that it's always co-apted with the hardware and all the ecosystem. So even though we were to find a better architecture, it would be harder to prove its value because the the GPU, the inference, the optimization techniques, everything is tailored to the MOE. So and I think there's a lot of untapped potential in the MOS still. In particular, you can take this architecture and loop over it, make it a bit more recurring or like go through diffusion models. One doesn't prevent the other. So I think there's still a lot of indepent and that's the direction we're exploring it at this time.

35:29 >> Yeah. And deepseek is as well, right? So you mentioned up front the open source ecosystem sort of influences itself through various players. did you guys borrow some stuff from Deep Seek? did they borrow some stuff from you? How does that work? >> Yeah, totally. So the they certainly borrowed and I think the the closed source lab also borrowed the very first mixture of experts that we find we did in the open in Fantasy 3 and for instance we're going to open weight ML4 in the end of October. so in in a few weeks you can see that this model also borrows the variant of this deepseek sparse attention because it's a great contribution that fed a bit to us. It also has a great feature that distinguish it from these models in particular it shares KV cache among the layers. So it's it's a bit more efficient as to memory management would say and it's a bit bigger as well than than GMO. So GM is like 700 total, but it's also way smaller than Kim than Kim.

36:37 Kim is almost 3D total while being super competitive with them and again better insider manufacturing finance and a bunch of other interesting benchmarks. >> Great. Speaking of sharing KV K cache, one interesting thing is that you guys have one model that does both quick answers and and long chain of thoughts and you can have had two and you choose to have one. Why is that? >> I guess it depends what you mean by two models because here there's it can be like a completely different model or it can be a model that is a bit different for which the inference is going to be a bit easier. So I think the operationally it's way easier to have one single model to rule them all and I think the the key is to bake in into that model different reasoning modes we call them which basically calibrates how long we want the model to think to to a question righting mode low is essentially simple question simple answer and then reasoning mode high or x high is basically think as hard as you can even if it's not the most efficient token wise because I want to be to get the best result. So it's really a way to navigate what we call the paral frontier. So the trader between the speed or the cost and the quality of the answer. I think there's we don't want to train one model pering mode. I think the the idea to have one model for all for two reasons. So the first one is operational easier to maintain and to to ship to customers. We do ship model weights to customer tenants. So they're able to actually do the inference on their ecosystem again sovereign no access to the API including ours. So that's one but the other reason is also that there's synergy between the modes. one of the big questions and big challenges in reinforcement learning so the last phase of training is to actually teach the model to reason at different things. So we have some what we call length penalty that is tailored to different reasoning mode and basically we tell the model okay here you have this kind of budget please respect it and do as as good as you can but then when we increase the budget the model can already build on top of what it has learned on the lowering mode right so I think there are synergies to be taken into account and that's also why we train multiple reasoning modes on the same model and again part of Big part of the reinforcement learning marvel is to tailor them properly with product information for instance.

39:15 >> Talk about the multimodal aspect of Mistrol 4 in terms of what it's able to do and the unique challenges of training a native multimodel model. >> Yes. So I think that's that's key to to master multimodel and that's key to do it natively. So what does it mean? First to to do it natively there's two ways to you can can there's two ways to add image as input to a model either from scratch or or afterwards. If you do it from scratch, it means that all the tokens that the model is going to see during the training are going to be a mix of text and relevant images which will allow the model to learn in context very very very very soon and to learn entire image domains right satellite images technical doings for instance or you can patch the model afterwards and when you patch obviously it's easier from an point of view because you have the complexity under the carpet but obviously it's some optimal In terms of performance at Mrol, we we love to to hit the bullet and basically to to do the the right thing first even if it's like the most complicated one because it's principle and it's the right thing to do eventually.

40:35 So we train the trunk ML4 native multimodel which means that the model is the best to understanding special images. So to to basically for instance identify objects in a satellite image we back in grounding in the model. So the ability to detect and tell you in which part of the image some objects lie. It's also great for industrial manufacturing. So for instance you have like this technical drawing a bit like a manual where you have like parts informations about the parts and then you can ask the model to reason about the technical drawing for instance find me the the total weight of all the parts in this object that are on the on this side for instance and this this capability I think is key because the world is natively multimodel and you need both text and image to do so.

41:33 And on image again we we are the frontier if not the best on many many many domains which also unlocks further further that further aspect of the model in particular interacting with a computer that's essentially a bunch of pixels if you think about it at a very level >> computer use. >> Exactly. >> Okay. Speaking of our diagrams and satellite images, how does the whole data aspect work? so specifically for those questions, but in general, right, there's been this narrative that we've we've the world has run out of internet data. and you know, we need to create data, then we need to acquire data. How did the data effort to work for Mistrol 4? I think it's it's great to underline because it's also unique skill we have at Misol which is to actually process the data in a way that is the cleanest for the model. So when you train a model you want the tokens to be as efficient as possible. So you don't want to always compute on tokens that the model has already seen or that has a bit that are become too ambiguous such as the model doesn't maximize the learning signal. So there's the notion of token efficiency that is one of many aspects that we optimize to be able to actually train beasts like ML4 that are at the frontier on such a small amount of comput.

42:58 Our our chief scientist is super known for eyeballing at data and finding exactly I don't only like a six sense for that and finding exactly what's wrong about the data and it has amazed me for years still. So that's I think the first part is to curate the data and process it in a way that is the most digestible and efficient for the model. But maybe the other part is as you said there's a finite amount of data that we're allowed to use.

43:29 Then what is the next frontier? I think the next is pretty much synthetic data but it's it's also a double-edged sword because if you feed an LM with the LM generated data then you might essentially have a mode collapse. So basically you might have the model actually tell you that it learns it but just memorize perfectly and not generalize roughly speaking. So it's it's to be it's it's harder but there's many ways to do synthetic data. You can have real world seeds. you can rewrite you can rewrite such some like new and generated content in a sense data augmentation right many versions of the same text and all these kind of aspects I think you can push and that's basically one of the next paradigm and next frontier that we working on and it took care obviously way more compute because you rewrite with a very good model and you need to rewrite or to at least generate or experiment on dozens of trillions of seconds very easily.

44:36 >> So synthetic data is a major effort just to play it back. And so you have a team somewhere in your 300 researcher that's specifically focused on creating synthetic data that's efficient and that's not going to create the problem that you described. >> Yeah. And that's a super we love this kind of problems because it's it was very hard to define what is a good static data. It's it's also expensive. So they need to find clever ways to to basically predict at smaller scale what will happen at large scale. So the scaling essentially there's many many aspects of it and we have a great specialist in house Arthur our CEO is one of the first first the scaling paper back in the time so we draw for everything of course they don't predict always what happens at large scale that's the beauty of it as well you can always run into unknown unknowns at large scale because again those models are beasts so to to better understand them But that's usually a big focus of area for us. Y >> great. Let's talk about the the compute aspect to this that we alluded to a couple of times but in in greater detail. So you mentioned it's a small amount of compute. I mean 3,800 Nvidia Grace Blackwell sounds like a large to me but I guess in the grand scheme of things it's it's small. walk us through anything that you've learned, anything that you expected to work that didn't work or or vice versa to which to to the extent you can share.

46:08 >> Yeah, I think that's that's also the beauty of being open weight. You can be transparent and I think it's important to be transparent about what it takes to to train such a model and it takes I think an extreme dedication from the team around the months. So it's not you press on the button and it works. Much more subtle than that. You need to monitor the model and to prepare for the next steps and the next generation.

46:34 So it's it's a super important dedication from the team and I I'm super grateful and by all the work they they put in maybe to zoom in on one particular aspect of of the training. That's again the reinforcement learning part. So to summarize, reinforcement learning is pretty much like you being an active student. So you've presented with a problem, a complex one, the day of the exam, and you can try as many times as you want, say 64 times to get to the answer. And if you get it right, so we have a way to verify the answer, then you get bonus points. And you will see if you get it wrong or if along the way you did some suboptimal behavior then you get penalty points.

47:20 And the beauty of reinforcement learning is that it works. The the difficulty obviously is to calibrate the difficulty of questions. If I don't know what you what was your major in in high school but if I present you with an exam that is not in your major you might not learn even if I let you try for a long time. And conversely, if I present you with a very super super easy exam on your on your major, then you will succeed in everything and you won't learn either.

47:48 So the big question is to calibrate the set of problems to a model you didn't train yet or you're currently training, right? Because we did it we did things back to back obviously and I think that what amazed me is that the reinforcement learning part worked one shot. So basically and what does it mean? It means that and we were at the global company of sites. So imagine like 1k people some of them partying some of them were in the war room to start the the reinforcement learning including me and the the marvel is that you start the training you present the model it's it's a very complex infrastructure again on on free GPUs because it's it's asing by nature we can can go back to that you start the training and then you see miraculously the reward or the success rate of models to these questions increase and that's really the beauty of it. It means that you calibrated every everything well and that the model works. And the beauty of modeling is that there's no such thing as saturation as long as you find complex enough problems that are of course relevant to your customers.

48:59 and the big difficulties to calibrate and to increase the complexity of problems as you as you go through the training. And again I spoke about you know pre-training stability the balancing and the dead experts before reinforcement learning is also kind of another beast in the sense that the training may be super instable. You might have the model can do since the model is allowed to explore in many ways, right? And you will reward or penalize it against it. Since the model is able to explore, it can do crazy things and it can basically crash the training. It can diverge the training. So there can be some precision problem or some divergence between trainers and generators. And all of this we monitor and we actually manage again we managed to make it work zero shot. I think it's a testimony of the things we're going to be able to make in the future with 10 times or 100 times more compute. So that's I think that's the thing I want to underline.

50:00 >> You mentioned the term calibration. Is that the secret? So because you use about a third of the compute that a deepseek or a moonshot use and you were able to extract all this performance. So is is the secret that the calibration is that is it the data and being efficient as you said is it the architecture is that the infra is it the calibration where does the the juice come from in terms of efficiency so the again the beauty of it but also the curse of it is that it's everything and so and but I would say that you need to be able to master everything to get to this level of performance so to give you an example you need to calculate the data perfectly problem that are not easy nor too and also relevant for what you want to do. You need to have a rocks already infra.

50:54 There's many moving parts in this line for learning in front because you don't know what the model is going to do in advance. So you need to be robust to that and the way you do it is basically to optimize for the throughput. So the model needs to see as many problems per day as possible. And I think here we have a like state-of-the-art throughput in this training because we spent a lot of time designing the the topology and improving many things on on the generator and the trainer aspects.

51:25 There's a science aspect to it. So you don't want the training to diverge. there's a reward shaping aspect to it. So you want to reward models properly. And here I think it's important to underline that such models minister of experts whenever you do something slightly wrong they take their fingers and tell you here you did something wrong I'm going to bug and crash the train that's that's for sure there are so much capacity so much memory that's also a curse and a blessing and we had like many many cases of the model hacking the reward or like the model going outside of the sandbox for instance in the training and all of that I think is super important to master to get to the the end training that actually works and that can run for days without saturation.

52:18 >> Great. I want to go back to this in a in a quick second but just on the reinforcement learning part itself I read somewhere that it's still running. So why did you decide to do this like launch the the public preview while still running RL and at what point did do you decide to stop running the RL? >> Yep. So it's a great question. I think we had a an incentive to prove to ourselves and to the world that we we train models. I think when we knew RL worked, we were super excited and we we actually previewed the model that was and we throw the checkpoint like two days before. So we we are super happy with with it. But also we wanted to give reinforcement learning its best shot and try to push it as far as possible as part of the release which also buys us time to make sure the model when we open weight can't you can't be used maliciously by by actors. I think that's important.

53:20 But yeah, AR is still running and shows no sign of saturation. And again, I'm amazed by the the work of the team because the model can do so many things wrong during that part that you need to be rocked on on the science and the intra to to channel the model to to pro to progress basically. >> And the RL is focused on on specific things, right? Coding, finance, is that is that correct? and what are the specific things it's focused on? So the the standard example of what is a naral instance we call it is basically some some code base and then a description of a feature to implement. Right? So that's the question. Then the model can do whatever it wants putting writing code and trying some things or using more tools if it wants and in the end you verify with fail to pass and pass to fail and test that the the feature is actually correctly implemented. so that's like the typical instance and it works well because it's I mean code is very easy to manipulate. It's easy to know at least it's not that easy but it's more easy than in other domains to know if the feature implement is correct but we don't stop there we have many more domains in the reinforcement learning in which the ground truth is a bit more subtle so for instance if you ask the model to compile results to to look for the to to look on the web and to search information and to build the one pager you can't deterministically evaluate the one pager Right. So for that the the beauty is to use a judge like the same model actually that judges itself on whether and gives points or awards or rewards as to some criteria on whether the report is good or not. So that's another kind of instance >> and that's for domains like finance and law. Take a step back like the the the one of the usual big questions in reinforcement learning is that for for coding and math you know what is right or wrong for finance or law you you may not.

55:27 >> Yes. And that's where you develop what we call the rubrics which are and rubric can be like in this one pager I want this amount to appear because that was the question this the tone has to be that way I don't want to to mention this because it's false let's check that everything is consistent like any grading points and it can be pages long models can attract that away for you so we have rubrics that we actually also generate with models everything is like loop at this point and it's really like as a judge glorified but that's really the idea that's how we make we can make finance and law work and all the other domains in which we're frontier so the cyber the manufacturing and so the finance on the cyber it's pretty much painting the model with a real life scenario in which there does an attack and the model needs to analyze logs toate assumptions ions and to basically propose and test a patch without disrupting the service. That's like big part of it. Basically any real life scenario for which you can verify it in a reasonable time like a few seconds you can crack it with reinforcement learning. That's again the beauty of it.

56:44 So in cyber we did a lot of efforts for real real life scenarios and in manufacturing there's this idea of having basically the model being able to design a chip for instance and part of it comes with agency. So the ability to manipulate the software on the computer to basically place all the components on the chip and part of it lies in the ability of the model to actually simulate for instance the heat on the chip. And if you don't simulate the heat of the chip and you play the components wrong then you might melt the chip essentially. So that's a bad reward. But if the model after trial errors in an energetic way manages to play the component and then to simulate the heat dispersion that passes like the criteria I would say on the simulation then it's then it's a it's a good point for reinforcement learning. So again any task any task that you can run in the sandbox that you can verify not deterministically but that you can verify through LMS or through means within a second.

57:52 This we can track with army and again we made a we made a few bets and on these bets we're frontier we don't have the same coverage as the the competitors and that's okay we on the frontier we're on the as best at the best open weight control on the capabilities and with we the best and some bets that we made and that we accel just for final success or does partial progress get rewarded as as well.

58:22 >> We also have a so when you it's like it has been the fear of reinforcement learning theorists for like years is to basically manually tweak the reward because here it's a bit like the game where you have some some animals and you have to hit them and you can't never you can never hit them at once. Whenever you try to patch something in the reward then you have another leak and you need to patch it and then you need to patch it and then you need to patch it again.

58:47 So it's it's super subtle science, super hard to do, but yes, since it's the right thing to do, well, we do it at NR. And in particular, one example is that you might know about doom looping. So the model being stuck in a loop or almost infinite loop. So that we penalize, but we also penalize during the trajectory if the model loops two times even if it recovers afterwards, right? in the end. So let's say the model loops times on the tool doesn't need to find the right answer.

59:21 In the end we tell the model okay you found the right answer that's good but be careful in this particular part of the trajectory you had a fail and please don't do that again or we will be very mad. So that's that's basically the idea of reward check here and we have a lot of uristics along the way and again the only way to learn about these uristics is to actually train models and gain experience as to all the crazy things models can do. Models are way more imaginative than we can be regarding the reward hacking part. So again that's that's a big part of the science we do at VR. Another classic question of reinforcement learning is how do you know that the reinforcement learning created the capability versus just emphasizing something that was already in the pre-training.

60:08 >> So it's an open question to be honest and the the I mean I will give you the order one answer. So the the the first order answer before investment learning what we usually do is to precondition the model. So to come up with a student metaphor before going to an exam and actually don't having no access to the answer what you might do as a student is read both a set of questions and answers from past exams right you don't learn as much at least in my experience you don't learn as much when you do so but that's a start right and that's what we do before in fast learning we precondition the model you might heard about mid training or SFT that's the part where we precondition the the the model to that >> which is supervised finetuning >> training of supervised fine tuning. It's essentially the the idea to show the model both the question and one answer that is that is correct and and basically the when you do so you have say what we call a pass at K. So you give a question and the model answers at least one time, right? If given like 64 times, right?

61:25 You are a student, you duplicate yourself 64 times with different seeds, you at least one of you will pass the answer. So that's pass at 64. What reinforcement learning would do is actually reinforce the best version of you such as it's not pass 64 but no pass at one or pass at so such as you're much more likely to give this answer. So in that sense it's magnifying or reinforcing a capability that you already had a bit by luck right you punch a bit above your weight before reinforcement learning and after reinforcement learning is putting work for you. So that's like the first order answer.

62:06 But then and that's again the beauty of it. You can also learn new things during reinforcement learning if you have enough budget and enough stability of the training enough days of learning to actually discover new behaviors yourself. and that's something that is super hard to do because you at the beginning you would waste compute, right? You would have the model or the student try 64 times and you don't succeed right on it. But then there's going to be some synergies. The model is going to learn on adjacent tasks like some some tips and tricks and then go back going back to the the big problem afterwards and then succeed. And again it's a subtle science of how to calibrate all the difficulties of instances.

62:54 Exactly like you as a student. You don't start with the biggest problem. You don't start with like the the hardest problem in the book. You start with the smallest ones. You try medium one. You don't you don't manage then you go to to easier ones but that are very very close to the medium one in a sense and then you go to the middle one. So that's again the the beauty of >> all right. I want to go back to some of what you said and and and talk about cyber for a bit in in in general. so you mentioned the model trying to get out of its sandbox which I think was reported by Reuters as as well. So you know models escaping sandboxes has been like the whole meme of the theme and meme of the summer with some very fun things on on X. What happened in your case and what can people learn from it?

63:47 >> So I think the first thing to say is that we didn't make a f about it. we just fixed it internally that is not fixed. And I think it's very important also to underline that these models they're super capable in particular in cyber that's great it's great to defend and to strengthen the defense of the entire ecosystem and as their capabilities increase you must increase the the guard rails or like the the sandbox around them right what we say that guarding these models is a software problem we very much warn people against anthropomorphizing models that want to escape. Some boxes in particular, they just follow instructions. Maybe illos instructions, but they follow instructions. And really my take is that if you take very capable cyber model and you have the same model monitor itself and on whether it's doing good or bad or manicious work, then you reach a good equilibrium and you can actually circumvent the model. So it's not a fatality. It's hard. I'm not denying it's hard but it's there's no fatality in being able to control and to guard models just as software and what we do at Mrol is not fear people just going step by step and fixing it and providing guarantees to customers that the first thing they ask guarantees and like best practices as to how to how to monitor basically the models and making sure they are used in in a good way. So that being said, we saw some behavior from the model during the the training.

65:23 I think the the the first one that we saw and that's like I think everyone saw is the model being in the sandbox and being able to ping different services in the training stack, right? And at some point our training crashed at small case. the before the final one. It crashed because in one sandbox the model tried to ping the trainer actually with a simple hello just to know what was going on here and actually it pings the trainer with the wrong kind of signature in the wrong way which was caught by code and then crashed the training. But that's typically the the thing that models do because the model was instructed in that case to build like some some web application. So it's it was natural for the model to try the ports also in the in the sandbox.

66:18 And there's a way to fix that, right? It's basically to prevent the model from or to monitor all the external connections and requests from the model to to to the rest of the world. and we fixed it and then we started the training and then we had the model able to intercept all the messages and to include them into the phone block. So we had leakage between various problems as well in the in the training. but again we and again we we spend the summer also that we spend the summer and we we made things right to strengthen our our training to make sure we have the proper guides and we do the same for our customers. And my message here is that there's no fatality. It's going to be hard, but we must absolutely invest into making sure these we can deploy these models in a reliable and way and to be able to monitor them. And that's why open weight is so important.

67:14 When you open write a model, you give the ability to the entire ecosystem to audit the model to make sure it make sure it's not malicious, but also to experiment themselves with the model. And hey, I tried to use MR4 at home. Here's my scenario. Here is here's how I monitor the model. Here's what I learned. That's like the perfect feedback for us. And that's what the community has to build as well. So that's why we care about weight as well.

67:44 >> Great. And to unpack some of this, so there is this whole discussion of the role of open source and open weight models in cyber and the tension is that open models are great to defend, but the part of the problem is like to know how to defend you sort of have to learn how to attack. And so there's this whole concept of of refusal and we had Thomas Wool from Hugging Face on this part a few weeks ago and he was talking about you know that whole story when the OpenAI agents started hacking them they had to defend themselves. Yes they had refusal they couldn't use Enthropic to protect themselves therefore they had to use I believe GLM 5.2 two. So that was great. But you know presumably you can also use GLM5.2 to learn how to attack or create a a cyber attack. So how do you guys think about this?

68:46 >> I think we must be honest and the reality is that there's a trade-off for for open weights. We believe it's in favor of open waiting in white but it's really a trade-off. So what what we do is basically to bake in refusal guard rails into the models we open weights that we test internally and also we trusted the partners and governments that way also we open way the model after a few weeks so we them into the model and FYI we refuse way more medications than the Chinese ones for instance so that's great but the thing to say as you said is that essentially if you're good at defending, you have the potential to be good at attacking. The question is now, what strength do the guards have, right?

69:37 And the the answer is that they're necessary but not sufficient. There's always a way to take the point model and spend a significant amount of energy to ungodly it if good at cyber. But then if you spend as much energy on guard railing the model as it takes to actually fine tune the model or train it yourself then it's not the same trade-off right because the guard rail is a potential barrier essentially it's not 100% safe if the model is open weight but it ensures that you need to spend way more effort to actually use the model in a malicious way so I think that's that the thing to say now I believe The trade-off is still very positive and very much in favor of open marketing in part because of the open open in particular because of the open open in particular because of the face incident the I want each and everyone to be able to defend themselves with a model at home essentially or in the ecosystem.

70:43 If you depend on the third party API, you can get cut off. As simple as that. And you can also hit refusal that you can't see comments. And usually in the API, the the refuses are a bit more more important. And by opening models, you also basically empower the entire community to accelerate on the defense part. Again the open source the beauty of open source that it accelerates research. People often use this analogy. The web and all the Linux servers rely on open weight libraries that have been tested over and over and over again and open source for years.

71:26 And really on the model side, I think there's a huge advantage in enabling each and everyone to edit the models and to learn together to defend themselves but also to bake in the right guard rays. I I'm not comfortable living in a world in which one or two companies are promising me they can guard the models themselves. I'm very much not comfortable in that. I prefer very much the aspect the option in which decentralized and real and actually people are actually comfortable for their models.

72:05 >> All right Pierre as we get towards the end of this conversation I I wanted to quickly touch on a couple of things. so we talked about lonk mistrol 4 but you also shipped time stroll which is a time series model talk about this quickly what is it what does it do and perhaps why can it does it need to be separate? >> Yep. So the one of the key strength of is not only to train frontier flagship models like ml or lunk that are the best but also to train specialist models. So we have great OCR models so to be able to extract text from images. We have great and state-of-the-art audio models vual audio in audio and we also have great specialist models in particular time tool. So here the idea is to be able to predict the future given the time series. So given some climate fluctuation, some energy consumption, some finance indicator for trading. And here I think the key part is that this model is so small you don't even need a GPU. You can run it on the CPU and the prediction for a meaningful period of time will only take like a few hundred millconds. So that's why it's so small actually. And this model is something that is asked by a lot of our customers because what we bring to the table is not only a model but also the entire expertise to transform a business with with AI and for that you need a flagship model obviously. You also need all the missile expertise to turn any enterprise knowledge or domain knowledge into specialist models that go the extra mile and that are fine tuned for the use case.

73:48 And we do that in for planes, for chips, for cars, for all the heavy industries where you actually need to design very compressed components. And that's also the beauty of of it. I always say that at least we are scientists first and foremost. We saw the AI revolution at some point. So we saw the light in the in the behind some door. But we have we are physicists or mathemat mathematicians by training. And that's also the the one of the key characteristic of of >> Yeah. Do you guys use your own product to start doing some you know AI research? So perhaps the general theme of RSI and AI building AI is that what's the reality of this at Mistro today?

74:35 >> Yep. We are pretty much investing on it. RSI is like a I mean a is saying they're doing it. The reality is a bit more nuance. Obviously, what works super well is to augment the researcher by running experiments end to end with all the infra by ending all the infra failures automatically. I think that's great and it accelerates our most scenar with our models and then I think the meta question is like can you go one layer up and basically task a model to improve itself over long periods of time and that's what we call the flight time currently the flight time so the time it takes for a model to do meaningful work is like a few hours tomorrow with like ML5 is going to be a few for sure of meaningful work without the moving and like and also with a swarms and again this new paradigm is something we deeply believe in because it will allow us to close the gap with the the best labs even faster.

75:44 Don't forget that humans are very bad at appending exponentials. We don't know it works like we can't comprehend it. So we're only a few months behind the best pier models. It's distorted here but for an on index ML4 is on the level of GPT6 L and it's like way better than all the previous OP models. so again we have a few months delay only and we count on the next paradigms that we discuss in this podcast to actually close the gap and we prog them because the am to power the frontier and to define the frontier of of tomorrow.

76:22 >> Yeah. And on that note, as as a last question, so reports say that Mistrol large five already be in pre-training. What does the next model need to do or need to prove that this one did not? >> So yeah, the the never stops. So indeed, ML5 is already in the stage. It's much bigger than than ML4. So it's it's chunkier or faster, I would say. And really for this model, the thing we have is so for instance we were talking about cyber take the example of like a cyber incident that happens in your company and you are a security engineer and you want to cross reference tons of logs. some of them might be false alarms and to actually find the root cause and patch it while ensuring business continuity.

77:16 The the bad news is that there's no playbook for that. I mean there's no way I can write you like steps a prior to do that. The good news is that with models like ML5 they won't need a playbook and they will be able to actually act reactively to to the the information they have. And here to to to to for cyber for instance, I would expect ML5 to be able to work for a full day of course in a swarm with multiple agents to identify major security incidents and to patch it patch the root cause while being able to ensure business continity and while being able to communicate on the progress thoroughly and that's like I think what the the compute the size and also the the science we put in this model will will be able to unlock on cyber but also an agentic AI for engineers for instance designing a cheap end to end like sounds hard but sounds also super or being able to do some meaningful productivity work on finance so I really I pretty much look forward to to seeing what this can do >> all right Pier it's been absolutely fantastic Fantastic. Thank you so much for spending time with us. Really appreciate it. Congrats on Lonk. Very excited to see what you guys do next.

78:42 >> Thank you. >> Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.

Summary

Pierto, the head of the research team at Mistral, discusses the launch of their latest AI model, Mistral Large 4 (Lunk), a trillion-parameter model trained in just two months. The conversation highlights the importance of open-source AI, the challenges of training large models, and the implications of AI access on global sovereignty. Mistral aims to push the boundaries of AI while ensuring that technology remains accessible and controllable.

- Mistral Large 4 (Lunk) is a trillion-parameter model trained on 3,800 GPUs in two months, showcasing Mistral's capabilities in AI development.
- The recent shift in the AI landscape sees a resurgence of Western open-source models, countering the previous dominance of Chinese AI.
- Mistral emphasizes the importance of open-source models for ensuring sovereignty and access to AI technology, likening it to electricity.
- The model employs a mixture of experts architecture, allowing it to utilize only a fraction of its parameters at a time, enhancing efficiency.
- Reinforcement learning is a key component of Mistral's training process, enabling the model to learn from complex tasks and improve its capabilities continuously.
- Mistral's approach to AI includes a focus on multimodal capabilities, allowing the model to process both text and images effectively.
- The company is committed to maintaining a balance between open-source access and security, implementing guardrails to prevent misuse of their models.
- Future models, like Mistral Large 5, aim to enhance capabilities further, particularly in areas like cybersecurity and productivity tasks.

Questions Answered

Why is it concerning to have limited access to AI?

The speaker expresses discomfort with the idea that a few companies could control access to AI, comparing it to electricity. They emphasize the need for a diverse and open AI landscape, highlighting the recent resurgence of Western models in the AI space.

How do different AI models compare in performance?

The speaker discusses the challenges of evaluating AI models, noting that while some models are performing well, others are lagging behind. They highlight their model's superior performance on benchmarks compared to competitors.

What are the challenges in training AI models effectively?

The speaker explains the delicate balance required in training AI models, likening it to a doctor needing to see patients to learn. They emphasize the importance of calibrating the complexity of problems presented to models during training.

How does reinforcement learning contribute to AI training?

The speaker describes the successful application of reinforcement learning in their model training, noting that it allows for real-time adjustments and improvements in performance. They highlight the importance of presenting complex problems to maintain learning momentum.

What measures can be taken to ensure AI models are safe and controlled?

The speaker addresses concerns about AI models escaping their operational boundaries, emphasizing that these issues can be managed through proper software controls. They advocate for a proactive approach to model safety and monitoring.

© transcribe · For agents Built with care and craft by Gokul Rajaram