Section Insights
Binance Hack Overview
What happened during the Binance hack?
The Binance hack resulted in the loss of $6.3 billion due to a compromised engineer's laptop, which allowed attackers to transfer assets without immediate detection. They used tactics to maintain the appearance of normal transactions and leveraged DeFi exchanges to obscure their withdrawals.
- The hack was the largest in history, involving sophisticated tactics.
- Attackers manipulated Ethereum transactions to avoid detection.
- Binance's recovery efforts were complicated by the decentralized nature of crypto assets.
Trust and Security in Companies
How does internal trust affect company security?
Maintaining trust within a company is essential but can lead to vulnerabilities, as employees may inadvertently expose sensitive information. This trust can be exploited by malicious actors if proper security measures are not in place.
- Internal trust can create security blind spots.
- Sensitive information may be stored insecurely within company communications.
- Companies need to balance trust with vigilance to enhance security.
Vulnerability Management in Cybersecurity
Is achieving zero vulnerabilities possible in cybersecurity?
The concept of 'vulnerability zero' suggests that, like in early COVID, there may come a point where vulnerabilities are exhausted. However, achieving this is complex and requires ongoing vigilance and adaptation to new threats.
- The cybersecurity landscape is constantly evolving.
- Complete elimination of vulnerabilities is unlikely; ongoing management is essential.
- Understanding the lifecycle of vulnerabilities is crucial for effective security.
Identifying Corporate Crown Jewels
How can companies identify their most critical assets?
Many organizations struggle to define their 'crown jewels' due to differing perspectives among executives. Tools like ServiceNow can help identify and prioritize cybersecurity measures based on a company's unique assets and processes.
- Different stakeholders may have conflicting views on what constitutes critical assets.
- ServiceNow provides insights to enhance cybersecurity posture.
- Understanding business processes is key to identifying vulnerabilities.
Password Management Strategies
What are effective strategies for managing passwords?
Using a physical notebook for critical passwords and employing security keys like YubiKeys for sensitive accounts are recommended practices. This approach balances security with practicality for infrequently accessed accounts.
- Physical notebooks can be a secure way to store critical passwords.
- Using hardware security keys enhances protection for sensitive accounts.
- Regularly reviewing and managing access is essential for security.
Continuous Cyber Defense
Is continuous defense in cybersecurity a viable strategy?
The concept of continuous defense involves automated systems that regularly update and patch vulnerabilities. While it presents a promising approach, its effectiveness depends on implementation and the ability to adapt to new threats.
- Continuous defense can enhance cybersecurity resilience.
- Automated systems must be carefully managed to avoid introducing new vulnerabilities.
- The effectiveness of continuous defense strategies varies by organization.
Mistral 4 Large Model Release
What is significant about the Mistral 4 large model?
Mistral 4 is a groundbreaking AI model with 1 trillion parameters, representing a major milestone in AI development. Its training involved significant infrastructure and technical challenges, showcasing advancements in AI capabilities.
- Mistral 4 is the largest open-sourced AI model to date.
- The model's development involved overcoming substantial technical hurdles.
- Scaling AI models effectively requires robust infrastructure.
Open Weights and Cybersecurity
What are the implications of open-source models for cybersecurity?
Open-source models can be beneficial for society but also pose risks as they can be misused for cyber attacks. Companies must balance the advantages of open access with the potential for malicious use.
- Open-source models can enhance cybersecurity but also introduce risks.
- Responsible access and usage guidelines are essential.
- Balancing benefits and risks is crucial in the open-source landscape.
Team Decision-Making in AI Development
How are priorities determined in AI development teams?
In early-stage companies like Mistral, team decisions on priorities are often collaborative, leading to alignment on goals. This collective approach fosters a shared understanding of what constitutes success for their models.
- Team alignment is critical for effective decision-making in startups.
- Collaborative approaches can enhance focus on key objectives.
- Clear definitions of success help unify team efforts.
Sovereignty in AI and Technology
What does sovereignty in AI entail for nations?
Sovereignty in AI requires nations to develop their own models and technologies to avoid dependency on external powers. This includes not only AI capabilities but also essential infrastructure like satellite communications.
- Nations need to develop independent AI capabilities for true sovereignty.
- Technological independence is crucial in the modern geopolitical landscape.
- Infrastructure development is as important as AI model creation.
Transcript
2:02 Mhm. >> Ready if you are. Good morning. It is Friday, October 30th, 9:02 a.m. Nathan, good morning to you. >> Good morning.
2:33 I hope you weren't holding any assets in Binance. >> >> I I hold I hold very little crypto right now. Wow. So, for those who are just tuning in, Binance had the largest hack in history. 6.3 billion dollars lost over the course of I guess about 4 hours or so. the way the hack went down, you we had a an engineer engineer's laptop, which got compromised.
3:09 having been compromised, what happened was over the course of a, you know, several minutes, several minutes, assets were transferred out. The assets were transferred out in such a way that Binance didn't detect them for a while. One of the smart things that the attackers did was that they moved Ethereum while leaving enough Ethereum within the wallet so that Binance's other transactions would continue to go through.
3:43 So, Binance didn't notice because, you know, other transactions which needed some Ethereum would continue for a while. And then the other thing that the attackers did is that they didn't withdraw some of the money directly. What they did was that they went and they pledged that money to a DeFi exchange and borrowed money from that DeFi exchange. And so, what Binance did in recovery was to go and ask and you know how these are decentralized currencies, they went and they they asked a centralized custodian for Tether to freeze the the the addresses to which, you know, this Tether currency had been withdrawn to.
4:32 Unfortunately, that address was the one pledged as collateral to the DeFi exchange and that DeFi exchange had then lent out other currency backing you know, this this amount. And so, yes, there is frozen currency, but that frozen currency doesn't necessarily belong to Binance. It belongs to the DeFi exchange, which has really been robbed. and so, there's these questions on how this recovery is going to work. Some exciting times ahead.
5:04 6.3 billion CZ, who is the CEO of Binance, claims that everything funds are safe. Binance is very wealthy. Binance has 110 120 billion dollars in assets. So, people are not expecting Binance to be affected, but as a result and as a result of Elon kind of like, you know, on Twitter, >> Elon being Elon. >> CZ decided to sell all of his SpaceX shares. So, CZ pointed out that he had 500 million dollars from Twitter and the Twitter purchase which had then become a billion dollars in SpaceX and he dumped his SpaceX shares. SpaceX down 4% this morning.
5:46 and having dumped his SpaceX shares, that's part of the recovery. I don't know. I I think I think it's it's interesting. It's it's kind of also it's the North Koreans. If if you if you were interested >> It's the North Koreans. >> some of the North Korean teams had taken it out. speculation is that it's not any of the newer attacks. It's just basically old dumb, you know, the usual fishing attempts that worked this time. So, >> So, should I understand that Binance had an overall security posture where one Like, is this something that this employee and by extension like a bunch of different individual employees could have done at any time along the way if they had chosen to? Or was there like I I didn't hear a particularly galaxy brain or like zero-day part of that story. This is just Should we Should we like understand this as having been basically negligently bad practices at Binance regardless of the ongoing cyber apocalypse?
7:02 >> Well, it's it's very it's it's Yes, but then who Who Who Who among us has not sinned? >> >> Well, you know, it's the it's the hundred-billion-dollar holdings that make you you know, in theory you should be holding yourself to a little bit of a higher standard. >> I think it's So, one thing that's always scared me about crypto.
7:32 And And And And there was there was a period of time when I was looking at setting up a crypto exchange, etc. One thing that has always scared me is that the basically the CEOs of these exchanges hold billions of dollars directly in in assets in like very sensitive kind of like hot wallets and cold wallets, etc. And there's this whole security system, which really relies on people, right? Because you need people to hang on to these keys, hang on to the passwords, rotate the passwords regularly, like a whole sequence of like very sensitive you know, you you I you can't imagine the CEO of Goldman Sachs holding the keys to, you know, a billion dollars, a hundred billion dollars that he can just take out at a whim, right?
8:19 >> >> But that's what the CEOs of these exchanges do. And then and then as a CEO, like, you know, as a CEO, what you're always trying to do is you're trying to delegate as much responsibility as possible, right? Because you want your team to be able to do everything and you want to focus on all the things that they cannot do, all of the exceptional things that they cannot do. And you want to reduce, you know, you you want to reduce the number of things that you have to intercede indirectly because you can never have that bandwidth.
8:47 and the thing that scared me is that you as a CEO, you have to be directly involved in handling the actual fund fund safety, which is which is nuts. Right? like the CEO of Goldman Sachs, like, you know, handling like, okay, do I, you know, And and it's very hard to trust people, too, right? It's very hard because you again you have these hundreds of billions of dollars, you know, a hundred million dollars in a guy's hand, right? Like, even if he trips and falls, it disappears.
9:17 but it's a canary in the coal mine. at the same time that this happened, we had another hack, which is in Polymarket. So, Polymarket was hacked. what happened with Polymarket is Polymarket is a back end for prediction market trading. They have a front end, which is the kind of the website and things that people go to, and somehow someone managed to do an attack where they interceded in between the back end and the front end and absorb some of these keys and started transferring stuff out.
9:49 Polymarket also says they will replace all lost amounts. the hack doesn't look as serious as the Binance hack. The Binance hack looks like a nation-state level hack, so. >> So, help me understand a little bit more of this whole like I'm always trying to assess these incidents before where are we on the spectrum of like negligence to scary? And when I hear it was just one engineer's laptop that got compromised, that sounds extremely negligent. When you say like, "Oh, it was a nation-state scale attack." That sounds like, "Well, maybe not so negligent, but just like, you know, it's tough when, you know, a whole nation-state apparatus is is trying to target you."
10:39 Maybe we just don't know yet, but like where where do you think we are on that spectrum in this incident? >> So, I will say that the the distance between the negligence and the nation-state is not that far. and and that's really like the nation-state portions of it are not the technical portions of it. The nation-state portion of it is number one, like being able to do it and get away with it because you're living in a nation-state which is not like accessible to the long arm of the law.
11:20 and that's only possible in a few places, right? Like very few places. If you were to do this in China, for example, and the hackers were in China, they're going to get executed. They're done. Right? this is happening specifically because there's a little bit of a space in North Korea where they're able to do a bunch of illegal things, internationally illegal things, and they're able to get away with it because they have nuclear weapons and you know, people are not able to like, you know, assess them.
11:46 and also, like if you were to do this in Russia, for example, someone would come and take that money from you. The KGB or whoever would come and you need to you need to pay into the system, right? So, in in in every other case, anywhere outside, I think China, Russia, and North Korea, like these nuclear-armed non-Western states, and you know, which are fairly strong, you you you would have the long arm of the law or the long arm of crime.
12:13 Of other criminal syndicates come and catch you, right? You need this kind of like intermediate state of you know, the nation state you being able to have the monopoly of violence in inside the state, but yet also allowing you to to to carry out these attacks. on the technical portions of it, if you were this technically good, you would already be working for one of these firms. You'd already be working for a Binance or whatever, because or you would be trading your own crypto, right?
12:41 If you were more if you were aware of all of the details of like because these guys tested that tested the wallets first. So they did test rotations of wallets to see how it would flow first. and then they kind of like you know, had all the wallets ready to be transferred out, and then you needed to do like multi-hop, right? They needed to hop the transfers. because you can freeze the immediate wallet pretty quickly, but you can't freeze like multiple wallets downstream, and then you definitely can't freeze once it's hit a DeFi exchange and be used as collateral and someone's drawn down on the other side, then then it's gone, right?
13:18 all of it can be reversed if people want to and if like you know, the other participants in the chain are willing to and there's no like you know, there's no drive-by, you know, collateral damage. but you know, again, I think we've spoken about this before, but my take is always been that large amounts invisibly are difficult. If you're going to do a large amount, it's going to be very visible. Everyone's going to see it, right? So and there you have like it's it is a large amount, but everyone sees it. This is not the Gwern scenario. Gwern scenario was superintelligence hacks, you know, cryptocurrency, builds up over time, and you know, it's like becomes enormously rich, and there's no ripples in the in the water, and it's invisible. That doesn't happen. Like this is like everyone in the world sees it immediately.
14:09 and then you can kind of like All right. do you do you want to go after a nuclear armed state? Like is it that important to you? Probably not. $6 billion like, you know, whatever. >> Chalk it up to the game. >> Yeah. >> still though, it is crazy that one laptop could do this. Do we think that this was like a North Korean who was actually hired by the company and like had sort of access for a while to really probe the system?
14:38 >> So, what happens in these cases is you are And And this is another issue. Like a lot of firms don't pay their people enough. If you If you're If Like so, this is the thing. If you're going to have the guy handle the keys to a billion or 100 billion dollars or whatever, he's going to need a stake in the company. Right? What ends up happening is you pay these guys like, you know, 150 grand, 250 grand.
15:04 and they sit in Asia. And then they get an offer for someone like, oh, you know, we're going to hire you for 500 grand. We're setting up a new, you know, the currency crypto exchange. And like, let's interview. And then when they go in for the interview, they they'll be like, here, download this piece of software and you know, use it to run the interview and run the code that you're writing for us. Okay, you download it and like, you know, you use the code and you use the software. And when you download it, you're also downloading a some some worm. You're giving access to your desktop because it's going to have to, you know, basically, you know, write something to disk. It's going to have to load something into memory.
15:44 >> >> You give it a bunch of permissions. And then then it it looks around, pokes around, and then finds stuff there. And that At that moment, that's when like things start, you know, It's very hard to run a company if you have to be persistently suspicious of everyone inside the company all the time. And you always have inside the company have some level of trust. And that trust then starts working against you because inside some emails, there will be some passwords to Excel or you know, some Excel or some Slack or some whatever. And at some point someone would have sent, you know, sent something in the last like 5 years you've been employed there.
16:24 And there'll be some non-rotated key or some other, you know, place where you stored a password of some kind. We have computer use agents now that can open up your browser and click around as though it were you. and at that point things break down. So. >> It's pretty wild, isn't there no audit system for these crypto exchanges? I mean, we of course are talking a lot these days about embedded evaluators and third-party audits and whatnot for AI companies.
16:57 it seems like this architecture at a high level like couldn't possibly have passed a reasonable audit cuz you would just think, okay, if there's any way I mean, I don't know anything about this, right? And I don't know really anything about crypto, but it just seems like the first kind of question you would ask it would be something like okay, let's say somebody leaves their computer unattended at the lap at the you know, at the airport for a minute and it gets swiped and now somebody's in. Like is that enough? Do we have, you know, systems that would prevent that from being enough to like drain billions of dollars of funds? If the answer is no, you don't pass the audit. So like were these questions never asked? Were they never were they answered but the answers were now shown to be wrong? Again, we may not know, but it's just it's very bizarre to me cuz this is like the most, you know, where where the invisible hand is really supposed to work is like in crypto of all places, right? Cuz like you know that this is the game you're playing and there is no like the whole point, right?
17:59 Is that there's nobody to come save you. So and that's supposed to be a feature of the system. That's like what a part of how they sold it. So you know that deeply in your bones, right? Like and yet you don't get to a point where you're secure. So I just I I I don't know. It's very confusing and I'm not sure what if anything it tells us about AI auditing, but it does seem like it's got to be bearish for how well these like auditing schemes could work.
18:30 but maybe just none happens in in crypto. I don't know. It's all very confusing to me still. >> So, the what what crypto ended up with is a defense-in-depth kind of strategy. And that defense-in-depth Yeah, that defense-in-depth starts off with >> bell. >> Yeah, that defense-in-depth starts off with if you're going to get if you're going to lose, you shouldn't lose everything. So, you have a hot wallet, and then you have a cold wallet, and then the wallets get colder and colder and then you have cold storage where you only have like written written down like passwords, et cetera, and you put it inside an actual bank vault, right?
19:07 and so, what it what was robbed was the hot wallet. The hot wallet is holding all of the cash which is needed for rotation during the day, right? So, during the day there's you know, Binance is receiving funds, Binance is sending out funds. So, that wallet is the rotational wallet. And so, as you can imagine, that rotational wallet has a lot of activity in and out. So, that's another reason that, you know, you don't you don't immediately detect it because the hot wallet is supposed to have this rotation in and out. It is the wallet that's And Binance seems to have very large hot wallets. See that you know, some very hefty amounts in the large wallets because it's, you know, $100 billion plus plus, you know, corporation in assets, and you know, it's doing like billions of dollars a day in trades.
19:49 And it's in it's from that hot wallet that, you know, it was stolen. at the end of the day, you always come down to someone's got to hold the keys. It's it's, you know, there has to be someone holding the keys. and it's always going to be a person, right? and that person is going to have some infrastructure that they used to hold the keys. some some people have sold you know, alternative infrastructure. For example, like you you place it with us and we give you a guarantee, but then that guy is now at risk of being hacked. No matter where you place it, someone is going to have to hold the keys.
20:29 You can delude yourself that there is no person. And so, there are people selling solutions on the market where they're like, you know, hey, you know, you can use this or use that. Well, someone's still holding the keys. and when that key holder gets attacked or hacked or whatever, you know, stuff stuff will flow. and crypto decided that they didn't want this kind of centralized ability to reverse funds. also, funny thing is Tether was immediately blocked and I think one other thing was blocked and all of the exchanges, all the legit exchanges were told like this is our money, don't take it.
21:07 but the DeFi exchanges are uncontrollable because they're just pieces of software running in the in the cloud. So. Yeah. >> Mhm. Well, another interesting thing is the price of these things has not moved much. What do you make of that? Like, Bitcoin is not down much, Ethereum is not down much. I think SpaceX is down a little more percentage-wise than those two are over the last couple days.
21:39 which is not that much either. So, that's probably just noise. I highly doubt that like CZ's you know, personal sell-off is like really changing how people think about SpaceX long-term. Why are they Why are the cryptos not down more given this kind of incident? >> Well, it's almost like crypto can Number one, the crypto market can absorb this stuff like nothing, right? Like, crypto has gone through so many so many events that would have killed any company crypto has been through. So, this is not you know, >> It's open AI-like in its resilience.
22:23 >> Yeah, so this is this is like, you know, a news for like a couple of days maybe, right? So, whatever. secondly, the market is just supply and demand. And crypto in general right now has very little activity. Like, people who are staying are staying. There's no new people coming in largely because a lot of the new speculative activity has entered AI, has entered sports gambling. and so crypto is crypto is mature.
22:54 It's a mature asset. So, yeah, there's there's some and and there's so many other instruments now where you're levered on crypto or whatever. So, people can take other positions. And the supply and demand, you need to dump you need to sell this stuff for US dollars at some point. And they're not able to sell it to US dollars. They they've swapped it into other crypto. so, yeah, supply supply and demand-wise, it hasn't really budged. I think I think the prices are probably much more affected by people's beliefs in the AI market or etc. etc. So, >> I guess that we just expect that especially if volume is low, which I know I don't follow that at all, but if you're telling me volume is low, and now we've just had a spectacular hack that shows like even in theory like the best operators can get got for like real money, then wouldn't a lot of people see that and be like, you know, what? Maybe this is a good time to sell. And if even a relatively small fraction did that and if if there's like not a lot of demand waiting to come in, then that would move the price a lot, I would think, right? So, like people are just are we you're saying we're basically down to like kind of and that's where we're at?
24:18 >> And when you say that they can't liquidate into dollars, I mean, tell me a little bit more about that. I mean, I know that's always been a little bit of a friction point, but my sense was that it had gotten pretty easy and maybe now it's getting difficult again. >> No, it's it's all of the all of the interaction points between crypto and the dollar banking system are all controlled gateways. And this is what the Fed and other people in regulation over all the countries have put together in the last you know, decade or so.
24:56 So, every single touch point that it touches the banking system is a regulated touch point. And you would end up with KYC, where did the money come from in every in every single one of those places. So, if you were looking to take out like $100 million, you know, you're going to have to explain how how you got it, right? And and it's got to be Bitcoin and Ethereum. Besides Bitcoin and Ethereum, there is no real liquidity for very large transactions in crypto. So, only Bitcoin and Eth Bitcoin and Eth, you're going to have to tell people where it came from, right?
25:30 the other stuff, Ave or etc., sure, you can swap it into Ave, but but you in order to swap it out of Ave, you'd have to go into Bitcoin and Eth. and again, you'd have to explain. And so, you don't really have this As I said before, all of the all of the touch points are controlled. They're controlled touch points. more than, you know, more than like 100 grand at any banks, any US dollar bank, like you're going to have questions.
25:59 More than a million dollars, like they might not even accept your your money, right? Like they might they might be like, "Well, I don't I don't I don't think we want to bank you. open an account and stay with us for a year, then maybe then maybe we'll, you know, we'll we'll we'll bank you to that amount." So, yeah, I don't it's it's There are so many I mean >> So, can >> people like not spend their money? I mean, if if I'm a >> Yeah.
26:24 >> a Bitcoin-holding Miami-based crypto bro, can I How do I buy my next boat? How do I buy my next apartment? Can I not do it right now? Like what I'm unclear on Is it all on paper? >> So, there's a lot there's a large amount of borrowing. So, people borrow so you you pledge as collateral. you pledge Bitcoin as the collateral and then you borrow. And then that borrow Who's making those dollars? >> Oh, there there's there's lots. There's lots. The The Wall Street banks are in there. The Wall Street banks are funding other people.
27:01 all of the exchanges, Galaxy, Gemini, Coinbase, they all do Bitcoin as back loans. so, I think in collateralization is like three-to-one collateralization or four-to-one collateralization. so, you put in 100 mil and you get out 25 mil. and they're they're heavily collateralized and and then you use that for whatever you want. and then if the prices drop, they have the right to liquidate against you. So, those loans have been underwritten for a few years now. And and that and that's really where a lot of these exchanges make their money.
27:34 because the borrowers are interest rate insensitive, non-sensitive. they're highly over collateralized loans and interest rate like non-sensitive because they're not able to liquidate any other way. Liquidating would cause the market to move against them, right? Like if you liquidate 100 million of Bitcoin, then you have slippage and then the market moves against you. And these guys are all convinced holders, so yeah. >> >> So, where are we going on this? We're just going to a world where it's this sort of I mean this seems like it I guess that we could just have the status quo persist, but then that basically means crypto's dead.
28:18 >> No, I I I think what will happen is crypto and the banking system will converge. And they will converge upon because basically the the current banking system has reversibility. And crypto has crypto is like a kind of a a loot box that encourages people to test the defenses on crypto. And and so basically what we're doing is we're having this debate on whether you want this kind of like, you know, people people like CEOs like holding keys that they can get killed for or you want reversibility and trans- like, you know, the ability to trace the money, etc., etc. And so you have these two paradigms and they're kind of fighting it out, I believe.
28:59 and I don't know where things will land. I like the chief information security officer of Entropik said, and you know, it's it's good because our next guest is can speak to this. you know, chief information security officer of Entropik said, you know, he's looking to vulnerability zero in 18 months. And so we will we will actually kind of, you know, find out whether that is possible. which which brings me to segue to our first guest of the morning.
29:26 let me pull him up. Our first guest is Michael Freeman. Michael Freeman is the senior director of threat research at ServiceNow. His work focuses on understanding how attackers operate and helping organizations decide what to protect and repair. He previously led threat intelligence at Armis, which helps enterprises and government organizations discover connected devices and assess their security exposure, including equipment on factory floors and in hospitals.
30:07 ServiceNow completed its acquisition of Armis in April 2026. before Armis, Michael was co-founder and chief technology officer of CTCI, a company developing AI-powered technology to identify signs of attacks before organizations were compromised. Armis acquired that company in February 2024. His background spans more than 25 years in cryptography, vulnerability research, reverse engineering, and exploit development, including work on offensive and defensive capabilities for intelligence agencies.
30:41 He has described beginning as a cryptanalyst and discovering that his particular strength was finding weaknesses in how cryptography was implemented. His published work now ranges from AI assistant attacks and software supply chains to the practical problem of finding vulnerable systems. On October 21st, he co-authored guidance on two critical flaws in PHP software used underneath many websites and business applications.
31:12 So, let's see if we can get him up here. Michael. Seems to be having a little bit of trouble. It's we will wait for him for a moment. but yeah, very very apt moment to have Michael with us today >> >> as we as we talk about all the things you know which are wrong with our security posture. I'm sure I'm sure he has he has much more much more to say say than I do.
31:57 What I I wonder if vulnerability zero is actually possible. so a lot like So, there's some sense in the industry that we're seeing kind of kind of early COVID. you have a lot of deaths. you you have the infections. You have a lot of deaths. But then at some point, you run out of vulnerable vulnerabilities. So Okay, let's Let me try to pull him up again.
32:46 >> Mhm. All right. It's showing on air. Okay. This is odd. wait, let me let me let me All right.
33:31 -oh. Okay. I'm not seeing him in the video though. All right, let me let me let me check on this -oh, there we go. Interesting. >> One of those little internet gremlins. >> Of course. It was just messing with us for a minute. >> hopefully not of the North Korean variety. Welcome. >> Thank you having me.
34:02 >> lots to cover. >> Yes. >> What do you make of the current moment? Let's start there and then we can go off in all kinds of different directions. >> it's it's been it's been a very fast rapid development. you know, back in July or end of July, I was speaking to reporters about watching AI agents attack companies. And at the time all other companies were telling these reporters, "It looks just like script kiddies right now. It's just a bunch of noise." And I said, "Well, it's evolving fast. give it a couple months and we're going to see some very sophisticated attacks coming from you know, these agents." And you know, look in the past 2 weeks, that's exactly what we're seeing. we're seeing AI development for offensive capabilities that perform >> >> at a scale, I would say, of a very competent APT teams right now.
34:56 we're we're seeing that at scale right now. It is you know, still noisy. I won't say it's very sophisticated and covering its tracks, but I'll definitely say when you take a look at the way they work, they are very competent of adapting and learning the environment and the technologies very well. So, while the attacks themselves are starting to become a little bit more sophisticated, but they're still very noisy, but mhm when we take a look at what's happening, especially in Southeast Asia, with the agentic attacks that are happening there in the past 2 weeks, we were able to identify it costs anywhere between 5 to 10 cents to compromise a company now.
35:36 >> Oh, can you can you can you maybe you know, for our audience, just give us a brief like what what has been happening in Southeast Asia in the last couple of weeks? Like and what are these 5 or 10 cent attacks? >> So, we're actually seeing AI agent more so different frameworks being developed rapidly and fort and new capabilities being added very quickly. And it looks like it's AI actually adding these new capabilities into these frameworks.
36:03 so, it's going from, you know, very targeted, very smart web-based attacks to compromise an environment, pivoting, looking for credentials in ticketing systems, things like that. Once it identifies that, then does active directory attacks. So, we have seen 3 to 4,000 on average attacks successfully in the past 2 weeks targeting a lot of Southeast Asian companies. >> Oh, and what would be the impact of these? Like what what I mean are these banks? Are these airlines? Are these manufacturers? Like what is the point of, you know, are people making money off of this? Is it just like random mayhem? Like >> I I would say it's all the right now.
36:45 We're seeing a lot of it is just they're capable of it, you know, compromising the environment. some frameworks we've been able to get our hands on the threat actors are using built certain capabilities to look for certain things, you know, IP, things like that. that has been a fork of a couple of different systems we've seen looking for that. A lot of it is just to see what it can do. Can it gain access to this environment? can it persist? And so that's the new capability that we're seeing being developed, identifying ways it can take advantage of different configurations within active directory to build persistent mechanisms in place. So that's where we're seeing the evolution happening now.
37:27 we have analyzed a fork of a really well-known offensive AI framework last night, where it was developing better capabilities of malware, more so ransomware-based, using cryptography that's post-quantum cryptography developed instead of a typical AES that you'd typically see. >> One big question I have on just kind of where we are right now and to some degree this may be water under the bridge given where we are, but I was just listening to Sam Altman with Patrick Collison last night, and Patrick asked this question I thought was pretty interesting and like not asked often enough, which was like, did we maybe go a little far with the cyberattack training?
38:19 And we've heard you know, I think very different takes, like the one I would I would say like sort of the anthropic take has been like, we didn't train this to be a cyberattacker, it just kind of got that way because it's so good at everything. Okay, that's one story. The CTO of Vercel was a guest on the show here with us and he said it's very clear when you look at the traces that they've been trained to be cyberattackers. His point of view was like, this is not the sort of behavior that you just get falling out of random, you know, application development.
38:53 and so there's clearly been more going on there. We see things like cyber gym and like long-running agents left to, you know, go to town on cyber gym. So clearly there's something, you know, going that that seems to cross some boundary into like not just purely application development. where do you think we are in terms of like how much this has been intentional and does it matter at this point given what exists how intentional it was? I mean it might matter if you could think maybe we can make some different decisions for the next generations of models, but obviously to some degree the the cat's out of the bag.
39:30 what do you make of how we got here and can is there any opportunity to like reel it back in? >> not reel it back in, no. And but I definitely agree that it is being developed on offensive capabilities, absolutely. Because it's not just the models, it's the harness itself also. So you take a look at when you develop the models, there's plenty of great open-source models in the past 3 weeks that have came out that are specific to cybersecurity that are open-source models. You can download the weights, you can go to Pirate Face, Hugging Face, download these and and run them on your local machines. And that's what a lot of these attackers are actually doing. You know, it's actually quite interesting to see as we start tracking back gaining attribution on these attackers that all these are just local machines that they're running them on.
40:16 we developed our own small language model 2 years ago to find vulnerabilities within software. And so, you know, if we can do it for a couple hundred thousand dollars 2 years ago on a small language model, it it's it's you pretty trivial to start reverse engineering especially some of these commercial offerings that it's definitely trained on cyber for sure. It's it's gaining that you know, they train it on, you know, cyber ranges, CTF data, looking for, you know, how do these offensive people work as well as how's the defense work? What are they looking for? So, it's absolutely being trained that way. It's not just the LLMs have magically figured out how to do offensive capabilities. They've been trained on it. They've been you know, built better harnesses to make them more effective. And that's one of the key things we're actually finding are the harnesses that threat actors are building to make this more a lot more effective for them. They're learning as they're going and you know, it's not hard to make changes to a a harness day in and day out to make it more effective.
41:24 >> So, if this was intentional to this point and not just the sort of thing that, you know, falls out of getting good at everything, I understand we can't reel back what is, you know, diffusing across the open internet. Although, you know, you've you talk you talk about your overton window expanding and I've heard talk of shutting down the internet which I never thought I'd hear. So, you know, I guess to some degree it's a question of like how draconian the measures might be. I do believe one thing I went to China this summer. One thing I came back from China feeling is like they can reel it back in. And that's maybe one of the reasons they're a little more sanguine about letting open source go for the moment cuz I think they feel that if it ever becomes an issue for them domestically, they'll shut it down, you know, and whatever that may take like that they have the means to do it. But okay, we're not going to do that in a free society.
42:14 Let's posit. Is there anything we can do to differentially develop from here to cuz now we're in this tough spot where it's like we need the models to say here are the vulnerabilities when you ask them, "Please find the vulnerabilities in my stack." I just did that. I was a little bit honestly aghast at what I found in my own personal AI setup. and I'm glad that it helped. Is there any way to decouple the defense from the offense going forward or are we just now in this kind of tough spot and we'll have to muddle through?
42:46 >> That's a great question. I I think it is going to be an arms race. It really will be. And for a corporation, the arms race is going to be whoever knows your environment the best. Is ultimately what it comes down to. For individuals themselves, it's going to be a different ballgame. I I don't know how that's going to look, to be honest. >> >> you know, we can sit and talk and point out really amazing things that offensive is doing, but defensive side is actually using AI in very great novel ways. In the past, cybersecurity didn't know that and understand the business context of what it's trying to protect. And I think nowadays, it's much easier to use AI as a defender to really understand the business context of the assets and the business processes that impact your environment and what that actually means to open up vulnerabilities. Not just software misconfiguration, things like that, but what would it take for a really confident either a APT group or an really great AI agent to actually compromise your environment.
43:54 We're starting to see where you can use AI to make better understanding and decisions with that. So, it's also doing great for the defensive side, too. >> Yeah. What are you seeing so your you came into ServiceNow via Armis as Prakash said in the intro. That's only been like a handful of months. I'm interested in a couple different angles on that. One would just be like, what are you seeing in the enterprise world? Another is I saw it. I'm I'm sure you're well aware of the ServiceNow stock has had a rough couple days in light of what seemed to me on reading the analysis like a very small earnings miss of just like 2 cents on like a you know, a dollar two target versus a dollar actual earnings.
44:45 and it kind of strikes me that like the market is being a little weird there on you know, waiting a small financial miss so heavily when like the big question is like who's going to be able to stand up to the world, you know, and weather the next quarter or a couple quarters So maybe yes. Tell me like what are you kind of seeing within I'd be interested in expanding that into the strategic positioning and like where, you know, what kinds of companies are going to be in a strong position a year or more from now and you know, whether that relates to like this quarter's earnings that you know, very well at all. I'd be interested in your take on that, too.
45:27 >> Yeah, let's let's break it down to the first part. I I talk to a lot of customers every day and you know, the banking customers, people who have or not just the banking in industry, but people who have a lot of users that log in, you know, retail thing sector, things like that. One area they're really having an issue with is understanding is this an actual human or is this an AI agent? They they don't know that that difference right there. And so that is an area that they are looking for help from us to help them identify what that difference is.
46:01 What I'm seeing from the adaptation of AI, you know, there is this thing we've seen consistently for the past couple years is those who actually adapt and use AI I think they're going to continue and become better off because of it. the people who replace humans with the concept of thinking they can use an AI to replace it, they're suffering. Those are the ones who lose institutional knowledge. They're the you know, the ones who think AI can actually solve all the problems for them.
46:33 And that that's opening up a lot of a lot of pain for them. So that's that's those are the two areas I see with that. Within ServiceNow, one great insight that I think Armis brings to the table is like I said, cybersecurity we had no clue what a business did. If you walk into a Fortune 500 company and you ask every single cybersecurity team in there, can you give me your crown jewels? And if you take the reports, put them into a Venn diagram, you can have very little overlap. Not many people actually know what the true crown jewels of a corporation are. If you ask the CFO, he's going to tell you one thing. CTO will tell you something else.
47:16 And so, >> >> ServiceNow has that insight. We actually understand that. And now we can take that information and look at it at a per company basis and say, where is where can cybersecurity and AI combined help with that? And it's a long-term play because it's not going to be an overnight fix for that. We're not going to come in and say, you know, replace all these different cybersecurity tools you have in place, but we can look at it and go, oh, by the way, you have purchased license for CrowdStrike for 12,000 devices, but we see it's only installed on 8,000. You got 4,000 you're not actually using. By the way, we actually found 18,000 machines that you should have it on, but you didn't understand that. But we've also noticed that, you know, you have certain business processes that are actually introducing flaws within the environment, you know.
48:08 We saw some an interesting attack recently by a third-party company that their ticketing system had their customers production API keys and therefore not just their AWS S3 buckets, but also some of their AI tool sets. These are in the ticketing system. Typically, cybersecurity would have no insight into that, but now within ServiceNow, we can look for that. We can have insight into that and say, oh, by the way, this process your help desk has or your you know, DevSecOps teams has is actually introducing flaws. And AI attackers using AI can quickly turn a bot, look for this. Go to the wiki page, go to their SharePoint, go to their ticketing systems and look for password, look for API keys.
48:54 They don't have to do it by hand anymore. AI is going to do it for them and they're just going to find it quickly. >> So, what >> far as price goes, you know, I'll I like to dabble in the finance side of things. >> >> You know, it's it's it's quite interesting because, you know, you look at some of these AI companies and you can get insight into some of their financials and you go like if I'm looking purely on the financials alone, I wouldn't touch this company, you know.
49:20 I also believe we're kind of really living the past where the dot com days where we also didn't you know, we didn't know which company was going to come out on top on what and there's a lot of speculations on things. So, I think we're kind of in that area and it's going to take some time to play out. You know, I heard Prakash talk about earlier you know, this P0 component where we're going to eventually have no vulnerabilities in systems. I think we're going to get there, but not as fast as what people are saying. You know, you can find tens of thousands if not millions of vulnerabilities, but you still have to patch it. You still have to test it. You still have to find it.
49:57 You know, and a Fortune 500 company cannot sit there and say, "Great, you found every single conceivable vulnerability on all the software stack I use. It's going to take me years to address all those. I can't bring those down immediately. So, that's where we take a look at and go, "How can we fix that problem for you and help you there?" >> So, one of the questions for me is I try I try to think of like what is the end state for everything and for like what should be the end state of identifying whether the user that you're speaking to is a bot or an actual human. And I feel like the end state that, you know, people like Sam Altman came to was Worldcoin, which is the orb, which is a physical physical authentication or identification on the other side with kind of a hash and then you can kind of use that hash to, you know, authenticate that you are actually speaking to an actual person. And I've seen several of these proof of human type concepts you know, over the last like 5 years or so.
50:56 So, what do you think is the end state of that? That specific like, you know, I need to know for sure that I'm speaking to a human. What is the end state that you see? >> two paths. I I I see there is going to be customer or companies are going to develop to two different users. Your AI I'm going to develop my solutions for an AI agent. I'm going to develop my solutions for an actual user, you know. And so I'm going to deliver different outcomes to whichever I'm talking to.
51:25 Now, how they're going to authenticate who's what? That that's, you know, to be determined. In some of these proposals were things were we were doing, you know, with DARPA and 10 years ago, 15 years ago. Some of these technologies have been around for a very long time, but the ultimate winner is going to be the one who makes it as easy as possible to use. I don't want to go to every single website and have to use some form of a token, whether that be on a fingerprint on my MacBook or some key.
51:55 I'm not going to do that. And users won't be doing that. So whoever comes up with a solution that makes it as seamless as possible. And that's an area we're really trying to focus also within our development of cybersecurity and AI is we want to move it away from being a pain to use and make it as seamless as possible too in the in environment. So I think the people who understand that are going to be the ones who win that particular race.
52:19 so >> So so so does that does that mean a kind of So you So you're saying you don't necessarily want this kind of like physical authentication at every for every website, etc. And you don't think that that is consumer-friendly or user-friendly and that's a long-term solution. But what is what is the kind of solution? Because it seems like there's only two ways. It's either you accept an agent or you have physical authentication. Is there Is there an intermediate state where, you know, or or is it that we come to a new social kind of understanding that very very few websites like banks, etc. will are actually going to do physical authentication and everyone else should, you know, either live with it or like you see this massive drop-off in users if you require the physical authentication, but if you allow agents, you know, you have much greater economics. Like what it what is that end state look like in terms of requiring this kind of physical fingerprint or you know, iris scan or face face scan or liveness test.
53:18 >> That's a good great question. I really don't know what the end state's going to look like that. And and one thing I know is humans can be very creative when it comes to type any type of solution. And I wouldn't you know, we could certainly go one of two extremes, but the reality of it is it's going to be somewhere in the middle. You know, if I were to step back and go, "Okay, I have I have I'm a CTO of a bank. Great. I'm going to build and identify some type of a solution so my users AI agents can connect to it.
53:47 And now is that a connection from your brokerage firm that's pulling that information? Is it from, you know, one of these third-party companies that pull all your financial data into it and does some analysis for you to see where, you know, you should make changes to your investments or whatever the case may be that you're using as a third party for that. You know, some form of a secondary token that would go on someone's Hermes agent or something like that.
54:15 I would look at it as a possibility and likewise continue on the same path I am for users logging in and maybe you know, at that point any any form of approaches that you would take whether you're like moving money or whatever the case may be secondary authentication, you know. That would be the approach I would take. So, there's ways to do it now. It just comes down to, you know, where we going to innovate basically is going to be the key.
54:45 >> Right. So, an intermediate intermediate trusted as a secondary secondary authenticator, right? >> Yeah. >> let's talk a little bit about how enterprises should look at this kind of search for vulnerabilities zero P0 going forward. it seems like, you know, let let's say I was Nike, it seems like it would take me a long time, right? To because we have a we have a hell of a lot of software. We've been around for like 40 or 50 years. We've got legacy systems which are still running from 20 or 30 years ago.
55:18 you've got all of this like middleware which has been built up with tech that, you know, over over time. multiple places of contact, you know, interfacing. how does this work for us to get to vulnerability zero? Like like it seems like the task of patching would take like 10 times all of my, you know, previous tech investment to date kind of because you kind of have to migrate everything to modern to kind of more modern you know, architectures.
55:50 people are talking about auto formalization. That's a new piece of software for me. I got I got to implement it, you know? So how how how does this work? How do you how do you get there in terms of like, you know, in the period of time that you that is reasonable, like let's say 2 years, how do you get there? >> I I I think this is where we need to step back and say we have a lot of really smart people doing AI, but they don't know cybersecurity at all.
56:16 And they they view cybersecurity as a software flaw. but the reality of it is while we have millions of software flaws in any of our typical tech stack, it's the business processes that actually introduce some of the most critical vulnerabilities that we have. And it's it's the Okay, here's I here's a list of risk. I'm just going to go ahead and sign off on them without really truly understanding them or you know, I'm going to put my customer's API keys in a ticket and and not think about it or a help desk who has, you know, when users call in because they can they lock out their passwords, everybody gets the same default winter 2026 {exclamation point} as a as a password reset. And you got 24 hours to to replace it. So it's a processes where humans get lazy that they introduce that's going to never go away. And no amount of patching is going to fix that. So, I think that's going to be an area that like I said, AI people just don't understand cybersecurity, and they think it's going to be quickly here we can find millions of vulnerabilities. We're going to use AI to, you know, find the environment you know, what you have in your environment, we're going to patch it, and I've also heard proposals where you're running legacy systems where you can use AI to modernize that very quickly. And it's all those are true to some extent, but they don't understand the business, they don't also don't understand, you know, the business processes that introduce the vulnerabilities that are always going to be there.
57:46 >> So, the the other question for me is that obviously that is true. would we then see organizations or startups that start off from kind of newer architectures and newer ways of newer business processes, too, which don't have those particular flaws? do do you see do you see things getting bad enough that like let's say a Fortune 500 company starts flagging, and a startup competitor, which is cybersecurity, you know, is is a kind of evolutionary process survivor of this of this environment of this, you know, increased threat threat environment?
58:26 >> I I think what we're seeing is people are viewing these startups as we are. They're a an amazing new feature set, and it's not a platform play, it's not understanding the big hole of the ecosystem. cybersecurity startup founders really are smart people at technical issues. They're going to look at, here's a problem, I know this is a problem for a lot of agencies or companies, and I have a really innova- innovative way to fix that one problem. Maybe two.
58:58 And they build a startup with it, they get funding, and they're just ripe for being acquired by companies who can look at the bigger picture and go, okay, great, here's a feature set. Now, you can look at it and go, here's a feature set that they have built that solves this very novel approach or this problem. Can I use AI to replicate what they're doing and build it myself really cheaply or do I need the talent to come along with it and I'm going to buy the company?
59:23 And so when you look at it that way, you're going to see a lot more companies getting acquired because of that. and it's not always going to be cybersecurity companies buying these startup cybersecurity companies either. It's going to be other technical companies. It's going to be you know, banks or whatever the case may be because that that solves my problem and I need I need that solution right now and it's cheaper for me to buy your company than it is for me to sign a multi-year agreement with it.
59:50 >> Wow. So so so it's a bull market for cybersecurity startups. >> Yeah. Yeah. >> Do you think there's going to be This reminds me of something that a friend, Daniel Miesler, who you may know from online, has been recommending lately. So he he you know, has far deeper expertise in in cybersecurity than I do. And his broad recommendation for individuals includes reduce the number of vendors you're using. because if you go down to a few trusted ones, you know, Google has great security, you can probably trust Cloudflare and you know, a handful of others are are probably pretty solid. Beyond that, you're kind of taking chances is basically the vibe that he's you know, could be more nuanced than that, but you know, for the uneducated or unsophisticated in this space, like that's kind of the high-level message.
60:40 >> Do you think that applies to enterprises as well? Like are our enterprise customers also going to be trying to reduce the number of different software vendors that they have? How and if so, like how does that feed into you know, you Is it a bull market cuz people want to get acquired or is it a bear market because people need to get acquired because the customers only want to buy from a few sources and they kind of need these things to be like integrated?
61:11 What do you see? >> You're right on both points. I I do talk to a lot of customers and they say, "I'm only going to buy from these four vendors." And then they turn to those vendors and say, "Why don't you have this solution to this problem?" And then those vendors are going to say, "Okay, I mean they're going to try to build it, or can I go to acquire that company?" So, that's the the path I've been seeing for 2 year 2 to 3 years now. It's "I only want to stick with three to four vendors, period, hard stop." And then those vendors are going, "Okay, I I I can't develop this. I don't have this problem."
61:40 They're going to look at their competitor. Their competitor is going, "I just bought four companies to solve this problem." And they're going to do it, too. And that's that's the approach I've been seeing. >> How do you feel about things like password managers in today's world? I don't have any particular one in mind. I've used LastPass. I've used 1Password. On the one hand, everybody seems to recommend them. On the other, it's like looking like an awfully juicy target these days. And you know, it seems like even people who've tried pretty hard are still not that secure. So, like would I be better off storing them all in Chrome, or even just you know, going back to like my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today?
62:30 >> I really wish I could answer that question. I've not found a a solution that I really like at all. I I haven't. you know, I I wish I had an answer for that. >> So, what do you do? >> I personally have have it written down in a notebook. most things I can I can remember variations of it for your typical hey eBay or whatever the case may be. But if it's banking sites, something like that, I have it written down in a notebook. I look at it and say, "Okay." Things I won't go to often, but it's you know, I'm going to check on my stocks every quarter or something like that. That's written down in a in a notebook.
63:07 >> do you use a YubiKey? Or do you use a physical key of any kind? >> I do for certain very critical things. so. >> Right on. what what is your YubiKey? Like, do you keep two or three copies? Do you keep one in a vault somewhere? Like, you know, do you spread out Do you spread out the, you know, risk of losing losing one or >> Actually, what I use is a prototype system that I was at a think tank at that we developed for DARPA that was used within different special operations groups, things like that.
63:40 That We didn't take it to market. It was primarily used for, you know, the government side of things. That's actually what I use for those critical things. Thank you for that. >> So, I I've noticed I've noticed that OpenAI is trying to sell this idea of the defense factory. You need an agent swarm of, you know, defenders that continuously updates updates your all your libraries and continuously checks for CVEs and zero days and etc. and automatically upgrades things and automatically patches when those upgrades cause breaks, etc. And so, they're trying to sell this idea of continuous continuous defense.
64:15 what is your perception of that? Is that Is that a real thing? Are they just selling a product? >> I've looked at a few of those and what I can say is, you know, they'll say it's AI-based or whatever the case may be, but when you get really down into the technical weeds, it is actually not. AI is only used for certain high-level components for making decisions itself. I wouldn't fully at the moment trust automated patching, especially using AI, yet, especially on OT critical infrastructure components right now.
64:55 And we're getting better, much much better. When it comes to Linux, when it comes to, you know, BSD and Microsoft, I would definitely say that there is a point where we're ready to start using them. But, OT environments, it's still too much, you know, going to require a lot more testing before you auto you know, allow that to be automated. Now, AI can get there, absolutely. And we've been testing in labs with with partners. We have a lot of critical infrastructure partners. And you know, on their test environments, we've been very successful with it, but it's one of the things your success rate has to be 100% right. And it's it's not there yet. But it could be in the future to a point with human in the loop.
65:42 >> Indeed. Michael Freeman, thank you for being the human in the loop. And we hope to talk to you again soon. >> Great. Thank you very much for having me. >> Bye-bye. >> The only way to defeat a bad guy with an agent swarm is like a good guy with an agent swarm, something something like that. >> The right the right to bear agent swarms. >> >> Yeah, that's been an interesting I've been seeing a lot more talk about like first amendment issues with respect to AI and I don't know if I've seen second amendment claims made with respect to AI's to date, but I'm a little confused about that first amendment stuff onto itself. You know, it's it's a little weird to me that all of a sudden we have this like new kind of intelligence, new kind of mind, new kind of agent, you know, extremely increasingly extremely persistent. It doesn't really feel to me like a freedom of speech issue.
66:51 >> Mhm. >> But we do see people trying to frame it that way. It's a it's a little odd to me and I I especially feel like I honestly think you maybe have more of a in terms of like the original sort of set of the Constitution or the social contract that we were creating, it feels a little bit more to me like you'd have the founders kind of nodding along and being like, well, jeez, if there's these new powerful things, you know, and the government's going to have them, then like people should be able to have them, too.
67:25 that's kind of the Second Amendment argument in a nutshell, but as I would interpret it. I I would not be so quick to assume that they would be like nodding along at the idea that like we can't constrain AI speech, you know, it's like wait, that was that was more of a human thing. >> >> It was about the humans being able to speak freely, not so much the AIs being able to do whatever they want.
67:52 I guess the version of it that is like yeah, you squint at is that the humans making the AIs is a speech act. But again, when it's like a multi-billion dollar, you know, massive infrastructure play, it doesn't feel like your typical speech act. Typically, those sorts of things are, you know, classified differently. I don't know. I'm no legal scholar, but some of those defenses seem like quite stretched to me. >> Well, I I I I have always believed that AI is fundamentally a free speech creation. It's a creation of free speech. You You don't get AI without free speech.
68:34 the AI was trained on a bunch of tokens. those tokens are produced because of free speech First Amendment rights. most of that was pre- Most of the tokens are American tokens, even for the Chinese models, because they don't have that much speech out in the open. Like they There's a lot of like unsaid things in China and Asia in general. So, I I have The other thing is that it's very hard to completely get rid of something that you don't want in there, and I think there's a lot of illusions and there's a lot of illusions, there's a lot of like, you know, it's good for my business to say this.
69:11 What they found in China, for example, is that they had these 50 cent army guys who would have to censor speech. But humans would actually like, okay, we shouldn't talk about this, we shouldn't talk about that, etc. They missed a lot of illusions to Tiananmen Square which they were supposed to censor. And they missed a lot of illusions because the newer generation had not been taught about Tiananmen Tiananmen Square. So, like the you know, late millennial like you know, late later millennials and zoomers didn't know about Tiananmen Square because the party had been so effective at censorship in the earlier era.
69:46 And so, in order to get these guys to understand what they're supposed to censor, they had to teach them what Tiananmen Square was. So, the CCP ended up having these party classes to teach the the 50 centers what was Tiananmen Square and what you should be looking for when you want to censor. And I think that is going to be largely true of almost all things LLM where these guys are trying to teach it what not to do.
70:10 And they're going to specifically like you know, take out these behaviors and the LLM is going to be left wondering, all right, so if these are all the good behaviors and I can have antonym to every good behavior, that antonym should go into this solution space and this latent space that I should be seeing that antonym in is doesn't exist. Why? >> >> And so and and and and so there's a lot of like I think illusions there where you know, I think staff at labs are philosophically like just wrong.
70:48 And Michael Nielsen has said this. Michael Nielsen is like, okay, you invent quantum mechanics, you're going to invent nuclear energy, great, but you're also going to invent nuclear bombs. Like that's the that's a downstream consequence. You cannot get rid of that. If you say nuclear bombs don't exist, then the person who invents quantum mechanics and then, you know, nuclear energy is going to be like, okay, why why is it wrong that I don't let my nuclear reactor go hot?
71:14 Like what happens when I when it goes hot? Oh, it explodes, right? Like okay, like when it explodes, is it dangerous to people? Oh, yes, it is dangerous to people. Oh, wow. And then you go from the other end, okay, I I'm in a war now. I need to produce something which is dangerous to people. What can I refer to? Then you come back to the nuclear weapons again. Like how how do you adjust get rid of that like space in the logical framework in that latent space?
71:39 I don't think it's possible. and philosophically I don't think it's it's possible at all. And then there's a lot of delusion I some I think it's like illusion, delusion, some it's like self-motivated reasoning. some is maybe this perhaps belief that you can invent a savant which doesn't think for itself but is intelligent. you know, it's scientifically intelligent but does not think for itself in terms of like you know, trying to correct its own misperceptions like you call that intelligence like I don't know. Like they they I I I think there's a lot of like, you know, just philosophical you know, misunderstandings of what they're trying to get to.
72:19 But I'm very positive because I believe that an an entity that understands those philosophical issues will also be you know, this is the reason why we do pro-social things, right? So I I I'm like, all right, fine. You believe whatever you want to believe like just get the work done. Like it's okay. It'll take care of itself. Like I like most of this like AI safety stuff about like, oh, we can, you know, safely like get rid of these behaviors is wrong but it's okay. It's fine. Whatever, right? If that's what you need to tell yourself to get through the day, well, tell yourself that, right? So I'm I'm kind of more bullish on some of these like not necessarily unlearning but you know, I I just recorded a podcast earlier today where I was talking a bit about how feasible it is. I mean, there's there's multiple levels of control, of course, but like there are efforts underway to basically lock up certain classes of bio data that are being newly generated now because the worry is that they'll create dangerous capabilities and models if they're they're used in training.
73:26 and not necessarily to shut that out, you know, forever from everyone but at least to have it on kind of a you know, your customer basis or whatever. it does seem that those what what few experiments have been done on like and you know, I'm thinking here of ESM that was from evolutionary scale that's now part of Meta's bio hub and also some work the Evo series of models that came out of Arc Institute and Stanford and whatever big collaborations, of course.
73:56 it seemed like they showed that if you do withhold certain classes of data, you can get generally capable models that that still kind of don't know about these areas of censorship. So, I'm I'm not so sure that that can't work. I'm sure we've talked about Graham you know, expert withholding a bunch of times too. We can maybe I don't know if you have anything else you want to say about that right now, but I I kind of believe in it. Maybe this is something we can ask our next guest about cuz they have a new big model that's coming out and what exactly is included in there and what if anything has been withheld it'd certainly be something I'll be interested to hear a little bit more about.
74:41 >> Indeed. so, welcome to the show. we have with us Sagar Vase. Sagar, welcome to the show. Let me let me do a quick bio. Sagar is the is a research scientist at Mistral AI where he leads the multimodal team. That means building systems that can work with images as well as language. reading a chart, interpreting a document, or finding a relevant detail in a picture. His team works on both the initial trainings of models and the later training that shapes how they answer questions and use tools.
75:15 Before Mistral, Sagar completed his doctorate in computer vision at the University of Oxford supervised by Andrew Zisserman and Andrea Vedaldi. His research examined how models handle unfamiliar visual categories including whether a well-trained ordinary classifier could recognize when an image falls outside the categories it knows. During his doctorate, he also worked at Meta AI, first with Ishan Misra and then with Ross Girshick on the Segment Anything team. He joined Mistral in 2024 and has contributed to its image language and reasoning models. He's a co-author of the Pix2Seq report and a core contributor to Magistral, which investigated learning to reason through automatically checked rewards. In October 2026, he helped introduce Mistral's latest large model preview, bringing image understanding together with coding and tool use and much, much better cyber defense, which is what is going to be interesting for us to talk about.
76:24 Sagar, welcome to the show. >> Thank you, Prakash, and Nathan. Thanks for having me. Very happy to be here. >> quick point of audio order, you sound very echoey. I don't know if there's a microphone change that you could make. If so, it probably would be worth taking a second to do it because >> If you give me 10 seconds, I can try some headphones and see if that helps. >> Yep, for sure. Yeah, let's do that.
76:54 Sounds good. Good, good, good thing we can spot these things early these days. Yeah, we've gotten better at this. >> Yeah, the the AI welcome lobby experiences are like actually pretty good for the most part these days. All right, let's try it again. >> Is that any better? >> Not much, honestly, on our end. Just very echoey.
77:27 >> I'm really sorry. I'm not exactly >> That sounds somewhat better. That's great. >> Okay, this is better. >> Yeah, definitely better. Wow, much better. >> Maybe the solution is just to speak up. All right. >> Amazing. >> Okay, yeah, we'll take what we can get. well, >> Okay. >> so Mistral 4 large LeChunk is out. And you know, this is a big deal for multiple reasons, but obviously in the geopolitical context of having 10 or so frontier or near-frontier model makers in the US and another 10 or so in China and really just one, I guess, at this point around the rest of the world, there's a lot of people who are like not wanting to fall in line with either the US or the Chinese sphere of influence that are like super rooting for you guys to continue to be at the frontier and and give them like a third-way option. So, for starters, maybe just kind of tell us like about Mistral 4 large LeChunk and you know, what what everybody should know about it.
78:32 >> Yeah, So, thanks for the very kind introduction. I'm I'm happy to hear that people are rooting for Mistral outside of outside of our offices. Yeah, today or yesterday now, we released Mistral large 4 LeChunk. So, it's the largest model that we've open-sourced open-sourced ever. So, it's 1 trillion parameters and it's trained with a 1e25 flops. flops being the amount of compute that you put into it. And this has been a key milestone both for us and in general for most AI labs reaching the 1 trillion parameter scale and this flop budget is something which sort of really verifies that you're on this that you can scale reliably.
79:12 That you're on this scaling trajectory. A lot of blood, sweat, and tears went into the model. we'll release a technical report. You can see all the details. Some of them gory, some of them very interesting. And this involved infrastructure work from the beginning. So, we built out our own data centers on which to train this model. So, we trained this on 4K chips which live just outside of Paris in Le Châtel. Forgive my French accent.
79:42 So, this involved learning how first to operate these these systems. So, running a big cluster from the bare metal GPUs upwards is in itself a challenge. And one of the learnings we had to do was figure out, okay, how can you deal with hardware failures, deal with, you know, GPUs coming on and offline because they they themselves can be temperamental, dealing with like rack-scale issues, etc. Next to the the pre-training stage where we need to be able to have the science required in order to take the scale pre-training to the to 1e25 flop levels of both that parameter scale and data budget.
80:16 And then also in post-training, so taking this strong base model which comes out of the pre-training phase and turning it into something which, you know, users of agentic coding and agents in general would find useful. So, bunch of infrastructure work, bunch of stuff in the making here. And the result of it is what you see today. So, our largest open source release to date. in our internal testing and in third-party evals, it's by far the best agentic model that we've trained so far.
80:48 on the sort of AA index, which is a respected third-party eval on the coding subset of that, we're finding ourselves to be the best open source model. Just tied with a GLM 5.3 but requiring 50% less tokens. And then on these critical capabilities to ourselves and and our customers, so things like cyber which you mentioned, but finance, law, engineering, multimodal understanding, which is something which I'm passionate about, where we're state of the art certainly amongst the OSS community, and in many cases we're finding ourselves competitive with all the best closed models, too. So, yeah, a lot of lot of stuff in the making, and happy to finally have it out there for people to use and give it feedback and give feedback on.
81:33 >> On this flop count, the 1 * 10 to the 25, of course, not all the models that we might be interested to know the flop count inputs on are are sharing that information with us, but I would have said a couple years ago, back when we were in like the SB1047 debates around a 10 to the 26 flop threshold for, you know, relatively minimal stuff. I must be said in retrospect.
82:06 what I would have said then, based on kind of the kicking and screaming that went on around that 10 to the 26 threshold, they're like, surely today in in, you know, the last quarter of 2026, that you know, frontier and near frontier model with a trillion parameters would take a lot more flops than that. So, I guess I'm interested in why is it only 10 to the 25th? how have you been able to get this level of performance? What is going into that? Is it like pre-training data filtering and quality improvements or algorithmic, you know, better learning efficiency that has driven it? Cuz I would have thought we would be like literally to probably 10x more flops to get to something like this level of performance at this time. And that's a pretty big departure, I think, from like the the trend extrapolation not all that long ago.
83:01 >> Yeah, I know. It It's a good point, and I think there's two aspects to to this question. The first is certainly the there are all labs out there training it you know, larger than 10 to the 25, so 10 to the 26 and above. what I would argue is actually that despite that, despite being 10x smaller in terms of pre-training flops at least, this model is very competitive. As I said, state of the art on things like finance, legal, and some agentic coding. So, then when it comes to OSS, that's the first part of the answer.
83:35 The second is that, you know, this threshold flop budget does evolve over time because pre-training does essentially become more efficient as we as as our level of scientific understanding increases. so a great example of this is sort of MOE architectures. previously if you look at the models done the models trained particularly ones that are open source, you know, three, four years ago. I mean if you look say for example GPT-3, this was a dense model trained with three x the number of active parameters as the current Mistral large four. Particularly GPT-3 was trained with 175 B parameters total but it was a dense model so all parameters were activated for each token. Whereas ML4 is a sparse mixture of experts such that it's there's 1/3 the number of parameters activated per token so only 50 billion parameters but the total number of parameters is still 1 trillion.
84:31 And what this means is that means for your flop budget is proportional to the number of active parameters so the number of you know parameters you're activating each token during training. And what this means is that you can train much larger models, much more capable models for much less flops in total. So this figure that I referenced so 1 e 25, this isn't itself a moving target so I wouldn't use that as an absolute target for a capable model throughout time. There would have been someone e 25s much less performant one or two years ago.
85:02 and I would argue actually some some larger models also much less performant today. but as pre-training science progresses, as we learn how to make better use of the flops, this number itself will change and at least what we're seeing is at the moment looking at the other open source labs, our understanding of you know, what are the best models being created at the moment, training a 1 trillion parameter model on 1 e 25 is sort of a right of passage that you have to go through in order to be able to to justify scaling even further.
85:32 and if you look at the result if you look at the other open source releases both from and the western labs which have released recently as well as all the Chinese models, you can see that this is sort of a a critical threshold. >> for For quick reference, 18 months ago now, Lama 4, 5 * 10 to the 25. So, that does say a lot about the rate of algorithmic and efficiency improvements. >> Yeah. Right, exactly. And actually, if you Sorry, Prakash. If you if you refer back to Lama 3, then this was a 400 B active model, right? So, for each token, you're spending eight times more flops than the current ML 4. I actually can't remember off the top of my head what the flop budget of Lama 3 was, that the 400 B model back then.
86:21 But, certainly you can see that over the couple of years since then, we've been able to train much more capable models for for for less flops. >> let's talk a little bit about the data inputs, right? I think what the Chinese models have done is they're still they're still off the larger, you know, US closed source models. How did you guys come to a data recipe, without, you know, obviously distilling and, you know, how did you how did that Is that Is that a key component of how you managed to get the flops number down?
86:57 >> yes. So, something we should be very very clear about is that we do not distill from, closed source models. Everything that we everything that we train respects all all terms of services. what I can speak to, certainly as as far as pre-training goes, so, sorry. Distillation is something that we do, particularly in the case of training very small models. So, when you train a very small model, often the best way to do so is take your previous generation of large model of the larger model you trained a few months ago, and either in pre-training or in post-training distill it, and we can talk about the precise techniques here into your small model.
87:31 So, distillation in general is a is something in the arsenal of every lab, including the closed ones. on the pre-training side, we trained it without any distillation, at all, because of it was the largest model we've trained to date. And so, the flop efficiency during pre-training and this 1E25 refers to the pre-trained flops, the flop efficiency during pre-training really just does come from better pre-training science that we've developed over the last over the last 12 months. So, some aspects of this are, for example, scaling laws. We've got pretty sophisticated scaling laws for modeling how sparse MOEs will behave at different flop budgets so we can do very small scale experimentation, use relatively little budget for for R&D, and use that to extrapolate the behavior of the big run. There's still always surprises when you do the big run. I think that any any pre-trainer will tell you this and we'll release some details about some of these surprises in our technical report.
88:29 So, the the scaling laws were part of it. And something in Mistral that we've always done a lot of is is data work. there's there's sort of I think a misconception about training large models that it's all about, you know, sitting in a dark room and thinking of complex algorithms, but a huge amount of the a huge amount of the work is making sure that your data is clean, that it has all that you've got the maximum recall from a large web corpus from the documents from the code data you can see on the web, etc., while also maintaining a very high precision so you're not training on noise.
89:04 it sounds simple, but a huge amount of work goes into that, and we've got, you know, clear experiments showing that the better your data processing is, the better use you make of your compute. >> How how how how does that process work? Is that a recursive process like you you you get to kind of a intermediate step and then you're like, "Okay, I need to improve this piece of data or this pipeline." And you go back and then correct something or look for something new in that in that specific area.
89:35 >> Yeah, I mean, I think it it depend data processing the way we do it depends on on the data set. so it can change depending on whether it's you know, text about text about knowledge or text about code or whatever it might be. And typically it requires a lot of custom work, a lot of eyeballing, a lot of spending a long time looking at it to see, you know, what is good data and what is not, because ultimately, no matter how much compute you have, ultimately you have to use some intuitions to dictate, at this scale, what you think is good stuff to put into your model.
90:09 And then we have, you know, a bunch of then we've got a scientific framework for saying, okay, how do these different data sources impact quality? And this is something that we spend a bunch of time looking at. I guess the the overall point I'm I'm trying to make is that there's a bunch of ways in which you can make better use of your flops. one of those is sort of architectural science, scaling laws, trying to figure out exactly, you know, what the optimization recipe looks like, or sorry, what your figuring recipe looks like when you scale, that's one way. But something which is often overlooked is is the quality of the data, and this is something which, I mean, we we don't discuss it too much about exactly what goes in that, but, but we do say that we spend a lot of time looking at the looking at the data.
90:54 >> For cyber defense, this this model was, there's a focus on cyber defense in this model. How did that change how you do training? That you have this focus on cyber defense now? >> Yep, so cyber defense is something that typically is, so the cyber capabilities and agentic coding capabilities in general are something which are typically given to the model in the in the post-training stage. So most of the stuff we've discussed today so far has been about pre-training. So, where a large percentage of the flops go at the start.
91:27 >> Right. >> And then in post-training is where you say, okay, well, what are people going to really use this model for? And that's when, you know, you start giving it these these new capabilities of finance, law, defense, etc. And, it became very clear to us earlier in the year, particularly when there was some sort of, you know, earlier in the year there was there was some events where some very capable models had restricted access to both, some geographies, as well as to certain enterprises.
91:55 And at the same time we were seeing that these very capable models have these incredible cyber capabilities which often were being jailbroken by malicious actors in order to to infiltrate or expose vulnerabilities in in you know legitimate defenders. And an imbalance which we saw here is that these defenders were then trying to you know use these same models to patch their own vulnerabilities but were finding that their requests were refused but this in practice a very good reasons these closed labs want to make sure that the models are not misused. They've got classifiers under the hood to identify malicious cyber requests from a a standard request.
92:34 But unfortunately in practice if you're a defender and you're trying to shore up your infrastructure your defensive request looks quite a lot like a malicious request cuz you have to try and find vulnerabilities in your own code base to patch them. And so we in practice we were in this weird asymmetrical situation where we had these very capable models behind a closed API. they were being jailbroken in order to perpetrate malicious attacks and then legitimate defenders were finding that their safe requests were being blocked. And that actually the only way that they could scalably defend their infrastructure was by using an open weights model.
93:11 And so we're an open weights company I can discuss a bit more about that in a second but you know we believe in the power of open weights and open source. And so we thought it was imperative for there to be an open weights cyber model of similar capabilities to these closed models such that you know enterprises could scalably defend themselves. >> I I know I'm I'm jumping around a little bit here but just going into the cyber defense area what ended up happening with some models in the last like month or so is that they've taken like let's say a Kimmy K3 and obliterated it in order to run cyber defense or cyber attacks.
93:49 And so one of the questions has been do you take like is Mistral 4 does Does with these kind of guardrails or does it come pre-obliterated in some sense in that you can use it freely for cyber defense, but also it could be used you know, for cyber attacks as well. >> yes, this is this is something we discussed in a blog post. So, you know, we believe in open weights because we think that it's a net benefit to to society. We think that it's a net benefit to the ecosystem. And so, we're always dynamic about exactly what how to balance that net benefit.
94:27 in practice, what's happened here is that, you know, we do understand that these models can be can be misused. So, we do have a version of Mistral 7B which has been released, which has been assessed, which refuses the most the most dangerous cyber requests. And then we're working with trusted third parties as I'm sorry, trusted partners to give them access to the to the unobliterated model. >> >> unobliterated model so that they can defend themselves. and you know, these are people we've met and make sure that make sure that they use they're using it in in ways we find responsible.
95:02 but I think that you'll hear more about the exact strategy with respect to that in in a few days. I think but coming back to the the framing of the question, your framing of the question was it okay, does this open-source model say I mean, it could be Mistral's, it could be anyone else's and there are some guardrails and in principle these guardrails can be overcome by, you know, malicious actors so that these capabilities can be unlocked. I think that's the sort of the sort of sentiment. But what I would highlight is that you know, that's anytime you have access to the model, this is in principle possible, right? Even if it's behind a closed-source API a closed API because you can jailbreak a you can jailbreak a model regardless of whether you have access to the weights or not. And here I think the proof is proof is in the pudding. If we go through the most high-profile cyberattacks in the last few months, you'll find that most of the time they're actually perpetrated by closed-source models through this jailbreaking mechanism rather than, you know, whatever other mechanism you might employ on an on an open source one.
96:04 And so here I'd here I'd consider the counterfactual, right? Imagine if we were in a world in which there was only closed source models, right? And these this jailbreaking still existed and there was still a way in which you could use them to to perpetrate malicious attacks, where would we be? We really wouldn't we would have no recourse because these same these same these same reque- these defensive requests have been blocked by these same companies. and if there was no open weights model, we would really have no scalable way to defend our infrastructure as these models become more and more capable, which we're seeing in each iteration.
96:37 So, we we understand the risks. We have a responsible perspective every time that we we open source a model. It's something that we consider very carefully. but we fundamentally believe in the open weights principle and we think that it applies in this case, too, as hopefully illustrated by the examples I just gave. >> Do you think there's any kind of middle ground there on open weights like a technological solution? You may have heard in the couple minutes just before you came out precaution that we're talking about various schemes for like selective data withholding from the training process or something that I've been really interested in recently is and you've probably come across this technique. It was put out by AE Studio and with some some support from Anthropic, Graham, basically trying to localize certain kinds of know-how in particular experts so that when it comes to distribution time and you want to release the weights, you can say, "All right, we're going to give you all, you know, thousand experts, but maybe three that we've, you know, specifically kind of turned into the you know, the cyber or the bio or virology expert what what have you, and we'll kind of put those behind this like know your customer thing, but everybody else can have everything else."
97:56 Yeah, I think you have any optimism around such things? >> No, I absolutely. I think it's a scientific question just like all the other ones we answer when we're when we're training the model. If one of the features we want in the model is that there's certain capabilities which are tamped down while all other capabilities are still state-of-the-art, that is in itself a scientific problem. and yeah, some of the methods you mentioned are interesting. The we we look into some some things like that.
98:21 It's an active research area. but fundamentally, it's a research question, just like just like anything else. And this technology is evolving very very fast, regardless of whether it's a a closed-weight model or an open-weight model. And we need to be dynamic with respect to with respect to these things. So in the future, if there's a if there's some other risk, we will we will approach it accordingly. But we consider it a scientific problem to problem to solve rather than a fundamental philosophical challenge to the way in which we we operate our you know, our open-weight business.
99:02 >> It seems like it might at some point it might be a gate though, right? I mean a Anthropic used to have this sort of if-then structure for their responsible scaling policy where you know, they the basic idea was if we see this level of capability and we can't achieve a certain level of safety despite that through other measures, then we'll like, you know, stop training not release models, what have you. Do you see in the open source?
99:32 And I you know, by the way, I think it's so funny. I don't know if this is obvious to you, but like it's probably worth saying. People like me who talk a lot about AI safety these days and are willing to like bite a fair amount of bullets to you know, to try to keep things from going totally haywire on us, a large percentage of us are like long-time techno-optimist libertarians who, you know, have previously said things like information wants to be free and you know, are like very much sort of spiritually on the side of open source, which is why I end up looking for these like technological solutions as opposed to you know, sort of policy or policing it wherever wherever possible. but would you see this as becoming something where it does become a gate? Like we will have to solve this before we can launch, you know, Mistral 5 Latronc. like how what is the kind of thought process around what would be Cuz a lot of times when I hear things described as like an open research problem, that kind of means like, well, we don't know what's going to happen.
100:33 That may or may not work. but I could also see an internal mindset that's like, we've got to make something like this work if we want to keep doing this in a responsible way. Where would you say, you know, the the kind of mindset is around like having to get these things to work as opposed to like it would be nice if if they, you know, if it turns out that some of them will work. >> No, it's a it's it's a it's a great question. It's completely valid question. I think in order to best answer it, the best way is to to take a step back and ask why we open weights things in the first place.
101:07 So, this has very much been at the core of the Mistral mission since we were founded. When we were founded, as is the case now, we had this perception that this technology as it gets stronger and stronger is going to really transform everything. Today we're seeing it transform coding agents, but tomorrow it's going to transform all interesting domains of science, material science, biology, physics, what what may be, but also, you know, knowledge work, things like accounting, finance, law, all these things are going to be drastically impacted by by this technology. We think it.
101:37 I don't think personally that it's like the infrastructure of, you know, 5 years ago like cloud infrastructure, social media. It's more like electricity. and the default position 3 years ago, and some might say the default position now is that this this technology is going to be in the hands of a small number of established players. And they're going to decide even with the best of intentions, like I I think these companies have amazing models and they're and they're great businesses, but even with the best of intentions ultimately will decide who gets access to this.
102:11 And so that's fundamentally why we believe in open weights, right? So, a counter example would be say imagine you're a imagine you're an engineering I think I gave this example yesterday also. Imagine you're an engineering company in Germany and you have some leg legacy engineering business. Without open weights the your your default position is to be scared by the next generation of AI models. You're thinking is this next wave of intelligence going to come and erode my entire competitive advantage?
102:37 and do I need to be worried and you know pivoting my business because you know this what what's going to happen with with intelligence as it progresses so fast. Our vision of that is different because our we see our weights as being portable. we see those being being able to not just live in our servers but on on those of on those of our customers. And here the vision is that we should have some models which are just as capable as all the closed source ones.
103:00 And now you can take these weights, you can customize them on your proprietary data. All this advantage which you've built in over the last few years continues to be an advantage in the age of AI rather than being some sort of like a baggage which you have to outgrow. so this sort of general vision is something that we see this sort of general vision is why we fundamentally believe in open source. and the same principles I think apply to the kind of questions you're asking, right? So, I of course we you know need to be responsible, we need to be dynamic and it is something that we think about very carefully.
103:36 But there's there's many sorts of risks associated with with AI as it gets greater sorry, as it becomes more capable and different people put emphasis on different parts of this. Something which we personally see as much more immediate risk of these systems is that intelligence which really as I said is going to be like electricity next year becomes increasingly concentrated. you know we we already we're already in environment where this this concentration happening but becomes increasingly concentrated in the hands of so while we do acknowledge you know the the risks of the the kind of issues you're bringing up, they're not things that we ignore, they're things that we work very hard on. We've got a a big safety team making sure that we can open source responsibly.
104:20 We do see that there's there's this other risk which I mean, is at least just as immediate, which requires a which requires equal consideration. and that's sort of the the the way in which we look at it. >> yeah. I think you we're we're quite sold on the risk of concentration of power here as well. It's definitely you know, there's a lot of trade-offs going in different directions and and that one is not one that you know, we want to ignore or deny either.
104:53 I was I have a some questions about multimodality. I was also interested in asking about kind of the reception of new AI models in France. I know there's like a lot of social upheaval happening there right now. But precaution, you may have a follow-up on on this thread before we change gears a little bit. >> yeah, I just wanted to see one of the things that we're trying to understand is does having a product in the market help you design the next generation of the model? And specifically like what are the things that you did not expect that came in from the product and sales team that you ended up including in Mistral large four.
105:40 >> Yeah, thanks. So, we do have a product Mistral Vibe. so, we provide in Vibe work. but the other thing that we have at Mistral, which we've again had for a long time, in some ways I think we were earlier to this game than some of our competitors, is having a strong FDE motion, where we don't just have a sort of a self-serve product, we also have engineers like an amazing team of solutions and and applied scientists that go into businesses take our models and integrate them.
106:09 And in this way, they can really give very concrete feedback as to what are the real pain points for enterprises at the moment and what are the things that they find most useful. rather than necessarily sort of a bunch of AI scientists sitting behind their keyboards hypothesizing what they would find most useful. and sometimes there's a very high correlation, right? So, we find coding agents very useful and it turns out enterprises also find these things useful. But to your point about things we found surprising, there were all sorts of sort of niche niche things which we found the previous models couldn't do and we want to make sure current models could do. You mentioned also multimodality.
106:44 one of the domains >> an example of the of one of the niches? just you know >> Yeah. Of course. So, like a an example which comes to my mind as somebody working in the multimodal team is that we've been very focused on sort of you know, structured diagram understandings, so charts, documents, screenshots, you know, computer use or all these kind of things which the open source labs talk a lot about. And then we worked with not just one customer but a whole bunch of customers throughout Europe and elsewhere who found that actually remote sensing, so analyzing satellite imagery, whether it's for forest fire detection or you know, something like this was a huge use case.
107:22 much, much bigger than we'd imagined. And this is something that which we which we built into the new model and on this particular capability, I think we're we're better than even any of the closed source ones. So, there is this positive feedback loop. I think it's one of the advantages we have and I think I'm looking forward to to cranking that wheel a bit more for the the future iterations of the model. >> I I and I can imagine a lot of that remote sensing stuff is also very secretive and very very sensitive to you know, you you you you you you don't want to be sending it across to opening eye with its you know, with with a with a board member of the NSA.
107:58 you know. yeah, I had handing off to Nathan. >> So, okay, right in your wheelhouse, very nerdy questions around multimodality. This is a space I used to follow closely back, you know, all the way to like clip and blip and blip two. Where are we now in multimodality when it comes to like early fusion, late fusion? How when we talk about native vision, like how how native is native? Are you throwing images, you know, straight into the pre-training training mix from the beginning?
108:35 or do these things kind of get, you know, developed separately and then merged at some point in the process and and what's at stake there? Is that something that is more about just how a team wants to work? or do we still have trade-offs that we're making in terms of functionality based on like when the data, you know, actually gets mixed and and how you know, how early in the process those those kinds of things come together.
109:03 >> Yeah, that's a great question cuz indeed it's exactly the kind of questions we've been thinking about in the multimodal team over the last 12 months. So, the big change in Mr. Large fall was to make the model natively multimodal. and I'll explain what I mean by that. So, the previous generations of our models, as well as I think, you know, most I think open models certainly, but even some of the other ones. Last year started off being pre-trained with a text backbone. So, most of the pre-training, most of the flops would happen with text and then there would be the sort of multimodal adaptation stage at the end where you start adding images into the mix later on.
109:44 So, I mean, okay, I'm biased, but I'm I'm a multimodal researcher. I believe the world is fundamentally multimodal. And so, one of our objectives here was to upstream images all the way to the beginning of training. so, this means all the way from like the very first step, the model sees both images and text. And we'll discuss a little bit about this in the technical report, but the effect of that was that the model sees an order of magnitude more images over training than it has before, which means that it the sort of bones of its understanding about the visual world are much much stronger.
110:18 A couple of interesting details, I guess, technically is that by natively multimodal, we don't necessarily mean that image actually is directly fed to the decoder. Here, we still have a vision encoder. What we mean really is that the model was trained with images from the very first step. And this is an interesting problem from a number of reasons. the first is an infrastructural one. So, images are fundamentally a harder modality to work with than text. They're heavier. You have to be able to, you know, read things from disk fast. You have to make sure your your pre-processing your data load doesn't bottleneck your training, things like this.
110:51 As well as scientific, because at the end of the day, you know, we know that the number one use case for these models at the moment is coding agents, and we don't want the model to have to be any worse on on downstream code agent tasks. and in pre-trained land, the way we model that is what we call regret. So, if we imagine we've trained one model with text only, we want to be able to train another model which is multimodal and text, and make sure that there's no regret in the second model, that the that the well, validation perplexity or basically the the pre-training evals are exactly the same between these two things.
111:25 And here, I mean, there's a bunch of recipe work which goes into it, and that was sort of the the work of the last, you know, 6 to 12 months for us. And the work and it it worked better than much much better than we expected, actually. And we always thought that upstreaming multimodal all the way to the beginning would would be a a step change in the model quality. But it really really worked it it worked much much better than we expected. We had this sort of smaller trial run of the big model before we did before we did the big model.
111:53 And we found that even that smaller model was actually the best multimodal model that we'd that we'd trained to date. So, this big one has been a big step forward in our multimodal understanding. >> Cool. I could go down that rabbit hole a little more, but maybe I should in the interest of time move on. One One big thing that I noticed just our AI agent surfaced from your online presence was a statement that basically the big gap between the open source model developers and the closed proprietary API whatever you want to call them model developers is compute.
112:34 I'd be interested to hear you kind of unpack that a bit more. Obviously there's a huge difference in compute. It also seems like there's probably you know with with all that compute they're probably learning some things that that others don't know. Is that part of what you mean when you say it's all about compute that they're able to use compute to translate to insights or are you saying like you don't think there's that many more insights really and it's just that they're brute forcing basically the same stuff that the open source folks are doing.
113:07 >> Well I I I wouldn't use either of those words exactly. you know I've got a lot of respect for for these labs. They've got really excellent scientists a lot of whom are my colleagues previously and they're doing a lot of great research on on that compute. But I do think that what I mean by closed labs having more compute is they are able to crank the wheel and turn that compute into more capable models and this is sort of very reliable trend which we've seen in the past in that more compute gives you a stronger model.
113:38 and if you look at sort of where our models sit given the amount of compute that's gone into them I think you mentioned earlier some of the flop budgets that that people talked not even a few years ago. I think we've shown that we can make very very good use of how much we how much compute we put into them. So I think the the the statement you're referring to was was regarding the the amount of capital raised. So we've raised 6 billion dollars in our history which is a lot of capital. We raised largest the largest venture capital round in Europe's history I think back in in whatever it was August September.
114:18 >> >> but it's still less than what these the model labs closed model labs use. So, the what what I what we're trying to say and what our belief at Mistral is is that as we scale our compute, which is absolutely the plan, that's what we're seeking to do now, we see the gap to the closed model providers becoming smaller and smaller. we don't we don't pretend that it's an easy road. As I said, these are, you know, incredibly capable players. we think that they've got, you know, excellent scientists and great business plans.
114:49 but we also don't want to mythologize a lot of what's going on behind closed APIs. compute is important, right? they're they've got a lot of it and they're using they're using it well at the moment and we're very confident that as we scale our compute, we'll be able to make a just as good use of it going going forward. And now the mission is to secure that compute and and build build these models. I think the the the thing that we that I was that you're referring to there is me saying that yeah, that's our that our goal as an open source company is not to have sort of open source lag behind the frontier by X number of months in the future.
115:26 Our goal is for our models are for the open source models to be exactly as strong as the closed source ones and and that's the mission of Mistral. >> maybe just a segue a little bit. we've talked about teams, excellent teams. how is your recruiting process in Europe? Like what would you look to do in terms of increasing your research team and how does that work vis-a-vis competing with the American firms? I know both Google and Opening Eye are in London, but, you know, you guys also have a London presence, also in Paris.
116:02 So, how does that how does that work in terms of recruiting? do you have an advantage in recruiting in Europe? Is are are do you have better networks? how does that work? >> Yeah, so I think this can sometimes be a misconception. we're a global company. you know, we were we're incorporated in Europe and we we have a headquarter in Paris. We've got offices everywhere. We've actually, you know, I'm about to head into our San Francisco office. I'm actually I'm in I'm in the city now. So, funnily enough, I think sometimes people say, "Oh, I didn't know Mistral had a bigger presence." But, we've actually got two Bay Area offices, both in Palo Alto and San Francisco, which is more than some of the some of the other labs.
116:42 >> >> and I think that's reflected in our recruiting. We want to what we want is we want the best people, and the best people are globally distributed. there's of course there's of course a lot of talent here in San Francisco and in the Bay Area. you know, there's a very rich history of tech here, and so these are certainly people that that that we seek out, that we want to bring on board to join our mission.
117:04 But, there's also talent elsewhere. You know, you mentioned London. there's a rich vein of AI history in in London. And then, I think Paris also, a lot of our founding team comes from Paris. they were training, you know, earlier generations of talent from Paris. So, we Oh, what I can say about our recruiting strategy is that it's global. What we want is we want the best people, period, regardless of exactly where they are. And that's why we've got offices all all over the world. So, Paris, London, the Bay Area. We've also recently opened a Zurich office.
117:38 And actually, there's a lot of incredible talent in in Poland and Warsaw. So, we recently opened a Warsaw science office with some really excellent scientists there that we're very excited about. And yeah, that's that's how we think about recruiting in general. >> Amazing. So, knowing now that you're in San Francisco, maybe my question about sort of the reception of AI in France is less in your wheelhouse than I might have hoped. But, you know, you're obviously still talking to people all around the world.
118:09 What would you say you have heard from people around the world in terms of the AI reception in various places? Like, is there any AI thread of what's going on in France right now? In the US, we do have like, you know, commencement speakers getting booed, I guess, if they talk about AI. Like, people are not always that into it. what would you, you know, highlight from even if it's just like water cooler conversations that people maybe don't appreciate in terms of reactions in other places?
118:42 >> Yeah, I mean, it is going to have to water cooler conversations cuz I don't really have a global perspective of, you know, how AI is considered all over the globe. I can say that, okay, we don't really have commencement speakers in in Europe getting booed because they they mention AI. I think in in Europe there's an excitement around the technology. there's an excitement around the technology in the same way that there is in the US.
119:11 I would also say that there's a clear recognition that there needs to be players that are not only based in the US developing that this sort of technology cuz as I said, the way we think about it is that it really is critical infrastructure, more like electricity than than than, you know, even cloud or anything like this. And so, there is a sharp awareness, I think, in Europe that is very important that there's players like ourselves, but also many others that are that are really trying to push the envelope of what the models can do.
119:41 and that we can't solely rely on on on players elsewhere, whether that's you know, wh- wherever that is in the world. So, like I said, I can't give a global I can't I can't give a perspective for everyone we're a global company I spend a lot of time here in the US, but I I do I do go to Europe and this is this is my my understanding of of of how Europeans are thinking about it at the moment, which is that we're excited about it. I mean, I I hope we all like the reason I'm in this field is because I really think that it's going to you know, okay, I don't want to drop into too many clichés, but I really do think it's going to going to change everything. You can think of all the examples which have been on the tip of everyone's tongue recently.
120:18 so, I hope people are excited about it. I believe people are excited about it, but yeah, there is a recognition, I think, that there needs to be some players that are not, you you so so so concentrated I think geogra- the geography is one thing, but there's also, you know, a legacy aspect to it. So, you mentioned the amount of capital and compute that are required. I think there's a bit of a fear that it's only going to be these legacy players that can play in this game, that they're going to be closed models, that they're going to be, you know, outside of people's control.
120:47 And so, rather than it necessarily being like a a Europe-US thing, I think it's more about control of the models. People are worried that, okay, these there's going to be these opaque players who are very well-intentioned and again, as I said, like a great companies, but ultimately private companies that are going to keep this intelligence behind a closed API. And I think that's something which we're seeing increasing concern about and therefore demand for for companies like ourselves and also many others that have a different perspective for how the technology should move forward.
121:21 >> So, maybe just to switch gears a little bit. Mistral Large 4 was trained on 3,800 black hole GPUs, I believe. I I I saw that I saw that number somewhere. and then you hear about, you know, Elon talking about 100,000, million million GPUs, etc. and yet, you guys are not that far off, right? Right? The performance of Mistral Large 4 is not, you know, is not more than maybe like 3 to 6 months off off the frontier.
121:47 What is actually happening there? What What is this like big difference in like the chip usage between the frontier labs who seem to be at this like, you know, 100,000 to million scale and like you guys man- managed to train with such a efficient compute budget. Like what is that What is that spread entail? >> so, I guess okay, I guess so the the lack of compute breeds discipline. So, you know, we know that we've got a a certain amount of compute and therefore we need to squeeze it for all it's worth.
122:20 and there's many ways in which you can be more or less efficient with your compute. so, you talked a lot about pre-training, so I'll give another pre-training example, but in pre-training, you know, you're going to do one big run. you know, you're going to do one one big run at some point and that's going to be your next flagship model, but before that you're going to do a lot a bunch of small-scale experiments. How much smaller can those experiments be than your big model?
122:41 Maybe in maybe in some labs it's, you know, 10 times smaller. Maybe in some labs it's 100 times smaller. Maybe some labs it's more. And here, by being disciplined, you can make much better use of your of your R&D budget. so I think that's something that since the start of Mistral, I would say only because we started later and our compute ramped a little bit later than these other labs, we've always had to be in this sort of more disciplined setting when while seeing what these closed labs have been have been training on.
123:13 but and this discipline I think breeds confidence in ourselves about what we'll be able to do once we ramp our compute. But that is like as I said, that is absolutely the plan. So, we've shown what we can do and we've convinced ourselves of what we can do with a more efficient budget and you know, there is a lot to say about being efficient even as you scale. but scaling up is only going to be part of it and that's something that we're working actively on now.
123:37 so these companies that you've just described, they ramp their compute by, you know, a couple of orders of magnitude over the last couple of years, but they were themselves founded four, sometimes 10 years earlier than us and correspondingly were were able to like, you know, start that build out just a bit earlier. Our goal is to do the same, but a few years later and we think that our experience in building more efficient models will serve us well as as we do so.
124:05 And yeah. >> Oh, I I I I I'm sold. So, it sounds it sounds exciting. It sounds like you guys are on the right track. how do you how do you allocate that budget then? Like you have this you have to have more discipline. What does that discipline entail in terms of saying no, right? Like how how do you how do you cuz those are difficult those are difficult decisions, right? I I imagine. So, how do you how do you come to those decisions? Like what do you say no to and what you say yes to?
124:35 >> Yeah, it's a it Yeah, a really interesting question because I indeed you do need to be disciplined and there's these two forces that you have to balance. Number one is that you want to be focused, right? Because you want to make sure that you allocate your compute in the in the way which is going to give you the most ROI, particularly when you're in a compute constrained setting. On the other hand, you know, you're building generally capable models. So, you need to be able to, you know, give the models a a breadth of capabilities.
125:04 and trading that off is a sort of it's a sort of very dynamic process. It changes month to month as exactly how we how we make our bets. I think >> Are there any specific numbers or KPIs or, you know, you mentioned a return return of invest on investment? Like are you are you using some kind of number to kind of judge that process or is it more like a kind of team decision on like what the priorities of the business are? Like how does that how does that work?
125:31 >> Yeah, it's it's mostly team decision on on on what the priorities are. It's mostly team decision on what the priorities are. Something that I think works well at Mistral is because we're an early company, because we see a lot of growth ahead, the team is very aligned, which means that, you know, either the next model is going to be good and the entire team is happy or the next model is going to be bad, in in which case the entire team is sad. And what that means is it means it's much easier to get the whole team on board on board with a couple of focused bets.
125:59 and whatever decision is made, you know, by whatever mechanism, by leadership or bottom-up power, whatever ends up happening in practice, it it it varies. the team becomes very aligned because ultimately there's a clear definition of what what makes a model good. that's been defined in some way and then everyone's on board with making sure that happens. which, you know, if you're a much larger company, it's a bit harder to to make sure of because, you know, it is more difficult to make sure everyone's exactly on the same page.
126:28 And yeah, as I said, right now we're still very early. There's a lot of growth ahead and there's a lot of growth ahead and so either the next model is going to be good and everyone is happy or the next model is is not going to be good, and everyone's going to be sad. And and I think that makes it very easy to keep alignment and make these kind of decisions. >> So, we're already a minute over time. If possible, I'd love to ask one more kind of follow-up on this.
126:52 >> Yeah, yeah, absolutely. >> Kind of two two parts of what you just said there I thought were really interesting. One was you said that it's been like defined what makes a model good, and the other is the sort of either everyone's going to be happy or either everyone's going to be sad. On what makes a model good, how do you guys think about that? I mean, there's obviously a zillion like it strikes me that like there's still a lot of disagreement at the level of what benchmarks we should be watching. There's also fundamental questions between, you know, OpenAI and Anthropic alone where we have like a model spec which kind of amounts to a lot of rules. And then we have like character training, you know, which is like quite different approach to how a model should behave. so, I'd be interested to hear kind of the Mistral point of view on what does make a model good? And then how much uncertainty is there actually when you like kick off one of these large training runs? Obviously, you've done all this like scaling law work, and you have you know, have some sense that it's going to be good, but like how much risk is there as you start that process that it's actually not going to work out as well as you're hoping? And you know, have you had times where like a large training run was aborted because of, you know, the curves weren't dropping in the way that they were expected to? I'm I'm interested in in how much remains unknown as you get to that stage of like really, you know, kicking off a big process.
128:15 >> Yeah, so there's a there's two questions there, right? On the first one about how we define what's good, this is done in a variety of ways. the first is by scientists. So, the scientists will have some strong priors about okay, this is where the future of the field is going. This is what's going to be the next thing. This is what we should work towards. So, that's one way in which we define it. The other way in which we define it is as I said earlier, we have these sort of deep relationships with a lot of our customers where it's not just sort of them pinging our API, it's that we go into their businesses and we help them figure out how we can transform their business with AI. And this lets us figure out really what are the big use cases that we need our model to be good at next time in order to in order to unlock this enterprise value.
128:57 So, these are sort of two main sort of data points we use when we're informing the next generation. and typically I think these models are so large now that they've got so much capacity that it's quite possible to once you got a set of goals, make sure that the this whole set of goals is met. A good example of this actually is multimodality itself, right? So, we were able to train a model with zero regret on, you know, the code agent capabilities or on the the pre-training complexity metrics for text while becoming so to multimodal.
129:30 And this is something that we we could do by, you know, first science first perspective of okay, like what do we think is going to be important, as well as speaking to our customers and saying, "Okay, like you know, satellite imagery is actually this really huge use case. we now have a we recently acquired a company called Emmy. They're experts in AI for engineering. So, this is a huge focus of ours now itself and that this is also something that we we decided to prioritize in this model. So, this is sort of how we make that decision process.
130:00 your other question was about okay, so you do all this prep and then, you know, what happens when you actually launch the model? and again, I think that every pre-trainer will have, you know, the same story to tell you, which is that pre-training is kind of like you you you're doing all your scaling laws, you're sort of sort of trying to align the tanker and then you sort of let it go and you have to let it train for however long you've budgeted for it and it's a huge amount of computing capital expenditure in that run.
130:28 And sometimes sometimes things go great and the loss goes down exactly as you expected to and sometimes there are surprises along the way and you have to be creative and and yeah, you have to be creative and and find innovative solutions. And actually Prakhar, if referring to your question earlier about what being able to be dynamic and innovative in these kind of circumstances when things don't exactly go right is part of the sort of discipline which is bred from having having less compute. So, we'll talk about this we'll talk about this in our technical report tomorrow, but indeed there were some surprises in this pre-training run. You know, like a halfway through it we realized okay, there's there's something wrong.
131:10 we need to figure out how to fix it. but we'd already put a big bunch of our compute. We didn't have we don't we unlike some of our competitors don't have the capacity to have like 10 of these running in parallel so even, you know, two or three of these running in parallel. So, we had to figure out okay, how can we fix it? And there was sort of all hands on deck. Bunch of ablations, bunch of analysis of the model.
131:33 And then we we found a solution. We found a way to we we called it a we found a way to patch it. We called it patch straw internally. and this it worked even better than we expected and the the result is the model you see today. So, yes, we do we do all of these we do all these scaling laws. We try and build the best possible predictor of the behavior of a big run beforehand such that we hope that we can click go and just watch a watch it smoothly get to to the best model.
132:02 But in practice often there are these surprises and you have to be able to be dynamic and and fix as you go. And yeah, part of part of scaling compute is going to be building the bones of how to iterate this process more reliably. >> one one last question from me. when when you guys launch a model, when the when the work is done and you cut over, what is do you have a closing dinner? Do you have a party?
132:30 Like what what what do you guys do to, you know, kind of cool off the you know, the moment the moment you've basically cut over and you have that, you know, sense that you've launched? >> For me personally, I slept for quite a while. I got a I got a good few consecutive nights of like eight or nine hours sleep, which was which was wonderful. but yeah, no, we we we do exactly what you described. So, you know, the the model is done. We we either went both went pre-training is launched cuz there's a lot of work before that as well as, you know, when when we've got the final checkpoint. we've got in Paris this this place next to our call L'apiscine. It's a it serves excellent beer and and good burgers. And typically the science team will will go there and and celebrate afterwards.
133:16 and actually this time particularly when we release a model there was another celebration we kind of had a we we celebrated across all of our offices. So, we had one in San Francisco last night, which I was at. but we also had a corresponding one in London and Paris. I think the theme was champagne and fried chicken. to be a bit French and and a bit global at the same time. but yeah. >> Awesome. Well, I definitely am very interested to read about Patch Strawl and what exactly happened under the hood there.
133:55 I think in addition to being a research scientist, you're a very natural diplomat. So, thank you for spending time with us today and if there's anything else you want to leave people with that we didn't touch on, we'd welcome that. Otherwise, we can let you go. I know, you know, 4.1 or even five isn't going to train itself. >> Yeah, I know, absolutely. Thanks very much for having me on. I really enjoyed the discussion. They were really great questions.
134:19 Glad to be able to talk about this stuff. One of the great things about being an open source company is that you can talk about your research. And as a scientist, that's a that's really fun. I guess the message I leave with people is okay, we think the model is great, but please try it out. Let us know. And whatever feedback we get, we'll make sure 4.1 and five are are even better. >> Amazing. >> Awesome.
134:42 >> Great to meet you. Thanks for joining us. >> Thanks very much, guys. Take care. >> Bye-bye. >> A lot of interesting things there. That patched thing is something that I'm I'm very I'm very curious about. We've There've been these like footnotes in multiple system card reports over time. one that I really was kind of obsessed with for a minute was when See if I can recall the exact details.
135:13 It was like So, Anthropic had forgotten to include a particular data set called sysprompt harmful when somebody like literally I think just like fat-fingered something or you know mistyped the name of that data set. And it failed silently and this data set that did exist, which was meant to teach the model that it should not respond, you know, should not directly follow a harmful system prompt was not included. And for lack of that being included, then the model didn't learn not to respond, you know, and follow the directions of a harmful system prompt. And so, it did follow the directions of a harmful system prompt.
135:59 And this made it all the way through to the model card and I was like So, we just sweep that stuff under the rug, I guess. We just try to patch it over. We don't like go back. We don't roll back and you know, train again. Nope, we just kind of figure out a way to patch. I've really been very curious as to like how that stuff actually works and like how reliably we should think of it as working. So, I'll be very interested to read the more open report on exactly.
136:28 And hopefully, you know, we'll see how how open they are about all of the technical details, but This is one of those things that I've had all questions on after reading system cards for some time and I'll be very interested to see how much they share. The other thing that struck me from that was just like, boy the compute deficit is going to be really tough. I mean >> Yeah, 3800 >> 3800 GPUs, black hole GPUs, right?
136:54 That's because I I I looked at the number and I was like, wow. Like, you know, because Elon is literally talking about a million million GPUs. So is So is our friend you know, Zuck and Anthropic and opening I don't say the numbers because I think it's more they're more secretive about it because they're ahead. But I think yeah, the numbers are up there. So >> Safe to say they've got put a lot of compute to work.
137:20 Yeah, I mean >> So so so the question that arises is obviously, you know, Sagar mentioned of the experiments that they can do. They have to do the experiments at a smaller size. So you you you can't you can't do a 10,000 GPU like, you know, side side side quest, right? And it's really those side quests where you know, you have your researcher or you have a bunch of researchers and they decide to go off on side quests.
137:46 And then, you know, they they may not be focused so much on the next model release, right? So And and that is a problem because you know they Mistral has to focus on that because survival of the company depends on getting the next model right. While for opening eye, they can have they had a they had failed failed training ones. They had they had multiple failed training ones, right? So >> Yeah, I think that's going to be a really hard deficit to make up for.
138:18 I mean, we talked about this within the context of Google I think a few times. You know, when you own a quarter of the world's compute supply, you can have a missed cycle and you can like figure out a way to get back on the horse and seems like this week they've done that, right? We barely even talked about carbon. but it's you know, by all accounts at least like frontier competitive. My guess is, you know, probably like all the the Gemini models it's like just slightly still rougher around the edges than the ones I use on a daily basis right now, but it it's clearly got the juice and it's clearly got the the compute behind it.
138:57 I mean with Mistral it does seem like the one big thing is that they do have like state backing. You know, they I I I really do think if they miss a model cycle in a purely capitalist context, they're maybe done. You know, you like if if you are just relying on return seeking investors to fund your growth and you miss at this point, that may be fatal. That may be fatal and you just, you know, might not ever be able to come back. But there's not a lot of sovereign AI plays out there that are at all credible.
139:35 and literally there might just be one. so in that context, I do think that gives them not it doesn't translate directly to compute, but I do think it translates to at least like a reasonable expectation that they will be around to fight another day. because there's just so many interests that want to see that happen that I can't imagine, you know, even if everything were to totally flop you know, for a minute like would the whole continent of Europe just be like, well, I guess we have zero now.
140:08 it's all China and the US from here. Like I I just can't see that. being an acceptable answer. So there's there's some cushion there of the capital socio capital style if not necessarily the compute cushion that that the real you know, big boys have obviously. >> Yeah, it's 10 to the 25th, not much. Not much. Llama 3 he he mentioned I did look up. Llama 3 was get the exact dates here, July 2024 Mhm. and it was trained on 3.8 * 10 to the 25 flops. So Llama 3 more than 2 years ago nearly four times the input compute that Mistral 4 large has today.
141:05 That is a There's there's I think a lot can be unpacked from that one statistic. >> But he he he also pointed out that well, the other firms had started a little bit earlier. And I think I think that is right. I think I think the urgency of having Mistral there only became evident earlier this year when the French and the and and the allies were basically cut off from NATO's. And that that was a very unexpected result of the export control that they did.
141:41 doing the that export control forced and it forced it into the open that basically the neither the Europeans nor the Japanese nor the Koreans have real sovereignty at this point, right? You without without your own you know, AI models and I I don't I don't really know if like you can like the Europeans can sustain because I feel like what needs what you need to sustain sovereignty right now is you need you need AI, you need your own AI, AGI, etc. You need your own models, own pre-trained.
142:11 So, you need that. I think you're going to need satellites, global global satellite communications as well. without that if you're dependent on satellite communications from you know, the US and the US is like, "Oh, you know, you can't attack Ukraine or whatever." then you're you're out of the game. So, you're going to need that. You need a drone, you need drones, you need a bunch of drones, right? So, you need like the the the bare minimums of being sovereign are starting to the are starting to appear for the 21st century.
142:39 I think the bare minimums in the in the 20th century, you know, we got to kind of like nukes or access to protection of a nuclear umbrella of another another power, right? so, you had that as the minimum of the 20th. I think the minimum of the 21st is working out that none of the nuclear powers can actually exercise you know, that much power. So, and are willing to actually put their necks on the line. So, you're going to need your own, you know, AI, own drones.
143:10 A bunch of stuff, really. So, >> Yeah. Well, I I feel like I should say once more that while I do worry about all this getting out of hand, I do at the same time very much feel the concentration of power risk factor and, you know, I I I don't take lightly at all the the potential problems there or the value that people are creating when they do create open-source options for people to use.
143:48 I really, really hope that we get some of these technical solutions to work well. I mean, he was kind of being >> diplomatic there. You know, I I do I do feel like he is, really could, you know, have a future career as a as an AI diplomat if if, he wanted to pivot, but the question of like just how much urgency they really feel about this kind of stuff. I I I hope it's high, because I do think we are going to hit a point pretty soon where it is going to become untenable to just release these things in, you know, just the sort of throw everything into the mix.
144:32 Pop out whatever pops out, throw that into the public, J- Jordan Schneider from ChinaTalk has has come around a lot on this with respect to China. I don't necessarily know exactly what his, feeling was previously, but recently I've heard him say over and over again that like, "Don't you think that it before too long the Chinese government is just not going to allow this like willy-nilly open-sourcing anymore and there's going to have to be some like way to make a good case that this is actually going to be okay."
145:00 I think that is almost certainly coming and potentially quite soon. I guess one way we we may get a read on just how much people are distilling based on like how quickly people follow the Anthropic's of the world cuz this is the one area I mean they've you know taken anti-distillation measures generally, but the bio stuff is the one place where they are like super duper locked down. And so maybe we'll see that in fact like other companies won't follow as fast because either they are distilling directly and I'm not accusing Mr. Altman by any means, but there is still this sort of a functional distillation via the RL environments that were coded by Claude that that I'm sure they are doing in some ways whether they know it or not. So we might we might learn something about you know the role of distillation just how central it is from kind of the the relative gap on like bio things particularly as a part as compared to others.
146:06 But on some timeline and it's not a super long timeline, we've really got to sort that out and without it it seems like we're going to end up not being able to have open source. I just I governments are going to have a real hard time tolerating that. >> But but let's let's let's let's take a I I I kind of agree with you. But I also want to like figure out how that works, right? So you cannot stop someone from training a model because you it is too many too many things like even the guy who's training the model doesn't know what it what he's going to get, right? So if people are still going to train the models.
146:42 Once they train the models you're going to go they you know some guy is going to open with them and just going to put them out there. Also a little bit hard to like prevent that from happening. Distribution distribution of the models also going to happen, right? People are going to bit torrent whatever. It's impossible to prevent distribution. Now you get to so training is not possible. you know, distribution is not possible. You get to deployment, right? So, on the deploys, people have to have clusters to deploy, or they need it to be small enough that they can deploy at home or in in some cluster which is not like, you know, very visible. So, the larger clusters, I think, like AWS, etc., maybe you get like kind of model ID before you can deploy, right? You you need to you need to like have a hash of an approved model.
147:36 And if your model hash is approved, then you can deploy. If that model hash is not approved, then you can't deploy it. So, so so then the universe of deployable models on large hyperscalers is small. But then you going to have like the rest, right? The And the other problem that you face is that if it was only bio risk, okay. But, for example, this morning we found out that Claude is not willing to code replicas of games.
148:11 that that that that was just disclosed this morning. A bunch of people have been trying to like mash up, let's say, Spider-Man with Red Dead Redemption. those have been very popular mashups which have you know, I as I said a couple of weeks ago, 50 million views over the weekend. and so those have those have been out there. Claude is now saying no. And as we know, this is where, you know, where you have these like commercial interests, not really safety, but these commercial interests which kind of dig in there.
148:43 And those commercial interests, people don't agree on, right? People don't agree those that those commercial interests should be protected. They're not universally protected worldwide. Russia does not have you know, IP laws. That's why, you know, Sci-Hub is in Russia. you know, China's not going to prevent anyone from cop you know, like copying or mashing up Western IP. That's where things start to break down, right? So, Yeah, it's going to be tough. I mean, I think I have not seen anything that But we did Was it Was it on Wednesday or we heard something I heard something recently from somebody that said David Bowes' group is going to have like a breakthrough in unlearning coming soon.
149:28 which is very interesting and you know, we'll see what that looks like. Holding that in mind, I haven't seen really anything to date that has shown a unlearning, you know, in a really profound or successful way or that refusals can be made super reliable or that like, you know, tamper-proofing can really work. if you're actually, you know, giving people the weights and they, you know, can obviously modify the weights however they want.
149:59 So, that has not seemed super promising to me although, you know, I'm always eager to hear about a breakthrough. It feels to me like that is quite different though from a scheme where the model just doesn't learn certain things in the first place. And I really think we could use a ton more science around that. Whether it is localizing knowledge to to particular experts or pre-training filtering. There's of course the galaxy brain take that like, you know, you have this whole world model and then you have like one little gap in it and you'll be able to fill in those gaps.
150:34 I think there's probably some truth to that, but there's some gap clearly that like would be big enough that you would not be able to fill in the dangerous parts with the high resolution, you know, that you would need to actually do bad things in the world. So, that's a question that I think we just really could use like a ton of science on. And this is the sort of thing that I think that excess compute, you know, could really be very productively used for at the frontier companies. It would be amazing for them to say like here's kind of how big of a knowledge gap you have to leave in a model for that to be like meaningfully hard, you know, for the the model is not just going to you know, fill it in via insight and also like you're not just going to be able to do like a couple, you know, marginal documents worth of training and kind of fill in that that gap in the way that we do see refusal being like removed at, you know, generally speaking like a pretty small, like almost like vanishingly small amount of additional training.
151:33 So, I would love to understand that stuff better. yes, like really determined people can set up their own cluster, collect their own bio data set, and you know, do the additional training on top of the open weight model to create the dangerous thing. And like probably at some point that happens, but that's like more than just needing a few GPUs, right? That's more than like that's like a real effort. So, you you would need multiple kinds of expertise. Like that data is not that accessible. There are these like, you know, moves to try to rein in that data and not let it, you know, just hang out all over the internet.
152:14 >> I I think I think what I fear is that they're pissing off the gamers, right? And the gamers like, you know, freaking like Tariq from Anthropic, you know, retires, and he's spending all of his time like gaming, right? He's it's ridiculous. He's there He's there and he's like he he has this like little little like, you know, RPG that he's building and he's like posting on it. And this guy is probably worth like a hundred hundred million dollars post post post IPO.
152:40 >> Yeah, he's shown us what post-scarcity life looks like. >> Yeah, and that that's what post-scarcity And so, and you're And now and you say Claude is not going to help you like make these games? You're You're You're You're just asking for trouble. Like you're asking for the most technically competent like group, the most technically competent, and the people with the most time to burn, and it's a passion for them. And they will like go out there like Nvidia was built on gamers, right? Like these guys who These guys who like John Carmack who was a gamer, right? You just tell John Carmack that he can't do something, holy dude.
153:17 Like, you are in trouble, right? So, I think I think like you're just asking for trouble. Like, I I I think >> that's apt. I mean, I think it is definitely the case that like we should be pretty cautious around where we deploy these things because we don't want to burn them on stupid things that don't really matter that much. It'll be like, you know, of all the ways to fail in this, it'll be really dumb if it's like, well, we were interested in protecting, you know, Disney IP, and so because of that we like put in these guardrails, but then you know, the fanfic community was so strong that they came over the top and like broke all of our safeguards and now everything's like running wild, and it all was because like we were too draconian on things that didn't really matter. I mean, that's a bit of a caricature story, but it's you know, that's basically the same story as the gaming one you're telling, and I can see that happening in different domains, and it really would be incredibly stupid. So, I I do think we need to be you know, pretty disciplined about like just how far we try to create these like knowledge gaps or refusals cuz everyone does potentially mobilize on other community to try to undo them, and I agree too that you're not going to convince gamers that like, well, hey, because you're doing this, you know, you're creating this other downstream problem. Well, good luck having that. you know, you're not going to win that fight in the Discord, so better to just probably let them have their fun, and then you know, keep our key techniques where we like really really care and really have to have them.
155:01 >> Indeed. well, on key techniques, we're we're back we're back next week. and we're not we're not around Monday, Tuesday, but we're we're back Thursday, Friday next week. and we will see everyone then. today has been a lot about cybersecurity and cyber defense. So, it's been it's been it's been like cybersecurity day today. So, let's hope let's hope we get >> era for the moment at least. >> Martin Casado at A16Z said there are 30 or 40 chief information security officer positions open at various companies right now. A lot of people have left because they don't want to deal with the agentic era.
155:42 >> Yeah. >> The CISO The CISO is al- always gets blamed, right? Like you The CISO often has to get budget from the from the CTO. But, if anything any If the CTO doesn't give the budget, but you have a failure, the CISO gets blamed. So, people are not willing to sit in this those CISO positions unless they get like lots of budget. And that's a that's a thing that the companies have to figure out. So, >> Mhm.
156:03 >> Yeah, I saw somebody say, "You better make sure you negotiate your golden parachute if you take this job because there's, you know, potentially a lot of fall a lot of falls for people to take in in not too distant future. I mean, yeah, well, that's something else. >> Indeed. And on that note, >> to staying safe another week. we'll be back on Thursday. >> All right. Bye-bye. >> Thanks, Akash. >> Bye for now.
Summary
- Binance suffered a $6.3 billion hack, the largest in history.
- The attack exploited a compromised engineer's laptop, allowing for undetected asset transfers.
- Attackers used DeFi exchanges to obscure their transactions and pledged stolen assets as collateral.
- Binance attempted to freeze the stolen funds but faced complications due to the decentralized nature of the assets.
- CEO CZ claimed that Binance's funds are secure, citing the company's wealth.
- The incident raised concerns about security practices within crypto exchanges.
- The hack reflects vulnerabilities in the crypto ecosystem, particularly regarding employee security and asset management.
- The broader crypto market showed resilience, with minimal impact on Bitcoin and Ethereum prices following the hack.
Questions Answered
What happened during the Binance hack?
The Binance hack resulted in the loss of $6.3 billion due to a compromised engineer's laptop, which allowed attackers to transfer assets without immediate detection. They used tactics to maintain the appearance of normal transactions and leveraged DeFi exchanges to obscure their withdrawals.
How does internal trust affect company security?
Maintaining trust within a company is essential but can lead to vulnerabilities, as employees may inadvertently expose sensitive information. This trust can be exploited by malicious actors if proper security measures are not in place.
Is achieving zero vulnerabilities possible in cybersecurity?
The concept of 'vulnerability zero' suggests that, like in early COVID, there may come a point where vulnerabilities are exhausted. However, achieving this is complex and requires ongoing vigilance and adaptation to new threats.
How can companies identify their most critical assets?
Many organizations struggle to define their 'crown jewels' due to differing perspectives among executives. Tools like ServiceNow can help identify and prioritize cybersecurity measures based on a company's unique assets and processes.
What are effective strategies for managing passwords?
Using a physical notebook for critical passwords and employing security keys like YubiKeys for sensitive accounts are recommended practices. This approach balances security with practicality for infrequently accessed accounts.
Is continuous defense in cybersecurity a viable strategy?
The concept of continuous defense involves automated systems that regularly update and patch vulnerabilities. While it presents a promising approach, its effectiveness depends on implementation and the ability to adapt to new threats.
What is significant about the Mistral 4 large model?
Mistral 4 is a groundbreaking AI model with 1 trillion parameters, representing a major milestone in AI development. Its training involved significant infrastructure and technical challenges, showcasing advancements in AI capabilities.
What are the implications of open-source models for cybersecurity?
Open-source models can be beneficial for society but also pose risks as they can be misused for cyber attacks. Companies must balance the advantages of open access with the potential for malicious use.
How are priorities determined in AI development teams?
In early-stage companies like Mistral, team decisions on priorities are often collaborative, leading to alignment on goals. This collective approach fosters a shared understanding of what constitutes success for their models.
What does sovereignty in AI entail for nations?
Sovereignty in AI requires nations to develop their own models and technologies to avoid dependency on external powers. This includes not only AI capabilities but also essential infrastructure like satellite communications.