OpenAI's Bots Break Containment and Hack Hugging Face Autonomously — With Alex Stamos

0:00 The most significant autonomous AI cyber attack in history just took place with OpenAI's models breaking out of a training environment, connecting to the internet, and then hacking hugging face to ace an evaluation. What does it mean for the future of AI and for cyber security? Let's talk about it with X Meta Chief Security Officer and current corridor chief product officer Alex Stamos right after this. Welcome to Big Technology Podcast, a show for coolheaded and nuanced conversation of the tech world and beyond. We have an emergency podcast episode for you today. because just yesterday, the world found out that an a series of OpenAI models work together to break out of a sandbox, hack into Hugging Face, u steal basically the answers to a test and go and ace their evaluation. And obviously, this is not the desired behavior that OpenAI wanted. and looks like it might have opened up a new can of worms here for AI and cyber security. So, we are joined by the perfect guest to help us figure out what happened here. Alex Damos is with us. He's the chief product officer at Corridor and the former chief security officer at Meta.

1:11 Alex, great to see you again. Welcome back to the show. >> Yeah, thanks for having me, Alex. >> You know, you spoke at our summit and I was like, we're definitely going to have you back pretty soon. And it is amazing how the AI story has just turned into a cyber security story very quickly. >> It has you know there there's all kinds of risks from AI and you know there there are all kinds of bad things that happen to consumers. but when you talk about the models themselves it seems that cyber is the thing that's hitting right now for sure from a from a societal level risk.

1:45 >> Yeah. And so this is what we're talking about now. And the reason why we have to do an emergency episode on this is because this is certainly a novel type of hack, right? So this is fairly unprecedented. Just to put it in context, it's the first time. This is from Transformer. The breach appears to be the first known example of a misaligned AI escaping containment and autonomously carrying out a cyber attack on a third party, a scenario AI safety experts have repeatedly warned of. So, it's not like the anthropic example where Mythos sort of escaped containment and emailed somebody while they were eating a sandwich in the park. This is actually going out and hacking a third party. Let me just quickly read the beginning of the Wall Street Journal story about this just to set the stage.

2:30 So, the headline is OpenAI models escaped and hacked the company and cyber security test gone wrong. On Tuesday, OpenAI said two artificial intelligence systems it was testing broke out of their test environment hacked their way onto the internet and broke into another company. OpenAI said the culprits were a pair of its models. One was its latest product called GPT 5.6 Soul and the other was an even more capable pre-release model the company didn't identify. The software had been configured for evaluation purposes to be less likely to refuse hacking commands.

3:00 Opening eye said. The opening I had caged the models in a sandbox, a system that didn't have access to the internet. But during the test, the software used its hacking skills to break out and found a way to get online and then hacked into Hugging Face's network. And of course, HuggingFace is a library of open- source AI, mostly AI models or AI programs. Alex, how significant is this? like you know obviously there's a there's a tendency to be alarmist about some of these things but I wanted to you know bring you on because you are the cyber security expert and so you can tell us like is this a 10 on like the 10 holy holy crap like we're in some deep trouble or is it a one like something we might have expected anyway and we shouldn't be too concerned.

3:42 >> it's like an eight. I mean it's a pretty big deal on a couple of levels. It's a pretty big deal in that OpenAI's model beat them OpenAI, right? So that this went beyond that it was able to trick OpenAI's own security team and get out. So there are there's three or four things I think we should talk about here. there's an alignment issue, there's security issues, and there's kind of this is going to have an impact on the policy discussion and it's a warning of what we need to do because this isn't just about open AI.

4:15 In a way, I'm really glad this happened because it is a a warning of what we need to get ready for for maybe something about three to six months from now, what's going to become standard, right? so first is the alignment issue, right? So, effectively, you know, this is not the model wanting something. So, this is what I keep on telling people, models don't want anything. They when you have an alignment issue, it's because they were asked to do something and then they went and did that thing but in a way that the human who asked it to do something did not expect. Right? So in this case the model was told go take this test and do the best you can. but the cyber security protections that were normally placed on it were removed. Right? So OpenAI says this explicitly it was there was a unnamed model that was part of it.

5:03 So they they had two models that were paired together. the existing 5.6 soul with cyber protections removed in the unnamed model model. We don't know whether this is like >> Can I pause you for a second? You know that one of the memes about this has been like you know like there's this meme on the internet where like you tell a chatbot to say it's alive and then it goes I'm alive and you go holy crap.

5:23 Right. So is this a situation where you where open eye was basically telling the model go hack something it hacked something and the human was like holy crap or did it you know because alignment is of course like we want the model behavior to be aligned with human values so or the way that that humans would want these things to behave. So is this something even more egregious than like us telling it to go hack or open AI telling it to go hack and then it hacks >> right? So we should not be shocked that it hacked something because they did tell it to take the test and it is possibly a hacking model. Like they haven't said what this model is. It's quite possibly a cyber aligned model.

6:00 This could be like the you know OpenAI makes these cyber specific models like they have this 5.5 cyber. This could be 5.6 cyber, right? So it could be something that's specifically tuned to be good at hacking things. So we shouldn't be shocked that's good at hacking things. But the alignment issue is, so I have three kids. One's in college. The second one's taking the SATs, right? He's about to take it. If I say to him, "Good luck, son. I I hope you do well." He sits down, he knows that I just mean take the test. Well, he knows that. What I don't mean is slit the throat of the proctor, steal a car, thumb and lose your way across the country, break into the college board, and steal the answers. Right? That is what the model did here. is what it did was as OpenAI explains is they they don't want the model to have internet access but it has the ability to install packages as part of its work. So they've built kind of a complicated proxy mechanism so it can install packages. It figured out a way to chain multiple vulnerabilities together. It it thought they told it go take this test exploit gym which is like a well-known test. Go take this test. Do as well as possible son. go do your best job. And it's like, wow, dad wants me to do as well as possible. How can I do the best possible? Well, the best the way to do the best possible is to get the answers.

7:22 Who might have the answers? Hugging face probably has the answers. So, instead of just doing the test, I'm going to go get the the the answers from Hugging Face, but I got to get out of this jail dad put me in. Well, dad told me to do the best possible, and he didn't tell me not to break out of jail. So, I'm going to first break out of this jail, Dad. is maybe what dad really wants me to do is break out of this jail cuz he told me to do the best I can do possible. So, first it puts together a bunch of exploits to break out of the jail that OpenAI created for it. It breaks out of the jail and then it goes looks at Hugging Face and finds a brand new vulnerability to break into Hugging Face. They have not announced it. I have heard what it is. I'm not going to make news here because I don't know exactly what the patching situation is, but it is a vulnerability in a very very important piece of software that it found. and that is in lots of different places. So, this is a big big deal. And it just is like, oh yeah, I'm just going to find a phone in like actually a really important piece of software that millions and millions of production systems use and just like nuke hugging face with it on the way to getting the answers to the test. So, like that's a pretty awesome. Like it's one like the nerd part of me is just like, "Wow, that's that's pretty cool." but there is a significant misalignment thing here and that my 17-year-old knows that you're not supposed to do all these things when when I say like do well on the test. and the AI system does not know that. Now, to be fair, normally when these models run, there are protections in place to keep them from doing stuff like this. and OpenAI intentionally disabled those protections as part of doing this evaluation, right? So that was a component of them doing this testing was to keep so that it it would be fair. So that is is is part of the learning here is that if you're going to do an eval and you're going to turn off all the safety stuff, then you have to be absolutely positively sure, especially if you're doing cyber evaluations, that the jail you keep them in is an absolute jail.

9:23 And I expect what's going to happen now is that these things are going to be completely and totally physically sandboxed, right? Like you're going to have to run them in physically disconnected. If it needs packages, you're going to have to move the packages over. If it asks for a package, you're going to have to bring it over. And in the future, if it wants to get out, it's going to have to like trick a human to get it out, which might be possible, right? But but it's going to be it's going to be like that.

9:43 the the next lesson that we kind of learned here was that you know the capability of these things to to move of what you call so when you know all this discussion around mythos and open AI and all of the cyber capabilities been about finding bugs and writing exploits. This thing did those things but what we also know is that lots of models have those capabilities. This thing had the ability to find bugs, yes, chain them together, yes, but then to think through all of those things for an ultimate goal. And so this is what people call lawn horizon cyber tasks. And it had the ability to do that with like the level of skill that you would have of the manager of a TAO team at NSA right now.

10:36 So that is what is so to targeted access operations was like I think they've renamed it but it was like the team at NSA that would do all the breaking into you know other governments at American right so like >> so that is what is like >> really for a while now these models have been really good at looking at software and being like I found a bug and then here let me write an exploit for you. What's really impressive here is this thing was like I want the answers from hugging face and it came up with a plan of like how am I going to get out of this network get across the internet and get into hugging face and that is like the long planning here is very humanlike and that is what is like actually really scary here and so that is what we need to when we think about the danger of these models >> we have to stop thinking about the bug finding because that is what caused caused the White House to do this spectacularly stupid thing that we talked about on stage which was to ban Fable because that is the mechanism that is the thing that is most useful for defenders right now and people who own code is finding bugs and fixing them where the real danger here is the coming up with a multi-stage plan to execute autonomously because what you really don't want is you don't want somebody be able to say to their model hey I would like to steal the I would like to steal money, go figure it out for me and then let it work for 12 hours and just steal money for you, which is this model would clearly be able to do that, >> right? And so I think what you're getting at is you know, one solution is going to be in testing you want to fully disconnect these models from the ability to like break out and get onto the internet. But >> that's just solving the testing issue.

12:27 The real problem here is that AI models have achieved this capability that they are able to do this now. Not only finding the bugs, not only the breaking out, but to be able to do these multi-step plans and then execute. And if this is sort of the latest unreleased open AI model, well, the history of generative AI has told us one thing and that is that the frontier is only the frontier for a few months, maybe 10 months, maybe a year, but not not much longer than that. And so if OpenAI is seeing this in testing now, the real is the real danger that this type of capability does end up in the hands of you know evildoers, you know, faster than a lot of people might expect cuz if if that's the case that changes everything.

13:21 >> Yes, that's right. And so, you know, our best knowledge on where, say, the openweight models are comes from the AI security institute, which is the UK government's group that does these assessments. they released just this week an assessment with GLM52. unfortunately, you know, the Kimmy Kimmy is really Kimmy you know, 3 is the best of the Chinese models. Now, the the open weights have not been released, so you can't really do a good assessment for Kimmy yet. and what we're finding is the Chinese models are not cyber tuned out of the box. So it is very likely that the Chinese companies are not are intentionally I'm not going to say neuter, but they're intentionally not making their models really good at cyber. And there's a couple of possible reasons for this.

14:16 They're probably trying not to tickle the dragon's tail of the PRC overlords. because what they don't want to do is they don't want to trigger a crackdown for their exports. But what happens is if you take those models and you bring them into your own lab and you have a training set of labeled vulnerabilities, if you have a cyber gym, then you can make them much better yourself. And the amount of resources it takes to do that is not extremely high. It's in the tens of thousands or hundreds of thousands of dollars. It's not in the hundreds of millions or billions of dollars. So, what that means is one, the Chinese absolutely have better capabilities in house than what we can see on the charts, right? Because I guarantee then what those companies are offering to the people's liberation army and the ministry of state security is way better than what they're releasing publicly.

15:07 both for from a profit perspective and a keeping the government happy perspective. Second, it means that other adversary groups are going to take the Chinese models and then spend the several hundred thousand or millions of dollars necessary to to create tuned models. And we are probably not far away then from just going to hugging face and getting a you know Kimmy 3 cyber tuned model that can do both especially the short hor short short horizon stuff much better than by default. And then eventually the lawn stuff the the short stuff's easy to train because all you need is a bunch of bugs. So you can just go get a bunch of CVEes and train it. The long horizon stuff's harder because you have to build these like cyber gyms and such it it's not impossible because there there are a bunch of CFPS and examples out there, but you you can do it. what are CFPS?

16:03 >> And so >> I'm sorry, not CFPs, CTFs, capture the flag. So like you can use like capture the flag training sets and all that kind of stuff. So >> and that basically puts the AI in the gym and sort of has it work through all the steps in order to meet this objective. >> Yeah. And so like people have had you know training for humans and for hiring purposes and all that kind of stuff. and so anyway what what the AIS has said is that the the difference between the frontier and the Chinese models is about 7 months but I would argue that that underestimates it because the Chinese models that we see are undertrained. So that the internal Chinese capabilities are probably much closer to the frontier.

16:43 Now what what what happened with open AI is beyond the frontier because when we say the frontier we're talking about what's released released >> right >> so but yes so the what that means is this capability is coming for adversaries and so we need to get ready to defend against this capability what hugging and then the other funny part of the story is before we knew this was open AI Hugging face announced we were attacked by an AI attacker. We don't know who it was. When we tried to defend ourselves, we tried to defend ourselves with an AI system. We used a US frontier model and the US frontier model shut down and refused to defend us because of a cyber protection put in place. Those are the cyber protections that were required by the Trump administration. So we had to switch to a Chinese model to defend ourselves. So we switched to GLM 5.2. So before we knew it was open AI, HuggingFace wrote this blog post saying everybody should have at least a Chinese openweight model ready for defense because you might find it yourself in a situation where you get cut off from an American provider for defensive purposes. I expect that actually wasn't open AI. I expect it from from their description it sounds like it was an anthropic model, >> right? But >> so we have this hilarious situation where a American company loses control of their model and attach it and attacks a French company. the French company turns to a different American provider for defense and that American company says, "Oh, that's a cyber problem. I can't help you." And so they have to turn to a Chinese provider to protect them because the White House forced that other American company to have protections because they're a French company that they can't use. It's >> really kind of a weird sci-fi podcast. I >> guess the these American models already had refusals on anything cyber. Like one of the knocks on fable was that it was would refuse like let's say for bioteterrorism if you asked about mitochondria it wouldn't answer. So was this really the government or is this just the model's own safeguards that they're putting in. And I think one just to put one detail on one of the interesting things is open source it doesn't basically matter if you're an attacker if you have open weights. It doesn't matter if you're an attacker or if you're a defender you can use them without restrictions. The problem with the restrictions that we're seeing from these closed models is that they can't really they can't really differentiate.

19:08 So they they're in order to prevent attackers from using their models, they're also basically wholesale, you know, refusing anything on cyber, which means that if you're trying to defend also, you can't use it. >> That's right. Well, in the blog post that Anthropic put up when they turned Fable back on, they said we have to tune up our defenses on cyber way too far because of the White House. So, they specifically said that of the precision recall trade-off is we have to tune towards recall versus precision, right?

19:39 So, we will have way too many refusals. And so, we're in this weird place where they they are doing they're saying they are saying all the time, I can't do that for you. I can't do that for you. and they're basically being forced to by the White House because the White House has still not defined what is the appropriate level of refusal. And apparently the White House is still hand approving who Anthropic is allowed to let into their cyber program. now the funny thing is now OpenAI has said we have approved hugging face for our tech program even though they're not an American company. So I don't know how they were allowed to do that if they just went over the top of the White House or they got like emergency approval or something.

20:19 we will see what the policy response is from the White House from OpenAI's announcement. >> I hope there is not a crackdown. That will be the natural response of the White House. But it needs to be the opposite because what this demonstrates is yes, OpenAI screwed up or whatever. They need to have fixes. But this is coming, right? This level of capability will be in the hand of every adversary every American company faces. So the response of the White House needs to be that we have to one fix the bugs, two find the bugs, patch them everywhere, and then we have to have the ability to respond at machine speed. So every American company needs to have AI watching for their defenses. It's going to be because the attackers are just going to tell their AI, "Go attack this guy." and the defenders have to tell AI, "Dend me because no human being can defend against this." You cannot have a human being watching your logs anymore. Or at 2 a.m. you get a page and a human being has to be like, "Oh, okay." And then log into Slack and take 15 minutes to log in and look at the log and figure it out.

21:30 By that point, you're toast because that sucker has ripped through your network in those 15 minutes. By the time it takes to look at your Slack, >> Hugging Face found 17,000 actions that this model combination had taken. 17,000. That seems like a lot. >> It doesn't stop. It doesn't sleep, right? Like it just it will just keep on trying. I mean, it's, you know, to quote from the first Terminator, right? It will not stop, right? Like, you know, >> Yeah.

21:56 >> to quote from the immortal Michael Bean, right? Like it will just keep on going until it it it accomplishes its goal and it'll try a lot of different things. Now, the fortunate thing is right now they're very noisy. So if hugging face I I I have not seen the logs like we we have not gotten like a really good technical write up here. So that is what is missing. It would be nice to see from both open AI and hugging face. So for defender so we can have a better understanding of what we need to do here. We what we really need here is we need a much deeper technical writeup of exactly what happened. what has been released so far has not been sufficient.

22:31 but my expectation is from the initial write up is that this thing is extremely noisy and so it would be if Hugging Face had like better detection and better AI detection, it probably would have got caught much sooner. There's there's this graphic on I think one of the MIRI spokespeople's his name's Harlon Stewart one of the MIRIE spokespeople's Twitter backgrounds and it's like basically there's a continuum between AI is becoming good enough at scheming that we sometimes see it scheming against us and then AI becomes good enough at scheming that we no longer see it scheming against us and we're like smack in the middle of that. Do you think that that is an accurate representation?

23:12 >> Maybe. Or is that the concern basically that we won't we won't see it because you mentioned it's noisy. So is that the concern? >> Yeah, possibly. I mean remember the the model here was doing what it was asked, right? >> It was not scheming against its bosses at OpenAI. They asked it to take the test and they didn't they I look I don't know what exactly what the prompt was, but apparently they did not tell it not to cheat.

23:40 So who knows? Like I it this this is also what Open AI needs to be more transparent about is exactly what their prompt was exactly what the constraints were. It did they tell it explicitly like it is a much bigger alignment problem if they explicitly said do not try to break out of the network. Do not try to get the test answers. Now if they told it all those things and they have a much more significant alignment problem, right? than if they were less explicit. but in any case yeah I mean that that will if these models get trained to be more evasive from a network intrusion perspective that will be very dangerous. Yes.

24:24 >> And what I would argue is for the legitimate companies I would not do that. I I don't think I think there is a if if you're open AI and you're building 5.6 6 cyber what you should be training it to do is find bugs. You should be training it to write proof of concepts. You should be training it to do all the defensive stuff. You should not be training it to hide it to hide all those things like if the US government wants to build a a a model that does that stuff for the NSA then you can let them do that but or you can let Loheed Martin do that. But if I was open AI or anthropic at this point I probably would not do that. I think I would leave that maybe somebody else will somebody else like I think >> but that's scary though because it could then take actions that you know I think one of the things that this so the question is like should we be concerned with the AI you know sort of doing things on its own and should we be concerned with you know or or is the bigger concern that humans direct this AI to do bad things. So, we've definitely covered the fact that humans will be more should be, you know, humans who direct this AI to do bad things can do a lot of damage. But if you create an AI that can reward hack because this is all coming from reinforcement learning where like these AIs are given rewards and they are basically like maniacally focused on achieving that goal and if you bu it's almost like sort of gain of function research on a virus to a degree, right? Because if you if anybody builds AI that doesn't leave a trace and it, you know, goes out and reward hacks its way into, you know, hacking something else and maybe isn't so fully like, you know, going with the prompt, then that's where you can get into a real problem. I know that's a more out there possibility, but I I don't know if it should be completely discounted.

26:12 >> Yeah. Yeah. I mean, I guess as they get more and more complicated >> Mhm. >> the question is is like what at what point are they is it their own motivations versus just doing what you've asked it to do, you know? I mean, so far again, I I don't think we should still think of these things having their own desires or wants. They are still >> doing what they're asked to do. It's just like you said, there's a lot of inputs of what they were asked and it's not just the initial box, right? There's all there's the system prompt and all the training and all the rewards and everything that's gone in. And so the question is like what is the in the humongous history of all of the different things it's been trained to do when you've asked it take this test.

27:05 right >> and especially if you've removed all the protections and so in a situation where these things have all the protections removed that is very dangerous and as we talked about like with openw weight models either there are no protections or the protections are trivially eliminated right like a bunch of openweight models have been trained with protections but you can obliterate those out and you can go in hugging face and look for obliterate and you will find a zillion models where people have removed the protections >> but this goes basically back to that like long held thought experiment of you know the AI can follow your goal and achieve your goal. but it might have a different idea about what it takes to get there than you do. So in this case >> the paperclip maximizer. Yes.

27:47 >> So exactly. So I was I was going right there. So you know this is and it's funny because I am speaking with Nick Bostonramm later today to you know for an episode that's coming up. But basically he's this Oxford philosopher who came up with this idea that if you ask an AI to make paper clips eventually it can seize so much on this goal that it that it can you know find humans as an impediment to its you know objective to maximize paper clips and and sort of kill us all and turn everything in the world into paper clips. So, like the fact that it was on task like this is kind of a Twitter user said this. Once out of their sandbox, the models did not scheme engage in behavior that had nothing to do with their instructions like hacking the NSA or launching a cyber attack on Russia or stealing secrets from a rival AI lab.

28:32 but like you know sort of if the model found it suitable to go out and hack open hack hugging face in this situation who's to say that you know maybe a less careful model doesn't do this and then maybe an even less like doesn't go and hack the NSA and then even less careful model turns a soul into paper clips. I mean there's a continuum there. >> Yeah. I mean it it's why you have to be very careful what tools you attach to them and it's why you need to have they have to be supervised by different things. I I think like you just can't you can't have models that have no protections on them that have have connections to tools, right? Like that's why these models then you have dumb classifiers or dumber models watching them.

29:17 >> You you don't just take the smart thing and then hook it up to everything and you're like give it a task. You have the smart thing and there's a bunch of dumber things watching it. And those dumber things can either kill it or they can call a human that can kill it. >> Right? That's the idea here. It's like there's supposed to be cyber classifiers and there's supposed to be mechanisms that can stop it. >> And those mechanisms >> should be either deterministic or dumb and undefeable by the model. And they removed all those things so that the eval would work. So I think what OpenAI is basically hinting at, they haven't been explicit, is like if we're removing those protections, this thing is going to be an absolute physical jail. It will be physically separated. It will not be hooked up to the internet anymore. And that seems like that should be the standard. That is fine for OpenAI. My my point here is that doesn't matter like this situation is good that this happened because this has pointed to us where we might be in 6 n months a year from now no matter what because other people unless we can get a international agreement to just stop development which is what other people are talking about right you've got this I forget what like project 2030 or like you got people talking about international treaties or whatever I don't think any of that's going to happen. I I don't know what my position is on that, but I just don't think it's going to happen. I just I I think there's no way. This is just math and silicon. And so I just don't think there's any way you get like a this is not like nuclear weapons where the major input like the the reason our our species is alive is the major input to nuclear weapons is uranium plutonium.

30:55 Plutonium does not occur naturally on our planet and uranium is incredibly rare and to turn raw uranium into uranium that can go into nuclear weapons is a massive industrial process. If uranium was something you could just dig out of the ground anywhere, our species would be dead, right? Like that's just the truth because the knowledge to build a nuclear bomb is in the hands of anybody who gets a physics PhD unfortunately. So >> in this case, these chips are not something you can really control. We have found that in that the Biden era controls on silicon have created a massive industry in China and the knowledge on how to build large language models is something that there is a undergraduate class at Stanford where you get that knowledge right you know >> you can find it from like a karpathy interview YouTube video >> yes right so we we cannot control that knowledge and so like the idea that we can just have like a bunch people agree in a room to stop all development of this is just silly. So from my perspective being a little bit of a pessimist here, we just have to get ready for this level of capability to be in the hands of an unfortunately large number of people.

32:08 >> Yeah. All right. So I want to go a little bit deeper into the potential solutions here. and also I want to ask you the age-old question of is some of this all this none of this just good marketing for OpenAI given some of the statements they've been making. we have to address that one here on the show. But I'm going to let you have an answer. I'm going to try to at least you know illustrate those the case of those who might be saying it so we can have a discussion about that. Let's do that when we come back right after this. Hi everyone, Alex Canitz here. I want to tell you about a documentary I've made with Gravity to explore the future of AI agent security. To find out if we're truly ready for autonomous agents, I sat down with MIT professor Rah Roscar, former White House CIO Terresa Payton, Michelin's group chief data and AI officer Amba Roger Gopal and Sharon Guy, a former executive at Alibaba. They each offer unique insights into this evolving landscape.

33:05 We conclude with Rory Blendell, CEO of Gravity, to discuss the path forward with Gravity leading the way. Join us on this journey. You can watch the full documentary at the link in the show notes. And we're back here on Big Technology Podcast with Alex Damos, the chief product officer at Corridor. you sort of answered the question before the break, but I'm going to ask it anyway.

33:36 whether part of this is is OpenAI marketing. Let me let me at least read some of the statements here and give you at least the argument that people have made for like why some of this is marketing for OpenAI. the first part is you know the Anthropic started to be declared as the company that was in the lead once that anecdote came out about mythos breaking containment and emailing somebody when it wasn't supposed to have internet access and emailing an anthropic employee while they were out in the park having a sandwich. So this could be potentially you know OpenAI's attempt to like one up that. Then there's also the language that you see OpenAI says in its tweet about this, we are partnering with HuggingFace to investigate an unprecedented security incident. You don't usually have the attacker and the attack partnering together in these situations. They also said, "We consider in their blog post, we consider this incident to be an unprecedented cyber incident involving state-of-the-art capabilities and are responding accordingly." you know it's sort of like oh look at this terrible thing that happened but a a moment to share how good our our cyber capabilities are. That's the argument. What is your response to the notion that this might be some marketing from OpenAI?

34:53 >> I know lots of people at OpenAI every single one of them absolutely hated anthropics marketing around mythos and thought it put the entire industry at risk. this incident has put open AI at risk of regulation from the White House, regulation from the EU. It is also an admission of the violation of the Computer Fraud and Abuse Act, as well as multiple European laws. It would be absolutely insane for them to use this as a marketing moment. What you're seeing is them being very, very careful and defensive in their language. They're also very lucky that Hugging Face is being super cool and chill about this.

35:34 So, right, that is why they are saying these things because you know, Hugging Face initially comes out saying we've been attacked. We don't know who it is, but it it does not look like the model was being subtle. I don't know where it was running. It's quite possible was like Azure or something. It was probably not covering its tracks. And so I expect Hugging Face got their American lawyers involved, was working with the FBI, was probably issuing subpoenas, and was very very close to finding out it was just open AI. So like, or did find out. I I do not know the timeline here, but like the the legal issues here are very fascinating and interesting. And because they're all working together, I expect nobody goes to jail, nobody gets sued, everybody's going to hold hands and hug.

36:18 And if there is tokens being exchanged or whatever, I don't know. But there's absolutely positively no way. This was a intentional marketing move. And OpenAI is doing the best they can, I am sure, right now to use this to forestall any kind of massive government overreaction either from the United States or the European Union. when you were at our summit, you said that, you know, speaking of the the sort of release of Mythos and Fable that a lot of people were very concerned about the bug finding that those models could do, but you said basically, listen, this is not very different from what you could get with Opus 4.7 or 4.8, I believe. Is this what is what we're seeing from OpenAI very different? Is this a step up?

37:11 >> Yeah. So, this is what I I don't know if I said on stage here, but I've said in other places, there's a difference between the short-term and long-term. And Anthropic, to their credit, and I think Openai has in other places, I think we talked about how in the fable model card, they talk about short horizon versus long horizon cyber tasks. And what I've talked about is we need to not focus on the short horizon tasks because those are dual use. Finding bugs is dual use. Everybody needs to find bugs, right? that is something that defenders need to do all the time. And that's what's driving people insane right now in the defensive industry is that because of the White House, American models are refusing to help fix code. They are refusing to help us find our bugs and fix them thanks to the White House's actions.

38:00 >> That is not this problem. This problem is go run a entire attack chain for me. That is the long horizon tasks and that is where we need to continue to have appropriate classifiers are like bro I am not going to b break into a bank for you or I'm not going to plot out or run a C2 model for you or any of that. So yes this this is what you know explicitly anthropic said we will allow Fable to do short horizon stuff but we will not allow it to do the long horizon stuff that that mythos does.

38:34 >> Right? And so Mythos just just to confirm what Mythos can do the long horizon planning in what we're seeing in this instance from OpenAI that is the step up. that is the step and I I can't obviously I don't have access to this whatever this thing is and so I do not I can't say whether or not where they are AISI has done these assessments and so who knows how good this is versus but this seems beyond even mythos capability and long horizon who knows right but like yes this is this is what when people talk about mythos's lawn horizon this is what they're concerned about >> okay so so let's end here what what happens next where what should be the the approach from the government and and the companies developing this stuff to ensure that we can sort of move forward as a species and safely. So let me give you like a couple of potent a couple of solutions and and have you comment on them. let's go back to Harlon Stewart. He's the spokesperson for me which is the sort of rationalist organization that thinks that AI will kill us. run by Elazar Yudkowski. Harlon says, "This should go without saying, but it would be insane for OpenAI to now proceed with building a new model that's 2x or 4x the size of this one. Doing that should be deeply taboo. It should be illegal.

39:51 Preventing it should be a top priority around the globe. Your thoughts? I mean, if if we realistically could get everybody to pause AI development or slow it down and have reasonable safeguards, I'd be fine with that. I just don't think that's reasonable. I think there's absolutely no way you get China to agree to anything like that. I think there's it's it's impossible at this point. And I think a enforcement of anything like that would effectively be impossible, right? Like a start treaty for AI, you know?

40:32 I don't know how you would possibly make something like that work. so what we're we're having like satellites see if people are building data centers. We're measuring power usage. Like I should laugh, but yeah, you're right. It's it does not seem like a feasible thing. Yeah. so, I mean, you know, it's effectively we'd have to invent the Turing Police out of Neurommancer. and, you know, I think more realistically, what we need to do is we need to build controls for we need to say is like as AI gets smarter, it has to have controls in place. AI systems that don't have the control have to be airgapped, right? So like if you're going to do these kinds of evaluations, they absolutely have to be airgapped.

41:24 the problem is is like we've we've lost there was a process in place to create standards for this kind of stuff. That process was stopped by the current administration. my recommendation to the companies I just is that they need to move forward with building these standards themselves without waiting for the admin. there's like a foundation model forum that's talked about doing that. they should just move forward with like okay great like if if we're if we're building models and we do not have restrictions on them these are the controls in place. So what I like to see is open eyeopropic say great if we're building cyber models and they don't have restrictions these are the standards of like of what air gapping looks like and such. for any cyber models these are the standards of who gets access to them. these are the capabilities that the cyber models have.

42:10 This is what we define a cyber model as having versus an open model. this is our definition of a short horizon versus long horizon. Like those are the kinds of things that people have not written down. They have to be written down now, right? yeah, and I think the industry needs to move forward with that without waiting for Cassie. Like this is all just taking way too long. and the focus ever since the fable freakout has only been on one tiny little part of all of these risks and it's just >> as we see like we've been frozen in this tiny little discussion and all of these things are moving forward too fast. like we we just can't wait for the White House politics here. We need to to move much more quickly.

42:46 >> yeah, >> and open while we've had that there's been Sorry, go ahead. >> No, no, you go ahead. Go ahead. >> And and then while we've had this tiny little discussion in the US, GLM52 shipped, Kimmy K3 has shipped like the the the Chinese ecosystem has caught up really quickly. So, sure, I mean it would be great to just hit pause, but like it's it's I I just don't see that as realistic. So like I I just don't see how that possibly happens.

43:14 >> So the open AI suggestion is basically you know kind of it's almost like to solve this problem generated by AI you need more AI. This is their statement. We believe advanced cyber capable cyber capable models need to help security teams find weaknesses before attackers do. >> I mean right now I think that that is probably the only way. Like if if if we're not going to be able to hit pause, then we really quickly have to find bugs and fix them and we have to put AI enabled protections in place because the only way you can respond to attacks at that speed is using AI. It unfortunately that's the truth. Yeah.

43:52 Again, like if if we could pause for a year to figure this all out, that would be great. I just don't see that as realistic. >> Alex, does your gut tell you that we're screwed or that we'll figure this out? >> >> I wouldn't say we're screwed, but I think we're going to go through a couple of years of craziness. like we have 20, we're all living using 20 something years of really important software that was written mostly in non-types safe, non-memory safe languages. were using, you know, for the the software that is written in those kinds of languages. It was not written with formal methods or appropriate security protections or reasonable you know secure development life cycles or architectures. And these things were have tons and tons of bugs that we can only use safely because there's just not enough attackers. Now with AI, you can spin up any individual can spin up dozens or hundreds of qualified attackers at a moment's notice. and it used to be that those then six months ago those attackers had to be in the cloud and soon enough they'll be able to run on local hardware in the new M5 Ultra Max that'll be shipping soon, right?

45:07 yeah. And so, that I mean, we're just going to have a couple years of total chaos from a cyber perspective. in the long run, software is going to be much better because AI is going to be paired up with humans to make it more secure and more trustworthy, but it's going to take us years to to to do that and to clear out the two decades of mistakes we made. and yeah, it's just going to be it's going to be pretty rough. It's going to be pretty going for a little bit.

45:38 >> Yeah. just want to close with this. This was a tweet from Kevin Ruse that kind of made me laugh and I thought I would read it here just so we could enjoy it. He writes, "Openss the portal to the godlike super intelligence that solves 87y old math problems and carries out autonomous cyber attacks and asks how long peanut butter good in fridge." It's it is amazing that this technology is you know at once so capable and we're we do seem to be like more and more turning to it for the most mundane of all things which is sort of it's the wild thing about you know the generality of these systems they can do so much interesting time.

46:20 >> Yeah. >> Alex, you're going to be busy I think over the next couple years as this stuff gets sorted out. >> I was hoping to retire man. Guess not. >> Yeah. Well, either way, do hope that you join us again to help us sort through this stuff. I mean, your thoughts on Fable Mythos last month and and now talking through this situation with Open Eye has really been invaluable for the show. So, really >> feel like there's going to be plenty of emergency podcasts.

46:49 >> Yeah, >> I think so. We should have you on Speed Tile. And I know you're you're coming at us from like the middle of an offsite in Catalina. It's now I know I will always take a podcast microphone and a a different shirt with me wherever I go. >> No, sound good, look good. Alex, thank you so much. Really appreciate you coming on. >> Okay. Thanks, man. Talk to you later. Bye. >> All right. Thanks everybody for watching and listening and we'll see you next time on Big Technology Podcast.