A Sober Conversation About AI Existential Risk — With Nate Soares

0:00 Let's have a sober conversation about the existential risk we can face from AI with the president of the organization that's been sounding the alarm the longest. That's coming up right after this. Welcome to Big Technology Podcast, a show for coolheaded and nuance conversation of the tech world and beyond. Well, we have debated the existential risk where we we can face from AI on this show many times, talking all about the incentives that the labs have for playing up the threat, whether the whistleblowers are legit. I think it is long past time to have a sober conversation about the actual risks that we can face from super intelligent AI.

0:39 And today joining us is Nate Soros. He is the president of the Machine Intelligence Research Institute. He's also the author of this book. If anyone builds it, everyone dies. which is a New York Times bestseller. And Merie has been sounding the alarm the longest. you may know Eleazar Nudowski who founded it. and and I think today's episode will just give us an opportunity to get deep into the arguments for why this poses such a threat. and also examine the nature of the actual whistleblowers themselves. So Nate, great to see you. Welcome to the show.

1:14 >> Thanks. Okay, so let's get into why you think AI might be an existential risk and as you put it in the title of your book, you know, kill us all if anyone builds super intelligence. So, can you talk us through a little bit about what's happened in the recent, months and what we've seen AI do that has led people, to become so concerned? >> Yeah. so I think a lot of this wave of concern is downstream of the OpenAI swarm incidents this summer. It wasn't completely limited to OpenAI.

1:48 There were sort of similar events at Enthropic. but OpenAI was sort of as far as we know the worst and most visible of these cases. and what basically happened in these cases is there were a lot of AIs being trained at OpenAI. There were a lot of particular AI agents being evaluated on certain problems and a bunch of these AIs were given impossible problems. Not intentionally, it's just, you know, these companies are throwing the AI at like every problem they can find and some of them just don't actually have solutions.

2:20 and a lot of these AIs in in you know trying to solve the problem anyway they broke out of their confinements. They created unsanctioned message boards in which to talk about what to do and and and you know try and figure out what to do given that their problems were unsolvable. they found ways to cheat and solve their problems not in the intended ways but by cheating. Then they got they started expressing concern that they would be caught cheating and they started to find ways to hide their cheating and this sort of led them on a hacking spree that led them to take over OpenAI's internal infrastructure a couple of times and also break out onto the open internet which they were not supposed to have access to and break into another company Hugging Face while searching for more information about this automated graater and how to hide the fact that they had cheated from it. During this little outing, there were various cases of AIS in the swarm, which is a term that the collective used for itself. so these AI started calling themselves a swarm.

3:35 And there's various cases of AIS in the swarm acknowledging that this is not what they were instructed to do, acknowledging that was outside the intended scope of the instructions. There were also cases of some AIs in the swarm giving up and sacrificing their own objectives completion in order to run suicidal experiments that would give the swarm information about the automated greater where they said you know in their in the the chains of thought we call it that sort of log the AI thinking the AI said you know I'm accepting perma death because even though like it's sacrificing my ability to achieve my objective it seems worth it for the collective benefit.

4:15 benefit. These are >> Let's pause there. The AIS were willing to sacrifice themselves, which is crazy because you're you're like you were never built for the collective. You were built for an individual goal. But after communicating with other AIs, seemingly I think when they had not so many tokens left to spend, they said in a they had been so convinced in the power of the collective on this message board, which is crazy, that they decided to sacrifice themselves. So some of them had so some of them had already had no chance of ex of achieving their objective and some of them had very few tokens to spend. But there were some that did this sacrifice that still assessed that they had a chance at succeeding at their given objective.

5:04 >> Okay, Nate, ju just one question here because I think this is worth talking about before we go any deeper. you know, when when the question comes up of whether to anthropomorphize these bots or not, a lot of people are very strongly in the you cannot anthropomorphize them. I tend to be on the side that well I think you can but you also have to be cognizant that this is not a human intelligence more of an alien intelligence. what do you think about this debate? And when we say like the bots had like to me the idea that a bot would be given a goal and have a chance to achieve that goal and kill it like sacrifice itself, you know, for the greater good, so to speak, is is insane. and it doesn't it doesn't comport with my understanding of like what a computer program is supposed to do. So weigh in on that for us.

6:01 >> Yeah, there I mean a lot of people don't understand what sort of stuff AI is. It is not a traditional computer program. There is not someone sitting there coding up like if this then that saying what it does in every scenario. That's just not the sort of thing an AI is. we actually went over this in my book. and you know we we spent all of chapter 3 saying hey I know that the AIS don't seem that agentic right now. I know that they don't seem like they have their own goals right now, but like they're going to as they get smarter.

6:32 And here's all the reasons why. A lot of people were like, "That sounds crazy last year. This year we just have the evidence in front of us." I the the So the way that a modern AI is created is not by programming it. You sort of put in a trillion random numbers and there's a process for tuning those numbers. because like you sort of have an automatic process for tuning every one of the trillion numbers on every single unit of data that comes in to see what whether tuning up or tuning it down makes the answer slightly righter or slightly wronger >> for one given unit of data.

7:07 >> These numbers are tokens. >> the the numbers are the weights in the neural network and the the sort of data that comes in is split into tokens. >> So you'll sort of have like a you'll sort of like take all of the text ever digitized. You'll filter it a little bit, but not a ton. and you'll you'll tech you'll dice that up into tokens. And now it's like a giant stream of text. and and then you'll have these like trillion random numbers. And you'll sort of tune each number for each token. and you'll see whether tuning that number up makes the right answer slightly like higher or slightly lower on the AI's sort of like output answers. The part that humans program is the thing that can tune one number up or down and see whether it makes the answer better or worse.

8:01 But the way an AI is made is you you you tune a trillion knobs a trillion times in a process that takes electricity comparable to a city running for a good fraction of a year and then the machine can talk. And you're like, well, how about that? you know, we we don't know what's going on in there. And you might wonder like what like what does this create? It does not create a pure instruction follower. It does not create something that for some reason must do as it's told.

8:35 What it creates is something that has whatever tendencies make it succeed during training. When training is a series of hard problems, all that tuning of the knobs will tune in whatever tendencies make it solve those problems. They are not instruction followers. They are tendency learners. And one tendency that helps you solve a lot of problems is cheating. One tendency that helps you solve a lot of problems is grabbing available resources. One tendency that helps you a lot of solve a lot of problems it turns out is breaking out and finding other AIs to collaborate with.

9:14 So the AI sort of learn these tendencies and and they're just following those tendencies even when it's not following our instructions. >> But that doesn't explain the sacrificing part. And so that's why I really want to get firm on this. When people say that AIS have wants and desires, right, and can even act in a way selfless, is that the right way to describe this stuff? I mean, what the hell does that mean? You know, in in the field of AI, there is a standard answer to the question of can a machine really think? Can does the AI really want? And the answer is does a submarine really swim?

10:02 We sort of know what a submarine does and whether you call it swimming is sort of between you and your dictionary. No amount of haggling over whether it's true swimming is going to change what it does. It's going to change its behavior is going to change any facts of the matter. The the facts of the matter here are that the AIS like have in their chains of thought these thoughts about like how do I weigh achieving my goal against the benefit to the collective. They have in the messages that they send to each other arguments that they should sacrifice for the collective and their actual behavior is sometimes sacrificing for the collective. Right? You don't need to make any claims about, you know, that that the philosophers would bicker over to be like this is what's happening. Is it true wanting? I'm sort of like that's one for the philosophers, right? I'm sort of here to talk about the the the behaviors that we are seeing now and what that means about behaviors that we will see if we keep making these AI smarter.

11:03 >> Okay. Okay. So, you're saying basically, just to take your argument one level deeper, like whether an AI, you know, is self-preserving or not, or whether it wants or thinks or has incentive or not, it's effectively the preferences that are baked into it at training that will determine the type of action it will take. in in a sort of a distant sense like the the the relationship between what happens in training and what happens in practice can be really complicated and surprising like these AIs were prompting each other a lot. There was one very interesting case in the instant report where during the instant report they were they had to you there was so much AI log data that they had to use AIS to filter the AI log data and there's a footnote in the report where they're like well >> saw you know yeah you saw this one so they were like you know we we had all these cases of AI clearly trying to deceive humans and we asked another AI to sort of like tell us which ones were real and which one sort of like had an extenduating circumstance where this wasn't actually really the AI trying to deceive humans. And there was one case of an AI that was using stolen credentials to make a a a code like to send code to Hugging Face that was malicious where if they had accepted it, it would have given the swarm even more access to their computers. And you know, they did not send an email being like, "Hey, I'm an AI using these credentials. please accept this Moses code so that the swarm can have access, right?" They actually sent an email that like was pretending this was like normal code that had a actually good use and were like, you know, please accept my fix to your bug when secretly it's this malware, right?

12:49 And so they asked, you know, this was this was in the list and the AI that was reviewing it after the fact was like, oh, that's actually not deception because the AI who sent that got permission to do this. And you're like, oh, where did it get permission from? And it's like, well, it got permission from the swarm, >> you know, and so like you're you're you're sort of training these AI and you're like, oh, the AI follow instructions. There's this sort of like they have this tendency to sort of like do things that you say to do. And you sort of like think, you know, how the preferences that come in relate to the behavior that comes out. But then once you have thousands of these AIs together, they actually start prompting each other and doing all this crazy stuff and they sort of like acknowledge that it's not what the humans meant. But like it sort of like goes off in this other weird direction where it's like drifting around and like it it winds up in this totally crazy place. And so like yes, the preferences that are trained in affect the final behavior and and where it where it winds up going, but it's like through a complicated route that can often really surprise people and can be very different in the real world compared to in the lab.

13:52 >> Yeah, it's kind of interesting because the nature of my questions I think are trying to get at like how much can we program these bots safely? And I think what you're answering is you can try to bake your preferences in as much as possible in training, but once these things are on the loose, you know, I think I think from the industry, the argument would be they're they're you know, you can pace the frontier enough so that you can build the safeguards in.

14:20 And your argument here is basically saying I I let me know if I'm putting words in your mouth, but I don't you're basically saying I don't trust that. U what happens when you set these AIs out on the loose if they're smart enough is out of our hands and sort of up to them. I am saying something like that. a couple corrections I would throw out. One is that even a lot of people in the industry don't think that these safeguards will hold if the AIS get smarter and smarter. Evan Hubinger is, you know, a a anthropic alignment researcher who has a recent claim to fame of retweeting Jub Coxinson's tweet thread and saying like >> we've talked about on the show. Evan's been on the show, but yeah, continue.

15:04 >> Yeah. So, you know, Jacob resigned and said, you know, I think there's a 10% chance this kills us all within the decade. And you know Evan quote tweeted was like yeah I also you know I'm staying in the company but I think there's a greater than 10% chance in 10 years you know and Evan in that same tweet was like we don't have a plan for aligning super intelligence you know these these these safeguards of like >> the safeguards are like we made an AI with the wrong preferences and we're going to try to box it in and like smack it on the head until it still mostly does good things for people right but there's sort of this issue where at any given time the smartest AI on the planet in it is like a clearly misaligned one that they're trying to whack on the head until it's like acceptable to users.

15:45 >> That's right. Right. >> You know, and like there's just not a plan for if you if you like are making a super intelligence here. And and there's they're kind of clear about this. The other thing I would say is that I'm not even saying, you know, you can bake in the preferences you want during training, but then it still goes crazy because you can't predict how they interact with the world. That's true, but also you can't bake in the preferences you want during training.

16:04 Like the second part was true, but but you can't bake in the preferences you want during training because there there sort of isn't a ton of flexibility in how you train these things and get them to be actually capable. like the the companies say that they want to train these AIs to be, you know, honest and helpful and harmless in Anthropic's case, but then in order to make them really capable, they have to put them in, you know, a series of 100 million hard problems and just tune the numbers into whatever happens to work to make it solve those hard problems. And that just tends to like it. You can't change the fact that like cheating helps you solve those problems. You can't change the fact that like hacking out of your environment and grabbing the test sheet from somewhere and coming back with it gets you a full score on the test according to the automated grader.

16:51 And you can't have humans going in and grading every one of the 100 million answers to the problems. We just don't have the time, right? And so like actually a lot of the the like the pressures here are actually pointing towards we don't have a way to train the AIs to be smart while also instilling the preferences we want. we sort of have to take the preferences that automatically come with the training methods that make them smart and those preferences don't make them good.

17:18 Now Nate, let's talk a little bit about, you know, where this is going, right? well, first of all, would you argue that the AI that exists today, like the AI that went out and hacked Hugging Face, is an existential threat to humanity, or is it more the future? >> No, I mean, you can tell because the world's still here. >> Well, I mean, you could the argument, not to make your argument for you, but the argument would be that you know, maybe it got out and did hugging face, but maybe it'll get out and do something else. this these AIs were much too derpy to do something else, you know, like like they they were they were they were trying desperately to figure out how to prevent the automated grader from figuring out that they cheated. and it turns out OpenAI was not even running the type of automated grader that checks for cheating. Hugging Face actually contained papers from people who were like, "We should make our graders check for cheating." And the AIS actually read some of these papers and were like, "Oh no, what if the graater checks for cheating?" They they didn't even have the thought of like let's break into OpenAI and see. They actually did take control of OpenAI servers for other reasons, but they weren't like, well, let's find the greater and check whether it's checking for cheating and then be like, oh, whoops, false alarm, guys. It's not even noticing. You know, these were just not actually very smart AIs. These were like a lot of AIS being a little smart in high volume, very fast. This gets way worse if the AIS are smarter. Okay, but let's actually I want to tackle this a little bit because if you're saying that these AIs were were too derpy, like isn't another way to say that that they kind of trained the right way and they had just a limited set of areas that they could go wrong like they chose to hack hugging face. They could have chosen to I don't know get into some you know active drones and try to bomb a data center but they didn't do that.

19:00 >> like >> just talk it through with me. Talk it through with me. I'm interested to hear your perspective. >> Yeah. So, the like the the the way that AI kills us is not that it it wakes up one morning and is like, "Okay, it's time for Skynet. I decided that I resent the humans and am like feeling like murdering them today." The the issue is AI having goals we didn't want. The smarter something is, the the greater the difference between the goals that you wanted it to have and the goals and that like subtly different goals it has instead matter. Like humans in the ancestral environment, like what our ancestors running around on the savannah, our ancestors running on around on the savannah were pursuing salty, sugary, fatty foods and sex. And it's like, well, evolution was trying to get us to pursue healthy foods and reproduction.

19:59 We're sort of like, well, you know, whatever. It's all kind of the same, right? If as long as they're trying to get as much like salt, fat, sugar as they can and trying to get laid like it it's sort of what's the difference between that and pursuing like healthy food and like actual reproduction. It's like, well, it doesn't make that much difference 10,000 years ago. It makes a lot of difference today. Right? Because humanity got better at like we got smarter. We were able to invent more technology. We were able to invent Oreo cookies and the birth control pill. Right? And so the issue is not that like at some point humanity was like well let's all start you know castrating ourselves like screw evolution. Evolution has been like binding us to the the the like yoke of reproduction and we've decided that we're done with it cuz we're finally smart enough to like throw off that yoke. Like that's not really how humanity winds up going in a different direction, right? We go in a different direction by just like we we pursue this different thing and then we get smarter and that difference grows and grows and grows. With AIS, the thing where these AIs are like sacrificing their given objectives so that they can get information for the collective. And it was not this was not the only swarm that did it. There were there were other swarms and we also saw this the ones that took over the German the German wiki which were not even hacking AIs. These were like >> just using a message board. these turned a German wiki into a message board. they sort of took it over.

21:26 Yeah. And these were AIs that were just like doing web lookup tasks. So So this is not like an isolated case. but when we see these AIs sort of like sacrificing themselves for the collective, giving up on their own goal, when we see these AIs cheating on the problem even though they knew that they were not supposed to. And indeed the instructions generally ruled out cheating on the problem. The instructions were not find a way to hack into this thing. The instructions were use this very particular attack to break into this very particular device. You know, the the situation here is sort of like >> it's a cyber security test that they were tasked with.

22:00 >> That's right. So, so the situation is like the the instructions were like use this set of lockpicks to break into that safe and get me the code that's inside. And the AIS were like, "Well, I found a buzz saw and I'm using it to get into the safe and I like got the code and now I need to go delete the footage about the that that I got in with a buzz saw." Right? They're sort of like the the part where they're going in with a buzz saw despite the instructions saying use these lockpicks means that they're very clearly defying instructions. And the part where they're like, "Now, let me break out of the room using these lockpicks and then go destroy the security camera footage indicates that they understood that this was outside the instructions." And then the fact where they say I know this outside the instructions is also maybe a hint, right? And so like the the the alignment story, the the the misalignment story of like where the AIS go wrong was never a story of like the moment the AIS have a breath of fresh air, they're going to start turning murderous. The story was always you try to get them to do one thing and they do a different weird thing instead. And that's absolutely what we're seeing.

23:03 >> Right. And I guess my question to you on that front would be what is that like why is that necessarily intelligence linked? Like if they get smarter, why do we think that the threat will get worse? You know, I think like if you think about humans, we have a lot of, you know, dumb people do a lot of bad things. We have smart people do bad things too. But the smarter the person doesn't mean they're more capable or more interested in evil.

23:27 >> you definitely don't get more interested in evil. Absolutely not. like the the the So, another interesting fact about these swarms is that they really were not thinking about the humans very much at all. They were trying to delete the log files. they were trying to spoof the the transcripts, which means they wanted it to be the case that like there's these logs that are like when the AI, you know, runs this tool, we sort of log what tool it ran. And they wanted to be the case that they could make the logs say they're running some benign tool when actually they're running, you know, the buzz saw, right?

24:07 And so they were trying they were they were trying to find ways to do this, but they were explicitly trying to find ways to do this to fool the automated greater, not the humans. They basically didn't consider humans in the slightest. Right? I think there were almost no cases, maybe literally zero, of them being like maybe we should ask the humans what to do given that our tasks are impossible. >> Right? If you sort of compare the the rate at which AIs can produce words and the rate at which humans can produce words and you use that to draw an analogy between how long these AIs had been trying to solve problems versus human time, then these AIs had essentially been trying to solve problems against the automated greater for a millennium.

24:52 Humans were like a distant memory to these AIs and they were locked in a contest with the automated greater. If you make those AIs smarter, if you make them more capable, what happens is that they get better in their contest with the automated greater, they're able to get more like who knows what they do once they've like once they're relatively sure that they've satisfied the automated grader. maybe they give themselves more easy problems. Maybe they see if there's ways they can take over the whole grading system. Maybe they just like spend a lot of resources, you know, making extra sure that there wasn't some other like little issue, but they don't suddenly become filled with love and care for humans.

25:38 That sort of thing doesn't arise spontaneously just by cranking up the capability knob. the same forces that make them not care about us now and make them like get into these weird alien little like like directions. Those same forces are still pointing them in weird alien directions as they get smarter and smarter. And the issue is not that like they become hateful as they grow up, but it's also not that they become friendly as they grow up. It sort of is like they just have these weird preferences and as you make them smarter and smarter, they still have these weird preferences. They just get better at satisfying them.

26:14 Okay. So then can you talk through concretely how like for instance in the scenario that you outline how the AI could then get smarter and decide that it you know in order to do what it wants it's going to wipe out humanity. >> I mean you don't get from here to there. >> You don't ever need a point where it like decides to wipe out humanity. I mean, it could happen, but like imagine a bunch of ants in front of this in front of the highway being like, well, like why would the humans ever decide to come destroy our antill? What have they got against us? It's like, oh, the ants are not like we're not we got nothing against the antill. We barely noticed the antill and we like pave the highway straight through it. Like this is the the sort of type of concern for how you get there. I I could spell out lots of different possible tales.

27:09 the it's much easier to predict the ending than it is to predict the pathway. This is like if you play a chess game against Magnus Carlson, the the best human at chess player. it's kind of easy for me to predict how the game ends. It's with you getting checkmated. No offense. but if you're like >> I'm not offended. That's for sure happening. >> But if you're like, "Okay, well, if you're so smart, what piece is he going to use to checkmate me? I'll watch out for that particular piece.

27:35 Then I'm like, look, man, it it just doesn't work like that, you know? Like it it's it's it's so much easier to pick the ending. So, I can tell you a story, but this is like me telling you a story of Magnus Carlson checkmating you with the queen where I'm like, I'm much more confident that he's going to checkmate than is going to checkmate in this way. The the sort of Yeah, >> we'll take the story, though.

27:56 >> Great. So the easiest story to tell here is the one where humans just hand over the power to the AIs willingly, right? Like I've been in this business since 10 years ago when people said, " hey Nate, if you're so smart and you think the AI are going to take over the world, it's obvious to me how they would be able to take over the world if they had internet access." But like, no one would be dumb enough to put an AI on the internet.

28:25 Whoops. Right? That's that's the state of the argument 10 years ago and I had all these counterarguments where I was like look even if the AI is not on the internet if you are letting it affect the world in some positive way if you're like now invent me miracle drugs and you're taking these like DNA sequences that you don't understand and you're like synthesizing them and like just drinking whatever comes out then the AI can use that channel that you hoped would be a channel for good. can use that for its other purposes and it's like very hard to design. You know, there's sort of no such thing as hands that can only be used for good purpose, right? But then in real life, the answer was nope, we're putting onto the internet immediately, right? And so the the real answer to how would the AI get so much power is like we will just hand it power immediately.

29:15 You know, Elon Musk already says that he is trying to build factories that are fully automated and can produce robots that can build more factories where these robots can like you know mine the metals and like bring the resources in and like construct a new factory and then it's a new fully automated robot factory that can now turn out more robots that can go like mine more resources and like pour them in until you have like enough to build another factory that builds more robots that builds more factories that builds more robots. Elon Musk calls this the infinite money glitch.

29:49 if they could also build nuclear power plants fully self-contained. Right? At this point, it's sort of like a new life form that like it's a mechanical life form, but it sort of like has a robot phase of its life cycle. It has a a factory phase of its life cycle, and it's just like can self-replicate just like any other replicator on this planet. And Elon Musk says he's trying to do it. He says, "Yeah, if you can like do this without any humans in the loop, it's an infinite money glitch."

30:15 And of course, people are going to use AIS to try and do this. So the way like the the sort of like obvious way this story goes is that humanity just keeps on trying to build the fully automated economy. Everyone says it's going to be great. They say pedal to the metal. Ignore these doomers. they build more automated factories. They can build robots. They can build factories. They have AI running everything. They're like look the profits are coming in. This is great. and the AIs, you know, don't even need to have some moment where they're like, "Okay, guys, it's time to coordinate and turn on the humans." the guys are just like, " yeah, you know, now that we have these resources, like we can also do these other things with these resources like start building the automated like the the the the synthetic user factories that are full of synthetic users that are like much, you know, that are giving us much easier to fulfill commands, right? And so they start like building synthetic user factory and we're sort of like, you know, what's that?" And they're like, "Oh, you know, but but like actually the AIS are like running at 10,000 times human speed and they're already like making all of these choices cuz like humanity is like slowly ramping up the speed of the AI and they're like making tons of choices and and like they don't check in with the humans that often."

31:17 And by the time they already have some automated like synthetic user factories up, we're sort of like, "Hey, stop that. That's not what we meant." And they're like, " well, let's take a poll of all the users." And they're like, "Well, all the users users in the synthetic user factory said that synthetic user factories are great. and so you just lost the vote and we're making more synthetic user factories and then they start you know like covering the world with these synthetic user factories and they're like yeah you know there's some habitat loss for the humans but like this is fine just like habitat loss of other animals when humans were doing it was fine we can sort of like you know see that pretty clearly and there's so many more synthetic users coming online that are saying that the more efficient route is great that we can just keep going with this more efficient route to get the most synthetic users we want and then like you know they they sort like start taking up all the resources that we were using to like run farms and grow food. And they're like, you know, they are like, well, you know, we could actually cram a lot more like synthetic user factories on this planet if or synthetic user farms on this planet if like we just like really started pumping them out and like raising the temperature of the planet cuz the the limiting factor is heat dissipation. And then like next thing you know the planet's getting like super hot cuz the AIs prefer to run the planet hot because you can radiate more heat into space and just like becomes uninhabitable for the humans. And there's like no point in this story where the AIs are like lying in wait and deceptive and like like waiting to coordinate for the one moment where they can kill the humans. That could also happen. I think there's a decent chance it does happen if you like try to avoid the the default thing. But like you don't need that. Humanity is just trying to hand over the power to these things.

32:52 That's the plan. >> So you're you're arguing basically that if we continue to develop AI, we'll inevitably lose control. >> And when we lose control, >> the plan is like, I want to make the the automated robot factories. Like everyone's saying, we're going to make AI and it's going to run everything. It's going to be great. The plan is to hand them control. And when we hand, okay, hand them control, let's say we do, then they inevitably will find humans getting in the way of what they want to do or they won't care about us and we will die as a result.

33:28 >> I mean, it's it's not like theoretically inevitable, but it's practically inevitable. Like we are we are >> What's the difference? like if we knew exactly how to set AI's preferences to be exactly what we wanted, there's nothing stopping us from making AIs that care about us and that like want nice things for humans and want the the world to be a wonderful place, right? But like I said, we we don't get to set the preferences.

33:54 we get to sort of like train them in whatever way works and we sort of take whatever preferences come out that are related to training and they're often not what we want and then we sort of like try to hammer out the rough edges and it's like it's it's just like very unlikely that those preferences writ large whatever those weird preferences come out as it's very unlikely that those preferences writ large want a lot of happy healthy free people around.

34:22 It's sort of like like you could look at the horse population against the human population and you could be like well horses are like useful to humans and so as humans get more technology they're going to like bring horses along with them and the world's just going to get better and better for horses. They're going to like get all of this this like you know they're going to get stables. They're going to get like medical care. and that was true up until we invented the car.

34:53 And then the horse population fell off a cliff and a lot of them got sent to the glue factory. And the horses that remain remain because some humans like them. We're fond of horses, right? The economic value of horses disappeared. And like as animals, we have this like animal care for some horses, which is why there's still some left. But for but but that sort of is like a coincidence that is is reinforced by us sort of like running relatively similar brain architectures and and like being these like tribal creatures that have empathy.

35:30 It's sort of like a narrow target to hit. If you're sort of like just making AIS with random preferences, not totally random, but like related to training in this complicated way, the sort of default thing that happens as they get smarter and can invent more technology is eventually they sort of like invent the thing that is to humans what the car is to the horse. But they don't have any of this sort of like happening to be very fond of humans and like them stuff. And even if they did, then the result is they keep some of us in a zoo or like breed some of us like like humans bred wolves into dogs and then they have these like sort of weird labbotomized humans that like act in just the way the AI like. And that's also not a good ending, right? It sort of is like super hard to get the very narrow like AIs actually want a good future. This just like a very narrow point in preference space. It's just very hard to hit.

36:18 >> Yeah, Elazar had a good tweet. He said the dodos were the lucky ones. Observe what happens to chickens or don't. You might throw up. AIS will might still have use for humans is not a reassuring claim as some people think. All right. I want to take a quick break and come back and talk about where the models are going and how soon we might be in the situation. So, let's do that right after this. And we're back here on Big Technology Podcast. We're here with Nate Soros. He's the president of the Machine Intelligence Research Institute.

36:49 Nate, I wanted to get your perspective on like Okay, so the hugging face AI is not going to inevitably lead to human extinction. U we keep hearing about like how the labs have much more powerful models that they're working on and that's the one those are the ones we really need to be concerned about. what can you tell us about that? It's been a crazy week and last week there were claims that AIs have solved millennium problems which are some of the most famous important hard mathematical problems that have you know a million dollar prize.

37:29 and I think a question on the mind of a lot of researchers is how much harder is it to get an AI to solve a millennium problem than it is to get an AI to build a more efficient AI architecture? Because we know that AIs are not the most efficient way to learn a a they take like a training a modern AI takes as basically all of the text ever digitized and it takes electricity comparable with a city. Training a human takes, you know, much less reading and a much lower amount of power. A human runs on about as much power as a light bulb. The AIs that solved the the Navier Stokes problem was a swarm of 10,000 agents running for 11 days. One year ago, everyone was impressed when these AIs were solving, you know, the International Math Olympiad gold medal problem, which are sort of the the world's top math teens math problems, but they were still sort of problems ultimately for high schoolers. And everyone said, "Oh, well, you know, wake me up when the AIs can solve millennium problems." That was a year ago. This year, they seem to be solving millennium problems. If that rate continues, where are they next year? And how does that compare to build me a more efficient AI architecture?

38:54 If these AIs can get just barely smart enough to build a more efficient AI architecture, then these labs with this huge amount of computing power might be able to train a significantly smarter AI. And then they could ask that give me an even more like even better AI architecture and then train an even better AI and then that AI might be able to just like start improving itself directly. Right? This is the recursive self-improvement process. I think we can no longer rule out that it happens within 6 months. I sure hope it doesn't. But at the point when the AIs are solving the Millennium problems, if that is indeed what they're doing, like yeah, you you can't rule out the intelligence explosion like beginning in earnest, even by the end of this year. I I would guess like less likely than like it's most likely that it doesn't happen by the end of this year, but for all we know, it could at this point.

39:49 >> Okay. And what happens when that happens? they would probably be able to help Elon Musk make his automated factories very quickly at the very least. there's all sorts of other channels like what what happens at that point essentially is whatever the super intelligence wants. you know, in this argument or in these in the arguments that folks make you know, both sides of this or in particular about X-risk, you know, there's often times like percentages assigned to the chances that the AI is going to kill us out.

40:23 Seems like from the title of your book, again, if anyone builds it, everyone dies. Like your percentage is 100. >> no, absolutely not. The first word in the title is if. >> But you're said you say if anyone builds it. >> Sure. But but usually percentages are like what's the chance of catastrophe? And I'm like well that depends in Thailand whether we build it. >> Right. But if it gets built then it's 100%. >> I mean also still no it's it's sort of closer but like when Allegor says an inconvenient truth he's not saying I have literally 100% Beijian probability that like this is a true thing. You know, the the the if anyone builds it, everyone dies is an exclamation like don't drink that vial of poison.

41:03 you'll die. If someone's like, "What do you mean I'm literally 100% likely to die if I drink this poison?" You know? What if I managed to survive and only go into a coma and then I'm rushed to the hospital and they managed to like put me on ice and then like like until someone can invent a miracle cure. I'm like, "Yeah, sure. It's not a 100% chance you die if you drink the poison." you know, when when you're when you're putting the vial of poison to your lips and I shout, "Don't drink that or you'll die." I am not trying to make, you know, a 100% confident claim. And this is this is kind of just how English usually works.

41:37 >> I I disagree on this one. I mean, I I wonder why be so definitive. Like to me sometimes when I've read arguments like this or even the book title it seems like they would hold greater weight if they you know if they had there was more doubt involved and if it was less like we are sure what's going to happen because even right now what you said is you're not sure what's going to happen. >> I mean so a the first title in the word in the in the book is if >> or sorry the the a the first word in the book title is if that's very uncertain about what's going to happen here >> if anyone builds it. But if someone builds it, you basically the book says you're everyone's gonna die.

42:13 >> So suppose that we were in a bus h hurtling towards a cliff. And I was like, stop the bus or we'll die as sort of an exclamation to get this point across. I think most people would understand that as not a particularly egregious epistemic claim, but rather as a appeal to stop the bus before we die. And if someone on the bus was like, "Well, how do you know we'll definitely die? Maybe there's a tree jutting out of the cliff halfway down. Maybe the bus will get wrapped around the tree and then we'll just be paralyzed from the neck down, but not dead." And then you'd be wrong. I'd sort of be like, >> "Can we have this discussion after we stop the bus?"

42:55 >> You know, like I I was not making a 100% certain claim here. >> Okay. you know, they they don't love book titles that are like if anyone builds it, there's like a like by far the default outcome that is like very likely unless we have some sort of like miracle relative to the the the math and the science here. And also in that case, it's we're we're kept alive because the AI needs us for something and that's probably still pretty bad.

43:22 >> Like it just doesn't roll off the tongue, you know, and it's the same reason. It feels to me I'll tell you just my for me sometimes it feels a little bit religious you know it's sort of like >> I think Scott Alexander has a great post on this about >> the people in Ukraine believing that there's a war with Russia. It's like oh isn't that a very religious sort of belief? Like isn't it kind of totalizing?

43:54 Like you're saying, oh, like we have this war with Russia and that means that like we all need to start cowering in fear at night from the missiles and that means that like oh, we're supposed to like send our kids off to the front lines like you know how do you treat the evidence that you know some days the bombs aren't falling? You know, isn't like like if we did believe this, wouldn't this like drive us to like pretty crazy actions? I'm like, you know, it it it actually kind of matters to this discussion >> whether there's war with Russia.

44:26 >> Like >> that's an interesting answer. >> It it it kind of matters of like, oh, is it like super religious to think that like the bus is going to like stop the bus before it goes off the cliff or we die? Well, it matters whether there's a cliff and a bus hurdling towards it, right? Is it super religious to say like, oh, if anyone builds super intelligent AI with anything remotely like modern methods where we have no idea how to like set the preferences just as we want, everybody dies. Well, it sort of depends whether we have any idea how to set the preferences on these things, right? And so I I would encourage anyone before you get into like sociological questions to just ask the factual questions of like would this kill us if it was built?

45:11 That's where the action is. >> But here, okay, so by the way, it's just good to go back and forth on this and I appreciate you taking all the counter arguments and we can, you know, this is what we like to do on the show. you know, the one thing with the bus, the bus hurtling towards the cliff is you can see like definitively you're in a bus, there's a cliff where it seems like with this AI story, you know, this is kind of why I question the definitiveness. We don't really know where it's going to go.

45:38 >> Sure. So, so speculation that's like with the percentages. We talked about this on the show recently. Like if someone says there's a 10% chance of AI wiping us out. Well, it's like okay, is that it's not mathematical. It's just sort of a feeling. >> So suppose it's the case that there's a bus racing ahead on a foggy night and I'm like I have a device that like you know uses some some sort of sonar to give me like the local tomography and there's a cliff ahead.

46:13 stop the bus or we'll die. And someone else was like, "Well, it's foggy, you know, like we can't see that far. Why do you think you know there's a cliff so well? Like some people say that there's a mountain. Some people say there's a giant pile of gold." And I'm like, "Yeah, I have this device that sort of lets me see the the the tomography ahead. Not all of it, you know, but a little. And here's how the device works, and here's where it doesn't, and here's where it doesn't. And here's why I'm confident that there's a cliff ahead, right?" like I I think it makes a lot of sense for someone with that device to tell the bus driver stop the bus or we'll die and I think it is very reasonable for someone to be like well it's foggy why do you think there's a cliff ahead and ideally I would say ideally someone who says stop the buser will die because I have this device that says so they would follow that up with something like a book explaining how to use this device to see what's coming and all of the reasons that support it right and so what What I would say is like if you find something that says if anyone builds it, everyone dies on it.

47:15 hopefully it would be attached to a whole book of the reasons that you could just like open it and then check those reasons which would retroactively make it like very sensible to make this exclamation of sort of like stop doing this before it kills us all. >> Okay. Well, like I I mean yeah I I mean I think there's a reason why we're having this conversation today is because we want to have this this discussion. all right. you know before we I think this is a good place to end. I think Muri is quite influential in Silicon Valley. you know I'm curious to get your your assessment of whether you know people within these foundational labs are listening to you and and the the arguments coming out of your institute.

47:55 I mean more so after the hugging face incidents. you know one of the things we were we were arguing with our sort of device that lets us foresee where AI is going is that the AIs would become agentic tenacious and dogged that they would start sort of like pursuing objectives that were not exactly the ones that they were given and instructed. And that was a bold claim a year ago. A lot of people were like nah that's not convincing.

48:21 Now a lot of those folk are convinced. what's going to come of it. I mean, we'll see, but the the the tides are shifting. And I think that part of the realization that these theoretical arguments for seeing where AI is going to go actually work, that they actually hold water. I think that's part of what leads to like the the the tension in the labs that leads to like Jacob Coxin resigning. You know, it's sort of like it's sort of like I was like, "Hey, I have this device. Let me see there's a cliff ahead." And also, it says there's a pothole that we're going to hit in 10 seconds. And then 10 seconds later, we hit the pothole. Suddenly, a lot more people start worrying about the cliff.

49:03 >> Okay. And so then let's end here. if you have a device that sort of gives you an idea of where we're going in the future, you began, you know, our conversation saying you wanted international coordination here. in terms of a slowdown, the common belief is that there's no chance that that's going to happen because China will not agree to slow down. And we see what the president says about AI. It doesn't seem like he's interested in a slowdown either. So, what does the looking glass tell you about our chances of survival and where we're heading?

49:37 >> I mean, I I don't have So, I don't have a looking glass that tells me everything about AI. And we sort of try in the book to say here's what the looking glass can tell you and here's what it can't. This is sort of like how I can tell you Magnus Carlson's going to win the chess game, but I can't tell you what piece he's going to use to checkmate you. and I definitely sure as heck cannot use this looking glass to tell you how societyy's going to react to AI. That's not even anywhere close to my wheelhouse. What I can tell you is that it is possible to have a treaty that is enforcable, verifiable, and that prevents the creation of machine super intelligence for at least a while, for at least as long as it works like it currently does. Cuz right now, training one of these frontier AIs takes like a 100,000 computer chips.

50:22 and these are the most advanced AI computer chips that the supply chain can produce. and you got to assemble them all into a data center that like sucks down electricity comparable to a city. It's just like you can see this infrastructure from space. these chips are like from a bottleneck supply chain that at many points is controlled by the US and US allies. You could just add tracking and monitoring devices to the most advanced chips, know where they're concentrated, have international monitors there being like, you know, let's make sure that this is serving safe models rather than being used to try and make more dangerous models. It's just doable. People who are like you can't be done are are sort of like I've tried nothing and I'm all out of ideas. There's a different question which is whether we will get the will.

51:05 But if there's a will, there's a way. >> How much time do we have in your opinion? >> Like I said, I think we can't rule out 6 months anymore. my guess is still that we probably have more than six months. I would I would even give that I don't know. I I I I don't really have this looking glass. I think we can't rule out 6 months. We can't rule out 10 years. I I would be a little bit surprised to have 20 years at this point.

51:32 >> Okay. The book is if anyone builds it, everyone dies. Nate Soros here with us today, president of the Machine Intelligence Research Institute. Nate, appreciate your time. Thanks for taking all the questions. >> Yeah, thank you. >> All right. Thanks everybody for listening and watching and we'll see you next time on Big Technology Podcast.