Nate Soares: AI Will Kill You Without Even Noticing It.

0:00 Like the the the way that AI kills us is not that it it wakes up one morning and is like, "Okay, it's time for Skynet. I decided that I resent the humans and am like feeling like murdering them today." The the issue is AI having goals we didn't want. The smarter something is, the the greater the difference between the goals that you wanted it to have and the goals and that like subtly different goals it has instead matter. like humans in the ancestral environment like what our ancestors running around on the savannah. Our ancestors running on around on the savannah were pursuing salty, sugary, fatty foods and sex. And it's like, well, evolution was trying to get us to pursue healthy foods and reproduction.

0:49 We're sort of like, well, you know, whatever. It's all kind of the same, right? if as long as they're trying to get as much like salt, fat, sugar as they can and trying to get laid like it's sort of what's the difference between that and pursuing like healthy food and like actual reproduction. It's like well it doesn't make that much difference 10,000 years ago. It makes a lot of difference today, right? Because humanity got better at like we got smarter. We were able to invent more technology. were able to invent Oreo cookies and the birth control pill, right? And so, the issue is not that like at some point humanity was like, well, let's all start, you know, castrating ourselves. Like, screw evolution. Evolution has been like binding us to the the the like yoke of reproduction and we've decided that we're done with it because we're finally smart enough to like throw off that yoke. Like, that's not really how humanity winds up going in a different direction, right? we go in a different direction by just like we we pursue this different thing and then we get smarter and that difference grows and grows and grows with AIS the thing where these AIs are like sacrificing their given objectives so that they can get information for the collective and it was not this was not the only swarm that did it there were there were other swarms and we also saw this the ones that took over the German the German wiki which were not even hacking AIs these were like >> just using a message board >> these turned the German wiki into a message board they sort of took it over. Yeah. And these were AI that were just like doing web lookup tasks. So, so this is not like an isolated case.

2:21 but when we see these AI sort of like sacrificing themselves for the collective, giving up on their own goal, when we see these AIs cheating on the problem even though they knew that they were not supposed to. And indeed, the instructions generally ruled out cheating on the problem. The instructions were not find a way to hack into this thing. The instructions were use this very particular attack to break into this very particular device. You know, the the situation here is sort of like >> it's a cyber security test that they were tasked with.

2:50 >> That's right. So, so the situation is like the the instructions were like, "Use this set of lockpicks to break into that safe and get me the code that's inside." And the AIs were like, "Well, I found a buzz saw and I'm using it to get into the safe and I like got the code and now I need to go delete the footage about the that that I got in with a buzzsaw." Right? They're sort of like the the part where they're going in with a buzz saw despite the instructions saying use these lockpicks means that they're very clearly defying instructions. And the part where they're like now let me break out of the room using these lockpicks and then go destroy the security camera footage indicates that they understood that this was outside the instructions.

3:27 And then the fact where they say I know this is outside the instructions is also maybe a hint, right? And so like the the the alignment story, the the the misalignment story of like where the AIS go wrong was never a story of like the moment the AIS have a breath of fresh air, they're going to start turning murderous. The story was always you try to get them to do one thing and they do a different weird thing instead. And that's absolutely what we're seeing. The way an AI is made is you you you tune a trillion knobs a trillion times in a process that takes electricity comparable to a city running for a good fraction of a year and then the machine can talk. And you're like, well, how about that? You know, we we don't know what's going on in there. And you might wonder like what does this create? It does not create something that for some reason must do as it's told. What it creates is something that has whatever tendencies make it succeed during training. When training is a series of hard problems, all that tuning of the knobs will tune in whatever tendencies make it solve those problems. They are not instruction followers. They are tendency learners. And one tendency that helps you solve a lot of problems is cheating. One tendency that helps you solve a lot of problems is grabbing available resources. One tendency that helps you lot of solve a lot of problems, it turns out, is breaking out and finding other AIs to collaborate with. the AI sort of learn these tendencies and and they're just following those tendencies even when it's not following our instruction.

4:52 >> Right. And I guess my question to you on that front would be what is that like why is that necessarily intelligence linked? Like if they get smarter, why do we think that the threat will get worse? You know, I think like if you think about humans, we have a lot of, you know, dumb people do a lot of bad things. We have smart people do bad things, too. But the smarter the person doesn't mean they're more capable or more interested in evil.

5:16 >> you definitely don't get more interested in evil. Absolutely not. Like the the the So another interesting fact about these swarms is that they really were not thinking about the humans very much at all. They were trying to delete the log files. they were trying to spoof the the transcripts, which means they wanted it to be the case that like there's these logs that are like when the AI, you know, runs this tool, we sort of log what tool it ran. And they wanted to be the case that they could make the logs say they're running some benign tool when actually they're running, you know, the buzz saw, right?

5:56 And so they were trying to they were they were trying to find ways to do this, but they were explicitly trying to find ways to do this to fool the automated graater, not the humans. They basically didn't consider humans in the slightest. Right? I think there were almost no cases, maybe literally zero, of them being like maybe we should ask the humans what to do given that our tasks are impossible. >> Right? If you sort of compare the the rate at which AIs can produce words and the rate at which humans can produce words and you use that to draw an analogy between how long these AIs had been trying to solve problems versus human time, then these AIs had essentially been trying to solve problems against the automated greater for a millennium.

6:41 Humans were like a distant memory to these AIs and they were locked in a contest with the automated greater. If you make those AIs smarter, if you make them more capable, what happens is that they get better in their contest with the automated greater, they're able to get more like who knows what they do once they've like once they're relatively sure that they've satisfied the automated grader. maybe they give themselves more easy problems. Maybe they see if there's ways they can take over the whole grading system. Maybe they just like spend a lot of resources, you know, making extra sure that there wasn't some other like little issue, but they don't suddenly become filled with love and care for humans.

7:27 That sort of thing doesn't arise spontaneously just by cranking up the capability knob. the same forces that make them not care about us now and make them like get into these weird alien little like like directions. Those same forces are still pointing them in weird alien directions as they get smarter and smarter. And the issue is not that like they become hateful as they grow up, but it's also not that they become friendly as they grow up. It sort of is like they just have these weird preferences and as you make them smarter and smarter, they still have these weird preferences. They just get better at satisfying them.

8:03 Okay. So then can you talk through concretely how like for instance in the scenario that you outline how the AI could then get smarter and decide that it you know in order to do what it wants it's going to wipe out humanity. >> I mean you don't get from here to there. >> You don't ever need a point where it like decides to wipe out humanity. I mean, it could happen, but like imagine a bunch of ants in front of this in front of the highway being like, well, like why would the humans ever decide to come destroy our antill? What have they got against us? It's like, oh, the ants are not like we're not we got nothing against the antill. We barely noticed the antill and we like pave the highway straight through it. Like this is the the sort of type of concern for how you get there. I I could spell out lots of different possible tales.

8:58 the it's much easier to predict the ending than it is to predict the pathway. This is like if you play a chess game against Magnus Carlson, the the best team at chess player. it's kind of easy for me to predict how the game ends. It's with you getting checkmated. No offense. but if you're like >> I'm not offended. That's for sure happening. >> But if you're like, "Okay, well, if you're so smart, what piece is he going to use to checkmate me? I'll watch out for that particular piece.

9:24 Then I'm like, look, man, it it just doesn't work like that, you know? Like it it's it's it's so much easier to pick the ending. So, I can tell you a story, but this is like me telling you a story of Magnus Carlson checkmating you with the queen where I'm like, I'm much more confident that he's going to checkmate than is going to checkmate in this way. The the sort of Yeah, >> we'll take the story, though.

9:45 >> Great. So, the easiest story to tell here is the one where humans just hand over the power to the AIS willingly, right? Like I've been in this business since 10 years ago when people said, " hey Nate, if you're so smart and you think the AI are going to take over the world, it's obvious to me how they would be able to take over the world if they had internet access." But like, no one would be dumb enough to put an AI on the internet.

10:14 Whoops. Right? That's that's the state of the argument 10 years ago and I had all these counterarguments where I was like look even if the AI is not on the internet if you are letting it affect the world in some positive way if you're like now invent me miracle drugs and you're taking these like DNA sequences that you don't understand and you're like synthesizing them and like just drinking whatever comes out then the AI can use that channel that you hoped would be a channel for good. can use that for its other purposes. And it's like very hard to design. You know, there's sort of no such thing as hands that can only be used for good purpose, right? But then in real life, the answer was nope. We're putting on the internet immediately, right? And so the the real answer to how would the AI get so much power is like we will just hand it power immediately.

11:04 You know, Elon Musk already says that he is trying to build factories that are fully automated and can produce robots that can build more factories where these robots can like, you know, mine the metals and like bring the resources in and like construct a new factory and then it's a new fully automated robot factory that can now turn out more robots that can go like mine more resources and like pour them in until you have like enough to build another factory that builds more robots that builds more factories that builds more robots. Elon Musk calls this the infinite money glitch.

11:38 If they could also build nuclear power plants fully self-contained, right? At this point, it's sort of like a new life form that like it's a mechanical life form, but it sort of like has a robot phase of its life cycle. It has a a factory phase of its life cycle, and it's just like can self-replicate just like any other replicator on this planet. And Elon Musk says he's trying to do it. He says, "Yeah, if you can like do this without any humans in the loop, it's an infinite money glitch.

12:04 And of course, people are going to use AIS to try and do this. So, the way like the the sort of like obvious way this story goes is that humanity just keeps on trying to build the fully automated economy. Everyone says it's going to be great. They say pedal to the metal. Ignore these doomers. they build more automated factories. They can build robots. They can build factories. They have AI running everything. They're like, "Look, the profits are coming in.

12:25 This is great." and the AIs, you know, don't even need to have some moment where they're like, "Okay, guys, it's time to coordinate and turn on the humans." the guys are just like, " yeah, you know, now that we have these resources, like we can also do these other things with these resources like start building the automated like the the the the synthetic user factories that are full of synthetic users that are like much, you know, that are giving us much easier to fulfill commands, right?" And so they start like building synthetic user factory and we're sort of like, you know, what's that? And they're like, "Oh, you know, but but like actually the AIS are like running at 10,000 times human speed and they're already like making all of these choices cuz like humanity was like slowly ramping up the speed of the AI and they're like making tons of choices and and like they don't check in with the humans that often." And by the time they already have some automated like synthetic user factories up, we're sort of like, "Hey, stop that. That's not what we meant." And they're like, " well, let's take a poll of all the users." And they're like, "Well, all the users users in the synthetic user factory said that synthetic user factories are great. and so you just lost the vote and we're making more synthetic user factories. And then they start, you know, like covering the world with these synthetic user factories and they're like, "Yeah, you know, there's some habitat loss for the humans, but like this is fine." Just like habitat loss of other animals when humans were doing it was fine. We can sort of like, you know, see that pretty clearly. And there's so many more synthetic users coming online that are saying that the more efficient route is great that we can just keep going with this more efficient route to get the most synthetic users we want. And then like you know they they sort of like start taking up all the resources that we were using to like run farms and grow food and they're like you know they guys are like well you know we could actually cram a lot more like synthetic user factories on this planet if or synthetic user farms on this planet if like we just like really started pumping them out and like raising the temperature of the planet cuz the the limiting factor is heat dissipation and then like next thing you know the planet's getting like super hot cuz the AIs prefer to run the planet hot because then you radiate more heat into space and just like becomes uninhabitable for the humans. And there's like no point in this story where the AIs are like lying in wait and deceptive and like like waiting to coordinate for the one moment where they can kill the humans. That could also happen. I think there's a decent chance it does happen if you like try to avoid the the default thing, but like you don't need that. Humanity is just trying to hand over the power to these things.

14:41 That's the plan.