Why AI Labs Want You to Think They're Dangerous

0:00 this is from the Wall Street Journal. Anthropic researcher quits over outofcontrol AI fears. An anthropic researcher is quitting the artificial intelligence industry over fears that the lab and its competitors are racing to build systems they won't be able to control. A sign of mounting safety concern within top AI companies. Jacob Coxin, a researcher who specializes in training new a model AI models by having them consume vast amounts of data, said Tuesday he's leaving the company because he doesn't want to participate in an industrywide rush to build AI systems that can improve themselves. Worried such systems could spiral out of control and destroy humanity. He followed it up in a tweet said, "I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving super intelligence and gambling with their lives. Don't underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains and progress is not slowing. we're going to get into all the details here, but Ranjan, I mean, this this thing has gone just like absurdly viral. 65 million views to date and counting. what was your view of this when you saw it come out and are you surprised that it has hit such a chord that it's reached so many people in such a short amount of time?

1:33 I think the escape velocity side of it was a bit surprising because again we have been talking about this for a long time. In fact, I spent part of my Labor Day weekend reading the meter research and going through it and trying to understand the what kind of real vulnerabilities we actually have around, you know, rogue agents escaping containment and and then obviously to see this kind of take off. It is interesting to me. Part of me like the fact that this happened right after Labor Day when everyone's summer vacations are over and they're back fully at work. I do wonder if it has a little to do with it. Do you think is that a crazy theory?

2:18 >> well, I think the fact that this hugging face story found a way to have life over a month and sort of primed people for a moment like this, you know, might have had something to do with it. Here's a conspiracy theory that I like that I don't think is true. but it could possibly be that you know, the owner of X probably has something to gain by something like this going so viral either by forcing OpenAI Anthropic to slow down or potentially having regulation come in that sort of entrenches those that have built large data centers already and maybe hit a button at X headquarters to make it blow up. But I don't know that might be a little a step too far.

3:02 >> I mean, let's not, you know, like forget the fact that something going viral on X can very directly and easily be controlled by said entrepreneur. So, >> if you don't know who we're talking about, it's Elon Musk. >> Yes. Yes. Elon Musk. I mean I I it okay I think as we kind of unpack this entire story it is worth thinking through what are the motivations and possible incentives for every individual player.

3:32 I think what makes this such an interesting story is there's so many nuanced fascinating layers of incentive for the ways everyone is talking about it. I feel I mean for us neither of us have a ton of incentive either way. So, I think listeners, you can get a pretty open unbiased view in terms of I mean, I'm pro- non-human extinction. I'm I want to make that I want to make that clear. So, I'm biased to avoid human extinction. I don't want it. But overall, like >> you're pro nonhuman extinction. Sorry, like 15 seconds to figure out. You're antihuman extinction.

4:11 >> Antihuman extinction. Sorry, >> could have just said that. >> Good cl Let's clarify that. I wanted to It's kind of like saturating a test rather than passing a test. If you listen to last week's episode, it sounds fancier, but I I mean I let's start with the virality of it and does like what what are Elon Musk's incentives here? You you kind of listed out a number of them. It's entrenching existing leaders. I think we should definitely talk about is this positive in a weird way for Anthropic who is a core rival for him even though >> I have that I have >> so I have a list of 10 of 10 important nuances to go through.

4:55 >> All right. All right. >> And I I I'll take us through that and the anthropic thing is is certainly part of it because and I'll just preview it. there I mean it's so crazy. This guy was only at anthropic for like three or four months. So the idea that he's >> isn't it six weeks >> something in the you know this stuff can be fuzzy. something like not a long time. So this idea that anthropic or they hired him to fire him to market their company. I don't know if that makes 100% sense to me but certainly you know it's coming at a time when everybody was talking about OpenAI and the and the hugging face hack and how powerful Open AI's models were. And of course, what was it like the day the day of or the day the was it the day before in close proximity to the millennium problem being solved. So you know this sort of did put the focus back on anthropic. But you know what we're going to go through all this.

5:51 Can I and and I preface this like what's Elon's incentive by saying I don't really believe that that's what's led us here because if I'm being honest like I'm just going to take the AAM razor position here like what is the simplest explanation because that could definitely be the right one and that is people were primed to be freaked out by the hucking face thing. Not only does this guy Jacob go out and say that he thinks we're facing this risk, but Evan Eubinger like who is high up in anthropics alignment division said Jacob, this was after Jacob quit. He said, Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it's more than 10% within the next decade. I believe anthropic is trying its best.

6:33 And now here comes the real jaw-dropping statement. But we do not yet have a plan to solve alignment for super intelligence and are clearly not on track to I think the kind the one two where it's like the exanthropic guys said something but now somebody within anthropic you know comes in and not only backs it up but makes it even more concerning. So this is an earn ear earnestly held belief among a lot of AI researchers and that is why you know I've always been I guess a little bit on the side of maybe it's not marketing maybe they this is just people actually believing it and saying it and by the way there have been multiple other researchers that have quit since this this Jacob Coxin tweet came out. but okay, you know, as as we get into this, you know, while I was writing notes for the show, I just kind of sat down at the computer and wrote several points of order. Like that was the way that I, sort of headed this thing because I was getting a little bit annoyed, I think, in terms of like the way that the dialogue has spun out of control. And I think that like you've set it up perfectly because you know within AI circles like John was telling you people will have their probability that AI is going to kill us something called a P doom and they say it's 10 30 40 70 whatever 100% if you're Eleazar Hudowski and this is my first point of order and that is that it's a percentage so it sounds mathematical but it's not math right when we talk about this story it's This is just somebody's guess. It's speculation. It's not math based. It's not like they ran the numbers and they're like, well, you know, if we do this, you know, 100,000 times, you know, at a certain point, you know, only like, you know, 10 1,000 or 10,000 of them, you know, kill us. It's like, no, this this is a way to make somebody's science fiction guess sound more credible. And in fact, no, now obviously he has incentive for this for Jacob to sound like a dummy because he's been acquired by Nvidia, but the hugging face CEO Clem Deang had a very interesting tweet.

8:42 He said, "Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change. Not saying it's necessarily uninteresting and wrong, per se, but let's keep things in perspective to hear from the full range of expertise across the ecosystem. you know, obviously he's taking a shot there because he's like got a lot of Nvidia stock now. probably billions of Nvidia stock. He wants it to go go up and you need AI to keep progressing in order to go up. But like this idea that there's a 10% chance like show me the math. If you're going to use a percentage number, show me the math.

9:19 Otherwise, let's just say, you know, my speculation is >> well, I I think I think that's exactly the right right point that 10% is clearly being thrown around because it's kind of like kind of low, but not that low when it comes down to human extinction, but it's still like you say 51% suddenly, you know, it's a very different story. So it feels like those numbers are kind of being thrown around in that way. But I think on that idea of incentive. So on one side you have people who don't want to push the idea of human extinction because they are heavily invested in Nvidia for the research community in a weird way owning and controlling this like that is a thing of power to be able to and they must feel that and think that and you want to think that because that is powerful. you are creating something and maybe you're resigning but like everyone within the research community who's kind of in this direction that is increasing your importance like overall in a pretty dramatic way to say that you are kind of like guiding and trying to guide away from human extinction. I feel not all of us can say that >> if this is marketing this is the worst marketing of all time. if you've seen what's happened since this tweet came out or a series of tweets because it was both Cox and Yubinger that I think caught people's imagination, politicians far and wide have come out and started calling for hearings for moratoriums on data centers even to ban super intelligence. This is from Governor Pritsker.

11:09 he says it's time to sound the alarm louder on reigning in artificial intelligence. It's becoming more clear that the threat AI poses to humanity. More clear the threat AI poses to humanity. So I'm calling for immediate action from the industry in Washington. First, the big tech industry must stop intense lobbying against AI safety regulations and get out of the business of trying to influence elections. The stakes here are too high to approach AI as a conventional political issue.

11:35 That's a kind of interesting first point. Second Congress must start holding hearings on AI safety now. Not next year, not next month, now. Josh Holly, Republican senator. Who is held accountable when AI goes rogue? I'm launching an investigation into open AI. Americans deserve answers. Okay, so well, if you're anthropic, you're probably not so upset about that. Here's Bernie Sanders. You know, bring Bernie Sanders in. Mr. Coxin is right. The very people building this technology admit that it could threaten the future of humanity. That's why I will soon be introducing legislation to ban super intelligence and pause AI development.

12:10 I think in Bernie's bill if you work on super intelligence, you go to jail for like 20 years. talk about you talked about the IPO to go from where you are today to the IPO. We've talked about it. Everything needs to go perfectly for you. you're not interested in having and doing anything that could lead to data center moratoriums to AI kill switch legislation to getting Daario and Sam Alman shipped in front of Congress for hearings which we know there's a there's a chance they won't perform well at. So this idea that it's marketing, it's just it would be terrible marketing and and so dumb of companies that are generally filled with smart people to sort of drum up the fear that their AI could kill all of humanity to what? Sell more tokens to enterprises as their IPOs need to be perfect in order to succeed because of the amount of money involved. Okay, I'm going to make an argument that I don't really believe, but I still am gonna argue against what you're saying and I think hear me out. So, what is >> you out, but go ahead.

13:25 >> No, no. Okay, think this through. What do they need for their IPOs to be perfect? This the single biggest business threat to these companies right now is this idea that frontier models become less important, less relevant, cheaper open weighted models, cheap cheaper older models doing using the right model for the right task. All of these trends which have very strongly taken over within the industry have completely altered the economics of the story of their upcoming IPOs. that like those companies win when the smartest frontier model is where the value is accured. Now people more and more like Astra is you know you can build some cool 3D world models and Astra's I mean it's pretty good have been playing around with it like it's actually feels pretty powerful I don't know if it's AGI or whatever AGI is supposed to be fable five good very good but like that idea that you need to be only using the front the latest frontier expensive model has gone away But when that model and that research and only you guys are so advanced that you pose a threat to all of humanity comes back into the narrative, it pushes people back to the idea that something is happening in those frontier models that only these labs can do. It's so advanced and complex and smart. And then obviously there's going to be this kind of push back, but I don't think this is going to like what is banning super intelligence even mean? Even the pause of the AI like it still brings back this idea that frontier front like the the you know state-of-the-art frontier models are something different and special and these are the leaders in it and that's why they're going to be worth so much because otherwise >> the entire story starts to collapse.

15:24 That's that's my first like Yeah. No, I hear you. the idea of a super intelligence ban, obviously it's hyperbolic, right? Like the real threat here is the hearings with the CEOs in front of legislators, right? That's going to make potentially make this technology even less popular. but more than that, it's the data center pauses. And we're seeing real momentum for data center pauses. even the free enterprise Republicans in Texas right now who are like I was at an opening of a chip fab last year in Texas. You had Ken Paxton there talking about how great it was that Texas was all about, you know, free enterprise.

16:02 Fast forward, you know, just a few months, Texas is not interested in data centers anymore. So, you know, you can't now it's localized, right? So I'm not saying it's going to be over for AI companies if there are some of these bands. But as that risk goes up, your risk to expand because you need to build the big computers to expand goes up. >> Then how could they let I mean and maybe that's just the nature of like social media and the companies themselves, but like the idea that multiple employ current employees come out and start saying these kind of things that yes, I do believe that 10% greater than 10 like how how do you let that happen? I mean, I I genuinely Or how are you okay with that? Or or how do you not respond to that? Like, if that's if that's really the case, you would think there'd be like some, "Hey guys, we're not going to ex extinguish humanity." Like, they're wrong. Like, shouldn't you say something?

17:04 >> Let me bring you into the boardroom of companies or sort of the, you know, corner office. If we're going to if we're going to assume that this future that you're pitching or this this marketing rationale that you're pitching is correct. Okay. So the boss with the budget is sitting there at the table and he's instructed his team to use open router. AI champion walks into you know boss's office and be like hey boss you know I know we were on opus 4.8 8 and you had us go to open router.

17:42 So now we're using like I don't know some Gemini flash which has been fine. But do you see that this dude who was at anthropic for two months said that that technology might kill us. We really need Fable 5.1. Just switch us on to the murder bots. Switch us on to the stuff that is exactly >> as an enterprise that demands that demands that the data is protected. switch uses on to OpenAI Astra. You know, after their technology has shown that, you know, if you let their agents go rogue, they're going to or if you give your agent if you give their agents tasks, they might decide that the way that you want them to do it is not the way that they should do it and instead they should go commit felonies and hack another company. Like, where where in the world is this scenario going to play out? be because no one is going to have the conversation exactly as it like the their agents are going to go commit felonies and we can get into whether how correct that is but it's more there is this power that is going to be available to others including our competitors and if they use it and we're stuck here trying to count tokens and like automate as you said and and I consider optimizing business processes for enterprise work of importance So even if the researchers don't but I mean like that I I agree like you could sit there and you know token optimize and start routing to the right model and building your workflow or there is this thing over there that is so powerful that it's going to just cure cancer and revolutionize our business and if our competitors use it and we don't we're dead. Like forget we're business extinct as opposed to human extinct, but we're like either way.

19:33 >> Yeah, exactly. >> That might be the pitch. Go to the corner office. Hey, I know you have us on open router since we're going to die anyway within a decade. Can we get some more budget for the good stuff? >> Okay, that's the pitch. That's the pitch. But but I think it that this this isn't new. We I mean we've talked about this for years and like it actually annoys me the idea that anytime these stories come these stories have been coming out this kind of language it reached escape velocity but it has been present the entire rise of the frontier labs that all powerful potentially dangerous and they have very again the sandwich in the park saying that our models escaped containment like and then having a coordinated PR effort around that to push the narrative about a anthropic researcher eating a sandwich in the park. That was PR like they the timestamps of all the press releases and the tweets were coordinated. So like it has been good so far and I don't think it's different now. I think it's like you know people are coming the politicians are coming out against it but I don't think it's it's bad. I don't think I think how you posed in a weird way it can actually be correct >> works. Yeah. I mean interestingly Amjad Msad the CEO of Replet was speaking this week about the past warnings and I think he said something really interesting. He said with GPT2 OpenAI came out and said we've made a model that's so powerful we can't release it to the public because it was an open- source company and it was always open sourcing models. If we release it to the public it's going to wreak havoc on the internet and we're not going to do that.

21:16 and Amjad and you know then talk about >> that's GPT2 >> too right and Elia obviously like one of the things that freaked him out was that he saw reasoning within open AI and and then you know went to build safe super intelligence whatever it is and Amjad's calling it AI psychosis so okay I think I think I'm seeing what you're saying which is that there is an incentive to believe this because it will drive you know growth in the industry and if you talk about it in these terms people are going to start to take the technology more seriously. So I I do think that like and this has been the biggest criticism of Coxin through the week is if you take him at his word, you got to ask how this happens. And I think you know again going back to what I said at the very beginning this a lot of this is science fiction. It's happening in people's imaginations. that being said, you know, so so basically like it's going to take a lot of time for any threat anywhere close to this degree to manifest itself. But that being said, we are seeing these capabilities, you know, get much better like you were pointing out. the hugging face attack is one thing. The Millennium Prize, solving that is another thing. And I think if you were to dismiss the risks from AI, both near and long-term, out of hand, you'd be doing yourself a disservice.

22:42 >> So, who there are some risks? We don't know the percentages, though. >> So, who is best positioned to help prevent these risks from happening? >> Well, okay. So, that this is such a great place to end our show. Such a great place to end our show. because this is the one thing we haven't addressed. Is it even possible to pause this stuff? And the argument from labs like anthropic and open AAI for that matter have been that no you can't slow it down. So we must accelerate and accelerate on our terms. So basically going back to the only we can save you thing, right? If we are the ones to lead, this is I'm saying that this is what open AI and anthropics say, then we can steer this technology in a good direction.

23:32 That's obviously something that Coxin, you know, is coming out and saying that's not not the case. and it always felt a little bit thin to me that that was it. like, you know, we have this dangerous technology and we're going to become billionaires on on the way to >> and and also >> I just have to say I mean Sam and Daria like it to the vast majority of the world the idea that like going out and them being the ones to stu steward this technology I feel has never been to the and actually maybe from my side too now this was not a conversation for the mainstream public. I actually saw like Cheryl Crow tweeting about pausing AI development. So, we've hit that level of escape velocity. When Cheryl Crow's coming, you know, it's >> finally we have somebody we can take seriously in this debate.

24:23 >> You know what? The AI super intelligence needs a steward. I nominate Cheryl Crow. >> Creating super intelligence was Daria. >> She just wants to favorite mistake. I don't >> soak up the sun. Literally, an AI will just soak it right up. Kill us all. >> Oh, God. End on Cheryl Crow. >> Don't Don't dismiss Cheryl Crow's perspective on this. >> Dude, that fun guest. Cheryl, if you're listening, we know you are.

24:54 >> Come speak with us about AIS. >> We've said AI needs someone good leading it and kind of winning over the hearts and minds. Jensen has kind of gotten there a bit, but I think Cheryl Crow's the one. Cheryl Crow, >> Lisa, and I'll >> All right, last question for you. Are you leaving? Okay. are you leaving this week compared to last week more heartened or more scared about AI's risks and more heartened or more worried about the business prospects of this technology given what's just transpired?

25:29 >> Weirdly neutral. >> Okay. Seriously. You're just like where this was going to be a blip. >> This is a blip. I'm calling it. This This is going to move past us and we'll be on to the next thing soon. Unless the hearings thing is interesting. That's like the one thing that Congress can do very quickly. And that is something I agree could go terribly wrong because no one I mean I can only fathom the media training that's going to be like this, you know, like pulled together on Sam and Dario in front of Congress. But that's the only thing. If that doesn't happen, I see this kind of moving on. What about you?

26:11 I'm about about neutral in terms of like the risks of AI because though he quit, he didn't really give us any like this was not like a Snowden moment. Like it would have been if he had documents or anything like that to show why we were at risk as opposed to just to tweet might have changed my mind, but I'm neutral on that. I'm a little bit more negative about the business prospects than I was at the beginning of the week because, you know, I'm not saying that it will necessarily happen, but I wouldn't underestimate the backlash that we're about to see.

26:44 >> All right. Well, I hope humanity's around next Friday and we get to do this again. I hope so, too. you know, despite despite it all, I'm having fun. So, is it are my incentives bad if I tell people to give the podcast five stars, you know, on a week like this? I won't do it. Listen, only >> you are pro. >> You still have time left. Give us give us a five star rating. >> You're pro human extinction then.

27:12 Telling you. >> Exactly. All right, Ranjan. Lots of fun as always. Thanks again for joining. >> All right, see you next week. >> See you next week. Thanks everybody for listening and watching and we'll see you next time on Big Technology Podcast.