Anthropic's Existential Risk Warnings Hijack Larger AI Debate

0:00 Latest reporting and and, frankly, a great body of work trying to cut through the noise of the week. Davey, I'm gonna start with you. Just summarize what our story outlines and take us to the the origin, which is the hugging face incident. Yeah. Absolutely. So, you know, we did a deep dive into sort of the hugging face incident as an example of how a lot of the rhetoric that's been going around right now is talking about how failure in safety in in the sandbox not being properly cordoned off, that there wasn't warnings. There were warnings that the human employees themselves Right.

0:45 Did not take seriously, that is being framed as, look. We can't contain this AI. It's so dangerous. It's going to escape and that being an inevitability. And that just, you know, sort of works to sidestep the responsibility that these companies have to avoid safety failures. Hey, Rachel. You know what's weird about AI? Humans design the What's weird? Humans, peeps, design the experiment and designed the safeguards. Like, that's the takeaway I got reading your piece. Remind everyone that AI came from humans. Run with it.

1:26 Yes. Yeah. I think that's something that that I've been thinking about for a while since we heard about the hacking face inadvertent hack and talking to people about it. I mean, this comes up quite a lot that humans were very much involved with this. I mean, humans set a set goals for AI. Humans set up the situation here in the sandbox where the AI is working on things, and humans are training. I mean, they're humans up and down. The humans also noticed, some situations where they could have in during evaluations when the AI wasn't doing quite what it should have been doing and did not. And OpenAI has acknowledged all of these things, and then to its credit, it has said, quite a lot about this incident and also what it's doing to change things over there. But I think that that came up a lot with this story, especially, like, as we're talking to people ongoing that people have a lot to do with this situation. It's not as simple as saying, oh, well, AI went and did something.

2:25 Rachel Davey, we have some breaking news. Gavin Newsom, the governor of California, has issued an executive order on the creation of an AI kill switch. The order accelerates what he says will be independent oversight of AI. That story just hitting the terminal right now. Davey, this is difficult because it's fluid. Right? It's just hitting the newswire. You talk as well in your your parallel newsletter about this idea that there is plenty of elements of human control.

2:56 Hugging face was the case study for that. Take us there. Yeah. Absolutely. And, you know, with regard to this AI kill switch, we also cover in our story the language that, federal lawmakers have started to use to create policy, that would affect, you know, citizens. And and one thing that we point out is a lot of this sort of rhetoric of doom is migrating from the AI industry into the rhetoric of these lawmakers. So, you know, I would want to take a look at how Newsom is framing this kill switch. One thing that the researchers that we spoke to said was that when you start from a place of doom, you know, sort of this assumption that AI is going to threaten all humanity, the policies may not match the actual harms that AI is causing right now. So when we're talking about misinformation, surveillance, these things that are happening right now, that is, what is sort of getting lost in the focus on doom.

4:07 Rachel, on the show in the last twenty four hours, there have been plenty of voices. Let's take our researcher Andrew O'Day or saying this is way overstated. The risk to to human life to their mind is near zero. You're up in Montreal right now. Right? Like, who have you been speaking to? What are you hearing on the kind of counterargument, the anti demurism school of thought? Yeah. I mean, I think up up here, I'm I'm hearing a little bit of both.

4:35 I mean, I spoke to Yeshua Bengio, who is a legendary figure in AI and has long talked about what he sees as very real harms that that may come from AI. And so he sees this situation, the hugging face incident in particular, as when I spoke to him yesterday, was sort of sort of a mix of of a human and and machine problem, and he's very, very concerned about what may happen going forward. On the other hand, I've spoken to other people, such as the CEO of Cohere, Aidan Gomez, also a very key figure in AI. And he had a thought that was a little bit more like, look. I I don't think that this is going to kill people or kill us en masse, but there are some really real issues here, and those do need to be worked on. And maybe slowing down certain things could be helpful, but it's not a a situation where, we're at imminent risk of all being extinct.