"Do You Think AI Is Gonna Kill Us?" | This Week In AI

0:00 So, funny enough, I went on a date last week and when we talked about my job, first question she asked me was like, "Do you think AI is going to kill us?" >> The models that they are seeing must be so much better than the models we have if they're worried about it because I can't get my model to do a simple task sometimes. Welcome to Agents Hour. It is September 14th. We are on site in San Diego and we are going the to do the news like we do every week and it was an eventful week.

0:36 There was kind of one big topic. Let's not beat around the bush. Let's jump right in. Pacing the frontier. Should we pace the frontier? What the heck does that mean? This is the topic. So, it all started with this post from someone unnamed person. No one really knew who this person was. They didn't have hardly any followers. They had never posted on X before. Just to be clear, this post got 171 million views. That is insane for someone who's never posted on X.

1:01 It's honestly like algorithm breaking in a way. So, we can talk about why that is, but Ned says, "I resigned from anthropic today. I spent the last three years doing pre-training research at both Open AI and Enthropic. Neither company's acting responsibly. They are racing straight to self-improving super intelligence and gambling with our lives. More thoughts below." And then he goes on to talk about some of the reasons. He doesn't get into a lot of details, but basically talks about how people are not taking this seriously enough. Even people inside Enthropic think that this is going to happen or it's it's possible and no one's doing anything about it. So that was that's how it all started. And then Evan Hubbinger, I'm trying to know exact title. Alignment science lead is Evan's title. Evan says, "Jacob is correct here. We really do earnestly believe AI could kill all humans."

1:45 Kind of like he's excited about it. I personally think it is greater than 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for super intelligence and are not clearly on track to. So, this was the thing I think it the original post was one thing, but this was the thing that got people extremely kind of bent out of shape for good reason because why like where did you come up with 10%. That's kind of important. Like what is the percentage chance that you think that we actually have we're all going to die and humanity? The way they talk about this is like extinction level ramifications.

2:20 So like Jacob went on CNN, talked to Anderson Cooper. One thing I'll just say I just noticed very much from him is it's all high level. No details. The threat to our existence isn't here today. But if we continue down self recursive self-improvement, then AIs will have ways to improve models themselves and then destroy the whole world. which you know if you go on CNN like I feel it's just like such a doomer thing because the people watching CNN are not in this industry deeply and I guess you know it makes sense to keep things general but it's a lot of FUD you know at least that's my opinion I do not think a 27year-old researcher who pro who is like 2 years old during 9/11 should have any like any you know point of view here but maybe he knows something that we don't you know >> yeah we shouldn't and we'll get into this a little bit we are not detectives.

3:13 We did our own research. You should do yours. But there's some kind of interesting sketchy things about how this became so super viral and how many people jumped on it so quickly for this to even happen. A lot of things had to go perfectly right with the algorithm. And so it was a little like skeptical. I think in the first 15 minutes there was like three major news article reposts. There was a Wall Street Journal article that actually came out a few minutes before the post itself. So this was all like coordinated in some way like but obviously the other thing is it clearly resonated with folks too and I think one of the reasons is just like human psychology everyone like fear spreads faster than anything so if you can spread fear and panic people will spread that faster than any good news like bad news spreads faster than good news right and so I think there was probably this bent up like everyone's on edge everyone hates data centers you know if you like talk about the just general perception especially across like the US so I think This is like there there's a lot of pent-up demand for people want looking for reasons to hate AI and this was like fuel for the fire so to speak. And the other thing about this Jacob Coxin is he did work at OpenAI for I think 3 years but Ananthropic I think he was only there for like six between 6 weeks and 3 months like it wasn't that he wasn't there that long. So it is interesting that he so quickly left and he is treated like he was at anthropic for a long time but he actually like couldn't have had much impact and I would question how much you can even understand about a company in just 6 weeks and maybe it was a little longer.

4:43 I think it's still to be determined. So then this person Parker comes out and this also and I'm only sharing posts that got a lot of you know a lot of views because there there was so many posts around this topic and it says this post looks like a start of a very sophisticated and wellunded PR operation. So this person with minimal followers and no previous activity goes to the Wall Street Journal publishes an exclusive within hours it has tens of thousands of reposts and then according to Grock some of the repost accounts were like a lot of like head of AI policy head of AI futures projects a lot of like people that are pretty big into spreading AI doom narratives across you know the industry and then a lot of if you track down where the funding comes from it's a lot of like people that are funding these AI doom NGO type organizations and this person doesn't really have that much of a resume. So, there's just a lot of if you if you kind of dig into it, it's interesting. It makes you at least think that maybe this wasn't completely organic. There was maybe someone like putting their finger on the scale, so to speak. You know, Bernie Sanders had an AI bill wanting to stop all AI, no AI progress at all. And one of the things in this is you get 20 years in prison if you try to advance AI at all. That's interesting part of the thing. And of course, if you do want all AI to stop, I guess maybe this is reasonable, but also seems a little wild of a claim. My thought is they put out this really like crazy stop everything and then maybe there's a middle ground. And I think maybe that's what they're going through.

6:12 Like here's the extreme scenario. Now we get some real regulation that finds some kind of middle ground. I think that's the idea. It's like you try to tip the scale by going to the extreme. And then this was the big response. So Dario from Anthropic puts out an essay that says we must pace the frontier. I've written a new essay on why the AI industry should slow down with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps.

6:36 You want to talk Obby a little bit about what were those three steps and you know like what is Anthropic committing to to at least start? >> Well before that I'd like to just talk about like history of Daario's views here to go back in time here. So Daario was at OpenAI. You know, one of the first interviews I was researching was him on Lex Freiedman's podcast and that podcast was really cool. Like it was very much optim a lot of optimism and a lot of technical deep dives into AI in general. And there was mention of having governance then. and then from founding Anthropic in like 2021, you know, Anthropic is a PBC, a public benefit corporation. It still is. you know, Anthropic's mission is to create AI in a safe environment. And then, you know, fast forward to 2022, you know, most researchers were doing RHF. And, you know, Anthropic was one of the first research teams to be like, we should take humans out of this loop and have the models critique themselves. but you need to have that in a very ethical way. So, they were like making sure that, you know, this research is happening both at a fast pace, but in a safe way. In 2023, he had his first judicial committee hearing where he did start talking about the misuses of AI. So there are biological threats, cyber security threats and he was advocating for external evaluations and he maintains that today. We're going to get into the the things that he he wants but even back then so it's not like some new thing. So 2023 he said all the problems that could exist and he his his first thing is hey we should have external evaluation on all frontier systems prior to it deploying in in mass quantity. later in 2023. This is where things kind of differ, right? Like he starts thinking about how AI can get poisoned by commercial incentives and you know then they should have this rubric of AI safety levels ASL similar to a biosafety level BSL1 to BSL4 that same type of thing there. Now we're in 2024 that ASL framework has evolved and it's something that is part of anthropics culture. safety of AI, no dooming yet. Everything up until this point I do respect a lot because it's saying there are potential issues. Here are some guardrails and things we can do. But then, you know, this last year, this year, we have obviously gone in exponential rates of growth. And I think that it's moving too fast for the ASL to contain. He went on CBS to talk about was let's say the first doomer take where he went on CBS Sunday morning.

9:13 talked about that you know the growth rate is so fast that you know the safety protocols are not catching up in time and the raw scaling will just outpace the actual safety that can come with it and that leads us to today you know we must pace the frontier the latest thing so a couple things that were established one there needs to be global coordination and that's a big contrarian view for a lot of Americans which is like US needs to own the AI boom but Daario's advocating ating for global coordination. So across many governments should own both the governance and the progression of this. And so if you're a democratic nation, maybe all major labs need to have safety baselines. Second thing is this embedded evaluator concept where there should be third party audits and government government regulation on any AI deployed. And then lastly, if there should be like you know there's not only compute constraints, there's these scaling constraints. People hate data centers, all this stuff. And that is an advocation of pausing development or slowing down. Now, when I go through the timeline like that, I'm not hating on Daario. Like, it is a natural progression of how this thing evolved. I think a lot of people saying that he's a doomer and he sucks and stuff is just because you're hearing what he's saying today. But if you do the research on his how his opinions have changed, it makes sense. So, like I'm not against him or anything. But the problem is, I think in America, we're very just worried about the Chinese lapse. And so any pausing is like telling builders to stop moving and making moves, which I think is what everyone's really tripping about.

10:50 >> I think if you were just to look at what Dario has said and you're completely new to AI, you have not been paying attention other than you know it exists and maybe you've heard about it, seen some ads, this is going to really scare the out of you. If you combine this with the previous topics, right? And of course, Jacob's going on all these shows, Dario's now going on CNN. I mean, everyone is kind of sounding the alarm and so yeah, I think a lot of people are going to be scared. I think Daario's actually coming from a good place. I think he genuinely does care about safety. I think the challenge that I have with Dario is I think he thinks that, you know, it's almost like main character syndrome in in a way where it's like he's the one or they're the ones that can do it and it should be in the hands of it's like the power should be in the hands of a few, which is always a scary thing. If you think about just governments and control in general is just if you want to control it, well then how do we know that you are not corruptible? Typically spreading control is a better way to prevent that kind of thing. It's not perfect. These none of these systems are perfect, but centralizing all the power behind one like government, especially governments, multiple governments, centralized power does seem challenging to pull off and potentially scarier than the alternative where everyone has super intelligence, right? So, I think that's the challenge that I see is I think Daro is coming from the right place, but I think he thinks that he's the guy and only he can do it. He does have the receipts though.

12:08 Like we're going to get into the other people who agree with him, but at least he has the receipts from the beginning and we'll see what the other other dudes are saying. >> Elon says Dario was right. And the funny thing is before this, some people that are really pro Elon flipped sides at this point. They were so anti whatever Daario was saying. Then Elon said it and they're like, "Well, maybe he has a point." It's so funny. So you see the people that like, you know, flipped their position when Elon chimed in. But Elon also has a history of you this is why he funded open AI at the beginning right he was worried that you know dominance by I think Google was a risk if Google controlled all the AI that that's a risk that having one company control everything and so open AI was supposed to be a nonprofit supposed to be open Elon has some of the receipts as well at least as far as like how he he thought about it I think he's always been worried that AI could potentially be negative right and we've all seen the movies right everyone knows what it could potentially do Clement from Hugging Face said, "It's now clear that alignment is critical and won't be solved behind the closed door of a handful of Frontier Labs. So today, we are launching the Open Alignment Initiative led by Tom Wolf, Hugging Face, and asking to be part of the embedded evaluators program that Daario just committed to. So let's make AI safer by making it more transparent. So more people with credibility coming in."

13:23 Sam Alman even came in and said, "I agree with Daario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employeeike access is a great idea and we will do the same. We will have more to share soon. So all you kind of have the big three I would say the big three are all in agreement that this needs to happen.

13:44 And then this came out. I thought this was kind of funny. You can't spell Daario Amado without AI doomer which is just funny that that works out that way. If you rearrange the letters it's kind of funny. Even if this is a very serious topic, it's it's a little humorous. We should we can all we should build a laugh even in times of grave danger potentially. We should build a laugh. And then so David Zach came out with a post and said, "Dario has written that we need to pace the frontier and Sam has agreed. People may be surprised by my response. Go ahead. You guys are the frontier by any reasonable metric.

14:13 Market share, revenue growth, model capability. The two of you have a duopoly on frontier intelligence. You've also claimed the lead is widening because of recursive self-improvement. I don't see what you see in the lab. If the unreleased models are scary enough that you think you should slow down. I support your decision to be responsible. On top of this, he also says, most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product liability exposure if your products enable a truly damaging cyber attack. The market already punishes models that behave in unpredictable or unauthorized ways.

14:43 After the hugging face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want. There's a bit more. It talks about Bernie Sanders's take and basically says you should go ahead and do it and if you do, you'll buy goodwill for the next conversation. If you don't, we'll know this was just another bid for regulatory capture or an election season SC up. So that of course got a ton of views as well. A lot of people resonated with that. meanwhile, this came out breaking. China's ZAI has raised 5 billion. The interesting part is where the money is going. They're saying 60% of the net proceeds will fund next generation GLM models in what ZAI calls its fully self-training system. Sounds like precursive self-improvement to me.

15:30 So the question is if China's already investing to do it, and they're investing a lot, can we actually pace the frontier? Maybe. But I think that's why a lot of people are worried because you can't actually control what everyone does. We can only control what we do. Gavin Baker came out and said, "While 24 hours and a lot of different proposals have been made, the only tangible new fact is OpenAI and Enthropic are going to have embedded thirdparty evaluators.

15:52 We talked about that." Dario floating meter as a possibility for one of those third party eval evaluators. Yan Lun jumps in and says, "Pessimists arc is right. Dario was already claiming that GPT2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now." somewhat related to pacing the frontier. You know, Tibo from OpenAI said that they're pausing signups to their 200 pro plan. Mostly I think this is compute related, but it the question is like are they just trying to improve or make sure that there's enough compute to go around? Are they running into scaling problems? Is this, you know, one of the reasons they want to like slow down the frontier a bit? Is there is it connected or not? I guess that's, you know, a bigger question. I'm not sure. What do you think? There is definitely some strain on capacity, but also like I mean this came out before all the doomer stuff or at the same time.

16:43 So, it's kind of it's kind of weird like I guess we should have all bought a couple more subscriptions cuz now we can't. But there's definitely something going on with capacity. Fable had the same issue, right? There was so much demand that they had to knock it off the subscription. This is unfortunate. I think there's a lot of backlash in the developer community, but what do you expect? you know, we've been subsidized for so long. >> Mark Zuckerberg chimes in, says AI development needs to move faster. He decided he's taken the opposite take of the other the big three, so to speak. If there's a Mount Rushmore of like AI, like frontier model lab figures, you know, Mark Zuckerberg is like not quite on the monument, but he's he's trying to get on it, I guess. And so, he just says, "No, we need to keep moving faster." So, I thought that was pretty funny. How do we want to summarize this?

17:26 What is your overall takes in should we be pacing the frontier? you know, for people like us, we're builders, we're out here using the things. Does this affect us? How should we respond when my mom or my grandma says, "I saw this CNN article. Is AI going to kill us all?" Like, what do you think? I'll I'll give my take. I will caveat this by saying, "We're not in the labs, right? We don't know what they've seen. We only know what we see in the outside world."

17:48 >> So, funny enough, I went on a date last week and when we talked about my job, first question she asked me was like, "Do you think the AI do you think AI is going to kill us?" And she's not in in tech at all. So like that is the general sentiment because the news comes from these, you know, the CNN's and everything. So from that perspective, it it's kind of like we're not at the point of this extinction level thing and a lot of people are worried about it. So I like just feel bad because a lot of people get anxiety from these news articles and things and it's just not an it's not an anxious point today. For builders though, you know, if the models stopped getting better, our life doesn't really change that much cuz we are building with the models at every part of its like growth. So if it stopped today, you know, we may not get anything better, but we're already building harnesses and the tooling around it to try to make it good enough. So I don't really think it affects us as much, you know.

18:44 >> Yeah, I do agree with that. If everyone decided, hey, we're going to define what the frontier is and we're never going to pass it. We we basically haven't figured out all the things we can even do with it today. There's a a lot that could still be done. Now, the challenge is I don't think it's actually going to happen. I don't think people are going to really slow down. I would say, you know, we've seen the hugging face incident. And if you really dig in, there's some interesting things there, but also I kind of would have expected that outcome given what the models have been trained on, what they would do. I think if we saw like a dozen hugging face level incidents, I would start to get more worried. I think only seeing you know a few of these that have come out I do think that the gap from going from there to like full scale AI takeover controlling everything all these external systems and of course there's no like perfect off switch to this but it's all comput and data centers right I don't think that we're at this point where this whole thing is going to be taken over in an uncontrollable way in the next couple years now maybe in the next decade could it happen I'm not an expert it's possible the models that they are seeing must be so much better than the models we have if they're worried about about it because I can't get my model to do a simple task sometimes. You know, you're just like missing obvious things, but yet I I think this model is going to fullon take over the world and kill all humanity. That seems skeptical to me, but again, maybe we don't see all the models that they have cooking in the labs.

20:01 >> So, I guess our tip to everyone is invest in a bunker and then you'll be fine either way. >> Yeah, I I will say I'm not investing in a bunker. I'm not overly concerned yet. I think we should be careful like this stuff is dangerous. We should have some safety guard rails. I think I am worried about too much overregulation in that like one controlling body controls all progress and everything has to go through that because I think if we slow down too much you become like some of these other overly regulated countries who can't really be competitive. And then I think if you're in the US you'd be concerned that well is that actually going to slow down other countries that are doing the same things and don't we want the best intelligence to protect ourselves potentially? I'm selling pre-sale tickets 2036 to Oby's Bunker.

20:45 You can Venmo me, Cash App, you know, but if you're really worried about it, I got you. >> Dude, did you subscribe? >> Dude, I host the show. Did you subscribe? >> Did you subscribe? >> Subscribe to Agents Hour every Monday, noon Pacific. >> Let's get off the Doom topic, even though this next one's kind of related. Safety and practice. Enthropic published a detailed threat intelligence report covers how people misused claw for cyber attacks, influence operations, surveillance, biology, and building weapons and how they found and stopped them. Zihad, I think is how you pronounce that, had a pretty good take on at least a summary of all the things that were in the report. If you don't want to read the full report, here's some TLDDR from it. There's a suspected Russian state linked group using Clawude across fishing intrusion, data theft, and ma malware development, including rebuilding malware after security products detected it. There's shiny hunters linked hackers used AI agents to scan 1.8 million Android apps for exposed secrets and help run breaches across multiple companies. A China based group built an automated exploit setup that could research vulnerabilities, build offensive tools, and keep working against target with little to no supervision. One consultant used it to build Lacana 360 for Mali's intelligence service, a system designed to monitor roughly 25 million SIM cards across the country, including calls, messages, voice interception, watch lists, and automated intelligence dossas. A whole bunch of things. There's a Yemen group, another Russia attack. I think they talk about distillation from Kimmy, from Deep Seek, Zai. They talk about all the different how Moonshot was basically distilled from a lot of these like Chinese models were distilled from Frontier and that is one of their one of Dario's takes and one of Enthropic's takes is if we're moving the intelligence forward and then we're getting distilled by these Chinese companies we're actually helping them move faster. so which I can agree with like that.

22:34 >> I think the the wildest thing was anthropic claims that moonshot deepseek were routing requests to claude instead of their own model for some users without telling them which I think people have said that's that's not true or there is true whatever but that is quite interesting. And then if you see all these like we you know weapons six weapons programs are using anthropic etc. like all these AI personas on dating apps. people trying to do these griffs with old people. Yeah, dude. Safety is important, you know. And I think Anthropic's looking at this list and saying we don't have an ASL to protect us from this right now. So, we probably should slow down because we have no guardrails for this.

23:16 >> OpenAI on September 10th said, "GBT Live 1 is now available in the API. bring Chat GBT's natural back and forth to your app with voice agents that listen while they speak and work with the models and harness you choose. So now you can get real-time voice right directly in your app. And I think this is something that a lot of people have wanted for a while because they interacted with chat GBT and the voice was pretty good comparably to a lot of things you could build yourself. And so I think a lot of developers will like that they can maybe just use their voice mode directly. Cursor introduced projects. This is from September 10th.

23:50 It's a new way of working in cursor. Rather than creating a chat for every task, you work with a coordinator agent in a single persistent thread. Like bot, your agent is always on. Proactively manage is work with sub aents and improves over time. On September 10th, we also had DeepSeek V4.1 Flash. Smarter, faster, more efficient. So, it's the smallest model in their new architecture family. And >> it's good. >> Yeah, it's pretty good. If you look >> If you don't get rate limited, it's great.

24:17 >> Yeah. For And for how small it is, too. Right. It's like impressive. It's impressive that you can get such good levels of intelligence on these smaller models, >> but you don't know if it's hitting Claude or not. >> Yeah, that that is true. Maybe it's just, you know, it's it's like the Scooby-Doo thing. You pull it off. You pull off the hood and it's like, "Oh, it's just actually Claude." Maybe that's why Cognition released Devon voice. Your favorite AI software engineer just got a landline. So, you say it, Devon ships it. So, it's powered by GPT live and our new 2 model. So, there you go. This is tangibly AI related. If you remember back, Tailwind had a lot of they had some layoffs because no one was going to the Tailwind docs anymore because AI was just reading the docs for you, right?

24:56 This was probably 18 months ago or so, a long time ago at this point. And so it's kind of full circle. I feels like Tailwind is kind of winding down, so to speak, and becoming part of Shopify because a lot of people use Tailwind and don't even know it these days, which is, you know, unfortunate for the Tailwind team because they built something that everyone uses but no one pays for. >> At least they have a home now to keep continue working on it.

25:16 >> Yeah. So congrats to them and I think that it is good because again so many people are building web apps these these days so many agents are building web apps these days and a lot of them are using Tailwind under the hood. All right this last one what is this Obby you want to tell us about it >> the death of agent plugins maybe in the latest MCP spec we talked about MCP's v2 this is another feature which is super cool because you can define extensions that load agent skills alongside the MCP server. So if you already have an MCP server, you can then bundle your skills within the server and then you can grab skill metadata. You can then use it as like you know how skills work is agents have to load skills into context to then use them.

25:58 Now you can do that through MCP. So why ship an agent plugin when you're already shipping MCP servers? Which also makes me think why Anthropic was not involved with agent plugins? And I would agree that this is a way better delivery mechanism in my opinion. I mean, we will see. Now we have like two standards trying to do some of the same things. We'll see who who ends up in >> MCP is having revitalization right now.

26:22 And this is a perfect like add some more fuel to that fire. >> A lot of people kind of lost sight of MCP even though enterprises didn't. It's still very well very widely used across the enterprises that I that I talked to. But I think people, you know, it was very trendy and then it kind of diminished and people, you know, were saying it's not actually that good of a spec. But, you know, with some of the new stuff they've released in the 2 2.0 spec, plus some of this, it's having a bit of a re revitalization. I think, you know, MCP is back, at least as far as like hype terms, as we kind of alluded to earlier. There's all these hype terms, and I think MCP is it's still it's still got a little hype behind it.

26:55 But that is it. That is the show today. Thank you for tuning in to Agents Hour. We do this every week. We talk about the news. We give our takes. Sometimes we're right, sometimes we're wrong. You be the judge. We'll let history decide. but you follow us at MRAA if you want to see what MRA is talking about, all the things we're shipping. We ship a lot. Check us out on YouTube so you don't miss any episodes. Please give us that subscribe, that like that helps other people find the show. We appreciate that. I'm Shane Thomas, SM Thomas 3 onx.

27:23 That's Abby at Abby on X as well. And with that, I guess we will see you next week on Agents Hour. Peace. See you.