transcribe

Spend the Lead on Safety: Biden AI Advisors Ben Buchanan & Tantum Collins on the Fate of the World

Cognitive Revolution "How AI Changes Everything" · 1h 48m · transcribed 1d ago
More from Cognitive Revolution "How AI Changes Everything" Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to the Cognitive Revolution

What is the focus of today's episode?

The episode features guests identified as China Hawks from the Biden administration, discussing their role in AI policy and national security.

  • The episode explores the intersection of AI technology and national security.
  • Guests Ben Buchanan and Tantam Collins discuss their new book, 'The Bitter Struggle'.
  • The conversation will cover the implications of AI safety and U.S.-China relations.
# 15:29

Ad for Claude by Anthropic

What capabilities does Claude offer?

Claude is presented as an AI collaborator that enhances problem-solving by understanding workflows and working alongside users.

  • Claude can assist in various tasks, from coding to business strategy.
  • It operates continuously in the cloud, even when the user's device is off.
  • The ad emphasizes Claude's role in extending human thinking.
# 30:59

Utilizing AI Agents in Production

How are AI agents integrated into the show's production?

AI agents assist in researching guests and managing PR pitches, streamlining the production process.

  • AI agents perform specific tasks to enhance efficiency in content creation.
  • The setup for using AI agents is user-friendly and adaptable to business needs.
  • Collaboration among team members is facilitated through shared AI capabilities.
# 46:28

Biden Administration's Approach to China

What was the response to U.S. actions regarding chip controls?

The Biden administration's actions were carefully calibrated, resulting in minimal retaliation from China compared to previous administrations.

  • The administration aimed to avoid backlash while implementing chip controls.
  • There was a notable difference in China's response to tariffs versus chip control measures.
  • The discussion highlights the complexities of U.S.-China relations.
# 61:58

Engagement with Competitors

What is the importance of engagement in international relations?

Engagement is seen as essential for achieving positive outcomes, rather than a binary choice of engagement or non-engagement.

  • The conversation emphasizes the need for nuanced engagement strategies.
  • Understanding the spectrum of engagement can lead to better diplomatic relations.
  • The importance of dialogue in addressing international challenges is highlighted.
# 77:28

Concerns Over AI Safety and Regulation

What are the risks associated with pausing AI development?

A time-bound pause could lead to a 'compute overhang', resulting in a sudden leap in capabilities when development resumes.

  • Regulating AI development requires careful consideration of safety and security standards.
  • Pausing development without clear safety thresholds may have unintended consequences.
  • The discussion calls for a balanced approach to AI regulation.
# 92:57

Risks of AI Misuse and Misalignment

What are the potential risks associated with AI and bio technologies?

The conversation addresses the risks of misuse and misalignment in AI, particularly in the context of cyber attacks and bioengineering.

  • Emerging threats from AI could include cyber attacks on critical infrastructure.
  • The discussion highlights the need for preparedness against potential AI-related emergencies.
  • Understanding the dual risks of misuse and misalignment is crucial for future AI governance.

Transcript

Speaker 1

0:00 Hello, and welcome back to the Cognitive Revolution. You're hearing my AI voice clone today. I recorded this conversation from the car on a recent family road trip, and though I'm home now, I got sick and I'd rather not delay the episode. My guests today say they are probably correctly identified as China Hawks from inside the Biden White House. They were part of the small group that pushed to cut China off from the most advanced AI chips yet.

Speaker 1

0:27 Both were also in the room when the United States sat down with the Chinese government to talk about AI safety. How those two things fit together is the heart of this conversation. They are Ben Buchanan and Tantam Collins, who goes by Teddy, and their new book is the Bitter Struggle. It chronicles how they made the case in government that AI would become central to national security and works through the hardest dilemmas it creates. And while they insist that democracies must lead, they also write that no single nation can manage this alone.

Speaker 1

0:59 Their theory of the case is this. Because the US and its allies control the supply chain for advanced chips, we can deny China computing power and build a lead without racing recklessly ourselves. Then, in the phrase they used in government, we spend that lead on safety. By Ben's estimate, we still have something like a hundred to one advantage in producing computing power and probably 15 to one in access to it. So the lead exists. My question is whether we know how to spend it.

Speaker 1

1:30 It's been 10 years since OpenAI showed a boat racing agent spinning in circles, crashing over and over to rack up points, and we still don't have a good solution to reward hacking. I also pushed on the better US than them framing. In some ways, the US is starting to look more like China in exactly the ways we say are bad. Teddy's response is that democracies have corrective mechanisms that autocracies lack, and that over the long run, concentrated power tends to end badly.

Speaker 1

1:57 I think that's pretty much indisputable. But I still worry about how our strategy looks to China. When I visited this summer, people told me the US is just trying to hold them down. What I call the Dario Plan, from Dario Amade's essay Machines of Loving Grace, is to build a big coalition, box China out, and eventually make them an offer they can't refuse from Beijing. I think that could easily read as a second century of humiliation.

Speaker 1

2:26 Ben made a point of saying, that's Dario's essay, not theirs. The model he prefers is America at its best after World War II, building institutions that seeded some of its own power and that other countries, China included could join without regime change. That struck me as notable, given the close ties I understand Anthropic to have had with the Biden administration and that Ben now advises Anthropic, my own view is that the real aliens here are the AIs, not the Chinese.

Speaker 1

2:57 I'd rather take my chances on cooperation than try to shoot the gap, building just enough of a lead and solving safety just in time. My shorthand for the deal I'd want is chips for controls. Ben's reply is that they tried. When the two sides met in Geneva in May of 2024, by his account, the Chinese delegation treated talk of shared AI risk as some kind of plot. Chips for controls, he says, is the end game.

Speaker 1

3:28 What they heard was sell us chips and we can talk about the rest later. Then he turned the question on me. Would I sell China chips as a precondition for talks? It's a fair question and you can judge how well I answered it. Still, I have to take their experience on board. They tried and found little interest. That sucks. But I suspect it might be different now, or very soon will be. Just as people who wrote off AI based on the original chatgpt have no idea what they're missing, we should be very open to the possibility that the Chinese government will update its worldview and priorities in light of new developments.

Speaker 1

4:07 Finally, Teddy lays out what the book calls the building bitter temptation. Even if we get everything else right, AI could make centralized administration so effective the democracies slowly creep toward autocracy on their own. I share his obsession with that question, and I've invited him back for a full episode. In the end, I heard more moderation than their reputation would suggest. Their chip policy is hawkish, but they really do want the two countries to work together where possible for the common good.

Speaker 1

4:37 They insist that China is no excuse to race ahead dangerously and that if they were in power today, they would keep trying to reach the Chinese government. I was heartened to hear that, and I hope the current administration follows their lead with that. I hope you enjoy this conversation about COMPUTE China and what it would really take to spend a lead on safety with Ben Buchanan and Tantum Collins, co authors of the Bitter Struggle

Speaker 2

5:04 Ben Buchanan, former White House Special Advisor on AI and now professor at Johns Hopkins School of Advanced International Studies and Tandem Collins, former Director for Technology and National Security on the NSC at the Biden White House and now co founder and CEO of Inherent. You guys are now the co authors too, of the new book the Bitter Struggle. Welcome to the Cognitive Revolution.

Speaker 3

5:26 Thanks for Asking for

Speaker 4

5:27 having us.

Speaker 2

5:28 I'm really excited for this conversation. The book is excellent. I did, of course, read the whole thing. I do my homework around here, and I think you guys cover really a. A full kind of gamut of all the main issues in AI. I think it's a. It's a great book. Not only that tells the story of your personal history and, you know, the roles you guys played in the Biden administration, but also just lays out for people, especially if they're kind of new to this at a high level in a very kind of digestible way.

Speaker 2

5:55 Here are all the big issues that we are facing and why this AI thing is such a conundrum. I think people should, like, give this to others in their lives who need that sort of primer. I would flatter our audience to say that they largely don't. For starters, one big takeaway, I would say, from the book is how much of the current situation a select group of people saw coming years in advance. And, you know, there were, of course, scaling laws, and some people have been willing, and yourselves included, to extrapolate these straight power lines on graphs.

Speaker 2

6:29 How did you get to that conviction? Like, what's the backstory of how you really locked in and said, like, this really is where the world is going and we're going to act on that belief?

Speaker 3

6:37 Well, we started studying AI together. We were in grad school in different. In different programs, but this is as far back as probably 2015 or so, and we'd often meet and talk about AI developments and so forth before AlphaGo. Teddy then went to work at DeepMind in the AlphaGo period. But. But the scaling law for me, I think crystallized in a series of events between 2018 or so and 2020. I read the Dario essay, the Big Blob of Compute, which certainly had an impact on me.

Speaker 3

7:09 Of course, the Sutton essay in 2019, the Bitter Lesson, which provides the inspiration for the title. And then I think the thing that convinced me those two essays were right was the release of GPT3 in May of 2020. And that's the technical story. I think what the book is trying to do, including for folks who are technical, is convert that technical story or show that technical story through the prism of the government. And how do you take these technical convictions and then turn them into government policy?

Speaker 3

7:41 And how the export controls are in the news now with Jensen's latest scandals and so forth. Where do those come from? What's the backstory for the government actually making policy on this? And at least for me, the turning point was GPT3, as I said. And then I started to try and make this pivot to tell the policy world this was happening. I wrote a piece in Foreign affairs the summer of 2020 called the US's AI competition all Wrong, which was about the rise of computing power and how important it would be to national security.

Speaker 3

8:07 And there was a small group of us who went into the Biden administration to put this into practice. And Teddy should speak for himself here, but he was absolutely a key part of that as well.

Speaker 5

8:17 I mean, maybe just one thing sort of from, I guess from my end, is that it was very interesting during this period of the late 2010s being at DeepMind, but having come from a non AI background, and as a result most of my friends not being in the AI world, and it felt sort of maybe almost like severance in that it's like I'd be at work, I'd be surrounded by all of these incredibly smart, incredibly convincing people who had, by and large, already priced in sort of the world that we live in now and to whom, give or take, this trajectory seemed inevitable.

Speaker 5

8:45 And I found that very compelling. And then I'd go back out into sort of the real world, as it were. I would talk to the rest of my friends, most of whom knew nothing about AI, and they would say, that's completely insane. You work at a cult. And I'd be like, oh, no, no, that's kind of compelling too. Like, you know, the historical priors here suggests that maybe the wisdom of the crowds is correct. Now, as it happens, the latter group has now tacked much closer to the former group than the other way around.

Speaker 5

9:11 And I think it was very interesting. Ben and I have been friends, as he mentioned, for a long time. We've been talking about these issues for a long time from the perspective of the tech industry and from the perspective of the policy world. And I think it became increasingly clear to us, and we write about this a bit in the book, that at the start we were like, these ideas are very interesting, but are they also a bit crazy?

Speaker 5

9:32 Could this really be true? We eventually became convinced because of some of the sort of trends that Ben mentioned, and I'll say it's a huge credit to Ben and some of our other colleagues, that from the earliest days of the Biden administration, even at a time when, and it's hard to imagine this now because now everything seems inevitable in retrospect, but back in 2021, especially in D.C. this was, I would say, like having bullish views on AI.

Speaker 5

9:58 Was fairly heretical. Ben and several of our other colleagues really had the conviction to kind of put their reputations on the line within the White House and say, like, this is actually the most important thing for us to do. And it was really a pleasure being a part of that.

Speaker 2

10:15 So on this topic of inevitability, I think on the one hand it's like, safe to say Kurzweil was right. In the presence of web scale data, web scale compute, we're going to get AI. It feels to me like that much is inevitable. But then there's a lot of things that seem like they might be a lot more contingent. Like the transformer itself, you know, is probably just one of a lot of architectures that could work. And then even in terms of, like, things like this week's Jensen fall from grace, you know, we have these moments where individuals make decisions, and it's like sometimes they may be super consequential.

Speaker 2

10:55 How much is going to be driven by fundamentals and how much do these episodic moments where all of a sudden Nvidia has a new CEO where they. Which they really had no plan for, it seems like, from one week to the next, like, how much do those sorts of moments matter? You know, kind of factually, if Sam Altman had not been reinstated by the board in that key moment, you know, how much would those things have mattered?

Speaker 2

11:17 I guess, you know, I'm very confused by this. I'm very interested in your perspective.

Speaker 3

11:22 It's one that we talked about, actually a shocking amount in the government. Teddy, I'm sure you remember some of these conversations. And I think it's ironic because I came to the government with such a strong belief in these structural forces and the scaling law, the better lesson, all of that. And I still do believe, I mean, technologically, I still, of course, believe those things, but I think after four years of trying to make policy, I left the government saying, like, wow, the role of individuals here is just so striking.

Speaker 3

11:49 Not just the five or six of us who had this conviction in 2021, but convincing people like Jake Sullivan, who I think you've had on the show, and Bruce Reed, the White House Deputy Chief of Staff and ultimately the president, that this was important to take seriously and to do. I mean, it really just comes down at some level, US Policy, the. The, you know, the direction this giant battleship that is the United States government heads.

Speaker 3

12:13 It really does come down at the end of the day to what do a couple people think? And certainly on issues that are novel, at least I think it Comes down to that. And of course, the semiconductor industry. We'll see if Nvidia is only down to Jensen or not. But I think I came in believing in structural forces. Of course, I still do. But I have more appreciation and honestly, probably a fair amount of gratitude for the individual people who were willing to give us a chance to make our case in 2021 and 2022, when running the world was time consuming enough for them.

Speaker 3

12:46 And this was still a speculative issue.

Speaker 5

12:48 One might be able to kind of differentiate between the arrival of a technology and what its impact is. And I think that the former feels sort of more inevitable than latter. We can imagine historical parallels here, like the space race or the creation of nuclear weapons. I think, give or take, it was inevitable that humanity would at some point in the second half of the 20th century, end up getting to space. I think it was inevitable that people would inevitably, at some point during the 20th century, create nuclear weapons.

Speaker 5

13:17 But would those weapons be created first by the United States or Nazi Germany? Hugely consequential for the world. And that I think one could argue there are structural forces there, obviously. Right. Like the US Just from a material and economic perspective, was doing way better. Germany had ejected a lot of great scientists. But I don't think it was completely inevitable. I think we could have lived in a world where that played out differently. Likewise, you look at the trajectory of the space race, and those things I think can be hugely, hugely consequential.

Speaker 5

13:45 For like, does this technology have unbalanced good or bad effects? And whether you're looking at it in terms of within government or in terms of the people who lead the companies and the decisions they make within industry, I think that those are the areas where it feels like you have the biggest potential. Sort of counterfactual discontinuities in terms of who's making their decisions.

Speaker 3

14:03 There's a scene at the end of our book where we are sort of trying to meditate on this question. And we say, almost to the point of your question, we say, this is about inevitability and choice. And the scene that follows that is we tell the story of Alexander Sachs, who was the one who delivered the Einstein Szilard letter to President Roosevelt that ultimately kicked off the atomic age in the United States. And sometimes folks said to me, like, do you think, you know, when you briefing the President in the Oval Office, did you think you were Einstein or Szilard with this letter?

Speaker 3

14:37 And the answer is no. I think insofar as we can substitute ourselves with anything, we and our colleagues are much closer to Alexander Sacks and the translator between these two worlds. And one way of looking at that translation is it was our job to find the structural forces, figure out what we believed about the structural forces, and then tee up the choice or advocate for the choice from the people who actually have the power. And that's probably how I resolved this dilemma at the heart of your question.

Speaker 3

15:04 But it's certainly, I think, fundamental to the work we are trying to do in the government.

Speaker 5

15:08 And as a general note, you know, we don't go around telling people we we think of ourselves as Einstein.

Speaker 3

15:13 Definitely not. It

Speaker 5

15:14 doesn't go down so well for some reason.

Speaker 3

15:16 Well, usually they say, who's Alexander Sachs? They'll say, oh, you're David Sachs. You're like, no, actually very, very much the opposite of David.

Speaker 5

15:25 Hey. We'll continue our interview in a moment after a word

Speaker 1

15:28 from our sponsors. Today's episode is brought to you by Anthropic makers of Claude. To show off Claude's latest cap capabilities, I asked Claude to take the lead in making this ad. Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation. You're watching my job. Give feedback and decide.

Speaker 1

15:58 Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude code routines, it keeps working in the cloud even after you close your laptop. For problems worth solving, get started with Claude at Claude AI tcr.

Speaker 1

16:30 That's Claude. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude AI TCR. This episode of the Cognitive Revolution is brought to you by Outsystems, the leading agentix systems platform. Outsystems is helping their customers build, modernize and operate enterprise systems starting from any coding tool in a govern agentic engineering model.

Speaker 1

17:04 With Outsystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem. Accelerate impact with industry proven agentic

Speaker 2

17:13 solutions that are pre built, governed and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control. Which is why thousands of enterprises worldwide trust unsystems for their mission critical apps. Teams any size can use Outsystems to build, deploy and manage AI apps and agents quickly and cost effectively without compromising reliability and security. With Outsystems you can accelerate ideas from concept to completion.

Speaker 2

17:44 It's the leading agentic systems platform that is unified, agile and enterprise proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. Outsystems own your agentic future. Learn more@outsystems.com TCR that's outsystems.com TCR it does sound like though, if you guys hadn't taken these roles in government when you did, we're probably on a quite different timeline, at least with respect to export controls.

Speaker 2

18:19 Like how different do you think the world is if you know, to use like a, a baseball analogy, if we just had sort of a replacement player in the role who wasn't so bitter lesson pilled as you guys were.

Speaker 5

18:32 I want to underscore that a lot of the force here and a lot of the original thinking came from some of our colleagues. Tarun Chhabra, Jason Matheny, Chris McGuire, Saif Khan, who I think would have have done, you know, similar work and advocated similar work. Regardless of if we've been there. I mean, I hope we helped lead to more calibrated policy and sort of push for better outcomes. But there are other people who are critical pieces of this too.

Speaker 3

19:03 I agree with that. I think if you took out the five of us, or the seven of us, depending on how you want to count that we're really pushing this, then you probably do get a pretty different story. I don't know that the two of us are the heroes here. I don't think. I don't think there are heroes here,

Speaker 2

19:18 but it's small enough for a movie cast. If we're thinking about it in cinematic terms, it's like it's not that big of a.

Speaker 5

19:24 Yeah. The most boring movie of all time. It's like a lot of. It's like a lot of very like, long bureaucratic discussions.

Speaker 2

19:31 Yeah. Well, I just went to Bletchley park and.

Speaker 5

19:34 Oh, nice.

Speaker 2

19:35 You know, they managed to make a movie out of that. So I think one day there will likely be a movie about this as well. I have to. If we're all here to watch it, I expect one will be made.

Speaker 3

19:45 I just hope they play with the hair I had at the beginning of the administration and not the total lack of hair I have after four years of serving in the government.

Speaker 2

19:51 Yeah, I hear it's pretty taxing. Okay. Further on contingency, one of the big observations that you make in the book, of course, is that the AI phenomenon is a private sector phenomenon. It's being led by companies. The government really didn't have a formative role in the way that it did in so many other technologies over the last hundred years. That means that the nature of these companies is also kind of a big factor, I would say, in determining where we are, where we're going to go.

Speaker 2

20:21 I've always kind of felt like the companies are, in a sense, ideological. Like the idea that we're going to make our mission to create something that's better than humans at all things is sort of, you know, that that's not like an assumption that everybody comes to, you know, the, the table with. And yet it has, like, really formed so much of not just how we think about it, but also like the AIs that we have. Maybe that was inevitable too, I'm not entirely sure, but certainly they were charging at that from an early phase in their development before it was at all clear that they could actually achieve it.

Speaker 2

20:54 Well, how do you characterize the companies? How do you think about the companies? I'm particularly interested in Dario's China Hawkism, which I have not ever been able to get a good explanation for. But, like, you know, what do you think are the sort of idiosyncrasies, Dario and otherwise, that are kind of most consequential and maybe underappreciated in terms of shaping the current moment?

Speaker 3

21:17 Teddy, you want to start here?

Speaker 5

21:19 This maybe date me a little bit, but I remember sort of being in university in the late 2000s, early 2010s, at a period when it felt like as you were looking at potential careers, every company had been told that they needed to obfuscate their actual goal, which was usually pretty prosaic, and it was about making money in some way, and dress it up in some kind of, you know, quote unquote, mission driven language.

Speaker 5

21:49 And so you'd have these kind of inspiring, very vague, essentially slogans that were sort of just rationalizations of a thing that was a way to make money. Broadly speaking, this is a huge generalization. I think the AI companies are unusual sort of the big labs today in that I think you're correct, Nathan. I think that they are actually, by and large founded and driven by a commitment to, in some cases a utopian view, in some cases a fear of a dystopian set of predictions about the world, but something that is, I think, obviously not wholly separable from.

Speaker 5

22:36 I don't want to sound sort of too disconnected from reality, like these companies are trying to make money. They are worth a lot of money. And many people it's totally reasonable to disagree about. Are they effectively pursuing their visions of utopia? And are their vision something that most people would actually describe as like, being that good? But the bottom line is, in my experience, sort of interacting with the leadership as well as the sort of rank and file scientists and engineers at these companies.

Speaker 5

23:02 Yeah, mostly they are motivated by sort of a vision of what this could mean for humanity in this broader systematic way. I think that's true.

Speaker 3

23:12 I think an interesting part of that vision is you get different takes from different folks in the ecosystem on how much that vision depends on geopolitical realities. And I think at its best, that vision articulates a view in which there's radical abundance. Everyone's, you know, Sam has some line like everyone in the future will be better off than even the best people today. And you have world peace and, and all of that.

Speaker 3

23:42 At its worst, it can. It can be kind of hopelessly naive about competition and geopolitical competition in particular. I think it. I think I read some quote that, you know, in the early days of the radio, people were speculating, maybe even some of the radio engineers, the radio inventors were speculating that this would make war obsolete because people could understand each other and communicate around the world. And why would you fight if that's the case? And of course, what happens is radio becomes a tool of war right away.

Speaker 3

24:06 So I think, you know, our job is on the National Security Council and to be a little bit more hard nosed about this and defend American interests and what we saw as democratic interests around the world. I think that's. We sort of lived in the middle of those two of saying, like, it's probably not going to be this idyllic place that you and your in your best aspirations want it to be. And because the technology's coming from the private sector, how do we from the government side bring it in and use it in a way that's consistent with our values, but also advances democratic interests and so forth.

Speaker 3

24:35 That's what the book's about, that sort of struggle from inside the government. But I think it's also something we wrestled with very personal, because we do think, I think Tay and I do believe that technology can make lives better, but we are probably a little bit more tempered in that belief than some of the most idealistic folks at some of the companies. Not all.

Speaker 5

24:51 One way of thinking about it is what is the maximal sustainable level of idealism? And I think that finding that sweet spot is a perennial policy question. Is especially relevant when it comes to AI. And I think often, I mean, Ben, you can say if this is consistent with you. I feel like often within the government context I was kind of like the crazy idealist one, sort of laying out schemes for how can this lead to world peace and so on.

Speaker 5

25:20 And I would say in the context of engaging with my colleagues when I was at DeepMind and with my friends in the academic and industry space subsequently, I was much more sort of relative to them on the don't be naive. There are real difficult trade offs here, which we write about a lot in the book that we have to take into account at sort of the realist geopolitical level. And it can be really hard when bridging these two worlds to find that right balance.

Speaker 5

25:45 And in some ways this I think has its roots in some sort of age old sort of game theoretic properties of like how can you. Is there a way out of some of these, let's say, repeated prisoner dilemma things? Toby Ord has a nice way of thinking about this in terms of actually maybe there are certain strategies in those types of settings that might challenge the realist assumptions of the other player in a way that then leads you to like a better equilibrium.

Speaker 5

26:08 And so there is some leap that you can do to get to some level of sort of idealism. But if you stretch it too far, then maybe it's hopelessly naive. And finding out exactly where that balance is, I think is very difficult and it's more out than science and something we thought about a lot in government.

Speaker 2

26:23 So maybe just lay that out for me. How would you describe your worldview starting at a high level? I mean, you know, we can get into all kinds of detailed policy questions, but obviously, you know, at the heart of the book and the bitter struggle itself is the idea that we have a system of government and a way of life that we think is good. And there's at least some worry that it may be challenged by the technology itself.

Speaker 2

26:48 It may be challenged by other countries with other systems of government should they get an advantage in the technology relative to us. You know, those are neutral terms. What are your kind of more colorful terms that you know, really motivate all of the actions that you tried to take in government in terms of the geopolitical outlook?

Speaker 5

27:07 Something I'd say is very important here and it's easy to lose track of in the context of especially, let's say a lot of news stories for the past week. I think that like the race framing can dominate and that can make things Seem very, very zero sum. You know, I think Ben and I both believe very strongly that the most significant implications of this technology are humanity wide, both on the risk side. And I think there has been a lot, there are a lot of really bad outcomes there which write about in the book, which are not just about some sort of horse race and capabilities between countries, but things that would be clearly net bad for just about everyone that we all have an interest in avoiding.

Speaker 5

27:46 And on the opportunity side, I mean, you know, I don't need to. I'm sure your listeners will be very familiar with this. I'm sure a lot of people have read, for instance, you know, well, a lot of the essays out there on all of the wonderful things that this technology could bring us in terms of scientific discovery and curing disease and so on. I don't need to list all of that. I think that that accounts for the overwhelming majority of the potential impact of AI.

Speaker 5

28:06 But in order to sort of navigate this trajectory of avoiding those clearly bad things and getting the clearly good things, you do have to grapple with some of these more hard nosed geopolitical realities. And so from my perspective, pricing that in and making strategy that is conditional on a realistic appraisal of what the stakes are and what might happen if you end up in a race where the margins are too thin, or if you end up in a situation where autocracies have a lead, I think it significantly increases the likelihood of being locked into one of these bad equilibria.

Speaker 5

28:40 And so when we write about these three central struggles in the book, and the main goal there is not to say we have to win because winning is good, but is to say we want to think about sequencing these actions so that we can produce outcomes that get the things that are clearly good for the world and avoid things that are clearly bad for the world.

Speaker 3

29:02 I agree. I think we could be a little more hard nosed about it. And the inspiration here is a speech that Kennedy gives in 1962 at Rice University. President Biden actually had this speech on the wall of his private study just off the Oval Office. And it resonated with him. I think for a lot of reasons probably not related to AI, resonated with me because it, I think, is the best insight into the AI policy we were trying to build.

Speaker 3

29:29 In this speech, Kennedy goes and says we're going to go to the moon and back within a decade. Everyone knows that part. Most people don't know. The last part of the speech where Kennedy makes the case to the American people for why space matters. And he says something like we don't know whether space will be good or ill for humanity, but only if we have preeminence can we help decide. The vows of this nation demand that we must be first, and therefore we shall be first.

Speaker 3

29:55 And you can sort of dial up the rah rah America as much or as little as you'd want. But I do think in doing the geopolitical calculation that Teddy was just describing, what we were trying to do was say, well, we think this goes better if America and really democracies have preeminence in this technology. And that worldview leads us to a policy approach which says because of the compute supply chain chokehold the United States and its allies have on China, we can deny China computing power, build a lead over them without being unduly accelerationist ourself.

Speaker 3

30:31 And then the phrase we used to use is spend that lead on safety with the big lead, with the arms race dynamics pushed aside. Go slowly and appropriately through the hurdles and the period that we find ourselves in right now. And that was the theory of the case, for better or for worse, from even before we showed up in the White House.

Speaker 2

30:47 So

Speaker 6

30:47 hey, we'll continue our interview in a moment after

Speaker 4

30:49 a word from our sponsors.

Speaker 1

30:52 The Cognitive Revolution is brought to you and powered by Tasklet. I've been running Tasklit agents for almost a year and they've become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all sources. Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from PR firms. And that's really the core idea behind Tasklet. Run your business on agents by giving them real jobs to do on an ongoing basis.

Speaker 1

31:26 Setup is incredibly easy. Just describe the work in plain English, research a new sales lead, update the CRM, and draft a personalized reply. Tasklit connects to your business tools and can run around the clock on a schedule or when something happens like a new email arrives. Each agent works with your tools and business context and remembers useful information across threads. You can also share agents connections and reusable skills with your team so the workflows that one person figures out become something the whole company can use.

Speaker 1

32:00 No developer needed. You decide what agents can do on their own and what needs your approval. And there are no per seat fees. You pay for usage, not for adding teammates. Start with one recurring job, like researching the people you're meeting and then just keep building from there. Visit Tasklet AI and use code COGREV for $50 in free credits. That's code COGREVasklet AI Today's episode is brought to you by Granola.

Speaker 1

32:34 The most valuable question I ask about my meetings isn't what happened, it's what am I missing? Granola is an AI powered notepad that makes your meeting notes useful. It captures what happens in your meetings and turns it into clean, structured notes with decisions and action items pulled out and made easy to find. Granola keeps my calls searchable in one place, so I run my blind spot finder recipe across recent conversations. It flags angles and issues I've been neglecting.

Speaker 1

33:05 Once you try it on a first meeting, it's hard to go without. You can try it totally free. Head to Granola AI tcr. That's Granola AI tcr. To get your time back, go to Granola AI TCR and try it on your next call. How do you think that's going? I mean, it is clear, I would say, that the export control policies have worked

Speaker 2

33:32 in as much as there is a huge compute advantage. There's, of course, been, you know, some gaps in enforcement and, you know, some stuff may have fallen off the back of a truck here and there. Whatever. Is the world shaping up in the way that you hoped that it would? I mean, one anecdote you tell in the book that I always use in my talks as well is the 2016 reward hacking paper from folks at OpenAI, including Dario and Jack Clark, where they, you know, show the boat going around in circles and just crashing over and over again to maximize points.

Speaker 2

34:06 One takeaway I would have from that is like, man, we're 10 years on from that and we still don't really have a good solution for reward hacking. And so, you know, we've got. We've got this lead. It feels like, you know, people are talking again about AI 2027. Like, now is kind of the time to spend it on safety. You have a way to spend it on safety. Like, are you optimistic that we're going to actually be able to, like, land this plane in the time that we have available?

Speaker 3

34:30 I would say technologically, the world's going basically exactly as we expected. And the primacy of computing power, the scaling law, the bitter lesson that of course is the inspiration for our title, the bitter struggle. All of that continues to hold, I think, in ways that were speculative in 2021. I mean, we thought, we thought it would be that way, but you didn't know for sure. So I think that's right. If you look at how the Policy is cashed out.

Speaker 3

34:53 Even with Donald Trump's pretty poor enforcement and in some cases direct reversal of the export controls, America still has something like 100 to 1 advantage over China or America and its allies. So something like 100 to 1 advantage over China in producing computing power and probably something like a 15 to 1 advantage over access to it. And if we close the remote access loophole and stop smuggling through trans shipments and so forth, that 15 to 1 could probably get closer to 100 to 1 or even more.

Speaker 3

35:19 So I do think the technological theory of the case was right. The geopolitical. Geopolitical theory of the case was right. What we have lacked, to be very blunt, in the last, you know, two years since Donald Trump took office was a conviction that this technology mattered. And my successor, of course, is David Sachs and he has this mantra, let the private sector cook. JD Vance, his first speech on AI gives a big thing. I don't want to hear about AI safety.

Speaker 3

35:41 I'm here to talk about AI opportunity. Future is not going to be won by hand wringing about safety. So I do think there was a very deliberate strategy to chart a different course that cost us time. And I am hopeful in this moment, post Jensen scandal, post hugging face incident, post reports about how fast RSI is happening, that we can capture a little more political urgency, maybe on the back of the election as well, and take some actions, regulate these companies, pace the frontier, build a big lead over China and do the things we were always setting out to do.

Speaker 2

36:17 Okay, I have like a thousand follow up questions on that alone. It gives me no pleasure to say this, but one observation I do make pretty often is that in some ways the US is starting to look more like China in the ways that we kind of think are bad. Right? Like we have the big man at the top now. We have like all this kind of arbitrary and capricious enforcement and lawfare. I went to China this summer.

Speaker 2

36:41 I talked to people there who are like, just trying to work on, like track to, you know, AI safety, you know, kind of research level collaborations. They are very afraid that the US government might crack down on them. And so they like keep a very low profile as a result of fear not of these are

Speaker 3

36:56 Americans. Americans

Speaker 2

36:58 Americans and some others. But yeah, American nonprofits trying to engage on an AI safety research level with China are afraid of our government, you know, weaponizing export controls against them. And I'm like, you're not exporting anything. And they're like, I know, but it doesn't really matter because, you know, this is how they, they operate. So on top of that, of course we have like, you know, elections and regimes change domestically. I think one could make the case that like maybe actually the Chinese system, with their continuity, while it like, comes at a significant cost in terms of the civil liberties that their people don't enjoy, that we get to benefit from, is maybe better suited to managing such an epochal technology as AI.

Speaker 2

37:40 Like, we're maybe just a little too all over the place to do it. Well, how would you respond to that criticism?

Speaker 5

37:47 You know, there's a quote that we have in, in the first chapter, no, in the introduction to the book, which is from William Seward, who was Secretary of State under Lincoln, who said there was always just enough virtue in this republic to save it, sometimes none to spare, but still enough to meet the emergency. And I think about that because, you know, while we were writing the book, it was not the easiest hour to defend America's commitment to the democratic values that we argue are very important and should guide the development of this technology.

Speaker 5

38:31 But I think it is easy, if you're caught up in the maelstrom of the day to day scandalous news stories about corruption and creeping autocracy and so on, to lose track of the fact that you do have deep structural factors that even Donald Trump is not able to gut entirely. And I think this is appropriate timing because you look at the election results from last week, and Van and I were texting about this, you know, that quote seems apt.

Speaker 5

39:04 There are corrective mechanisms built in here. Better late than never. At some level. And I think that at many times throughout history, people have made arguments that in a moment of some kind of crisis, geopolitical crisis, technological change, et cetera, an autocratic system might be better suited for some immediate tactical reason. It can mobilize resources more quickly. It can act in a more decisive and coordinated fashion. Sometimes up close, those claims are correct. But I would say that the broad sweep of history suggests that when you concentrate power in an autocracy, the long run results tend to be not great.

Speaker 5

39:42 That over time, liberal democratic systems tend to lead to outcomes that I think, you know, most people would look at and say maybe they're not perfect, but they are more just in the aggregate, fairer, more reliable. They are more likely to lead to the kind of world that we want to live in. I would say that that holds here. And the United States is not perfect. We write about this from the perspective of not just the United States, but also the United States and its democratic allies.

Speaker 5

40:07 How can we make sure, the democratic values are sort of paramount globally. And yeah, the system has its flaws, but there are corrective mechanisms built in that strengthen my conviction that it is better for incredibly consequential technologies to be developed under those auspices in democratic countries than in autocracies.

Speaker 3

40:24 I agree.

Speaker 2

40:25 How do you. So another big thing I heard from folks in China, and these are Chinese people this summer, was the perspective that the US Is just trying to hold us down. They're still very sore about how Huawei was excluded from Western networks on, you know, what they would say is kind of relatively thin accusations of espionage. And not too long after that happened, we had the Snowden revelations. And it kind of came to light that, like, actually the US Is doing exactly what we were accusing them of doing, including, you know, listening in on Angela Merkel's phone calls.

Speaker 2

40:59 And so they're like, so that was clearly pretense. And you just, you know, we got one national champion company that hit the top tier in a really critical industry. And as soon as that happened, you came down on us. And now you're doing the same thing again on AI. How would you try to defend the US Policy to not the Chinese government itself, but like Chinese AI companies, you know, that have that sort of vibe. Is there a message that we can.

Speaker 2

41:26 Because I do take the sense from the book that you think that we will ultimately need to have international cooperation, including like US China cooperation in the fullness of time. So how do we get from here to there without them like deciding, you know, what, this 100 to 1 advantage that we are on the wrong side of is just so existential to us that we need to like, escalate into other domain, like attacking Taiwan. You know, for example,

Speaker 3

41:52 not only do we think we should have engagement with the Chinese in the fullness of time, we spent a lot of time having engagement the Chinese in the Biden administration. And there's a line in the president's executive order that when you translate from the legalese says something like, the United States will engage on AI safety not just with its allies and partners, but also with its competitors. And three weeks after that, President Biden took that to President Xi in their meeting in Woodside, California, said, I'm very serious about this.

Speaker 3

42:17 The two leaders agreed to set up a AI expert group. We were both there. We met in Jenny with the Chinese in May of 2024. And the Chinese said side said, we're. We don't think AI risk is real, essentially. And we, we think this discussion of risk that including risks that will Affect all of us is some kind of, you know, plot and we don't buy it. We kept pushing. By November of 24, when Biden and Xi met in Lima, they could agree to the most basic thing, which was no AI nuclear weapons.

Speaker 3

42:45 So a starting point, I guess. But we really tried on this point and I don't see it as inconsistent at all to say, look, we're not going to sell you the most advanced technology humanity has ever created in these computer ships. But also we think we have shared risks here. And I don't think it's inconsistent to say as you develop international agreements and do negotiations, you want to negotiate from a position of strength. In fact, historically, some of the most in successful negotiations have come when one side has position of strength and uses it in a, a good way.

Speaker 3

43:17 Look for example at the Intermediate Nuclear Forces Treaty between the United States and Soviet Union in which the US built up a substantial capability is clear this Gorbachev that he couldn't keep up and they have this good deal. So we, I think it's not hard for me to come up with a vision that we in fact believed that we want to engage with the Chinese. We don't want to sell the most advanced technology. We welcome collaboration on issues that affect all of us.

Speaker 3

43:42 But also we are wary of the way in which China takes AI in a well documented fashion, uses it in its military, uses it in its cyber operations and uses it to oppress its people. And we don't feel like we should enable those things as the price of cooperation.

Speaker 5

43:58 Maybe one footnote here is the wording that was eventually agreed on. The nuclear point is actually even more limited than sort of the high level description of AI for nukes, which is specifically AI shouldn't be used in the context of nuclear weapons employment. And getting that exact. There's a whole, the decision making chain has many components when it comes to decisions around using nuclear weapons. And then there are all of these processes for monitoring them and developing them and designing them and so on and simulating them.

Speaker 5

44:30 And it took Ben several months just to narrow down that specific wording which I think the general public would look at and say that is the most no brainer thing imaginable. And even that is like a sliver of what at first blush looks like the most no brainer thing imaginable. So that isn't entirely because of, I don't want to say that that's wholly because of Chinese obstinacy or something. This is sometimes just like how diplomacy works and bureaucracies are Complicated and so on.

Speaker 5

44:59 There is a long way to go to get things over the line that feel like we should have done them with urgency yesterday. As Ben notes, historically, that is easier if you come into it from a position of strength.

Speaker 2

45:12 So a lot of that, I think, is, you know, basically indisputable. But still, I'm kind of like, if I try to take the Chinese perspective, and not because I want to defend everything the Chinese government wants to do, but just because I want to make sure, like, we're playing our cards right, I read something like the Machines of Loving Grace, second half. And I would basically paraphrase the China strategy as build a big international coalition, box them out, and then ultimately make them an offer they can't refuse, which sounds like borderline regime change.

Speaker 2

45:46 Certainly if I was them, I feel like I would be kind of reading it that way, or at least, like, wondering, you know, to what degree is this sort of setting up to be like, a second century of humiliation, like the AI edition, which they are very sensitive to, as I'm sure you well know how. I mean, should I be worried about that? Should I not be worried about that? It just seems to me like they are very sensitive to falling behind in technology because they know that that leads to very bad outcomes for them.

Speaker 2

46:10 And no less than Dario is basically saying, like, we'll get to them last and look like, you know, they're. They're going to give up on, you know, competing with a superior foe and, like, basically kind of adopt something more like our way of life. That seemed a lot to ask, and I wouldn't be surprised if we get, like, lashing out from them as opposed to, you know, things actually going that way if we indeed, like, you know, put pedal to the metal on exactly that plan for the foreseeable future.

Speaker 2

46:37 But it seems like you think I shouldn't be worried, or. Yeah, tell me.

Speaker 3

46:40 Let's be. I mean, we. We of course, worried about how our actions would be perceived, and we tried to scope the actions in such a way that would not yield this kind of backlash. And indeed, there was not much retaliation from the Chinese for the Biden administration actions on chip controls, because they were very carefully calibrated. There's far more retaliation for things like the tariffs, far more from Trump, far more lashing out than from anything we did.

Speaker 3

47:01 But. And you'll certainly not see us advocating for, you know, overthrowing the CCP or things like that. I personally don't think the CCP is. Is a moral actor, but we understand the principles of sovereignty Here. I do think that history here can provide a guide to us. And I don't think the history suggests regime change. I think the history suggests that when you have a nation with a lot of power, the long term smart strategy for that nation is to build institutions that cede some of that power from the nation to the institutions and provide a stabilizing force to the broader ecosystem.

Speaker 3

47:38 And oftentimes, though not always, those institutions relate to the management, the shared management of a new technology. I would argue throughout American history that this is the kind of thing America does at its best. America's got a very flawed history, we all know that. But if you look at the Post World War II era, American basically backstopped a series of global institutions that ceded some of its power to the rest of the world, including to China.

Speaker 3

48:03 The United nations, which China has a permanent veto, a permanent seat and veto, with the Security Council being one example, the creation of the imf, the World Bank, World Trade Organization. And the same thing happened in many similar contexts with space and building the space institutions with air travel and the air travel institutions with the Internet creation of ICANN and the like. So I think there's a strong history here of the United States saying, look, we're ahead and at our best we're going to recognize it's in our enlightened self interest to build some stabilizing institutions.

Speaker 3

48:35 We will give powerless institutions. You all can join and benefit from this system while retaining your autocratic nations at home. We're not going to stop you, we're not going to force regime change for you to join the wto and this sort of goes better for everyone. I think the history is pretty clear on that point. I think we saw our work as falling into that history.

Speaker 2

48:54 Let's flush that out then in a little more detail like that started to sound a little bit like the Demis plan. I don't know if you're going as far as like international development, which I think he's kind of contemplated at times, but at least it sounds like international governance bodies describe, you know, what, what do those look like and on what terms does China eventually get to join? You know, we have this lead like are we at some point making, I think kind of the implication of the machines in Lemongrass description is like at some point there is kind of a chips for controls sort of deal.

Speaker 2

49:29 That's, you know, that's maybe the more, the less regime change and more kind of positive read of that. I'd love to hear in more detail how you see that playing out and on what timescale too, because it does seem like we may be up against the clock even more than, you know, the, the short timeliners predicted a few years ago.

Speaker 3

49:48 Let me give one example and then tell you should weigh in. To be clear, we did not write Machines of Loving Grace. Machines Loving Grace is. Is Dario's essay. Not, not our essay, but we created something in the United States called the AI Safety Institute. And this was the first attempt to give the government real tech technical expertise to manage this technology. Coming from the private sector, we knew that even though the companies inventing this technology were primarily American, managing this technology was going to be a global endeavor.

Speaker 3

50:16 And there may be companies from other countries that join the conversation in a meaningful way. So we create a network of AI safety

Speaker 5

50:24 obligatory intervention that I'm right now CEO of a non US based AI company. So I certainly hope, I certainly hope that, that, you know, great companies in other countries like the United Kingdom get to play a role in this.

Speaker 3

50:36 Very, very fair point. But Teddy was grinding it out with us in the bureaucracy in 2024, had not yet started this future global European behemoth that would, that is inherent labs. So. But we created a network of international AI safety institutes and had a conference and invited, I believe, Chinese researchers to the conference to say we want to collaborate with allies and partners here and again also with our competitors on setting up institutions that work here.

Speaker 3

51:03 Now, your point about timelines, this is what worries me. And we were doing this in 23 and 24. We felt even then like time was short. And what worries me is we've, I think, gone backwards in some ways for a year or two. Maybe there's a pivot happening now which is good. And, and I have nothing reassuring to say about how little time remains, but directionally that's what we are trying to do. Teddy, you want to give some other examples or add?

Speaker 5

51:27 No, I think I just make one broader observation that there is often an understandable temptation to talk about all of these things as sort of matters of hard and fast principle. But in reality, as you learn, when you kind of get into the rooms where these decisions are being made, they almost all are a question of degree. And so, for instance, when it comes to the question of the relationship with China and how that is perceived, and I should say I have a lot of sympathy, honestly for the skeptical takes that people understandably have from within China.

Speaker 5

52:01 When it came to the rooms that we were in and trying to figure this stuff out, it was never the case that really anyone wanted to Go for either of the two most extremes. Either that we are, let's say, selling absolute, most capable technology at scale with no restrictions directly to the plaque, right. Or which is not new to AI. Right. This has been the case for sensitive technologies for as long as we've had them. Nor that we shouldn't let Chinese gaming enthusiasts have access to personal GPUs that they can use for better graphics rendering.

Speaker 5

52:39 And then the question becomes how exactly do you draw the contours of this? And these are all discussions that we had over many months with colleagues who worked specifically, you know, on the China Directorate in different aspects of economic policy and so on, to think through quite carefully what is the outcome we actually want to achieve here and what is the likelihood of retaliation.

Speaker 2

53:01 So as we approach potentially a, you know, 2027 recursive self improvement loop or intelligence explosion or who knows what, it does not seem like we are particularly likely to be able to spend the lead to solve all the AI safety issues that remain open. You know, just progress on reward hacking alone. Like, doesn't seem like, you know, we would need to see a real bend in that curve. If we're going to all of a sudden be like, okay, great, we've got solutions.

Speaker 2

53:32 Tell me if you disagree. But then if you would generally agree with that. In the absence of solutions and with the sense that like the Chinese ecosystem is, you know, often said to be six months behind or whatever, it does seem like if we're going to restrain ourselves from going as fast as we can in AI, we're going to have to convince them to do the same thing. And presumably that's going to involve some sort of deal.

Speaker 2

54:00 Do we have any sketch of what that deal could look like right now?

Speaker 5

54:05 I'll let Ben weigh in maybe with a more comprehensive take. But one thing I'll say is that this I think is one. I think I speak to both of us when I say that as Ben mentioned earlier, the timelines are really tight and that isn't great from the perspective of governments because even in the best of days, governments are just slow, tend to be slow moving institutions. The grappling with something that evolves as quickly poses real challenges.

Speaker 5

54:34 And I think because things move so quickly, it can be very difficult to say ahead of time. These are exactly the restrictions that need to be put in place in, let's say 18 months. But sometimes that's how long the policy making system could take. And so something that we pushed hard for when we were in government was less to lock us into specific policies and more to try and imbue the relevant bits of government with the insight and agility necessary to tighten those timelines so that you could move at something closer to the pace of the technology itself.

Speaker 5

54:58 So insight in the sense of making sure that government actually knows what is happening within the labs, how capable the models are, how quickly they're evolving, what the technical security measures are that are being put in place. And I think that incidents of the past couple of months demonstrate that it would probably have been good if the government had been a little bit more in the loop there. And Ben can talk about the specific mechanisms we put in place and which of those were subsequently removed.

Speaker 5

55:19 And then correspondingly, you want the government to be able to take action much more rapidly than the sort of default policy making process allows. I think that applies at the domestic level. And also when we think about what would eventual international coordination look like, you'd probably want something similar with the added dimension of some kind of verification scheme that each side can trust the other is enforcing. And this is an area where I think there is a long way to go on the technical front in terms of developing the types of capabilities that you need to do that sort of verification.

Speaker 5

55:54 And I think that that means that we can speculate a lot about what these measures would look like. How would we know that China is pacing? But putting that into practice I think will be really, really, really hard.

Speaker 3

56:04 I strongly agree with all of that. You speak for both of us there. I think given the spot we are currently in, we have to slow down in the United States. I guess my one point of optimism is if we did actually cut China off from computing power, I think they would struggle to move ahead of us. So I think we could put a little bit more time on the clock because of it. So we are, we are in a very unusual position, I think the two of us where we probably correctly identified as China hawks who favor a more aggressive chip control policy towards China.

Speaker 3

56:34 But we also say we need to have dialogue with the Chinese on this. And we also say that the China AI ecosystem is no excuse to race forward dangerously here in the United States. And I think those positions actually are very logically coherent given technical facts available on the compute moat and all of the like. But that is the where we find ourselves here. And what concerns me is just we don't have a lot of time and I think we need to act quickly to re establish an even larger lead than we have.

Speaker 3

57:05 And then we've got to slow down the United States to make sure we are spending that lead. Well, it will not be easy. And, you know, Teddy and I are here talking to you on a podcast, which is great, but that doesn't mean we have any actual power over the policy that's in other people's hands right now.

Speaker 5

57:19 I'd argue that those positions are not only coherence, but in fact, inextricably linked.

Speaker 3

57:25 Yeah, I agree with that. What I think is a baffling position is folks who say they are China hawks and then want to sell China chips or say, as the President sometimes says in this very kind of militaristic way, whoever wins, AI wins and then wants to sell them chips. And it just seems like the more logically consistent and technically correct position is to say, no, this is really important. As you say, Nathan, we've got real questions on safety.

Speaker 3

57:53 Therefore, we need to proceed judiciously here in order to create space for that. We need to make sure we're not in a perceived arms race with what most people say is our closest competitor.

Speaker 2

58:02 The position I take, which I think is coherent. Tell me if you think it's not coherent, but then maybe more to the point, tell me why you don't favor it is just starting with the observation that the real aliens here are the AIs, not the Chinese. I'm kind of like, man, we got two great civilizations. You know, China's been at the frontier of technology for most of their 5,000 years with a few blips where they fell behind.

Speaker 2

58:27 And I think, you know, learned the lesson of how costly that can be, and they're very determined not to let that happen again. So my position is kind of like, I'd rather take my chances on trying to find a way to cooperate than trying to find a way to kind of shoot the gap and, you know, develop the technology just powerful enough and have just enough advantage and then also be able to solve the safety problems just in time and then hopefully, you know, disseminate that to them.

Speaker 2

58:56 Which, by the way, I do think they would be very happy to take our safety solutions if we had them to offer. So that part of the plan I'm not so worried about, I guess. Why not take your chances with our fellow humans and try to have this sort of meeting of the minds that, like, look, this is a force that is bigger than all of us. Like, these guys understand, you know, sort of deterministic, you know, technologically driven stories of history, right?

Speaker 2

59:23 Like, they're. They're quite primed, I would say, more than us to believe that, like a technology revolution Will, like, fundamentally alter the state of things. Why not? Why not try that route,

Speaker 3

59:34 let's be precise, by what you're proposing, because I would say we did take our chances on negotiations with the Chinese, and we pushed very, very hard in face of Chinese opposition for those negotiations. And if you had us in power, you would see very aggressive. Aggressive in a good sense, very sort of highly prioritized negotiations with the Chinese, probably in the way that you want. The precise question I have for you is, would you, as a precondition to those negotiations, sell China access to GPUs?

Speaker 2

60:00 I think I would try to bundle, yeah. Chips for controls is kind of my general label for this. That's

Speaker 3

60:06 the end game. But what they're saying on the front end, what they said to us in Geneva on the front end is sell us chips. Right. We'll talk about other stuff later if you want, but start by selling us chips. And I think that's not a posture where. I don't want to speak for Teddy, but I'm pretty confident that's not a posture we were willing to take. And that's the. That's the policy choice you have to confront to the government is, do you want to sell them this technology in the hopes of getting a deal later, knowing that that will shrink the lead?

Speaker 3

60:29 If we could know with certainty that, you know, we'd have a great system of solving these problems and AI safety would be great then. Then, sure. But that's not how international politics works.

Speaker 2

60:39 Yeah, I mean, I. I do think attitudes may have changed over there quite a bit. I mean, how do you read the. You know, Xi gave his first big keynote on AI at the WAIC this summer. He talked a lot about forestalling loss of control, the need for international cooperation. Maybe one way to kind of square the circle between what you heard then and what I might hope would be possible now is just to say, like, a lot of people are a lot more AI pilled now than then, and maybe that extends to Chinese leadership.

Speaker 2

61:09 And maybe they'd be willing to entertain some things now that now that they've seen, you know, rogue agents, including at least a couple swarms from Chinese labs, from what I understand. Like, maybe they would kind of now say, yeah, actually, you guys were right a few years ago. We weren't ready to listen then, but now it sounds a lot more real. And so, yeah, let's, like, let's try to do a deal.

Speaker 3

61:31 I think that'd be the best news ever if they had that update. We're not in power so we don't know where they are right now. But I can tell you if we were in power, you would see us continuing to try to have these negotiations with them. The only point I'm making is I would need to see more than just rhetoric before we reversed the fulcrum of the policy, the fulcrum of the lead, which is the compute controls.

Speaker 3

61:51 But if China could come to us incredibly, say, yeah, we blew you guys off in 2024, but we've got a very different take now, you know, let's talk in a serious way without preconditions. Absolutely. I'd be, I'd be open to it and frankly, I hope the Trump administration would be as well. Do they have the capacity to have a meaningful air safety dialogue? I don't know. Maybe they do in the post SACS era. Teddy, what would you add here?

Speaker 5

62:16 There's a simplified version of this that is a matter of binary principle. You do engage or you don't engage. You, you know, you, you do work with your competitors or you don't work with your competitors. And I think a big part of what we're trying to say is like it's not a binary. There is this whole spectrum of exactly how do you engage, with what preconditions and so on. And that we did try to engage and that we think engagement is not just a good thing, but an essential component of eventually getting to a good outcome.

Speaker 5

62:46 And the question is taking into account various geopolitical realities, including a pattern of historical behavior and a generally, unfortunately low trust relationship between these countries undergirded in stuff the US has done, but also stuff that China has done. And we can look at the militarization of islands in the South China Sea or ongoing cyber operations where multiple times there were, I think, what a lot of people on the US side would say were good faith to the extent of being naive.

Speaker 5

63:11 US efforts to engage China across multiple US presidencies across the decades and say, hey, let's reach an agreement to stop doing this. Chinese counterparts committing to that and they're not doing it. China, in fairness, can point to lots of things that they see as duplicitous from the United States. Not saying it's one sided, I'm just saying that that historical context can make it difficult to, without preconditions and without good verification mechanisms, sign onto something blindly. And then I think the second thing I'd say is in all countries there is a gap between the median person or the medium subject matter expert and government decision makers.

Speaker 5

63:52 And so I think I will absolutely defend the fact that there are a lot of people in China who are deeply concerned about AI safety and who care about this being used for the good of humanity and avoiding all these bad outcomes. When we were in government, we talked a lot about who are the innocent people who will be caught up in these policy actions, who are the people who are just trying to found an interesting and good company, solve a fascinating scientific research problem, build a great app, who now, because they had the bad luck to be born and be based in China, will not have access to the compute that they would if they were in other places.

Speaker 5

64:27 And I think that that is a sad and unjust fact about the world. We spent a great deal of time thinking about how can we minimize those costs. And we spent a great deal of time also looking into the many sort of fledgling components of civil society in China that cared about the same things that we cared about. However, part of the reason why we're concerned about this technology being developed in China is because the gap between the median person and decision makers and government, because it's not a democratic society, is almost definitionally much greater than it is in countries that have more reliable democratic feedback mechanisms.

Speaker 5

65:03 And so it's one thing to say there are a lot of people in China who care deeply about these issues. It's another thing to say we can 100% trust the decisions that are made by the people at the top of the hierarchy when it comes to these kinds of commitments.

Speaker 2

65:17 I'm not so sure I'm sold on the idea that our feedback mechanisms are that much stronger, but maybe leave that as just exercise for the readers

Speaker 3

65:26 at a minimum. I think it's fair to say in the Biden administration, the folks who cared about AI safety probably had more access to the president than the folks who cared about AI safety did in China had access to Xi Jinping. I think that's definitionally true.

Speaker 2

65:39 Yeah, well, you guys are walking proof of that.

Speaker 3

65:42 But to your point from before, that's not the case now. Right. So, like there, our system does fluctuate, and I don't want to pretend otherwise. And, but I do think it's the case that, as Teddy said, there, there, There is a sense here in which there are some great track two dialogues, but during our time in the government, and again, that ended almost two years ago, we did not see that translate to concern at the track one level about AI safety.

Speaker 3

66:05 Quite the opposite. Emphatically the opposite.

Speaker 5

66:08 And can I just say something that's going to sound so, so, so dumb. And basic. But it can be easy to lose track of it, which is that having elections matters. I know that that's like democracy 101. And I know that American elections are full of money and they're not.

Speaker 3

66:24 Buy the book, everyone. Core insights like that.

Speaker 5

66:27 Exactly, exactly. Yeah, yeah. You can spend, you know, 300 pages reading about this in the context of AI if you, if you buy our book. But Ben and I can provide a litany of complaints about the current political situation. And a lot of my research interests in AI have to do with how can we build systems that significantly strengthen democratic decision making and update it for the technologies that we have now, as opposed to the technologies that were around 100 years ago when we designed these systems to make them more responsive and faster paced and less creaky and fairer.

Speaker 5

66:56 There are a million things here that we can do better. But also just as a very basic description, like having elections does matter.

Speaker 2

67:05 Let's stay for a second on this question of like, okay, we've got potentially short timelines. We've got increasingly credible signals. I would say for me the most credible signals of potentially imminent recursive self improvement or acceleration of capabilities, if not necessarily through some hard RSI mechanism, is just what we've seen in math and cybersecurity. It's like, if there's that much low hanging fruit in math and cyber security, from the model's perspective, surely there's a lot of low hanging fruit to be picked in the ML stacks that the companies have created.

Speaker 2

67:39 So it seems quite likely to me that we will see like fast capabilities advances in the short term. And it doesn't seem like we have that much farther to go before we're into superhuman territory. So with that kind of backdrop, we need to pace, we need to like define, you know, what RSI is so we can avoid it. We need to like operationalize pacing in some way, shape or form. If we think of the rivalries between frontier companies, which I think is like in some ways such a Greek tragedy, right?

Speaker 2

68:11 Like all these guys have like known each other for a long time. Somehow they've like come to hate each other in some cases, you know, it's Dario and Sam couldn't hold hands, you know, at the, at the one Indian, you know, summit. Like that photo is burned into my brain. I say all that to say, like, in some ways the distrust between the American frontier companies is kind of a, maybe a prototype problem for the US China distrust.

Speaker 2

68:37 So if we think like, okay, we've got two, three, five companies in the us that are pushing the frontier. What are our best ideas right now for how to govern that situation? Or get them to somehow govern each other, mutually police one another? What if anything, are you bullish on in terms of actually slowing down for real in the short term?

Speaker 3

69:02 Anthony, one very quick point before we answer that question, which is it's worth noting you mentioned cybersecurity as one of the areas that's updating you on AI that is definitionally a national security matter. The reason you know about the reason any of us know about AI's capabilities in cybersecurity is because anthropic and OpenAI and Google took those capabilities and essentially turned them over to the defenders this year. I'm willing to bet every dollar I have if China had access to Mythos level capability first, it would not be doing that.

Speaker 3

69:28 It'd be using them to hack American infrastructure and the like. So again, if you disagree with that, then we should talk about that. But this is like is is the crux of why we in general feel like it goes better if you have America at the at the preeminent frontier rather than an autocratic nation. To your question, Teddy should weigh in. I think we've got a lot of ideas here. I fully agree we should slow down.

Speaker 3

69:49 I fully agree we should pace. I fully agree we should have a regulatory system. How do we manage industries where you have bitter competition between companies and you want to avoid a race to the bottom with negative externalities? You regulate them. That is what government exists to do. And we'll be the first to say overregulation is a thing. There are real dangers. We spent a ton of time thinking about the ways in which overregulation harm nuclear power and all that kind of stuff.

Speaker 3

70:12 But smart regulation can actually move the needle in significant ways. The example we always give is the railroad from the 1870s in which the introduction of standardized braking and train coupling and track widths and time zones led to an industry that was safer and more trusted and faster as well as one that was more stable for for America. So a lot of upside there concretely. I'll give a couple of examples then Tay should should weigh in with others.

Speaker 3

70:38 I think you should have mandatory pre deployment testing by the government, ideally if not by the government, that by a credible third party auditor before systems are released to the public. I think you should have embedded evaluators at the companies to hold them to safety standards and security standards during the training and development process. Hugging face of course happened before the system was released to the public. I would be open to some government mechanism for pacing the frontier, making sure that safety and capabilities development move in similar places.

Speaker 3

71:10 We could talk about what those would be, but directionally I'm in favor of significant government action here. And we tried to set up the foundations for that action during our time in the government. Teddy should add more.

Speaker 5

71:20 All that's correct. One other thing I'd throw in is not only enforcing security standards, but providing government defensive assistance for the places that are developing the sort of thing that is of the most interesting consequence to actors that might have malevolent intent and try to secure that stuff, try to obtain it.

Speaker 3

71:43 Yeah, it's worth dwelling on this for one second. It's a really good point from Teddy. Nathan, you and your question pulled out one of the themes of our book, which is this is the first revolutionary technology the last hundred or so years from the private sector. I would argue there's at least three consequences to that. The two that are most obvious and the ones that we felt acutely in the government is the government doesn't understand it by default because the people in the government working on AI are not working on frontier AI systems.

Speaker 3

72:08 Doesn't mean they're not good people, just they don't have this kind of exposure. And secondly, the government can't shape it in the way that the DoD could shape nuclear or NASA could shape space. So you have to assert some role through regulation or whatnot for the government. But the third one is the technology grows up in a private sector that is not used to the world of national security and counterintelligence and foreign espionage and all that.

Speaker 3

72:31 And I think that leaves them very vulnerable. And given the stakes of what they're making, it's something we stressed about tremendously. And Teddy's absolutely right. It should be part of some kind of regulatory and government packages, not just regulating the companies to have security standards, but really helping them get there against what I expect will be very soon if they are not already top tier national foreign espionage threats.

Speaker 5

72:52 And Nathan, to come back to your observation that you know, in some ways this, this intercompany rivalry is a preview of the interstate rivalry that is accurate. Obviously in the case of anything that is domestic under the purview of a single government, the coordination problem should in the easier to address. And that is why when you level that up to the international level, figuring out good verification mechanisms is so important. And this is one of the reasons why I kind of underlined it before because you just have a much lower trust environment and there is no reliable supranational entity that can just tell everyone, don't worry, we have it covered.

Speaker 2

73:31 Not until the AI singleton takes on that role.

Speaker 5

73:34 Yeah, that's right. Yeah, yeah, for now. Exactly.

Speaker 3

73:36 To be clear, that is not the policy we were advocating. Let's not, you know, David Sachs would go nuts if he thought that.

Speaker 2

73:45 I mean, that kind of bundle of policy recommendations in some ways sounds pretty similar to kind of where we're at. Right. Like we're getting these moves toward embedded evaluators and so on. It does still seem to me like it's probably not enough for the current moment, especially if the, if we're about to get into such a steep part of the S curve. Are you open minded to or you know, willing to tentatively endorse maybe blunter restrictions like you know, in the.

Speaker 2

74:19 I wouldn't say this is the limit, but you know, one that is like pretty simple to define and presumably pretty simple to enforce is like no bigger frontier training runs than we've had so far for. You know, and this is where my sunset clause comes in. You know, I would think, well, I wouldn't want that to be set in stone forever, but you know, could we agree for six months or 12 months, no bigger frontier training runs than we've had to date.

Speaker 2

74:44 Do you think the moment is urgent enough where things like that should be under consideration?

Speaker 3

74:50 I'm not the CEO of an AI company, so. Teddy, why don't you start?

Speaker 5

74:53 One thing that we talked about earlier is this idea that in order to be able to make good policy, the government actually has to know what's going on in detail. And I think it is hard for those of us who are not up close and seeing the internal metrics at the companies that are developing the biggest and most powerful pre trained models to say exactly where the lines are of which things should stop and which things should continue.

Speaker 5

75:27 And how can we tease apart? Well, these specific types of pre training and post training are good for safety capabilities and inevitability. And these ones are bad. To what extent can we tease apart stuff that's good for stuff on the defense side versus the offense side? And so I think it is very possible that there are blunt instruments that would be significantly better than the most likely counterfactual alternative, which is nothing. But in order to know which of those instruments make sense and which ones don't, it's kind of important for the government to know things other than when they sort of tumble out into the open because some of your agents like do a bad thing that becomes Publicly known.

Speaker 5

76:06 And once upon a time Ben and I were in government and we had regular discussions with sort of the senior people across all of these companies. And that provided some informal sense of what was going on.

Speaker 3

76:16 And no, and in fact we had regulation that they had to tell us.

Speaker 5

76:19 Exactly.

Speaker 3

76:19 That was the executive order, right? Yeah,

Speaker 5

76:21 there were disclosures there. I guess my point is like we had informal channels that supplemented the disclosures in the O that gave us an even richer sense. Now we're not in government and you know, I do not sit up close next to the biggest pre training runs that are happening at the trillion dollar labs. And that makes it honestly pretty difficult as for all intents and purposes, a member of the general public, you know, an outside commentator on these things to say which blunt instruments make sense and when should they be put into effect.

Speaker 3

76:56 In general, I agree with that. I would say if I were to pick a blunt instrument, I would probably pick some definition of full RSI or closed loop RSI where you're sort of like you know, saying this is what we're not going to do. We're not going to just give the set up a data center and say go at it, make your successor and have no visibility. So that's probably the blunt instrument I would go for.

Speaker 3

77:15 I do think on this question of a temporary pause or whatnot, what I would want to know the answer to is are we creating a huge compute overhang? What are we doing with the time? And I could imagine circumstances where we say we want to not proceed to some level of capability without meeting safety and security standards. In fact, I probably support that a time bound pause for a pause's sake runs the risk of creating a compute overhang.

Speaker 3

77:37 Then when you do turn the machine back on, you get a big jump in capabilities. And I'm not sure sure that that is a good thing. So if you ever intend to turn the machine back on, if it's a pause as opposed to a ban, then I think I want the pause linked to safety thresholds and security thresholds rather than to which is what we tend to call pacing rather than to some time period. But look, I, I would, you know, do cartwheels if this was the kind of nuanced conversation that the Congress had.

Speaker 3

78:05 And I could be convinced a bunch of different ways. As you say in your question, we're not particularly close to that right now.

Speaker 2

78:12 Yeah, well even at the level of like anthropic, the previous RSP sounded a lot more like that and the new one is a lot less like that. So, you know, even among the kind of thought leaders in that space, we've drifted, I would say, the other direction. Do you have legislators that you're excited about? I think we're maybe both going to have a chance to say hi to Scott Wiener. This is weekend. Who's coming to Congress?

Speaker 3

78:37 Yeah, Scott Wiener. Manny Rol in Colorado is great. So I think there's more. There's more. I would look at it in two ways. There's more folks who, you know, came closer to running on AI safety or being deep on AI safety than ever before. We did not have a Scott Weiner type or a mini regional type when I was. When we were in the government. And also the folks who you might describe, as I use a term, with no judgment whatsoever, more normies than the rest of us over here are now elevating this issue.

Speaker 3

79:08 People like John Ossoff have done a great job, I think, talking about this in a way that resonates. Adam Hamilton in Kansas talked about it. Data centers, of course, were a big issue. So I do think there's, to use a political science term, a much higher degree of salience of AI than was the case when we were in office. And it was acutely obvious to us that when you asked people about our AI policies, they pulled like 85, 15 favorable, but no one was going to vote on them.

Speaker 3

79:35 It just wasn't that top of mind. I think that's different now. I think 2026 was. Was the first election which AI was a big issue. 56% of voters said it was a crucial or major factor in their vote. It will not be the last. My prediction is it will be the number one issue in 2028. It will be entangled with affordability and corruption, but it will be the number one issue in 2028. And the presidential nominee, certainly the Democratic Party, but probably of both parties, will be the one who can offer a theory of the case to the American voter as to why, you know, why they're taking a particular approach to AI.

Speaker 2

80:07 Yeah, the wave of ballot initiatives on data centers has been quite. Yeah.

Speaker 3

80:13 It's like, depending

Speaker 2

80:14 on how you

Speaker 3

80:14 count, it's like 43 out of 44 or something. On data centers,

Speaker 2

80:17 these waves seem to only really come along, like, every so often, and on issues that people very much care about, some of which in retrospect, maybe look quaint, but it's definitely a signal, I think, that it is reaching consciousness in a way that it certainly had not until quite recently.

Speaker 3

80:36 Yeah. And let's just connect it to China Here, because what you'll often hear, and certainly in doing a number of book talks and podcasts for our book here, what I've gotten asked a bunch is, isn't it just the case that the backlash against AI in the United States or data centers is some Chinese information operation? And maybe it is. I don't know, we don't have access to that kind of information in the public sector, that maybe China is trying to amp this up.

Speaker 3

80:58 But I think most of the reason people are worried about AI in the United States is people are worried about AI in the United States. And I think this is a case where sometimes the but China argument that you so frequently hear from people who actually, ironically enough, are not really China hawks just goes way too far in saying, you know, everything that's bad is because of China, and everything that's, you know, good is because of Silicon Valley or whatnot.

Speaker 3

81:21 And I think this is a case where certainly our posture was if the government did not regulate this technology in an appropriate way, it would not be trusted. And I think the result of 18 months of no real regulation is the technology is not trusted. Now the tact is changing. Sachs is gone, Clayton's in. And we may see a different approach here from the Trump administration, but I think we should recognize that we shouldn't be surprising that we're at this point with the public views on AI, given the approach the government's taken.

Speaker 5

81:47 The other China angle here is something that they've done very well with respect to on this front, which is like building out the energy capacity so you can build these facilities without having some of the adverse effects on local residents that people are understandably worried about. In the United States, it's not limited to energy costs. I know there are other factors in there, but that's a component of it.

Speaker 3

82:08 And we have a whole chapter on energy and permitting and so forth, because we felt like it was a really big part in an area where China's just done a much better job with the United States. And insofar as we're arguing for government capacity, part of the capacity we're arguing for is just like building more power in the United States, ideally clean power, so that people had lower bills.

Speaker 5

82:30 It's remarkable. You look at the numbers. I mean, China's become a renewable electoral state at an extraordinary scale and incredible speed. It's certainly super

Speaker 2

82:40 impressive. And being in the, you know, obviously, flagship Chinese cities this summer of Beijing and Shanghai, being around all electric cars is really quite a pleasure. Like, it's quiet, you know, the Air is. Beijing obviously has a reputation historically for bad air. Air was very good there this summer at the exact same time that the air in my hometown of Detroit was like, worst in the world due to a Canadian forest fire. But whatever.

Speaker 5

83:04 I spent a bunch of time in China, like over the years, and that was almost entirely at times when you literally. You never saw the sun in Beijing. I remember. I suppose there was one summer when I was there where there was. People got very excited because there was an eclipse. We all went outside at the eclipse, which was midday, and you couldn't. There was like. You couldn't see anything. You had like emotional darkening of the sky sort of.

Speaker 5

83:26 We were all squinting and it's extraordinary to me now to see photos of what these cities look like.

Speaker 2

83:33 Yeah, times have definitely changed. They're in that respect, for sure. Is there anything else that's top of mind for you in terms of things we should learn from China? One that's top of mind for me as just a prompt, is when you go there, everybody says it's totally safe. There's no crime, men, women, whatever. You can walk around any hour of the day nosing your phone. You'll be fine. Certainly that's how it felt to me. It does not fully feel like that in American cities.

Speaker 2

83:59 I don't want us to import Chinese surveillance and apply it in the same way that they've applied it. But this has me asking the question, what would surveillance with American characteristics look like? And I wonder if you have any thoughts on that or other things that you think we should be learning from them.

Speaker 3

84:17 Teddy, you want to talk about the bitter

Speaker 5

84:18 temptation? Yeah, exactly. To sort of a set of things that I think are deeply undervalued. And as Ben can recall, I like, incessantly pitched these to people when we were in government, and most people didn't find them that interesting. But this is to some extent an extension of these questions around how can we use AI to improve governance institutions, including democratic systems? And we write in the book about what we call a bitter temptation, which is this idea that AI could.

Speaker 5

84:51 Even if you line up everything else right, even if you crack safety problems, even if you establish good global governance mechanisms, even if you build the lead that is necessary, even if governments learn how to use this stuff to do the functions that they currently execute safely and effectively, there might be a slow creep within currently democratic states towards autocracy, simply because AI might enable, as many people have written about, much more effective centralized administration of stuff that to date has been essentially impossible to manage in A centralized way.

Speaker 5

85:30 We can think about it in terms of economic management, in terms of surveillance, in terms of the administration of social programs, right? Things like Social Security and so on. That is potentially a huge asset because you want your government to be able to do things and to do things effectively. But we worry about a world where that capacity rises very quickly while the mechanisms for democratic oversight by the general public remain as they are now. And you can think about this from an information theoretic perspective.

Speaker 5

86:00 At the moment we hold elections, let's take the United States, right? Like the big elections get held every two years or every four years, depending on where you draw the threshold of what counts as a big election. And you fill out a multiple choice form essentially that contains very few bits of information and at a very low rate compared to the frequency and density and consequence of decisions that the government then on that basis makes on your behalf.

Speaker 5

86:28 So you fill out this checkbox that gets aggregated with a whole bunch of other checkboxes. All of your reasoning about why you did that does not get folded into the formal process. Maybe it comes up in some polling, then someone gets elected, and they kind of interpret that mandate as they see fit. And then you can almost think about this as being comparable to a very low information RL environment. At the end of some period of time, that person maybe runs for re election.

Speaker 5

86:49 They maybe get reelected. Whether people ended up voting for them the second time around is due to many, many factors. And the credit assignment is really, really hard. That made sense at the time that we designed these mechanisms, which was hundreds of years ago, because there were all kinds of practical limitations on how frequently and how deeply and in how nuanced a way you could gather feedback from people. It was infeasible to hold elections or have ballot initiatives or what have you every day or every week.

Speaker 5

87:14 It was infeasible to have everyone write a detailed essay about how they felt and assume that you could aggregate those in some reliable way. You can imagine that now we sometimes use this thought experiment of like, let's say that we were on the first ship to build a human settlement on Mars, and we have all the technology of today. We're trying to figure out what's the best democratic system you could imagine, building much richer systems using AI that, for instance, do enable people to provide more frequent input, more detailed input.

Speaker 5

87:40 And so when we think about sort of like, what does it look like to implement broader government capacity in a way that doesn't trigger this creep towards autocracy, one way of thinking about this is to say you want to increase popular sovereignty. The type of binding control that the general public knows it can exercise over the government, in theory represents it. You want to increase that by a degree that is roughly proportional to how much government capacity has grown.

Speaker 5

88:13 And already government capacity has grown significantly without seeing huge updates there. So, yeah, now we have some ballot initiatives, but mostly you can look at the headcount, the budget, et cetera. The number of decisions the government makes now compared to 100 years ago has increased hugely, in part because of popular demand, because people want these things, and in part because of technology. As technologies have enabled the government to do things like administer Social Security, let's say people have called for it to do so.

Speaker 5

88:36 But it would be really nice if people also had a corresponding increase in their ability to guide what the government does. And so there are lots of more in the weeds things that we can discuss here. But in brief, I think that there is the capacity to have increases in sort of government capability so long as that is balanced by increases in popular sovereignty.

Speaker 3

88:55 This is Teddy's favorite subject in the world, so I'm going to let him just take.

Speaker 5

88:57 Yeah, people fell asleep when I would try and pitch this, you know, in the Situation Room or what have you, to all of our government colleagues. But Ben, Ben to his credit, was a very patient listener.

Speaker 3

89:06 As Teddy said, that was actually, believe it or not, the brief version of that answer.

Speaker 5

89:11 Yeah, that was. Yeah, you'll be shocked. For those who want more, you can find it in part three. I

Speaker 3

89:16 limited Teddy to just one chapter on this point. So chapter 14. For those who want more, I think it's an important chapter. But this is not the whole book. It's just one portion of

Speaker 5

89:23 it. Yeah, that's right. But lots of people out there in the world. The Collective Intelligence Project, some stuff we're doing at my company. There is lots of good work being done in this space and I hope it has an impact.

Speaker 2

89:31 I share your obsession with it, for what it's worth. And I would invite you to come back and do a full.

Speaker 3

89:37 You guys should do that. Be

Speaker 5

89:38 careful what you wish for. Nate.

Speaker 2

89:39 No, truly, I mean, I go around. I don't know if you've ever read the book. It's a very niche self published novel called Liquid Rain. But I recommend it all the time. It's basically a quasi utopian future where a form of AI enabled liquid democracy has been instantiated.

Speaker 5

89:56 Oh, nice. Cool.

Speaker 2

89:57 And he even has a kind of notion of surveillance with, you know, something like, if not American, at least like decentralized or federated characteristics, which I'm not sure if it's workable on like a cryptography level, but it's at least you know, a vision that I think I found, you know, certainly much more attractive than having no, you know, use of technology or having like the, you know, the full on Chinese model.

Speaker 5

90:23 We think about this, I mean, we are trying to put into practice, I hesitate to use the term surveillance, but at inherent, the company that I run, we are trying to put into practice essentially like what, what is like an AI native social contract for collective decision making and that include questions about how can you have these systems sort of be pervasive in the way that many people will write about. Oh, the company of the future, the nonprofit of the future, the organization of the future.

Speaker 5

90:46 You want to have agents doing all of these things all the time. You want to give them all of these affordances, you want to give them all of these insights. I think that if that works well, it's great. There are lots of potential problems from a safety perspective. The more affordances you give them, the more you want to be able to measure rigorously how aligned they are and the more data you want to give them to enable them to model and effectively fulfill human preferences.

Speaker 5

91:05 But then there is also on top of that, this kind of social contract element, which is how can you make sure that those things that people collectively trust them, that you think that they are weighing preferences across people fairly. This is kind of related to pluralistic alignment research and some of the stuff that I worked on in my DeepMind days. And I think in order to do this effectively, you do need to kind of design the institution from scratch around these ideas.

Speaker 5

91:30 And that is one strand of what we are working on at inherent. And I do think it's. I think that this stuff can actually really work. And that's what we're seeing.

Speaker 2

91:39 Let's definitely come back to it and we'll do the full treatment. I'll pitch you on my idea of the states as. Because we do have ballot initiatives in like half the states, the sort of natural unit from which to drive some of this stuff in the early days.

Speaker 5

91:56 The laboratories of democracy.

Speaker 2

91:58 Yeah, exactly. Maybe as we just look ahead to this kind of short term, potentially short timelines world, what do you think an emergency looks like if we have an AI Chernobyl for whatever that means? Like what, what do you think are kind of the most likely form factors that that would take? What is the response going to look like? Does the government just come down in A heavy handed way or like, you know, I really, I think we're obviously in uncharted territory.

Speaker 2

92:29 But you guys were at the White House. Different White House. I know that, but what, what do you kind of expect

Speaker 5

92:34 back when it still had an East Wing?

Speaker 2

92:36 Yeah. The follow up to that will then be like what can people do if they want to get involved and position themselves to be helpful? But let's first just kind of sketch out what, what sort of scenarios are you watching for?

Speaker 3

92:48 I think the two vectors that we thought as far back as 2021 and 2022 would be the most significant vectors for this kind of risk are cybersecurity and bio. And whether it's a misuse risk or a misalignment risk, it would, I would bet a fair amount of money that those, it's going to be one of those two vectors in the beginning. If we have an emergency and it could be a cyber attack against a bank or critical infrastructure by North Korea.

Speaker 3

93:14 We've started to see bits and pieces of that already, though not probably at the scale you're imagining. It could be a hugging face style thing or you have a bigger breakout and you kind of have a enduring form of agents that's hard to dislodge because they're more capable. You could have equivalent risks on the bio side again, I think probably a little bit further afield than on cyber, but that's probably the form factor and I don't know if I have a view on if it's misuse or misalignment.

Speaker 3

93:39 I don't know if you want to talk about immediately what to do about this. I think we've given you a lot of policy things that, that can be done about this. I'd like to shift the aperture to what can normal people do about this? And what did, what have we tried to do about this?

Speaker 5

93:54 And we should flag that. We're, we're, we're, we're normal people these days. You know, like we're all in the same, we're all in the

Speaker 3

94:00 same. We shouldn't have any power. I don't know if it's the same thing as being normal, but, but I, I do think the point of political salience is one of the most important in our conversation here. And one of the things that's most different about the moment, as we said before, from when we did have power and in some very real sense we wrote this book which is very challenging. I've written four books. This is the most challenging one to write because it's targeting two different audiences who I think have the same mission.

Speaker 3

94:25 And it's for people who don't understand the technology, who want to get an accessible introduction to all we talked about, misuse, misalignment, the scaling law, the compute supply chain and all that. It's meant to be accessible to that group of people. And it's also meant to be educational to people who do know all those things, who are probably regular listeners to the show here, but who want to say, hey, how does this translate to action in the world?

Speaker 3

94:51 We've moved AI. AI, whether we like or not, has moved from a domain of technology to a domain of geopolitics. We know the solutions here are going to at some level be geopolitical, if not also political. What's an introduction to that world? So I think the mission of both those groups is actually the same, which they need to understand the other side better. And for better or for worse, I do think we are in a position of being translators between these two worlds.

Speaker 3

95:15 And our articulation here, having left, is to say, can we show each side what the other is bringing and why the other is necessary to solve this problem? And then ultimately that has to translate to political action. And I would not describe our book in any way as a polemic. I think it's much closer to like a meditation. But it is the case that what we are trying to do here is to educate a political system that fundamentally is going to have to act to manage this technology and that requires people from whatever background and perspective being ready to contribute and push that system forward.

Speaker 2

95:49 So in short, political activism is the. Is the number one recommendation for

Speaker 3

95:55 maybe we would say well informed political activism. And that's. We've tried to do the well informed part here, but yeah, I do think this is a moment where political engagement, political activism is really important.

Speaker 5

96:05 And it's been noted. Yeah, we were just talking about. Right. We see this with the data center issue. This has become top of mind for people. I think as this stuff breaks out into the open more and more, I expect to see more issues that only US policy and AI nerds about care about a couple years ago becoming something that is really relevant to the general public.

Speaker 2

96:26 Do you favor well informed political activism? Maybe not quite a contradiction in terms, but it's like, it's often pretty coarse messages, right, that get sent. If somebody's like, okay, I'm gonna go join Pause AI protests, would you say, yeah, go do that? Or is that too coarse, not well informed enough for your taste? Like how, how sort of blunt, do you think people should be in their, in the signals they're trying to send to their representatives

Speaker 3

96:59 for better or for worse? I think you're talking to people here who spent a long time on the nuance of this, so you're probably going to get an answer from us, certainly from me, that says let's aspire to nuanced, well informed political activism. And in some sense, given that we've talked for almost two hours and we've written a book and spent four years of our lives all dedicated to the cause of democracy being the form of government that can manage this best, I do think I still have enough faith in the democratic system to manage this and that does, I think, require well informed political activism.

Speaker 3

97:32 So we've tried to do our part here to say yes, understand what's actually going on with the technology and the geopolitics, and I think this book is our contribution to that. But I wouldn't, I'm not saying you were doing this, but I wouldn't sort of write the American voter off and say, well, there's no way they can get the complexity of AI. They don't need to get all of it, but they can get enough of it, I think, to land in good spots.

Speaker 2

97:50 Yeah, I'd say it's the Congress as much as the voters that I'm worried about in terms of how rich those signals can really be.

Speaker 3

97:59 Well, I mean, you'll never find executive branch people defend the Congress, but I do think with some of the folks that have been elected this time around, and, you know, maybe it takes a Dem trifecta and the President who gets it more than the current one does, I'm enough of an idealist to believe, or I have enough faith in democracy to believe that that is the only path which we're going to actually address these issues. And I think it's fair to have a pessimism of the intellect about the Congress, but I also think you have to match that with an optimism of the will.

Speaker 3

98:28 And certainly this book is meant to inform that that optimism or that engagement and say like, yes, we can do this.

Speaker 2

98:35 If you do have time for a three question lightning round, I would do that. If not, we can skip it.

Speaker 3

98:41 Let's do it. Let's do it.

Speaker 2

98:43 Open source.

Speaker 5

98:44 We're government bloviators, especially me. So lightning is a relative term here. Ben, you can cut me off

Speaker 3

98:49 30. Let's make it actual lightning. Teddy.

Speaker 5

98:51 I'll do my best.

Speaker 2

98:53 Open source. Are we going to see a technological solution? Are we going to have to have a policy solution. Is China going to stop their companies from just releasing model weights? Yolo, what's the expectation?

Speaker 5

99:04 Ben, you can take this one.

Speaker 3

99:07 What a great lightning round. During our time in the government we felt that the benefits of open weight models substantially outweighed the risks. That might change at some point. I think it's probably going to be an open weight model from China that carries the most risk. I'm hopeful China does a better job than they have thus far at managing some of the cyber risks. We'll see in that answer. I'm informed by what seems to be the reporting that Chinese models are being used in cyber attacks and so forth.

Speaker 3

99:32 But I don't want to pretend to have done a deep dive on that.

Speaker 2

99:35 On biorisk. One of the policies you guys pushed for was this pre synthesis screening of DNA sequences. That seems to be one thing just about everybody can agree on. It still seems though like we are headed for a world where there are like very capable models in the bio domain soon and the world you know at large is like scarcely better off than we were going into. Covid I would say if you had the sort of benevolent dictator conk, would you put a pause on additional scaling or like closing of the loop with the wet labs in bio specifically until we're able to do some like hardening of the world or would you take some other approach?

Speaker 3

100:19 It's worth noting we had gene synthesis screening in the executive order of 2023. President Trump repealed it on the first day. They have since brought it back. But we did burn like almost two years in there, which worries me tremendously of not developing this policy. I would probably be very, very wary and would probably be comfortable banning the integration of AI into BSL 3 and 4 labs. I don't think I worry that much about BSL 1 labs because they're just not that dangerous pathogens.

Speaker 3

100:44 And there's probably upside for pharma research, but certainly for anything that is very dangerous. We should be extremely careful with that. And I'm absolutely in favor of radical investments in biosecurity and biosurveillance. We are simply not ready on that front. This was we did the max of what we could with executive authorities, but this is a tier one issue for Congress in my view. Teddy, you want to take. I've taken both lightning so Teddy should go here.

Speaker 5

101:08 The only thing I'll add here is that as many of your listeners may know, this is especially challenging because the biofocus models are often much, much smaller and require less compute train than the big general purpose ones, which makes any kind of restriction or even monitoring much more challenging because the number of actors who could potentially do this is significantly higher.

Speaker 2

101:29 Any advice you would give or is it just sort of out of their control to countries 3 through 193 around how they can shape AI? We've got like the Netherlands, you know, has maybe a claim. Taiwan has a bit of a way to influence things perhaps. But is there anything anybody can really do? Aside from.

Speaker 5

101:51 I'll take this one because I. Because I live and run a company in what most people would probably call country three here, which is the United Kingdom. And I think there are a lot of opportunities, especially in the vein of pressing on stuff like AI safety and security research, which I would say is under resourced relative to capability stuff, and where amounts of money that can be deployed by non US China entities can really move the needle.

Speaker 5

102:17 A great example of this is the AI Security Institute, previously called the AI Safety Institute in the uk, which I think has done really, really good work.

Speaker 2

102:24 Yeah, excellent. No doubt

Speaker 3

102:26 that was the first lightning answer I've heard from Teddy Collins in like 12 years.

Speaker 5

102:30 Yeah, we've known each other for decades and

Speaker 2

102:32 just snuck it in under the wire. Well, I really appreciate you both for your time and public service, for being ahead of the game on AI by, you know, a significant margin and also the, you know, just the kind of. While I sort of from the outside understood you guys to be pretty sharp China hawks coming into this conversation, I think I also heard like a lot of moderation that, you know, I'm sure was always there, but like, doesn't always reach the, the surface view that outsiders have.

Speaker 2

103:02 And I really especially hope that the current administration does what you say you would do if you were in, in power now in terms of, you know, continuing to really try to reach out to the Chinese government and establish what common ground may be established. And I don't know, you know, nobody obviously knows until we try what that is, but I'm heartened to hear that you would be trying if you were in there now, and I hope they follow your lead on that.

Speaker 2

103:29 With that, I'll give you guys the last word. Anything you want to leave people with before we break

Speaker 3

103:34 you first, Teddy,

Speaker 5

103:35 I think we've covered all the big things, honestly, great questions.

Speaker 3

103:39 I'll just say, you know, I've done a number of podcasts and book talks and so forth, and the thing that is greatest for an author is when someone has read the book carefully and engaged with the ideas even if they don't always agree with the ideas. So appreciate the the conversation here. I think this is a book that certainly we are very proud of, and it has a lot of nuance and but the opportunity to unpack at least some of that nuance with you has been a lot of fun.

Speaker 2

104:03 My pleasure. That's very kind. I very much appreciate it. Ben Buchanan, Tantum Collins thank you both for being part of the Cognitive Revolution.

Speaker 5

104:10 Thank you for having us.

Speaker 3

104:11 Thanks so much.

Speaker 6

104:29 We ran so we could walk, we ran so we could walk? Now the road is open wide so walk with me walk all night long we kept on running, never stop to feel our feet? Now there's no one at our shoulder and the morning's cool and sweet?

Speaker 6

105:02 You say faster, always faster. Someone's coming up behind? Tell me what was all the running for if not a little time? We ran so we could walk, we ran so we could walk? Now the road is open wide so walk with me, walk.

Speaker 6

105:54 There's a river up ahead now and the bridge is thin and new? I would rather cross it slowly I would rather across with you. Fast is how we got here, slow is how we stay? Being first is only good oh what you do with it today we ran so we could walk?

Speaker 6

106:32 We ran so we could walk, so we could walk? Now the road is open wide so walk with me, walk. We ran so we could walk we ran so we could walk? The whole wide day in front of us. So walk with me, you.

Speaker 6

107:24 Smiley.

Speaker 1

107:39 If you're finding value in the show, we'd appreciate it if you take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions and sponsorship inquiries either via our website Cognitiverevolution, AI or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a 16Z where experts talk technology, business, economics, geopolitics, culture, and more.

Speaker 1

108:11 We're produced by AI Podcasting. If you're looking for podcast production, help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement@aipodcast.ing. and thank you to everyone who listens for being part of the Cognitive Revolution.

Summary

Ben Buchanan and Tantum Collins discuss their new book, *The Bitter Struggle*, which explores the intersection of AI technology and national security, particularly in relation to China. They advocate for a strategic approach to AI development that emphasizes safety and collaboration while acknowledging the competitive landscape between the U.S. and China.

- The U.S. has a significant lead in AI chip production and should leverage this advantage to prioritize safety rather than rushing ahead recklessly.
- The authors argue for the necessity of international cooperation on AI safety, despite current tensions with China.
- They express concerns about the potential for democracies to drift towards autocracy due to the centralizing power of AI.
- The book emphasizes the importance of well-informed political activism to shape AI policy and governance.
- They highlight the challenges of regulating AI, especially in the face of rapid technological advancements and geopolitical rivalries.
- The authors stress the need for a nuanced understanding of AI's implications, advocating for a balance between technological advancement and democratic values.
- They propose that effective governance requires increasing public engagement and oversight in line with the growing capabilities of AI technologies.
- The conversation reflects a cautious optimism about the potential for democratic systems to manage AI responsibly, provided there is adequate political will and public involvement.

Questions Answered

What is the focus of today's episode?

The episode features guests identified as China Hawks from the Biden administration, discussing their role in AI policy and national security.

What capabilities does Claude offer?

Claude is presented as an AI collaborator that enhances problem-solving by understanding workflows and working alongside users.

How are AI agents integrated into the show's production?

AI agents assist in researching guests and managing PR pitches, streamlining the production process.

What was the response to U.S. actions regarding chip controls?

The Biden administration's actions were carefully calibrated, resulting in minimal retaliation from China compared to previous administrations.

What is the importance of engagement in international relations?

Engagement is seen as essential for achieving positive outcomes, rather than a binary choice of engagement or non-engagement.

What are the risks associated with pausing AI development?

A time-bound pause could lead to a 'compute overhang', resulting in a sudden leap in capabilities when development resumes.

What are the potential risks associated with AI and bio technologies?

The conversation addresses the risks of misuse and misalignment in AI, particularly in the context of cyber attacks and bioengineering.

© transcribe · For agents Built with care and craft by Gokul Rajaram