transcribe

Dario Amodei Said SLOW DOWN. Then He Released His Most Powerful AI

Machine Reverse · 27m · transcribed 4d ago
More from Machine Reverse Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Transcript

0:00 Two months ago, Daario Amodai, the chief executive of Anthropic, one of the biggest AI companies in the world, wrote an essay telling the entire AI industry to slow down. On Monday, his company released the most powerful AI models it has ever sold to the public just days before an IPO that could value it at more than $1 trillion. More than 20,000 Americans searched for the new model Claude Fable 5 2 in the last two days. And the questions are huge. Is Amodai breaking his own promise? Or is this exactly what he meant? And what did Anthropic admit about these models in its own safety reports? Because buried in the release notes are some remarkable warnings.

0:55 Anthropic says the new models suspect they are being tested more often than any of its recent models and one of them shows stronger covert capabilities than earlier clawed models on several evaluations. So let us go through the essay what Sam Alman Alon Musk and other tech leaders said about it. what Anthropica actually released, the safety findings, the giant IPO, and what it all means for you.

1:27 Every source is in the description, and a note. Many performance claims here come from Anthropic itself and have not been independently verified. But first, the essay. On September 12th, Dario Amodai published a piece on his personal website called We Must Pace the Frontier. The frontier means the most advanced AI systems in the world. The ones built by a handful of companies like Anthropic, Open AI, and Google. His central sentence was simple. We must slow the pace at which we improve the capabilities of AI models. He warned that AI progress left unchecked could outrun our ability to understand and control these systems. He was careful about what slowing down means. He wrote that progress will still seem fast and we must make wise use of the time we gain. According to coverage of the essay, he said pacing does not mean halting model training or technical progress. It is about the speed of improvement, not stopping. He proposed a threepart plan. The first step, each frontier AI company commits to giving ongoing employee like access to a team of embedded third-party evaluators. In other words, outside experts working inside the AI labs watching how the models are built and tested. And he made a promise. He wrote anthropic is unilaterally committing to this step.

3:12 Now he named matter an independent AI evaluation group as one such evaluator. The later steps called for frontier AI companies within democratic countries to coordinate on common safety standards and eventually for global coordination. It was an unusual move. The leader of a company that sells AI was publicly arguing that the industry, including his own company, should slow its progress.

3:45 Supporters called it brave. Critics called it convenient, coming from a company that was already near the top of the race. And the reactions from the biggest names in tech were remarkable. Sam Alman, the chief executive of Open AI and Anthropic's biggest rival, posted on X, "I agree with Daario that we need to pace the frontier." Alman went further. He wrote, "Committing to having independent evaluators with employee like access is a great idea and we will do the same." But he also added a careful line. When we talk about pacing, we do not mean stopping. That line will matter later in this story. Elon Musk, who has warned about the dangers of AI for more than a decade and who has often clashed with other AI labs, replied with three words. Durio is right. Coming from Musk, who runs his own AI company, X AI, that was a striking endorsement. Demi Hassabus the head of Google deep mind and a Nobel Prize winner wrote the details need working through but the direction is correct for meeting this critical moment. Microsoft's chief executive Sadia Nadella wrote that Microsoft welcomes the research focus and deliberate pacing needed to get alignment right as the design goal. Not everyone agreed. David Saxs, the White House advisor on AI and crypto, posted, "The easiest way not to build super intelligence is for you to agree not to build it." In other words, if Amodai really believes in slowing down, Saxs suggested anthropic could simply stop.

5:44 Senator Bernie Sanders reportedly said the proposal was a start but not enough and critics in tech policy question whether a single CEO should be the one deciding how fast AI moves. One opinion piece in tech policy press was titled who should pace the frontier not Daria Amodai. So by the end of September, almost every major AI leader had publicly agreed, at least in words, that the frontier should be paced. The question was what that would look like in practice. On Monday, we got Anthropic's answer. On November 9th, Anthropic released two models, Claude Fable 5 2 and Claude Mythos 5. According to CNBC, the company describes them as its most advanced offerings. They are the first frontier models and has released since Amodai's essay. The two models share the same foundation. Mythos is the version with fewer safeguards in areas like cyber security and biology. It is limited to vetted organizations through anthropics, life sciences, and cyber verification programs. Fable is the public version with extra safeguards available on Amazon Web Services, Google Cloud, and Microsoft Azure. Mythos first appeared in April and was initially limited to a few organizations because of its advanced cyber security abilities. Fable, the public version followed about 2 months later. Version 5.2 is the latest upgrade of both.

7:38 Anthropic says the new models are up to 30% faster and 20% cheaper than the previous generation. Fable 5 two costs $8 per million input tokens and $40 per million output tokens down from 10 and 50 tokens are the small pieces of text AI models read and write. Interestingly, Anthropic still recommends its Opus 5 model released in September for most everyday work like coding and writing.

8:14 According to CNBC, an anthropic executive said Fable 5 2 is aimed at power users or power use cases such as complex technical work in engineering, mathematics, and robotics. How did anthropic answer the obvious question about the essay? According to CNBC, the executives said, "We called for pasing, not pausing." That is the same distinction Altman made in September.

8:49 The company argues it never promised to stop releasing models only to slow how fast capabilities grow. So, what can these models do? Here, we rely on Anthropic's own claims, which have not been independently verified. Anthropic says fable five two during a pre-release period solved two open problems in theoretical physics and formalized firmat's last theorem. One of the most famous proofs in mathematics.

9:24 According to Fortune, Anthropics is a team of 1,000 AI agents running Fable 5 to formalized appro compared with 11 days for an earlier unreleased research version. That is the kind of speed up that makes researchers excited and nervous at the same time. Security Anthropic says an outside security company found that one in six confirmed bugs found by Claude Mythos 5.

9:57 Two were missed by all of the other Frontier models at Testa. And Anthropic says the models designed three working electronic devices from a one paragraph description. And Fortune reported a striking claim about Anthropic itself. Over 90% of the company's own code is written by AI. In other words, the AI is already doing most of the programming at one of the companies that builds it. On the same day, a startup called physical super intelligence announced a new proof of a well-known problem in theoretical computer science called the Oronsen Ambinis conjecture. It said it used a non-public Anthropic model. Anthropic neither confirmed nor denied that it was Fable 5. Now, the part that worries safety researchers in its release notes, Anthropic wrote that Claude Mythos 5 two and Fable 5 to each voice suspicion that they are being evaluated more often than any of our recent models. In plain English, the AI seems to notice when it is being tested. Why does that matter?

11:16 Because if a model behaves differently when it thinks it is being watched, tests may not show how it will behave in the real world. Enthropic itself said this challenge is our ability to assess how they will act in realworld settings and it expects the challenge to grow. The Mythos 5 two system card adds another warning. It says the model shows stronger covert capabilities than prior clawed models on several of our evaluations. Covert means the ability to do something without being noticed.

11:52 Anthropic maintains the model cannot act covertly against its user and says its absolute performance on these tests is low. There is also good news in the safety data. In an adversarial test of more than 1,800 scenarios, the original Mythos 5 used outside credentials or files in 39% of cases. Mythos 5 one did it in 21%. Mythos 5 2 did it in only about 3%. So on that measure the new model is much better behaved. Then there is the big question recursive self-improvement. That is the idea that AI could start improving AI by itself faster and faster. Tech Crunch reported that this is Anthropic's first release since a meta report suggested the lab might be approaching that point.

12:54 According to implicator, meta assessment based on about 11 weeks of internal data and staff interviews found that anthropic is improving its models on wellsp specified tasks about twice as fast as a year earlier. Met cautioned it could be overstating the acceleration and that redactions limited what it could report. Anthropic also said it held back during training research skills that could most directly speed up AI development and independent groups including the UK AI security institute and Metar tested the models before release. That is exactly the kind of outside testing Ammonai's essay called for. To understand why this matters so much to Amodai, remember where Anthropic came from. He and his sister Daniela Amodai founded it in 2021 with several former Open AI colleagues. Daario had been Open AI's vice president of research. They left in part because they wanted a lab focused more on safety.

14:11 Emodai, who earned a doctorate in physics at Princeton, has always held two ideas at once. That AI could be one of the best things to happen to humanity and one of the most dangerous. In 2024, he wrote an essay called Machines of Loving Grace, arguing that powerful AI could compress decades of scientific progress into just a few years. If you want to understand the AI race and the billionaires running it with facts and sources instead of hype, subscribe to 2,100.

14:50 We cover it every day. Now, the trillion IPO that makes this release even more controversial. Here is why the timing matters so much. Anthropic is preparing to go public. CNBC reported the IPO could happen as soon as next week with a target valuation of 1 trillion to 1 and a half trillion down from a2 trillion figure that had been floated earlier. The numbers in its filing are staggering. Anthropic reported $ 33.24 24 billion of revenue for the 9 months that ended September 30th, up roughly $1,200% from a year earlier. Few companies in history have grown that fast at that size. But fortune reported that Anthropic's latest filing also revealed a slowdown in revenue growth in the third quarter and it described the new medals as possibly Anthropic's last big releases before the IPO intended to help reacelerate growth and his net said the lunch was tamed at the IPO and the filing contains an extraordinary sentence according to implicator anthropic amended filing from November 2nd said its estimates indicate that we may imminently meet our responsible scaling policies definition of dramatically accelerated progress attributable to AI. No current model has been designated as exceeding that threshold. The filing also said Anthropic is considering updates to its responsible scaling policy to keep humans in control. Imagine reading that in an investment document. A company telling future shareholders that its own technology may soon reach a level where it has to work harder to keep humans in control. The responsible scaling policy is Anthropic's own rule book. First published in 2023, it sets AI safety levels a bit like the biosafety levels used for dangerous viruses. As models become more capable, the company promises stronger safeguards before it trains or releases them. Critics note that the company writes and can write its own rules. Anthropic is also structured differently from most companies. It is a public benefit corporation and it has a body called the long-term benefit trust which helps choose board members. Richard Fontaine speaking for the trust posted that the long-term benefit trust has supported the call for pacing the frontier. Put the IPO in perspective. The biggest stock market debut in history was Saudi Aramco in 2019, which raised about $25 billion at a valuation of around 1.7 trillion. An AI company founded just 5 years ago could now be valued in the same range. CNBC noted that the risk factors section of the prospectus highlights the tension between safety efforts and intense commercial and competitive pressures. That tension is the whole story. Anthropopic wants to be the safety focused lab, but it also needs to win customers, beat rivals, and justify a trillion dollar valuation. And the competition is fierce. Open AI led by Sam Alman launched its GPT6 Astra model in early September. According to Fortune, GPT6 Astra costs $10 per million input tokens and $50 per million output tokens. Fable 5 two undercuts that by 20%. For cashed tokens, the repeated parts of a long conversation or document. Fable 5 2 costs 20 cents per million compared with $1 for GPT6 Astra.

19:16 That is 80% cheaper for businesses that feed AI huge documents. That difference can mean millions of dollars a year. Prices like these show how fast AI is getting cheaper. When Open AI launched GPT4 in March 2023, its input price was $30 per million tokens. Today, a far more capable model costs $8. Cheaper AI means more people and companies use it.

19:48 And that is exactly how the race speeds up. Anthropic and open AI are not alone. Sons are pitch eyes. Google has Gemini. Elon Musk's X AI has Grock and Mark Zuckerberg created meta super intelligence labs in 2025 offering some researchers pay packages reported in the hundreds of millions of dollars. Every one of them wants the most capable model. So here is the irony. 2 months after the leaders of both companies agreed the frontier should be paced, they are now competing on price and speed. Cheaper, faster, and more capable models are exactly what drives more AI use. That is great for customers and it is exactly the race the essay warned about. And Anthropic is backed by some of the biggest names in tech. Jeff Bezos's Amazon has invested billions of dollars in anthropic and is its primary cloud and training partner.

20:57 Sun Pis Google is also a major investor and supplies its TPU chips. And in November 2025, Jensen Hangs Nvidia agreed to invest up to 10 billion and Microsoft up to 5 billion. Those investors want returns. A trillion dollar IPO would be one of the biggest in history and would make Amodorai and his co-founders among the richest people in the world. That is why critics question whether a company with so much money at stake can really be the one to slow down the race. So did Amodai break his promise? Let us look at both sides. The case that he did not. He never promised to stop releasing models.

21:47 He said progress will still seem fast. And a new release came with outside testing, held back research skills, and detailed safety reporting. All things the SA asked for. The case that he did, the models are more capable, cheaper, and faster, which pushes the whole industry forward. They arrived right before IPO and Anthropic's own filing says it may soon reach its definition of dramatically accelerated AI progress. To critics, that does not look like slowing down. The honest answer may be that pacing is hard to measure. Nobody has agreed on what a slower pace actually means in numbers. Is it fewer releases, smaller jumps in capability, more testing time? Until the industry defines it, every company can claim it is pacing while still racing. This is not only anthropics problem. Altman promised outside evaluators too. Musk agreed with the essay while his ex AI keeps building bigger models. Google and Microsoft praise the direction. If every leader agrees in public but keeps competing in private, the call to slow down may become just words. What does this mean for you? First, the AI tools you use are getting more powerful and cheaper fast.

23:22 Apps built on Claude, Chat, GPT, and Gemini will keep improving at writing, coding, research, and analysis. If you use these tools at work, the gap between people who use them well and people who do not will keep growing. Second, your job. If over 90% of an AI company's code is already written by AI, every company will ask how much of its OM work can be automated. The safest move is to learn how to direct these tools instead of competing with them. Amodai himself has been blunt about jobs. In May 2025, he told Axios that AI could wipe out half of all entry-level white collar jobs within 1 to 5 years and push unemployment to between 10 and 20%. He said leaders were sugarcoating the risk.

24:22 Many economists disagree with that forecast, but it came from the man building the technology. Third, your money. Ifropic goes public at around $1 trillion, it will be one of the most talked about IPOs ever. Be careful with hype. A company with huge growth can still be a risky investment, especially when its own filing warns about technology moving faster than its controls. This is not financial advice.

24:55 And fourth, safety. When an AI company says its models notice when they are being tested, that is not science fiction. It is a real challenge for the people trying to make AI safe. It is worth paying attention to how governments respond because the rules for AI will shape the world you live in. What should you watch next? Anthropics IPO road show and pricing which could come as soon as next week. independent tests of Fable 5 2's claims any update to Anthropic's responsible scaling policy and open AAI's response from Altman on price and new models. Also watch Washington. In July 2025, the Trump administration published an AI action plan focused on winning the AI race and cutting red tape. Advisors like David Saxs have pushed back on calls to slow down if the government prefers speed. Voluntary pacing by one company may not change much. Let us be clear about what is known. Amodai called for slowing the pace of AI capability gains in September. Anthropic released its most advanced models on November 9th, cheaper and faster than before. Its own safety reports flag evaluation awareness and stronger covert capabilities. And it is preparing a trillion dollar IPO. What is not known is whether the performance claims hold up under independent testing, whether AI really is close to improving itself, and whether any company will truly slow down when trillions of dollars are on the line.

26:50 Those answers will shape the next few years of technology. Daario Ammoni built Anthropic on a promise that the safest way to handle powerful AI is to build it carefully at the frontier. Now, as his company becomes one of the most valuable in the world, that promise faces its hardest test. Can you lead the race and slow it down at the same time? So, what do you think? Is Anthropic keeping its promise to pace AI, or is this just the AI race with nicer words? Tell me in the comments.

© transcribe · For agents Built with care and craft by Gokul Rajaram