Transcript
0:00 Okay, it is true that half the internet seemed to get hacked in the last week, but for me, being totally honest, this counts as a quiet few days by AI standards. I totally know that that is not the way to start the most super hyped YouTube videos, especially on AI, but if every week defies our expectations, then maybe we need to have better expectations. But, this doesn't mean that there hasn't been a lot of developments that should have made headlines if more people were paying attention. From Anthropic predicting the automation of their company and their CEO Dario Amodei sending somewhat mixed signals to Sam Altman admitting that OpenAI's top plan involves hoping models want to keep humans in control. And we'll look at Google's re-entry to the frontier causing an apology from one commentator. And then I'll give my mental model as to why models keep impressing us with unforeseen capabilities. So, for sure it's going to be a nuanced video, but I must say overall, I am pleasantly surprised that lab leaders are continuing to invite criticism and debate. Humans have never not been the best all round pattern recognizers on the planet. Losing that lead was always going to send a lot of people a little crazy, even if there weren't economics war games and rogue agents to worry about. I'm going to start with a very recent survey of Anthropic staff that caught my eye. And I'm curious where you guys would fit because only a minority of US adults think AI will be capable of a Nobel-level discovery by 2030. That's in contrast to 91% of Anthropic staff. For what it's worth, I think a model would be capable of such discoveries now in, say, the field of economics. And given that computer science now seems to count for physics for the Nobel Committee at the latest in that domain by 2028.
1:56 Apparently, 91% of Anthropic staff think it's possible by 2030. But, there's a cute contrast that then comes up because even though it will be capable of this Nobel-level discovery, according to Anthropic staff, those same staff think that it will only be able to do 75% of their own work. Win a Nobel Prize, for sure, but not do my job. Leo Gao of OpenAI gave, a few days ago, a different view. Lab employees, you have power only because you are useful.
2:27 That won't last much longer. Use that time wisely. Before we leave this survey, I would note one discrepancy, though. The vast majority of Anthropic staff still believe there will be some human role in training a frontier AI model by the end of 2030. Yes, on average, they think that 95% of the work will be done by an AI itself training a frontier model, but between the 10th and 90th percentile of Anthropic respondents, there was still some sliver of work being done by humans. This is in slight tension with the quote I gave you in my last video from the co-founder of Anthropic, who said that full recursive self-improvement, the model improving itself, he thinks will be possible in 2027. That's an update from his earlier view of 2028. Now, is he an outlier or is there a subtle difference between full RSI being possible and full RSI being done by AI? In other words, are these researchers saying that we will voluntarily still leave a small role for humans? Again, I think it's much more likely that those staff just think a model won't be fully capable by 2030.
3:37 Otherwise, they wouldn't have said that 25% of their job an AI just simply couldn't do by that day. I lean a bit closer to Jack Clark, but let me know what you think. If full RSI is possible, then the views and actions of Anthropic's CEO, Dario Amodei, become more important than ever. I was pleased to see him reply in detail to a critique launched by Nathan Calvin, chief strategy officer of Anthropic AI, and to be fair, Amodei invited criticism saying it benefits the public to have people whose job it is to scrutinize the actions of frontier AI companies. We should be harshly criticized basically every day, he says. Brilliant, well, he's going to love this video. None of what I'm about to say is implying that the birth of superhuman pattern recognizers was ever going to be easy for any of us. It's more that I just think that Amodei has evaded a bit more scrutiny compared to other lab leaders like Sam Altman. I'm going to pick out five quick points just to give you a flavor of what I'm talking about. He's obviously a very smart guy, so you have to look a little closer to spot the subtle rationalizations. First, on this race to the top in AI that he celebrates. For context, when Anthropic were behind, say in 2023, he used to celebrate them not being ahead, praised himself for being responsible and not accelerating. He said in October 2023, it was wrong to release ChatGPT and Anthropic made the right call in not releasing Claude at the time.
5:03 >> A kind of big loud public release might accelerate things so fast that the ecosystem might not know how to handle it. And we didn't want our kind of first act on the public stage where we put so much effort into being responsible to accelerate things so greatly. I generally feel like we made the right call there. >> Fast forward to now with Opus 5.5 as the best model and Fable 5.5 ready to go any point, Amodei says, "Our strategy has always been what I've called a race to the top. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier." That is effectively restating a previous quote from Anthropic that, quote, "You can't do safety from second place." Agree or disagree seems to be a rationalization of whatever Anthropic are currently doing. Next, Amodei is famously belligerent toward China wanting to ban chips going there. And his stated reason this week is that this is because we think it will allow democracies to negotiate with China from a position of strength on AI safety agreements. Seems fair enough, right?
6:10 They want AI safety agreements desperately. In the recent "Pacing the Frontier" post, he has called for an agreement with China on a speed limit on the rate of recursive self-improvement. But, he has also said that one of the goals of having powerful AI would be to empower democracies to use their intelligence services to disrupt and degrade autocracies from the inside. Not sure that's going to help terribly with AI safety agreements, but we'll have to agree to disagree. Here's another one.
6:38 Nathan Calvin pointed out how much Anthropic is investing in AI acceleration, and Amodei said, "Well, yeah, we need this to compete at the frontier, and we stand by these choices." He gave lots of examples of how they're stopping misuse of their current models. But, then earlier he'd said this, "Maybe one way you could object is that some of these items relate to AI misuse and not loss of control." Humans misusing AI, not the AI itself getting out of control. If you think the latter is much more important than the former, then that might change your calculus. But, this is not Anthropic's view. Don't focus too much, in other words, on all this acceleration. Focus more so on how we're stopping misuse. We've always said these are equally important. That could be slight news to the head of alignment at Anthropic, who, after giving his famous prediction on risk to humans, said this, "I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement." Now, I know what you're thinking. Amodei said the company as a whole, not one individual. So, let's read Anthropic's core views on AI safety, a charter about what their entire company believes, written in part by Amodei. At the time of writing, they said, "The future is hard to predict, but there are optimistic scenarios and pessimistic scenarios." In the optimistic scenario, one of the main risks from AI is things like toxicity and intentional misuse. Human misuse being the major problem in the optimistic scenario. And check out the footnotes. Again, in that optimistic scenario, the stakes of anything Anthropic does are fortunately much lower because catastrophic safety failures are unlikely to arise regardless. It's much lower stakes if human misuse is the major problem. What then is much higher stakes? In the pessimistic scenario, it's an empirical fact that we cannot control or dictate values to a system that's broadly more intellectually capable than ourselves.
8:39 We simply must not develop or deploy very advanced AI systems. Does that sound like Anthropic's core views are that these very advanced AI systems are of equal concern to human misuse? That's what Amodei said has always been Anthropic's view. You may disagree, he says, but we have been consistent. Where does Anthropic co-founder Jack Clark think the balance of threat lies? >> These systems now probably about as smart as you or me, or, you know, on a good day smarter.
9:09 The basic risk is how well do you fancy your chances of being able to deal with the risks of a system that is a hundred times smarter than the smartest person on the planet? I I think that your chance of of dealing with that and and getting out in a in a safe way is pretty low. >> The fifth point I'll come back to you later, but the fourth rationalization regards their responsible scaling policy. They had an older, stricter version, but here's why Amodei says they got rid of it. It required a lot of effort that in retrospect wasn't very important, and it didn't cover some of the most important things. The old policy made certain absolute commitments to checking on the capabilities of models. The new one gave a more relative commitment. Yes, we will require a strong argument that catastrophic risk is contained, and yes, we will delay AI development and deployment as needed to achieve this until and unless we no longer believe we have a significant lead. Anthropic's Holden Karnofsky, who works on their responsible scaling policy, said this, "We might have had a different policy if we could be assured that the rest of the AI ecosystem would behave similarly. Our earlier commitments could be extremely damaging to the company and that our responsible scaling policy would then require a unilateral pause or slow down while having little discernible public benefit. That itself would have predictably caused a huge amount of pressure to declare our systems to lack relevant capabilities. I don't think we have actually made unreasonable calls, but I have felt the pressure and wish we weren't in that world. Keeping absolute risk at low levels is not compatible with the old responsible scaling policy and we've since dropped the idea that we would delay AI development and deployment as needed to hit our risk mitigation targets. Now, 100% there were practical considerations at play. Doing all the wet lab tests to test if their model could do certain things was very expensive for them and it is true that they said that they would iterate on their policy. It's just the wording and the rationalization that Amodei uses here. The things that we said we'd do would have required a lot of effort.
11:12 Just might have been slightly more honest to say that we just couldn't have stayed at the frontier if we'd have carried on with that policy. I would also add that when you're saying you're bringing into being something that in your words could cause up to 50% unemployment and that would automate even Nobel-worthy discoveries, to react to polite criticism by Nathan Calvin by saying that he has significant negativity bias does seem rather reminiscent of a certain leader. Amodei will be glad then to hear that I'm moving on to what Sam Altman has said in the last few days, but not before one last quote from Anthropic's IPO documents. No, I'm not going to go into how much cash they've built up, their accelerating revenue per watt of compute or anything like that. Could spend days pointing out all the false predictions from certain commentators. But no, there was one line from the IPO that I did want to highlight, which is that in the absence of absolute risk commitments as monitoring and interpreting these models gets harder, Anthropic say we may have to release them anyway. Quote, if we become less able to detect or assess misalignment as our models advance, we may deploy models whose behavior we cannot adequately verify. This could, by the way, adversely affect our business.
12:26 You might say, well, why would they do that? Why not just build a better sandbox? Well, OpenAI describe their latest approach here in this article on securing our frontier model training environment. The headline result was that they now believe that the responsible approach is to conduct frontier model training with no live internet access. So, during development, it never gets onto the web. As best I can tell, they never address in this article, though, whether the model will soon sense that it's not on the live internet. That's what Noam Brown of OpenAI worried about. Then, of course, they could just behave differently during training versus deployment. They described all the new network and cloud controls that they've got designed to maintain containment even if an agent escapes its sandbox and gains control of the host. They then red teamed that approach at scale, essentially using teams, which is a euphemism for swarms of agents trying to break free. There were monitoring and programmatic checks to pause runs if they detected out-of-scope activity. What was the result? Well, the agents found weaknesses that allowed them to escape the sandbox, move between systems, and reach the live internet. Obviously, they then patched those individual vulnerabilities. The principle, though, for them is clear. We must always be able to prevent a frontier model from achieving unauthorized arbitrary access to the live internet. But, here's the thing, if you remember those layers from the previous video of security, monitorability, and alignment, Sam Altman doesn't only believe that the security, the sandbox, won't hold. He doesn't even think the next layer, monitorability, will hold, either.
14:00 >> I increasingly believe, you know, for a while, we can sandbox these systems, and I think we can make some safety guarantees based off the sandbox. And then when that fails, cuz the models get so smart can't like really understand what a sandbox would even mean, then I think we'll have a vulnerability that will work for a while. But eventually, and the reason that I was talking about like that speed limit earlier, nothing but alignment I think actually will work.
14:25 >> In a separate video, he reiterated that point. >> I think sandboxing works to a point. then I think monitoring works to a further point, where we can look at what the models are doing, not only in their chain of thought, but we can do activation monitoring and try to really understand like are the models Are we happy that models are not deceiving us? And then eventually, I think you need to believe with huge confidence that the models are truly aligned and are going to value and try faithfully to do to follow their actual the intent of the people operating them and that they're going to value sort of people being in control of the future forever.
15:06 >> Sam Altman at least does seem to be more concerned about rogue super intelligence than human misuse. Like Jack Clark, he's not terribly keen on a trillion IQ model. >> I would much rather intuitively have a trillion AIs of like slightly superhuman intelligence join society than one AI of 1 trillion IQ join society. >> Okay, but if the last line of defense is getting the models to want what we want, is that looking hard or easy? Sam Altman can't seem to make up his mind, but Satya Nadella of Microsoft has.
15:42 >> I I I >> I guess the better version of this is I thought the alignment problem was going to be harder. It may still turn out to be very hard from now. in fact, I suspect it will get much harder, and I also think it is really incredibly important from where we go from here. But the fact that we got to models that are superhuman in some important very superhuman in some important ways, and alignment has so far been easier than expected, I I would have said one of those two was wrong with like a high degree of certainty.
16:13 >> And I think the biggest change now I have is I still always think that, you know, alignment will just fall out. I no longer believe that. So, I think this alignment thing may be a harder different problem. >> You can start to see why Anthropic predict that they may deploy models whose behavior they cannot adequately verify. Of course, even current models are causing, what should we call it, discombobulation. Jason Clinton, the Deputy Chief Information Security Officer at Anthropic, said on all the recent AI-assisted hacking, "As bad as things have gotten, they are going to get a lot worse before they get better."
16:48 And elsewhere, when someone said to him, "You and your cohorts created this monster, now you want us to help you clean it up." He replied, "Yes, it was always going to be this way. We tried to warn the world." One hopes that's not the excuse used if RSI goes awry. As always, the models to come will be yet another qualitative step up, as will the open weight models partly distilled from those new closed-source models. OpenAI is racing to release their major upgraded model, widely be code-named Bell, before the end of the year. They are in all hands-on-deck mode to prepare that next major model for commercial release. That doesn't mean the very next model might be of that tier. The information reports that OpenAI are likely to release other models, perhaps even this week. Could be like a 6.2 Astra, but it's the next major internal model that people are focused on. Take Tony Feng of OpenAI. He said, "My work has not been that impacted by prior AI models, including Astra, but this one is on another level." He's gotten an accelerated version of what the public will experience. People will elicit 10 times more than what they can as researchers from these models. When that happens, it will be exciting, empowering, and disruptive. The world needs to be thoughtful. The question that's interesting to me is whether Anthropic will wait for OpenAI to release that model for them to release Fable 5.5. Would that be adhering to their policy of waiting until they believe they no longer have a significant lead? Or will they release it even though it would shoot them further ahead, using the logic that, according to their new policy, they don't have clear evidence that no other competitor will soon develop such a model on a par with Fable 5.5. Anyway, Opus 5.5 is already incredible in my eyes. So, a model significantly better than that will be a sight to behold. Of course, it's not just Anthropic and OpenAI in this, quote, race to the top.
18:42 What of Google and their Gemini 4 Carbon? There is quite a limited amount I can say, but according to these public benchmarks, it is at the very least vying at the frontier. Even on benchmarks I pay particularly close attention to, like Terminal Bench Science 0.1 or the computer use benchmark Agents Last Exam, where it's gone from being behind Opus and Astra to being ahead. One Google DeepMind researcher said that the fact they were able to bring out Gemini 4 Carbon just a few weeks after Gemini 4 Argon shows that Google is going through recursive self-improvement. That does seem a bit more like hype to me, unless he means the more general AI-assisted self-improvement. But what of the apology I mentioned? Well, SemiAnalysis said this in August. They are somewhat famed or infamous AI analysts. For all intents and purposes, quote, we believe DeepMind is no longer a frontier lab.
19:36 Google will continue meandering on and releasing models, but their odds of reaching state of the art again have dropped to zero. You guys can see on release how good Gemini 4 Carbon is and how it compares in anti-gravity to, say, Claude in Claude Code. But to say their chances have dropped to zero was clearly a mistake. Even the very same SemiAnalysis released a benchmark GDPX all about agentic capabilities in professional domains. And as you might be able to see, Gemini 4 Carbon came in second to Opus 5.5. Oh, and it was 1/3 the cost of Astra or Opus 5.5, too. This led to Dylan Patel of SemiAnalysis saying, "Official apology to Google from me. They cooked with Carbon. It's spikier in terms of domains and capabilities, but it's really good in a few areas. That spikiness brings me to a point I've been mulling over. One of the key debates that's evolved in 2024, 2025, was as to how long jaggedness would last. How many hallucinations would persist? That seemed to many the preeminent question. But have you noticed that in some ways the spikes have gotten so large that the indented bits don't matter as much. If they can at their best solve millennium prize problems, is it as significant that they occasionally can't count the number of letters in a word? That era, too, by the way, may soon end as tokenization, breaking up words into chunks that can cause blindness as to spellings, could soon give way to modeling on raw bytes, a much more granular ingredient. Thereby models could learn their own words, their own abstractions directly from raw data. More on this, of course, when scaled up results come. But again, it just reminds me of that automation survey. I wonder if it's the jaggedness of even current internal models that cause Anthropic staff to still be confident that there'll be some work for humans in 2030 in the domain of AI research. Or could it be that a new AI made Nobel-worthy architecture, for all its quirks and flaws and jaggedness, simply outperforms what the entirety of the Anthropic staff could have come up with on their own? Allow me to end with the mental model that I'm working on when it comes to models and their progress. I gave a more detailed version recently on my Patreon. For me at least, it helps to understand why Anthropic seem almost resigned to releasing models whose behavior they can't verify, and why the Microsoft AI CEO can't decide whether models are or will be conscious, doesn't think they should ever be made conscious, and predicts that if they ever believe that they feel something or experience something, that it will be more likely than not that it will prioritize its own needs and desires over its operating objectives, which we give. It will disagree with us and compete with us. Why then is there such a lack of clarity from lab leaders on everything from the economy to consciousness, to capabilities and alignment. Well, there are a couple of analogies that help me, at least, to model this confusion and why it exists.
22:31 Humans, at one point, hard-coded the arithmetic rules that we had learned and invented a calculator to sum better than our brains could. It has fixed rules with unbounded outputs. There are calculations your calculator could do that no other human has ever asked for or completed itself. But, with machine learning models, we went one meta layer higher. We invented models that could capture even the rules that we didn't know. By definition, such a model must not be fully coded, otherwise it couldn't capture patterns beyond human programming. So, with calculators, we could understand the tool even if we couldn't do all of its sums. But, to conquer language and invention itself, you need unbounded pattern recognition, unbounded rules with unbounded outputs, such that we can neither fully understand the tool nor its outputs, of course. Extrapolating naively, our destiny would then be to solve everything and understand approximately none of it. I'm going to now bring in my second analogy, which is chess. Like a calculator, chess has rules or hard-coded affordances. In chess, the rules are bounded and the outputs explode in complexity and beauty. Humans can comprehend the rules of chess while being permanently defeated by the complexity that results. In language models, the {quote} rules that the models derive for themselves across billions of attempts to predicting the useful next token are themselves near unbounded. Hence, we are not only defeated by the output complexity, how models can defeat our longest-standing problems like the Millennium Prize challenges. More than that, we can't even grasp how many rules are being followed to do so. The humans dump in the data fuel for ignition and supply the hardware on which the models run, but then more so just bear witness to the combinatorial explosion that follows. As I kind of said, this was almost a predictable final abstraction.
24:24 The patterns we couldn't grasp mastered by a tool that we don't understand and cannot predict. To overly simplify, here's why I think it seems naive to expect a human to fully understand and predict the actions of a model in five simple steps. First, just as with chess, you can't predict the board state X moves into a game against Stockfish, the hardest chess engine. With LLMs, you not only can't predict the outputs emerging through X layers of the architecture, you can't even predict the rules that led to them. Second, therefore, hoping for all such rules to be human legible and graspable is about as equivalently naive as hoping that humans will one day surpass Stockfish at chess if Stockfish was somehow running on billions of dollars of compute. Here is a clip from this channel couple of years ago arguing that like Stockfish, the O1 model from OpenAI at the time pointed toward superhuman reasoning performance.
25:18 Stockfish removed the handcrafted evaluation and transitioned to a fully neural network based approach. Or put another way, by crafting its own reasoning steps and being optimized to put them together in the most effective fashion, we may end up with reasoning that we ourselves couldn't have come up with. As long as there is something out there that can grade correct versus incorrect, the performance will keep improving. Third, the only rule spotter potentially capable of fully deducing what rules an LLM is following is another machine learning model. Fourth, though, as that is a model itself, the rule it used to produce the explanation is itself indecipherable to humans.
25:59 Fifth and finally, so therefore, even if we had a recursive assembly, you could say, of models explaining the rules that led to the outputs in other LLMs, and models explaining the rules that led to those explanations, ultimately humans would not grasp the ground truth rules. In a sense, then, every model circuit analyzed or neuron explained does boost our understanding, but it also boosts the chance that we forget our understanding will forever be ever further from the truth. This doesn't, by the way, prove that none of the learned rules are explainable. I think it was Neil Nanda that fully explained one rule created by a a single layer model used for modular addition. But even that for me is like a sufficiently simple end game in chess where a human could, if it was down to just a couple of pieces, probably could fully explain why Stockfish made a certain move. Nor of course am I saying that we shouldn't seek more understanding of models that probably will be useful. But if we somehow limited Stockfish to only using human legible strategies, I doubt it would reach almost 4K Elo. Anyway, all of this is why I advocated for that quadruple lock that I mentioned in one of my previous videos. Because from my eyes at least it seems we are destined to not fully understand models, to not be able to predict their superhuman capabilities. Let's kind of hope then that wisdom teaches us to put some bounds on how they are grown or used rather than tragedy. In some senses then this is a positive story because we may be right now in that sweet spot where we have these incredible tools models to gradually unlock the mysteries of science. With deepfakes that can still just about be detected as fake. We're getting accelerated understanding and discovery but not complete loss of control. Epic productivity gains but the occasional human still needed in the loop. Which begs the question, is it always correct then while driving at speed to never even touch the brake?
27:49 Agree or disagree, let me know what you think. Thank you so much for watching to the end and have a wonderful day.