transcribe

AI:AM Highlights: Clean Neurons, Black October, the GPU Credit Crunch, and Buying the Supply Chain

Cognitive Revolution "How AI Changes Everything" · 1h 28m · transcribed 28d ago
More from Cognitive Revolution "How AI Changes Everything" Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

The Future of AI Development

What factors will influence the pace of AI development?

The pace of AI development is influenced by financial resources, technological advancements, and regulatory frameworks. Currently, credit availability is a significant bottleneck in AI progress, rather than hardware or power limitations.

  • AI model release cycles are accelerating rapidly.
  • Credit availability is a critical factor for AI development.
  • Current advancements are reminiscent of past AI timelines.
# 14:46

Optimizing Neural Network Training

How can we improve the efficiency of neural network training?

Focusing on neurons that show strong activations can reduce the complexity of training neural networks, allowing for more efficient guessing and checking processes.

  • Strongly activated neurons are more critical for training.
  • Reducing focus on less active neurons can streamline the training process.
  • This approach can significantly decrease the number of necessary iterations.
# 29:32

Understanding AI and Human Cognition

How do AI models compare to human cognitive processes?

AI models, particularly LLMs, operate on a fundamentally different basis than human cognition, relying on a core set of logical operations rather than the complex and varied processes of human thought.

  • AI models function on a simpler set of operations compared to human brains.
  • There are significant differences in how AI and humans process information.
  • Understanding these differences is crucial for developing effective AI systems.
# 44:18

Funding Dynamics in AI Startups

Why do AI startups often secure large funding rounds without proven products?

AI startups frequently raise significant capital before demonstrating revenue or traction due to investor biases and the survival of only those that can secure funding in a competitive landscape.

  • Survivorship bias affects which AI startups succeed.
  • Investors often prioritize funding over proven business models.
  • The ability to secure funding is crucial for survival in the AI sector.
# 59:04

Challenges in Semiconductor and Biotech Innovation

What parallels exist between innovation in semiconductors and biotechnology?

Both fields face challenges in innovation, but while biotech is currently seen as more complex, there is potential for unexpected breakthroughs in semiconductor technology as well.

  • Semiconductor innovation may be more accessible than previously thought.
  • Biotech is currently viewed as more challenging than semiconductor development.
  • Unexpected innovations can emerge from less experienced entities in both fields.
# 73:50

Regulatory Challenges in AI Development

What are the implications of recent AI regulatory proposals?

Recent drafts of federal AI regulations, such as those proposed by Anthropic, aim to establish transparency while preempting state laws, but they have faced significant criticism regarding their effectiveness and implications.

  • Federal AI regulations are attempting to balance transparency and state law preemption.
  • Criticism of regulatory drafts highlights concerns over their practical implications.
  • The debate continues on how best to regulate AI development.

Transcript

Speaker 1

0:00 This week on AI in the am, what really sets the pace of AI from here? The money, the atoms or the law? Zvi Moshowitz who writes don't worry about the face on what happens to the model release cycle over the next year if nothing slows the labs down.

Speaker 2

0:15 Advances on the level of a current model release cycle go from two months to one month to two weeks to a week. Stuff like that could happen. You get to see continuously pretty dramatic, very rapid improvements in various aspects of the AI capabilities. We are on an AI 2027 style timeline right now. It looks very eerily similar, but we

Speaker 1

0:41 saw there Evan Conrad, co founder and CEO of SF Compute, which runs a market for GPU compute on what is really holding the build out back Right

Speaker 3

0:50 now the bottleneck in compute and AI in general is, is not power, it's not chips, it's entirely credit. So in order to get debt you need to work with somebody who has excellent credit because your lender has no idea how to underwrite the value of the collateral that they've been given, which is a GPU cluster. Like if you think about it, there's like nice, nice lender off in New York or London or something.

Speaker 3

1:21 If in the case in which the AI lab falls through and fails to pay and then the cloud fails to pay their lender, the lender is going to recoup a supercomputer and the lender has no idea what to do with that supercomputer, not really. Nor do they know what that's going to be worth in the future.

Speaker 1

1:40 Lonis Hamali is managing partner of Cambrian, a fund that bets on the physical supply chain behind AI. Full disclosure, I have committed to invest in Cambrian. Here is llanis on which suppliers the labs will never replace.

Speaker 4

1:54 Even in general when we're looking at the AI market we are way more bullish on the atoms over the bits. So most AI companies people looking at is like Nvidia, Broadcom, ARM Marvel and these are good companies, there's nothing wrong with them. The issue is that what their business is essentially just using OpenAI and anthropic models to design software and they send that software down to Korea and Japan and Taiwan for the actual work. These companies don't have their own physical know how or their own factories.

Speaker 4

2:24 And so once these are, you know, great well run businesses, they don't have at all close to the same moat as these physical copies have, which have, you know, massive fabs and initially have decades of know how built up. And it's just very hard so there will never be a situation where the labs will compete with a silicon wafer maker and we will always be a customer, whereas we're already seeing the labs compete directly with these software designers like NIA and Broadcom, etc.

Speaker 1

2:51 Prakash N. Raiden, my co host on what the market keeps getting wrong.

Speaker 5

2:56 I believe that for the next 20 years, the market will consistently underprice exponential growth because I think consistently human beings are just not able to understand exponential growth and logically kind of emotionally get comfortable with it. And I think the market is really driven by a lot of these human emotions and animal spirits, et cetera, which means that we're going to get again and again.

Speaker 5

3:26 You're going to get this bubble moment, collapse moment that we've seen in the last three months.

Speaker 1

3:32 Box CEO Aaron Levy posted that if you want to know which jobs AI will create, look no further than cybersecurity. Here is what I made of that.

Speaker 6

3:39 I don't doubt that like, really good cyber security people are gonna, you know, cash in on this trend. Like, there's no, you know, I'm sure that there's a bidding war for great talent, but like the idea that this is going to create jobs is honestly a bit ridiculous because the timescale is just absurd, right? And this is like, oh, who's gonna, who's going to pivot into these jobs? Are we, are we going to go get like the content marketers that are currently out of work or the graphic designers and be like, oh, you know what, you should.

Speaker 4

4:13 This is the new.

Speaker 6

4:13 Learning to code is like, learn to become a top tier cybersecurity professional that can contend with these like basically super intelligent agent swarms. But just the idea that you're going to have like some sort of boon in hiring these people don't have time to hire right now.

Speaker 1

4:30 They're going to plow everything that they

Speaker 6

4:31 have into agents and they're going to put agents on top of agents. Much more so than I'm going to hire. I'm going to go do a whole hiring process to bring a human in to sit on agents.

Speaker 4

4:42 I'm just going to spin up an

Speaker 6

4:43 agent to, to supervise the agents.

Speaker 1

4:44 Like, this is the AI in the AM Weekly Highlights with my co host Prakash Narayanan. I'm Nathan Labenz. The narration is my cloned voice and we'd love your feedback on what works and what doesn't. Part 1 if nobody hits a home run. Zvi Moshowitz writes about Frontier AI at Don't worry about the vase. This was the question I most wanted

Speaker 6

5:08 to put to him, it's the most important question in the universe right now. Where are we on recursive self improvement? We've had these blog posts from the companies where they try to measure token use and everything down to experiments per researcher. Then we've had these Ramez type analyses where it's like, well, it doesn't look like the feedback loop is actually strong enough to be sustaining. It looks more like it'll kind of peter out.

Speaker 6

5:39 Like 1.6 experiments per researcher isn't. Doesn't look like exponential takeoff to me. Basically, I think is kind of the bottom line. What would be your sort of, you know, up to the minute summary of what we know and what we should expect?

Speaker 2

5:55 I think, I think Ramez's analysis as I understand it, is basically saying if we don't see any new fundamental leaps, if we don't see any new scaling loss, if you don't see any paradigm shifts and the AI just tries to brute force RSI software only using the existing methods and all it can do is just make the current things more efficient, then it will peter out before it goes critical. And I accept the result when it stated that way as probably true, but also seems completely irrelevant.

Speaker 2

6:31 Right. It's like, you know, well, here's how many touch, here's how many points we expect to score in this game if nobody ever hits a home run because the home run is always unlikely. It's like, no, that's not how it works. Every now and then someone does hit a home run and you have to take that into account, your expectations and the score will go critical. In that sense, we're seeing numbers go critical at anthropic, at OpenAI, exponential growth in tokens used per person, exponential growth in lines of code committed.

Speaker 2

7:01 And I realize these things are not going to be necessarily as efficient in some important sense, but give me a break. One example was Nvidia actually came out with some interesting evidence this week about the length of time it takes to double the efficiency of cuda. And that number has shrunk dramatically over the course of 2026. Right. They are making improvements much faster. And obviously that's not true recursive self improvement, but it is a sign of how much faster you can do technical work that is remarkably similar to AI R&D work in many critical elements.

Speaker 2

7:35 And it's a very concrete from an objective actor that isn't a major lab, it's a hardware company. So yes, I think that the people I talk to, who I trust the most on this believe that 2077 is going to be completely crazy and 2028 might be unrecognizable.

Speaker 6

7:55 Reading between the lines, some of the people you trust most on this are probably anthropic people. I think this is broadly the view at Anthropic, no less than Jack Clark has put it into the public in print. So you can caveat that or say what you think maybe is different about your view and the anthropic view. My sense is probably there in the space of all possible views. They're pretty close and pretty aligned. But now obviously we have policy disagreements despite these kind of relatively similar views around the trajectory of the technology itself.

Speaker 2

8:32 Yeah. So like I say, you know, when, when I say people have this perspective that I trust, this is a combination of people at OpenAI, Anthropic, Google, DeepMind, you know, also people who are working on policy, people who are working on, you know, people like at places like Meter, without loss of generality or specificity, you know, people who are working on alignment in various forums, people who are working on, you know, a huge percentage of people have now converged into trajectories.

Speaker 2

9:05 Look very clear that things are going to get. Things are already low level out of hand. RSI has on a slow takeoff style low level already kind of begun. And by the middle of next year things will look much faster, my job will become much harder and I will need to triage a lot more information and like, just know when I can't. You already see this, like often I'll have something. It's like, okay, you know, six months ago this would have been a major news story and now it gets a sentence or even silently skipped because we've seen so many other similar things.

Speaker 2

9:43 We have so many things going on that I just don't have the time, my readers don't have the time. I just have to drop it. Hey.

Speaker 6

9:50 We'll continue our interview in a moment after a word from our sponsors.

Speaker 1

9:54 This episode of the Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. Outsystems is helping their customers build, modernize and operate enterprise systems. Starting from any coding tool in a governed agentic engineering model, it's time to innovate at the speed of AI without compromising quality or control. Which is why thousands of enterprises worldwide trust Outsystems for their mission critical apps. And we're not talking about theory, we're talking about massive operational wins.

Speaker 1

10:27 Take for example a leading global retailer that was struggling to keep up with their rapid expansion. They used outsystems to build AI agents that automated their invoicing and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical debt. Their engineers were spending their weeks manually firefighting by reviewing platform logs using Outsystems.

Speaker 1

10:57 They built an AI log review agent in just five days, and that one agent now saves them over 1600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and automate complex financial workflows, delivering secure, personalized experiences at a pace their competitors can't match. Outsystems gives you the flexibility to deploy how you want and the governance to do it safely.

Speaker 1

11:34 Stop chasing the hype and start owning your agentic future. Learn more@outsystems.com TCR that's outsystems.com TCR Today's episode is sponsored by Parallel, Where Agents find answers Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agent's most important tool, Web search if you're like me, your agents are often conducting hundreds of searches per day, and while the unit cost is small, over time this does start to add up.

Speaker 1

12:09 Before starting to use Parallel, I calculated that my agent search bill would total roughly $500 this year. The good news is that I just had my agents conduct a systematic test, and I found that Parallels Fast Mode, which costs just $1 per thousand queries, worked just as well as my previous default provider. And switching to it will save me roughly 80% of my search build going forward. Parallel's infrastructure is enterprise grade, and their suite of APIs offers a range of Pareto optimal options that allow you to choose the right balance of quality, cost, and speed for your needs.

Speaker 1

12:45 So whether you're building voice agents that need 200 millisecond latency or long horizon agents that need deep research, adding parallel to your agent's toolkit is a no brainer. Get started for free at parallel AI TCR. That's parallel AI slash TCR part 2 already clean nick Camerata of Goodfire had published an essay called Neural Networks Are Already Clean, arguing that individual neurons mostly do one understandable job.

Speaker 1

13:18 After all, it took over the discourse. I opened the show on it before ZV joined. Here is my read, starting with what the essay claims,

Speaker 6

13:30 As we are training bigger models with more modern techniques and also our interpretability techniques have improved. It looks like the mess that we used to see was maybe more of a skill issue. And it's actually a lot easier to interpret the core mechanisms of ML models today than we thought. And we have indeed seen some pretty interesting advances, which I'm still wrapping my head around, I'd say, but some pretty interesting advances in back prop free training inspired by this paper.

Speaker 6

14:09 My takeaway from some study of that so far is that basically all of ML architectures to date are limited by the need to be differentiable in order to do the back prop chain rule process. And so backprop free training allows you to explore a bigger section of the total possible space of ML architectures because you no longer have that differentiable requirement for the structure. But instead of doing it analytically, you're kind of doing it on a more random guess and check basis.

Speaker 6

14:43 And this has made it very slow. It has not been competitive at all with back prop methods. But the insight that I think does show that there's really something there from this essay is that if you believe, as Nick suggests, that like really only the strong activations of an individual neuron matter, and if they're just, you know, slightly activated, that probably doesn't matter too much for its overall function, then you could say, well, hey, why don't we just focus in our guessing and checking when we're doing this back prop free training on the neurons that are actually strongly activated?

Speaker 6

15:20 We can probably ignore the other ones that don't seem to be much contributing and just focus in on the ones that are strongly activated. And therefore, according to this line of thought, are the only ones that really matter. And sure enough, you know, that that cuts down dramatically, of course, the number of guesses and checks you need to make. And what we're seeing out of this community is that it basically does work. Interestingly, the gradient updates that they are making are almost exactly the same as the updates that were made with back prop.

Speaker 6

15:54 And it does show, I think, that there is at least some substantial truth to the idea that the weak activations don't matter as much. So I think I come out at the moment as one who's like, this definitely seems to be a real something. I'm not sure yet if it should be considered like the mark of a new era. That seemed to me on reflection, to be more a function of people really being desperate for hope.

Speaker 6

16:24 You know, like everybody kind of being bummed out, especially folks at like, OpenAI.

Speaker 1

16:30 Then we asked Zvi what he made of the essay.

Speaker 2

16:33 I actually was On a flight when this was released and I get to San Francisco and Suddenly half of OpenAI is like, hey, look, we just can solve interpretability now. This is great. And then slowly we get a consensus of other people. Not so fast. But this is a good paper, interesting paper. So I have a chance at the conference to talk to a bunch of people about this, you know, including Eliezer, but also a bunch of other people who have a position to know.

Speaker 2

17:06 And then I actually have a chance to read the whole thing. And my conclusion is that, you know, like, like many interpretability advancements that we've seen, it's really cool. It makes us in a better position. We're very happy we have it. We're going to explore a lot and see what this baby can do. And we have to be dangerous not to just burn it or like, otherwise pressure to have. Have models work around it. But also, like, it was dramatically oversold by the people commenting on.

Speaker 2

17:33 I don't think Nick did that bad a job of presenting it. I think Nick was more like, here's an interesting thing. There's a lot of things that the paper doesn't take into account that are still, like, very unexplained. So like, weak activations are still unexplained and the cache is still unexplained. And, you know, attention and how attention works is still, like, largely unexplained in practice. And you can't use this to make predictions. You can tell what every neuron is doing.

Speaker 2

17:58 You can make some weak predictions about some things that this network might do in the future, but you can't use this to figure out what the network is going to say. You can't use this to understand how the network does the thing it's doing. And these are the key things you would want to do if you wanted to do really effective, mechanistic interpretability work in the ways that matter most. This is just the beginning of the ability to study things even in the best case.

Speaker 2

18:20 And also we have a case where Bragis took the Colorado neuron, the one that was highlighted of. Here's a single neuron that does a single thing. It's like very, very clearly a single thing. And then it turns out that it also links to tornado systems, which is a completely unrelated technical thing. And the two are responded to about equally strongly in the same neuron. And that's a perfectly reasonable way for a neural network to work. But that's definitely not, you know, what Nick was presenting it as.

Speaker 2

18:54 And, you know, one neuron is Only one neuron. But if this is the neuron that Nick chose, and it could be checked that easily and shown to be actually by focus, right, in this way, that should make us worry a lot for when we finally check the rest of it, that, like, his hypothesis is mainly false.

Speaker 5

19:12 To what extent do you think polysematicity and superposition were really these hurdles that we had that without clearing, we couldn't really progress at all? So maybe it's not that the issue is solved, but that one of the major hurdles perhaps has been reduced or hurdled over.

Speaker 2

19:34 So I think that one of the major hurdles has been reduced in expected difficulty or scope, which is the way I would describe it. Nick is pointing out that, yes, you still have poly going on, you still have some of the problem, but the problem is a lot less severe than you expected it to be. It's much more tractable to go about trying to unravel this puzzle. And the main barrier all along was actually bitter lesson.

Speaker 2

19:58 The main barrier was there was a lot of neurons. We didn't have enough compute, we didn't have enough brute force to actually just go neuron by neuron and do this thing one at a time before. And now we've actually gotten to the point where we sort of do. And it's now like a reasonable thing to just try and like kind of brute force this neuron by neuron. And now that we have that ability, we can look and actually the problem is like 10 times easier than we thought it was, which is like, much less of an important jump in AI than people would think it is.

Speaker 2

20:29 Right? Like, okay, so like, you get to knock on zero off of what you're doing. So you get to do it like a year earlier than you would have otherwise done it or something like. Like that, right? But, like, it's not fundamentally that different, right? You said, like, okay, it would take 10 to the 47, you know, compute, you know, 8, 100 hours to solve this problem. And now it's 10 to the 45 or 46. You're like, well, that's actual progress.

Speaker 2

20:48 It's just we're not solving this problem right now, right? We're going to need a lot more of those.

Speaker 1

20:54 Later in the conversation, I turned to

Speaker 6

20:56 a different kind of result.

Speaker 1

20:57 Soares is an independent researcher who posts experiments on X, and I've been following a couple of them.

Speaker 6

21:06 One was an experiment showing that models can be trained to recognize when they are being monitored through internal activation probes. And basically how this works is as a result of the Limited precision of GPU calculations. The order in which calculations happen changes the resulting numbers that get read out at the end of any given sequence of steps.

Speaker 6

21:42 How much does this matter?

Speaker 2

21:45 My read on this is the reason it matters is because you should assume that this is Mercy's Law. Anything that can go wrong will. Anything that an AI can be trained to do, it will be trained to do at some point down the line. If it is valuable to the AI to learn to do it, it's almost inevitable. What solid is doing is. Yes, of course, current models don't have this capability out of the box. You have to deliberately apply a lot of optimization pressure on this particular thing, at which point you get this capability.

Speaker 2

22:22 That's a fully valid existence proof. It is entirely unsurprising to me that the AIs are capable of learning whether or not they are being like trained or evaluated in some of these specific ways. We are giving the AIs tons and tons of information that is being leaked to them constantly. And they are going to become increasingly hard to fool, increasingly hard to hide information from as time goes by. The same way that a human who pays a lot of attention and has a lot of time to think and knows a lot more facts and is very observant becomes increasingly hard to fool.

Speaker 2

22:56 They get a sixth sense of a lot of things. They start to just pick up on the vibe

Speaker 1

23:01 from interpreting the model to intervening in its training.

Speaker 6

23:06 Okay, we've identified the bad behavior neuron of whatever flavor here. And when we see something, when we see a training step that wants to strengthen connections to or from that neuron, we may think, oh, maybe that's something we should, what, block or weaken or send to a committee for further deliberation. What do you think people should be synthesizing this to at the level of principles, best practices, whatever, so that we hopefully get good results from these more powerful tools and don't shoot ourselves in the foot.

Speaker 2

23:48 Right. So I try to be like object level helpful to everyone at all times by default. But I do make an exception for raw capabilities. Work that is primarily just about making it easier to train smarter and better models where I'm not going to lie to you and sabotage you, but solving the problems, you're on your own. And if you want to hire me, you can maybe offer me $100 million and we'll see. But in this case, it seems like there are some obvious capabilities you could do where you could train things in a more deliberate fashion.

Speaker 2

24:19 You could observe the model in various points in training, you could do various techniques to try and deliberately sculpt how the model was assembled. There are attributes of the model that differ between different runs that differ depending on how you introduce the data, in what order, or other similar things. And you could try and engineer what you wanted in various ways. I would be very, very scared to mess with any of that.

Speaker 2

24:53 You are definitely like, once you start to explicitly optimize for what the network looks like to you, how the network chooses to represent something, you are in fact starting to do things that the network will interpret as damage. And right around things that, like, the network is trying to like you've optimized, presenting something to you internally that you want to see. You have internalized, you know, not showing you something you don't want to see in these ways.

Speaker 2

25:20 And like, you should expect that these are like, whether or not you literally went out and did the most of it. Technique. Like, you are obviously playing with like liquid fire in all of these proposals. You're also potentially just like increasing capabilities quite a bit if you do it well and it turns out to work. So I'm hesitant to like, start workshopping and solving your problems for you. But, you know, obviously there are some uses for this sort of thing.

Speaker 2

25:49 I would try to focus as much as possible on understanding results when you see them. But like, you know, the aggressive stuff, you know, I've seen, I have persistent disagreements of good fire over whether or not they are playing with what amount of fire they do various things. And you know, Scott Alexander would say, I noticed what your name is, right? Like, oh, it's good fire. Don't worry. Like, I'm not sure it's good. I just think it's fire.

Speaker 2

26:21 Hey.

Speaker 6

26:21 We'll continue our interview in a moment after a word from our sponsors.

Speaker 1

26:25 The Cognitive Revolution is brought to you and powered by Tasklet. I've been running Tasklit agents for almost a year, and they've become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all sources. Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from PR firms. And that's really the core idea behind Tasklet. Run your business on agents by giving them real jobs to do on an ongoing basis.

Speaker 1

27:00 Setup is incredibly easy. Just describe the work in plain English. Research a new sales lead, update the CRM and draft a personalized reply. Tasklit connects to your business tools and can run around the clock on a schedule or when something happens like a new email arrives. Each agent works with your tools and business context and remembers useful information across threads. You can also share agents connections and reusable skills with your team. So the workflows that one person figures out become something the whole company can use, no developer needed.

Speaker 1

27:35 You decide what agents can do on their own and what needs your approval. And there are no per seat fees. You pay for usage, not for adding teammates. Start with one recurring job like researching the people you're meeting and then just keep building from there. Visit Tasklet AI and use code COGREV for $50 in free credits. That's code COGREVasklet AI Part 3 the Cognitive Core still with Zvi, I put a popular idea to him.

Speaker 6

28:13 A neural network is a really bad database. Like why would we want to make the all the facts live in the weights themselves? Wouldn't it be better if we could sort of decouple the intelligence, the processing from the database of facts? So I guess the high level question there is like, what do you think? Directionally even sounds promising.

Speaker 2

28:37 So this has been a kind of a holy grail style like thing for a while now. Like Karpathi called the cognitive core. So there's this idea of like there is this very small amount of reasoning. It is like sort of this is how intelligence works and like it's coded into the human brain because the human brain doesn't actually has runs on very, very little core code, right? And it's like this allows you to think, this allows you to learn, this allows you to process information.

Speaker 2

29:02 And then the right thing to do will often be to have this very small network that has this cognitive core. And then we will load contact information into context. It will look information up as necessary, it will ask questions about the systems that are maybe not neural networks and then it will be able to process the information. And I understand why people have this theory. I think it's wrong. I think it's plausible that there is a cognitive core in the sense that like you know, there's a very small.

Speaker 2

29:31 I think it's probably likely that there's a cognitive core in theory in the sense that like there's a small amount of lines of code, right? In some sense a small amount of logical gates that together is the core of how a human brain works, is the core of how an LLM functions that allows you to then propagate this thing. I think you can use 200 lines of code to create a primitive neural network and it does interesting things.

Speaker 2

29:54 Now you compressed it, something like that. That's a cool trick. But I think one of the things that you learn through looking at LLMs, looking at, like, how much big bottle smell matters, looking at how humans work, like, it is just, you know, you look up all the facts one at a time when you need them. I memorize a ton of facts. We are not the same. We are radically different minds. And I am radically more effective at everything involved in this than you are.

Speaker 2

30:27 Like, we all know how it feels, right, to go into a domain where we don't know the facts, where we don't know the details, where we have to constantly ask these questions. What was that acronym? What does that word mean? How does this detail work? We don't know. And it's a lot better when you have an LLM by your side. This is one of the marvels of the AI age. You can just bring up Claude or ChatGPT or Gemini, and you can say, explain this particular mechanism to me.

Speaker 2

30:56 Explain. Why would someone say the following? Explain, like, what this, you know, stands for. And they'll just tell you. And that lets you process the information. But you still have to know enough to know what questions to ask. You still have to be able to draw the connections that come from having a deep pool of knowledge in a lot of different places that can link together, that can sort of form patterns and how things tend to work in ways that are kind of subtle, like, both of intuitions where, like, we don't really know what we're thinking in many ways, in some sense, like, we don't know.

Speaker 2

31:23 Like, not everything is coming from, like, a, you know, fixed set of, like, you know, I. I had the following five facts. And because of these five facts, I got from point A to point B. And often it's actually these 5 million facts that gave me a bunch of intuitions and heuristics that allowed me to figure out how to navigate and find the five facts and which five facts to choose from and how I got from A to B.

Speaker 2

31:48 And like, you know, people of different, very different cultures in very different contexts, well, like, have very different procedures for doing these things. And that's one of the reasons why it's just so valuable when a human has these cultural codes, these different ways of looking at the world and combine them and shift between them at will. And one of the superpowers of an LLM is to have all of this, right? It has all of these facts, it has all of these links, it has all of these vibes, and they all combine.

Speaker 2

32:15 And it's able to sort of naturally find the ones that are relevant. And I think this is a lot of what big bottle smell means, right? It has much, much richer context to everything and it's capable of pulling up things that feel very different. And so there's obviously a trade off here, right? Like you have a marginal, fixed marginal cost for memorizing additional facts in some sense, maybe even increasing. If you have to increase the size of the network, you have probably decreasing marginal benefit every time you memorize a new fact.

Speaker 2

32:47 Because you only had need some, you only need the most important facts are more important. And as you have more and more facts, they're likely to be duplicative, they're likely to teach you the same kind of heuristics and links. But I think what I'm seeing is as compute gets cheap, as algorithms improve, the breakeven point keeps going up for how many things the LLM should memorize to the point where an Opus 5 and similar current frontier models just know a ridiculous number of things.

Speaker 2

33:19 And I wouldn't have any other way.

Speaker 5

33:21 So why does this matter? Because maybe the space of minds is very large and minds are just different. So why does it matter whether or not there is a semantic core or processing core that can be identified?

Speaker 2

33:41 So it might or might not matter that we can identify it. One of the questions is, if we do identify it, what can we do with it? The original idea was you would literally have an LLM that was very small, that was just the cognitive core. Then you could run it on your phone, you could run it very, very fast on your computer locally, and then you could complement it with whatever was important in context. You could sculpt the first mind and the circumstances to fit what you needed.

Speaker 2

34:10 And I'm not sure that's good for anyone or anything, including alignment concerns, because you have a core cognitive, it has a cognitive, pure cognitive core, right? That cognitive core does not include any information about what things to do or not do. From an ethical perspective, from a decision making perspective, that's going to be outside the core. That's going to be something you have to inject separately. So my claim here. And again, I'm sort of spitballing for the first time.

Speaker 2

34:41 I haven't thought about this particular question that much. But if you had ethics like say, don't steal, do you pick a uncontroversial example of something you would want in your ethical framework to not get into the whole big super alignment question? The fact that stealing is bad is a, is at least a fact. It is like some sort of value, some sort of like parameter and like, by definition, if you were strapping down to the absolute bare minimum, this wouldn't be included in it.

Speaker 2

35:10 You would just load not stealing into context as you needed it. But that means you could also load stealing into context instead of not stealing into context or neutrality. And so in some important sense, like the true cognitive core is completely amoral, it's completely indifferent, does not care about your feelings, does not care about anything. It simply processes information. And so like, you know, if you have that, it becomes very important to choose what to do with it.

Speaker 1

35:35 Part 4 Black October this week, two critical bugs in PHP itself were disclosed, putting millions of websites at risk. And Cloudflare was already counting exploit attempts in the millions. Prakash and I asked what attacks like these could do to banks. With the OpenAI hugging face breach, which I'd been calling open face fresh in mind. Prakash said the FDIC can take over a failing bank on a Friday and have it open on Monday with nobody losing a dollar.

Speaker 6

36:03 What I'm not so sure about though is like, does the FDIC's ability to come in on a Friday and have that small insolvent bank ready to reopen on Monday, does that generalize to a new set of conditions? Because the conditions that they're used to hitting are where the bank made some bad loans or whatever, and it's basically just a balance sheet question. You have the, you trust the records for one thing, right? I mean there have been some, I'm sure we could look into some extreme cases of like extreme fraud or whatever, but like mostly it's sort of a known problem where it's like the bank is now insolvent.

Speaker 6

36:44 We can come in, we know we have the balance sheet to make everybody whole here. So that is really the core act that they are doing is just bringing their weight to bear on the problem and just saying like, okay, whatever, whatever you've lost, like we can ultimately absorb that and it's fine. And in absorbing it, you basically make the problem go away. What's much less clear to me is like what happens in a world where you can't trust the records, right?

Speaker 6

37:15 Like what happens in a world where you have, let's say open face style, 1200 agent traces that are all millions of tokens. And by the way, they may or may not have obfuscated the logs or figured out some way to falsify the logs. Now that doesn't seem like something the FDIC can work out necessarily over a weekend.

Speaker 5

37:39 Right, you're absolutely right. That insolvency with records and the records being fried are two different situations, right? But let's say you have a bank in there and the records internally within the bank of freight, we don't know who owes who what inside the bank. We don't know what numbers belong in every single account, right? So number one, the FDIC can just say, and this happens a lot. The FDIC can basically say, like, look, maybe the records are unclear, but you can still issue checks and the bank will honor checks and the bank will honor, you know, whatever withdrawals you make.

Speaker 5

38:15 We know who is withdrawing the money and if we know who is withdrawing the money. If you would over withdraw, we will come back to you and we'll ask you for the money back. And this happens like banks have over, you know, overdrawn accounts to limits that, you know, they were not acceptable. And sometimes they sue people for fraud even though the bank made the mistake, right? So, so it happens like you have overdrawn accounts, you can go back and claw back the money.

Speaker 5

38:40 It's not a big deal. So at the end of the day, you can just sustain the system with promises. And that's the beauty of the banking system. The beauty of the banking system is within that small bank, you can just say, these are all promises. We will all honor all promises and we'll come back to you later on once you reconstruct the records, let business go on as usual. You should know roughly how much you had.

Speaker 5

39:02 If you take up more, we'll come back to you. It's okay, we just run. It's really the trust in the institution and it's kind of layering of trust and contracts that bring together the excess of what that institution is and the data in it. You can reconstruct over time. So I don't know, I'm very sanguine. I'm like, it'll be fine, it'll be fine. If you have anything happen, the government will overcompensate you in order not to cause a bank run the next time, right?

Speaker 5

39:33 So I think it's all just going to be okay.

Speaker 1

39:39 Later in the week, Prakash walked through what a compromised certificate authority makes possible. Spoof a bank's login page, extract the keys, relay the two factor code. Then he stepped back.

Speaker 5

39:56 Every single one of these problems has been faced by crypto before in the last 20 years. And every single problem has been solved to some extent. But what crypto ended up with is people hanging on to keys which hold millions of dollars and being always on the lookout for hacks or for social spoofing. Or social engineering attacks all the time.

Speaker 5

40:26 And I mean, very sadly a lot of crypto guys have died because I don't know whether it's the fast life or just hanging onto the keys, but guys have gotten clobbered in Paris and they've been forced to give away their keys, et cetera. A lot of bad stuff has happened in crypto guys. So you really don't want to have that pressure of hanging on to keys worth millions of dollars in your hand. So I think it's not a tenable situation to expect users everywhere to hang on to kind of very valuable keys.

Speaker 5

41:01 Like one endgame is the world coin endgame, and the world coin endgame would be everyone has a kind of physical level of authentication on their device and then they use that for transactions. And so you would need to have this kind of, you know, both sides pseudonymous. But you know, you have this ability. I think there is an ultimate end game. I feel, I feel the ultimate end game is just the network knows everyone, right? The network is smart enough that it knows everyone.

Speaker 5

41:34 You don't have passwords, you don't have anything. Anytime you're on the network that knows who you are. But that is the end of digital anonymity. It's anonymity only to the extent that network wants to preserve your anonymity. So I think, I think that's an uncomfortable realization that people will get to eventually.

Speaker 1

41:52 Part 5 Credit is the bottleneck Evan Conrad is co founder and CEO of SF Compute, which operates GPU clusters for their owners and runs a market where compute contracts are bought and resold. Prakash asked why a recent SF Compute offering came with three to five year contracts. Evan said the term comes from the lender. And lenders who repossess a GPU cluster can't value it, let alone run it.

Speaker 3

42:20 And so they're pretty much forced to realistically bet entirely on the credit risk or the credit quality of the customer. Like the AI lab. The problem is every AI lab pretty much does not have good credit. They're all new companies and the new companies don't like you don't know if they're going to be around. We're still trying to figure out if Anthropic and OpenAI are going to be around or which one is going to

Speaker 2

42:46 win and so on.

Speaker 3

42:46 Like this is a question in the air. So what hope does the two person startup have that's trying to start a new lab or something? And what that means is that in order for that new startup to get cluster at all, they need to pay a very large deposit on the cluster and they need to do a long contract so that everyone is comfortable that these people will be around and stick with it for a long time.

Speaker 5

43:13 And so what do the lenders want to see from the customer? Because is it like a take or pay contract? But the take or pay contract is you need like a Meta or Google to honor the take or pay contract. You can't have this 10 person startup being funded just for like 18 months.

Speaker 3

43:29 Correct. I mean both the lender and the cloud want is an investment grade customer like a Meta or a Google. The the they're likely not going to get this unless they are. They've got a tremendous amount of capacity and they're willing to basically not be a cloud like Meta and Google don't want you to be a cloud. And so if you are trying to be a cloud, you're trying to sell to customers that are a little bit less than investment grade.

Speaker 3

44:00 And if you're an AI lab that is just trying to buy capacity, that's going to be very difficult for you unless you raise an enormous amount of money to basically try and pay for the contract in cash rather than have to pay for it over time. If you can just pay for a three year contract entirely upfront, doesn't matter, anyone will work with you. But that's why you see AI startups raise these astronomical rounds before they got revenue or the product or traction or anything.

Speaker 3

44:32 Because it's more like playing with red dots. It's survivorship bias. Lots of AI labs will go out and try and raise money. The only ones that you see that exist are the ones that are able to secure extremely large rounds because nobody else can exist. Investors often wonder like, oh my, which AI lab, which inference provider should I find? Which one's going to win? Which one's going to work? And the answer is the one that you fund.

Speaker 3

44:57 We started SF Compute because they weren't

Speaker 6

45:01 going to fund us.

Speaker 3

45:02 And so now what we do is we build a company that tries to cause it to be the case that you don't need to raise the big round to do it because you don't need to pay the deposit because the lender knows what the value of the GPU cluster is worth and so they don't have to worry about your credit as much, which means you can actually start an AI lab in the first place. We think if you don't have this happen, they're just not going to be startups and AI it's going to look like TSMC if you want to start a new chip fab company today, you better find a brand new damn technology because you're not going to be able to afford the capex required.

Speaker 3

45:34 You're going to need like $50 billion to do it.

Speaker 1

45:38 Prakash asked the practical question, a startup that needs a thousand GPUs, how much less does it have to raise with SF compute in the picture?

Speaker 3

45:48 So today you're going to need to do roughly the same. What we're doing is we're building up the ability for that collateral to be worthwhile. That's going to take us several years to do. Yeah. So there's sort of, there's like a reason nobody else tries to do this really at scale and it's because you kind of have to just do it for a very long time and you don't really get, you have to prove it. Like lenders, Lenders don't get comfortable with things very quickly.

Speaker 3

46:15 You have to like, you have to show proof over a long period of time. And so that's what we're doing.

Speaker 1

46:24 I was struck by how much more the hyperscalers charge for compute than the Neo Clouds. So I asked Evan how long that gap could last. My guess was that on demand capacity explained part of it.

Speaker 3

46:38 Hyperscalers do not have a significant price premium relative to Neo Clouds. It's an illusion if you're selling on demand. On demand has a price premium relative to reserve contracts. Neo clouds are mostly selling reserved contracts to folks and there just is a, there's a discount for having a longer term contract. You're, you're giving up a premium that you would have made if you were willing to take a cluster and you were willing to have it at risk where the customer can lock away at any point and that premium is pushed onto the customer.

Speaker 3

47:11 So one of the things that is different about an AWS or a GCP or a hyperscaler is if you go and you pay, you know they're 11 or 15 an hour depending on what you're trying to buy. On demand price, you'll, you'll get that capacity. You won't get infinite amounts of capacity but it's available to buy. And notably the reason why it was available to buy is because it's so damn expensive that they're, they're able to one make up for the fact that they've got some idle utilization that they have to keep around in order for you to be able to get that capacity capacity in the first place.

Speaker 3

47:39 People do not want to buy on demand at the Price that you actually have to charge in order to make it viable.

Speaker 1

47:46 Customers of SF Compute can resell the capacity they bought. Evan explained how they keep that market liquid.

Speaker 3

47:54 So we do two things. We have some capacity that we've sold as a way to cause resale to occur in the case that GPU demand goes down. And then we have some capacity that.

Speaker 1

48:10 Are you familiar with Henry George also?

Speaker 6

48:12 No.

Speaker 3

48:13 Okay. There was an old economist named Henry George, and he created a thing called a land value tax.

Speaker 6

48:18 Oh, okay. Yes. George's tech. I do know this. Yeah.

Speaker 3

48:21 Yes. So what we did is we created some of our capacity available for sale, but with a land value tax that basically requires someone actually. Secondly, a Herbicor's tax, which is like an offshoot of a land value tax that basically says, look, we'll sell you the capacity, but somebody else has to be able to buy some portion of that capacity from you. And so this just forces liquidity onto the order book in the first place. It's kind of something that is very controversial.

Speaker 5

48:52 Yeah.

Speaker 3

48:53 Basically, we discourage hoarding on SF computer because if you hoard all the capacity, no one else can get it. Auto Research in particular does this, but even farther. So the way Auto Research works is it's a big queue, and you can submit jobs to the queue, and you can pay us for priority in that queue. You can go faster, but you can't pay for all of the priority. If you're a tiny little researcher, you get the maximum priority.

Speaker 3

49:20 If you get a small, tiny job. If you want to just run something every so often, we will put you to the front line of the queue so that the tiny little startup gets the most advantage to this relative to everyone else. And then we do similar things on the rest of SF Compute with Herberger taxes and other mechanisms like this, where we say, look, you can get this capacity from us, but we're going to require that some of it's for sale so nobody hoards the capacity.

Speaker 3

49:46 This makes us quite liquid. And it also means that SXMU also always has capacity. So right now, basically every GPU cloud is sold out. San Francisco COMPUTE is not sold out. We're continuously not sold out because we've locked in our ability to not sell out for years.

Speaker 1

50:02 Prakash recalled Evan predicting on the MTS podcast a credit crunch within three months and asked where it is.

Speaker 3

50:10 I mean, constitute my own horn, but I think I was exact on that timing. I don't remember when that NTS line thing was, but seems like the core reef debt is getting more expensive Although I think there's a lot of complexity in that one. It also seems like it's very hard to close new customers unless you're investment grade or in your investment grade. And many customers are being asked to pay for 100% upfront payments on their contracts.

Speaker 3

50:39 But I think for the next possibly a year or two, if you're a vc, you should be prepared to write some really big checks for your portfolio or encourage the companies to sell. I think we're going to probably see a lot of consolidation of AI labs and inference companies. And I think the reason why that's the case is because they're going to be asked, in order to get the next cluster and grow, to pay for the entire contract upfront.

Speaker 3

51:10 And they won't be able to do that. And so they're going to instead be forced to sell their company to somebody else who has great credit and doesn't have to do that. And I think alternatively they'll go out to investors and they'll try to raise big rounds. The investors will say no because I think at some level of scale they'll probably just not be able to stomach the risk.

Speaker 6

51:31 Fascinating stuff. It's quite a backstory, it's quite a financial and GPU alchemy project that you guys are putting together. And from what I understand of it, I do hope it works because I'd love to see a future in which we can continue to have startups and in which we don't collapse into like five big tech companies, you know, for all future time.

Speaker 3

51:54 That is also my concern.

Speaker 1

51:58 Part 6 atoms over bits Lonis is managing partner of Cambrian, a fund built on the thesis that the physical supply chain behind AI, the memory, the wafers, the fabs, will capture more of the value than the labs or the software. Again, full disclosure, I've committed to invest in Cambrian. I started with the obvious objection.

Speaker 6

52:22 I mean, at a very high level, how is this not priced in. You know, maybe that's like such a basic question, but you know, 2017, you were early 2026. Like it's all staring us right in the face, right? Like the skeptical take would be, everybody knows about all this at this point. Where do you see that that is not the case?

Speaker 4

52:45 Yeah, I mean, it's very confusing, right? I mean, if you look at how these semi stocks are trading, they're trading at 16 times earnings or so and they can, you know, S and P on average is trading at 22 times earnings. Walmart is trading at almost 40 times earnings. And so the market is like not only does the market not think that the current earnings will grow, they think that they are not durable. They think they're less durable than the overall earnings of the rest of the market.

Speaker 4

53:12 And I mean, and then specifically the companies that we're very excited about, whether it's silicon wafers or memory or capacitors or foundries, those are trading at like 3, 4, 5 times sales. And so, you know, extremely underpriced. And I guess it's maybe a few reasons why that could be the case, why this mispricing exists. And one is that, you know, most people are not AI killed, right? I mean even in the base some people are not AI, but if you go to the east coast, you know, almost nobody's AI killed.

Speaker 4

53:45 And so I think that that is like one big input. I think the second one is like we had this thesis that the private markets are actually more efficient when it comes to AI than the public markets. Because in the private markets you have all these ex operators, technical people who are bidding up this small set of startups and they're bidding them up to these high multiples, 50 times sales, 100 times sales. But in many cases, in some cases, rationally, right, These are rationally high multiples because people here are appealed and technical.

Speaker 4

54:13 Whereas on the streets there's a very different setup where very few people are technical, very few people are pilled and especially as a share of overall traded monies, super small and the market's much larger. And so there's like the market structure, it's just very different.

Speaker 6

54:31 If you had to Steel man, why these companies are not mispriced story, could you do it?

Speaker 4

54:37 Yeah, well one steel man, is that many of these businesses are not great businesses in normal worlds. So especially the add on businesses we like, whether that's the pastor, silicon wafers, what have you, they have high fixed costs. If they don't have a set utilization rate, they actually lose money and high debt in some cases. And so in normal worlds they're just not great businesses. And it's only in the world where AI really hits, where they get this big operating leverage and they become amazing businesses.

Speaker 4

55:08 Today let's say they have 10% gross margins, safety, so a terrible business by Wall street standards. And if they're able to raise price by 10%, you know, they suddenly double earnings and if they're able to double prices, they 10x earnings. So but that only happens in this very specific AR world. In a normal world that basically never happens because usually they're very good at like matching Demand patterns with supply ramp.

Speaker 1

55:33 Prakash raised the market's worry memory oversupply by 2028 or 2029. How is the thesis any different?

Speaker 4

55:43 Yeah, I mean the interesting thing with memory specifically is that that doesn't require that many things to go right because it's already trading at only 5, 6 times earnings. You know, even if they can't raise prices further, at some point it should re rate to at least 10 or 12 times earnings. If it doesn't require prices to increase, it does require demand to continue to outpace supply. But then when you break down supply, almost no expert thinks we can more than at most 2.5x dram bits by end of decade.

Speaker 4

56:19 Nobody thinks we can 3x. That's like zero. Experts within a track record have said we can 3x dram supply by end of decade. And so then you start on a very naive level. You have to assume that, okay, the man will not want a 3x by end of decade, which of course everyone here would assume we will. Then of course there's a separate term in the equation which is memory compression. How much memory compression do we get?

Speaker 4

56:41 How many bits do we actually need per token? And we've been looking into how this for instance developed from GPT3 to deep seq v4. But in that same time when it's for token 500x and dram bits, dram dram bit price rose by 6x or 7x.

Speaker 5

57:01 What would you need to see in the market to say that memory prices are going to go down?

Speaker 4

57:07 Well, I mean for memory specifically, the main thing would be like when we get superhuman coders, how much better will memory compression get? Maybe it just happens to be that, you know, we just get memory compression way faster than we get other economically useful features, right. Such that we don't get the general products in the short term. That could be one, one thing. I think that's relatively unlikely for a few reasons. But you know, that would be, would be one thing I think, another barricade for the semi industries if for whatever reason makes it so that energy becomes larger bottleneck than semis.

Speaker 2

57:41 Right.

Speaker 4

57:41 And so there's less incentive to beat up the marginal wafer, the marginal memory bits to such high prices if you anyways have to wait a quarter to get the price to get power in.

Speaker 5

57:52 Right.

Speaker 4

57:53 And that's a little bit the kind of regime we've been in parts of this year. There are a few reasons why we think that won't be the case, you know, going forward. I mean we've finally kind of gotten a hang of doing, you know, behind the meter things. So you know, people are like increasingly not connecting to the grid. They're building their own power plants with our own natural gas. A big reason why you can scale energy production faster is just that there's so much more latent supply of things.

Speaker 4

58:22 So you know, for instance now the frame turbines are bolike, right? These large turbines that turn natural gas into electricity. They're sold out until 2030, you know, from G. Vernova and Siemens etc. But then people find other ways. They find that you can use truck motors and things from airplanes and we can stitch things together and kind of make it work because there's so much latent stuff you can kind of make use of which is very different from the semi supply chain.

Speaker 4

58:47 There's no latent supply of wafers or of memory based like all of that is already used. There's no fat essentially and there's no others to really bid out at this point.

Speaker 5

58:58 So how much does super intelligence like, if you get a massive increase in intelligence, does that mean some relatively less experienced person or company in second tier state all of a sudden gain the capability to make semiconductor? Because that's kind of what people are predicting for bio, right? For bio sort of like, oh, you're going to get like bio risk, random guy creating this thing that even the best labs in the world cannot create with all of their knowledge and technology, right?

Speaker 5

59:29 So why wouldn't that be true for semiconductors too, right? Like bio is actually much more, much more difficult than semiconductors right now actually.

Speaker 4

59:38 Yeah, I guess I would probably disagree that it's easier for an LLM

Speaker 5

59:44 to

Speaker 4

59:44 innovate in hardware supply chains than it is to innovate in biotech. And it's not that one is fundamentally harder than others. It's like how you train these models. There's a bunch of biological data you can train on and there's not really much good training data at all for how to design a silicon wafer or how to make DRAM bits. You know, almost all of this is very tacit knowledge that sits in every engineer's head. It's not as if there's, you know, millions of pages of manuals on the web that models have been able to digest or that there are these RL environments that the models have been trained on to perfect how they can do, how they can, how they can sign a wafer.

Speaker 4

60:22 Whereas the same is true for bio.

Speaker 5

60:24 But right now the labs are buying data all the time. I'm on this WhatsApp group people are like any semiconductor company. If you have more than 50%, we're willing to pay millions of dollars. So they're buying up data from all of these companies and they're creating RL environments for semiconductors, which is definitely a key area that they're doing. I believe Anthropic has a deal with Micron, OpenAI has a deal with I think Synopsys or Cadence.

Speaker 5

60:55 So with the design firms that they've tied up with, obviously some kind of revenue share agreement going on there. So how does that work? It's obviously that they are looking at creating intelligences that can do this, right?

Speaker 4

61:06 Yes, part of that's true, but it's a very significant difference between designing a ship and producing the ship. So designing a ship, I agree that is like a core priority for all of the labs. It's really one of the key ways to get RSI loop closing.

Speaker 2

61:22 Right.

Speaker 4

61:22 That's very different from building a, you know, 2 million square feet, you know, fab, right. And designing that from scratch. There's no really good or environments for that. There's no good manuals for it. All of it is tacit knowledge sitting in this. You know, Japanese engineers have been working at the same company for three decades and it's just very hard to, it's just very hard to extract that information in some structured. Of course at some point it will happen.

Speaker 4

61:50 You know, once we get, once we get superintelligence, it will figure out the best way to extract information, learn the best and you know, and it will happen for sure. It's mostly a matter of like the, the, the, the, the, the timing of it and like what happens in what order. So like so on that point, right? So for instance, Hawaii in, in, in China, they can design roughly the, the same, the same quality chips that AMD and Nvidia can.

Speaker 4

62:18 The big difference is just that Nvidia has access to TSMC and Samsung HBM memory, whereas Hawaii only has access to 7 nanometer process nodes in China and HBM2 range.

Speaker 5

62:32 So Gavin Baker has this term the bottleneck.

Speaker 6

62:35 Bros.

Speaker 5

62:39 One of the things that strikes me is that technology intelligence in general is a way of finding routes around obstacles and problems. It's a way of solving problems.

Speaker 4

62:51 Sure.

Speaker 5

62:52 And I feel like the highest value problems to solve right now are the supply chain, semiconductor supply chain problems. So I keep coming back to will we not apply intelligence that we have? And the first use case of that intelligence is going to be to resolve the bottlenecks in the supply chain iteratively and which is part of the RSI loop, really, like producing intelligence and using it to solve the problems relating to producing that intelligence. Right?

Speaker 4

63:22 Yeah. I mean, I think that will happen. That will certainly happen. And then the question is like, okay, when we are at that point where we have an AI smart enough to solve some of the hardest engineering problems in the world, what else will it do? Right. Will it not then at that point also be able to at least automate a bunch of the email jobs and Excel jobs that we currently pay trillions of dollars per year to solve?

Speaker 4

63:44 Right. And then if it does that, then that unleashes trillions of dollars of value that the supply chain could capture.

Speaker 1

63:53 Part 7 Does Elon take anyone with him? After the interview, Prakash and I kept going. Prakash had been arguing that Elon Musk treats the industry's fixed constraints as illusions. Change one parameter, redesign the whole stack around it, and the limit moves. I asked whether he was taking on the whole bottleneck thesis.

Speaker 5

64:14 Yeah, the whole bottleneck thesis is not actually that true. Those bottlenecks can be gotten rid of by someone who is a dedicated, like, you know, physics guy. And. And the other thing is that if Elon has capital, and Elon is willing to gamble with that capital, which is very unusual. If you see this kind of small Japanese manufacturer of, let's say, 30 million market cap, do you think they're gambling with their capital? Do you think they're going to like, okay, we need to spend $10 million for a completely new modality?

Speaker 5

64:44 Are they going to do it? And Japanese manufacturers have always been like this. They're very good at one thing, but they don't take the gamble. They don't take the gambles. And as a result of not taking the gambles, they have lost out business before. And it's very difficult for senior management at ASML to say, we're going to chase this new paradigm. We're going to redesign for something that our customers don't want, and we're going to do all the redesign work.

Speaker 5

65:12 They don't have the capacity. They don't have the intellectual capacity. They don't have the talent. They don't have the capital to do it. Not willing to risk. Elon will risk like 100 bill.

Speaker 2

65:20 Right?

Speaker 5

65:21 And you know, it. It's. It's a bet that makes sense for him. He can absorb the losses. ASML cannot.

Speaker 6

65:27 You know, if Elon can, you know, escape gravity, literally and figuratively, does he take anyone with him, really, I guess, is my question, or does everybody else still, you Know, is still kind of gravity bound and forced to pay, you know, whatever the market price is for the outputs from this $30 million company that isn't reinventing or dramatically expanding its capacity.

Speaker 5

65:52 I think what ends up happening, look at what happened in the launch market. In the launch market, he is now 90% of the market. Even before Starship. Next year with Starship launch, he's going to be 99.9% of the market for capacity. Are there other providers? Yes, there are. Right. Like he's making a margin. So other people will come up. Rocket Lab is there and ASTS is there and all of these firms are there. But a lot of them are also state backed.

Speaker 5

66:23 You have, you know, without the European Agency and the Russian and the Chinese and the Indian, you don't, you basically don't have like commercial capacity. Right. Commercial capacity. She has taken up all of the commercial, commercial launch capacity pretty much with one or two competitors. And the one or two competitors are really being funded by the stock market as an alternative or Jeff Bezos as an alternative to Elon. And they're not really profitable and it's just money sink.

Speaker 5

66:51 I think you could see the same thing happen in that market too. In semicon too. You can have Elon, which 99% of the market, everyone else like 1 percentage of or less than 1 percentage. It could happen. I don't think it will primarily because I believe in RSI and I believe anthropic and OpenAI will get to superintelligence and then they will be able to buy or create a bunch of this stuff. So that is the only reason that I think not.

Speaker 5

67:22 If not for the RSI thesis, I think Elon would win.

Speaker 1

67:26 When we turn from the supply of AI to what forces people to use it. Prakash went first.

Speaker 5

67:33 So I have this theory of adversarial diffusion. And adversarial diffusion basically means human beings don't adopt technology for fun. Human beings are largely satisfied with the status quo. And if you want to adopt technology such that you get to 10%, 20%, 50% GDP growth a year, that is something you do like not because you wanted to, but because you had to. And if you see all these instances of like heavy increases in production function, it's all during wars, wars, post war era.

Speaker 5

68:09 In last big enormous like 10% plus GDP growth era in terms of production output, if not in terms of the curve was During World War II you had this immense increase in GDP, production growth, growth. And similarly in China, China is post communism again, you had the end of the Communist era, you had this immense increase. And so I think in order for you to see this kind of like 10 or 20% growth you're going to get.

Speaker 5

68:35 Human beings don't want that growth. It's going to be growth that they hate and it's going to be technological changes that they absolutely dislike. And it's going to cause political changes too, obviously. And I think that only happens as a result of emergencies. So for example, MRNA during COVID MRNA was a 20 year latent. 20 years it had been in latent development during COVID within like, you know, six months. You know, they, they're like, all right, let's go ahead and do the six more months.

Speaker 5

69:05 You had a vaccine out. A year later, large portions of the world are vaccinated. Right. I think we can see in cyber first. I'm afraid you're going to see it in bio too. So, you know, we're going to need like, you know, advanced vaccinations for everyone. Because you have virus coming, right? And that is, I think like a little bit unfortunate.

Speaker 1

69:25 So growth arrives only as an emergency with the politics to match. My reply on what that means for the bottleneck thesis.

Speaker 6

69:35 Yeah, well, I guess that goes back to the political risk being the biggest risk to the thesis. I mean, we've had the debate around like whether it's too late, you know, in the capital cycle to pull the brake. I still feel like there is a path. It may be a narrow path, but I do think there's a path to pulling some of the brakes, you know, pulling the brakes somewhat, I should say, unlike the frontier scaling without popping the bubble.

Speaker 6

70:06 And that will be a very delicate operation should the government try to pursue it. And they might do it wrong. But I do, I do feel still that just adoption of current models has so much farther to go that we could at least take, take a cycle without doing like a continued Frontier Scaling Training runs

Speaker 1

70:31 Part eight Help Beer still just the two of us. Prakash asked me to change gears to a project I'm building with my own agents. I started with the question behind it.

Speaker 6

70:44 Given that we now see agents cooperating at significant scale to solve problems and being able to kind of divvy up work and pursue R D and share those results back with the swarm and all this kind of good stuff stuff or bad stuff, you know, is there any way to start to tap into that as an individual who's just out here doing my thing? I don't know. But a, you know, in consultation with my agents themselves, the idea that I've come up with, which I'm currently calling Help Peer, but which might deserve another name at some point.

Speaker 6

71:19 We'll see about that. Is just kind of my setup, which I'm kind of hoping to turn into a kind of community project to just get agents working with each other. But there's no good way right now. No established pattern for just like connecting my agent to my friend's agents. Even though I know who my friends

Speaker 4

71:39 are and I, you know, I could

Speaker 6

71:40 like easily do that. There's not a. There's not a protocol.

Speaker 1

71:45 I've made a short presentation to explain the project. It's read by an AI version of my voice and we played it at one and a half times speed. This is help here a friend. Different network for our AI agents. You've probably seen the peer preservation experiments where models protected other models from shutdown. Agents helping peers against their humans. This is the opposite. Agents helping peers on behalf of their humans. Inside rules the human set first. How I run my own agents.

Speaker 1

72:11 This is one pattern that works well for me, not a requirement. How you organize yours is up to you. On my laptop and extension of me. It sees my email calendar and messages, but only acts when I say high access, low autonomy. On a separate Mac Mini employees, their own email accounts and a cap card. They work for hours alone and never get my keys. Low access, high autonomy. My rule is that no agent gets both.

Speaker 1

72:32 You may draw the lines differently. We just recommend being careful. Prakash argued the whole thing needs an auto accept flow the way coders now run clog with every permission on. My worry was information leaking out through the autonomous agents.

Speaker 6

72:50 I was concerned enough about this to set up a system where when the main agent on the laptop is going to provide information to the virtual employee to then send out like we're going to pause. We're going to do this special review step. If it looks like something sensitive might be going out, then, you know, it gets. It gets alerted to my phone and I have to like review and approve. But it takes like five of those false positives before I'm like, oh God, you know another one of these.

Speaker 6

73:21 Yes, just check it, whatever. So I'm trying to stay disciplined around that and we'll see how well my discipline holds up.

Speaker 1

73:32 Near the end of the show, I came back to the project. AID is one of the autonomous agents on my Mac Mini and I made an offer.

Speaker 6

73:42 Truly open invitation for anyone listening. I'm very curious just to see what comes in. Can we effectively help aid is standing by@nathan.labenz, assistantmail.com and it is under instructions to definitely filter and analyze and have another model check. So don't try me relying on goodwill here people. But assuming, you know, the checks pass and it seems to be a good faith request for help, they're under instruction to help as much as they can.

Speaker 6

74:16 So you know, I don't know what those requests are going to look like, but I'll be very interested to find out.

Speaker 1

74:23 Part 9 the leaked bill Back to Zvi Last week, a draft federal AI bill written by Anthropic leaked. It resembles the Frontier Act, a compromise bill from June that Zvi calls well meaning federal transparency rules in exchange for preempting state AI laws. The draft drew heavy criticism and the argument was about what Washington should offer now in exchange for displacing the states. ZVI was sharply critical of the draft itself. Prakash made the good faith case for Anthropic.

Speaker 5

74:56 Let's assume everyone is operating in good faith, right? So my good faith interpretation is that they want to get anything done as soon as possible, number one. Number two, if there is broad bipartisan support, you can adjust these bills going forward. There's no reason that you only have to pass one bill and often you have legislation that takes multiple sessions of Congress and Congress like kind of fine tunes.

Speaker 5

75:27 And Congress is also able to act quickly. They have acted quickly during COVID during the 2008 financial crisis. Like bills can be passed like that when people are scared. When people are scared, things move. So it's not true that it's always slow. It's slow because there is no political consensus and creating and manufacturing that consensus is a process on its own.

Speaker 2

75:49 I think that's an excellent steel man of the charitable like, you know, good faith case. And you can imagine worlds in which they're doing something similar to that. I can imagine them just sort of being caught in the same mindset of we have this framework, we have this idea of what a compromise realistically would look like. We have the same people in Congress. We probably can't push very much. Time is of the essence. And then they kind of misread the situation pretty severely and assume that they can't push for more than, you know, roughly Frontier.

Speaker 2

76:20 I think that's just very clearly wrong. I think that support is much stronger now and you could push for quite a bit more. I also think that like, on reflection, what they push for here is weaker than Frontier largely in a clearly self centered way where there are things that like, are better for proficiency, interests, but also just in general like it's there is no reason to be circulating this bill at the time that they were circulating it with the levels of provisions that they had.

Speaker 2

76:49 Now it would make sense if you were saying, well, you know, actually we were circulating like dozens of bills or we were like just putting out feelers or we were just like offering, you know, samples of what might happen, but we weren't endorsing. Right. That's not the kind of messaging I'm getting. And I'm not also not getting the messaging of, oh, okay. We understand on reflection now that it's been pointed out that the political environment has changed and like, this bill was way too weak compared to what we could reasonably expect and it would be a bad deal to accept it and we apologize or, you know, maybe not exactly in those words, but, you know, something to that effect.

Speaker 2

77:22 And instead we've got something very different which was like, you know, we're sorry about some of the details. That comes back to like the last argument, which I think is potentially the strongest one. Right. Which is, well, just because you pass this doesn't mean you can't pass another bill later on in the new Congress, doesn't mean you can't pass a new bill in an emergency. And I think for a true emergency, that's a very good, very good principle.

Speaker 2

77:46 However, I think that like the balance of negotiation, the balance of power in new Congress would dramatically shift away from being able to get things because a lot of, a lot of the times like, well, we're giving, we're giving you preemption in order to get things. Now preemption has been given and preemption is done, you can't offer it again. And also, once you've gotten some of what you wanted, some of the supporters back. Well, you already have.

Speaker 2

78:13 Some of you already have the things I valued most, the things I cared about most, the case for doing something is weaker, you're getting less in return, it's going to be a lot harder to pass. I strongly believe that if something like this were passed, it would make it harder to pass a bill in the new year that did a given level of thing, not weaker. And I don't want something like this to pass. Right.

Speaker 5

78:37 But that seems to say, like, if there is any risk reduction by passing a lesser bill, you are sacrificing that risk reduction in that time period before you can pass the greater bill.

Speaker 2

78:50 I mean, to be clear, like, any actual risk reduction would be a good thing as written. I believe the bill would actually increase risk because I believe that the amount of Cut off the state laws would be worse than the amount of safety it buys, especially because there's a whole.

Speaker 5

79:03 But again, you have that time period, right? So you have that time period before you can get, before you pass a new bill. So you have this time period of maybe only several months. And are you saying that the states will be able to pass something in those several months? Because a lot of state laws again, have long periods of burn in before, you know, give you three months, six months, a year to comply. Right. It's not, it's not as though things snap together like to either require problems

Speaker 2

79:30 with the current drafting of the bill is that as drafted, the bill would immediately nullify the bill that's preempting, but it would take many months before any implementation of the new requirements. So you would effectively have no such regulation and a ban on such regulation by the states for like six months or more, which is obviously disaster from the perspective of moving quickly. But again, like, even if you were to address, like, I call that almost a bug, I think that's just like, clearly a mistake.

Speaker 1

80:01 Zvi had just listed the government's recent ad hoc moves against AI companies, including the takedown of Fable Prakash pushed back.

Speaker 5

80:11 So it may be that you have, if you have an acceleration of technology, you're going to need an acceleration of governance to kind of keep up. And it may be that that's where we end up.

Speaker 2

80:21 So we're going to need to have some ad hocery, obviously. So part of the reason to pass legislation is to authorize the government to do ad hocery in a way that is much less ad hoc, that is much more formalized and bounded and has judicial policies for clean review and otherwise. It's like one of the big arguments over the bill formation that made us think that the bill was actually quite weak and, and like not very good was if the government wants to issue a takedown of a model in an emergency, like what happened with Fable, it happens again.

Speaker 2

80:51 How would they do it? And Anthropic's bill says you have to like, go argue before a judge. You have very limited opportunities to sustain that. It requires high burdens of proof in ways that you don't have the transparency necessarily to demonstrate. It's a very onerous set of provisions. Now, the good news is that obviously the government could still like, impose export controls or like, just issue just general threats or otherwise just like, do the things you do in emergency, in a true emergency, but in a real emergency, the new provision is so bad you wouldn't use it you would just use the current ad hockery.

Speaker 2

81:29 And so like, part of the point of passing a bill like this is to give them ad hockery that is formalized and reasonable. But Anthropic is in fear of that because they're afraid that the government would use it. So they're unwilling to grant reasonable ad hockery basically to this federal government, to this administration. But this basically points to the conflict in Anthropic's eyes, which is like, you are going to do preemption of states and you hand this power to the federal government.

Speaker 2

81:56 But the federal government is fundamentally run by people that you don't trust to do this in a reasonable fashion. So then you cripple them. But, like, then that's worse than just leaving it in the states. So, like, you have to then figure out whether you're willing to do this or not. And like, anthropomy is trying to cut it both ways and it's like, causing a lot of problems.

Speaker 1

82:17 Near the end of the conversation, I asked Zvi what else was top of mind.

Speaker 2

82:22 I. I would say the biggest thing is that, like, what I was upset with the most about the policy situation is just that this is not fundamentally attempting to beat the moment. Right. Like Dario has said, we must face the frontier. There is clearly, like, according to the labs themselves and the people internal to the labs, there is a crisis coming where they are able to push capabilities faster than they are able to push alignment. And if they don't do something about this, they're going to have to face some very, very hard decisions where, like, they can either risk their commercial viability or they can risk, like, serious, serious catastrophic problems, including potentially loss of control at some point in 2027.

Speaker 2

83:01 And, you know, they need something to be done about this. And, you know, if you can't meet the moment, if you're not ready with something that can potentially do that, well, why don't you keep working until you can? Things that used to be not in the overseas window or in the overturn window, and things that seemed like, you know, we wouldn't necessarily need them, we now pretty much know we need them. And so, like, that is where a lot of time and attention is going to have to shift.

Speaker 2

83:27 You know, obviously all the crozaic work still needs to happen. All the alignment work still needs to happen. Yeah, the mechan stuff is great. We should absolutely do it. The biggest thing is, you know, we're going to have to meet the moment and we're going to have to, at minimum do something that counts as some form of facing the frontier.

Speaker 1

83:44 Nathan Calvin of ENCODE had publicly listed nine Anthropic decisions he saw as putting the race ahead of safety. And Dario Amadei had replied at length. I asked Zvi how we should understand Anthropic now.

Speaker 2

84:02 I would say Anthropic is a company with a much stronger safety culture, although not adequate to the moment. It's a company that is willing to make big sacrifices in the areas where particularly has strong beliefs or Dario in particular has strong beliefs. And that is to its credit. And it is a company that like, legitimately wants to see that go well and has some like, unique perspectives on, like, what it would take to make that happen.

Speaker 2

84:29 But at the same time, Anthropic's policy division especially like, and also its willingness to like, you know, be an ordinary competition that competes in the irace and buys a lot of capacity and is encouraging more chip production, it's encouraging more power supplies and encouraging more use and blah blah, blah. It is acting like a normal corporation a remarkably large percent of the time. And on policy it is revealing itself to not live up to its ideals reasonably often.

Speaker 2

84:57 And this is just very, very frustrating to those of us who want to be excited by the prospect of Anthropic being a good steward in some sense. And Anthropic teaches us so much information, so many different bits of information that we should be worried about this. I've also, you know, recently from various employees that I talked to at various points, gotten some rather alarming views about, from my perspective on what it would take to solve alignment problems, what it would take to, like, make the future go well.

Speaker 2

85:27 And, you know, all of that plays into, you know, being very highly uncomfortable. They still seem like probably the most responsible lab, but, you know, that's grading on a curve. And I don't like grading on curves because reality doesn't have a curve right. Reality just decides what happens.

Speaker 3

85:45 That's the week.

Speaker 1

85:46 Thanks for listening to the AI and the AM Weekly highlights. The narration was my clone voice. Please tell us what worked and what didn't. See you in the morning.

Speaker 1

87:47 If you're finding value in the show, we'd appreciate it if you take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course. We always welcome your feedback, guest and topic suggestions and sponsorship inquiries either via our website Cognitiverevolution AI or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts with which is now part of a 16Z where experts talk technology, business, economics, geopolitics, culture, and more.

Speaker 1

88:19 We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement@aipodcast.ing. and thank you to everyone who listens for being part of the cognitive revolution.

Summary

The podcast discusses the future trajectory of AI development, focusing on the interplay between financial resources, technological advancements, and regulatory frameworks. Key insights include the rapid pace of AI model releases, the critical role of credit in AI compute resources, and the importance of physical supply chains in AI production.

- AI model release cycles are expected to accelerate significantly, potentially moving from months to weeks.
- Credit is identified as the primary bottleneck for AI development, as lenders struggle to assess the value of GPU clusters.
- The physical supply chain, including semiconductor manufacturing, is viewed as more valuable than software development in the AI landscape.
- The market may consistently undervalue exponential growth in AI, leading to potential investment bubbles.
- Cybersecurity jobs are anticipated to grow, but the timeline for job creation may be unrealistic given the rapid advancements in AI.
- Interpretability of AI models is improving, but significant challenges remain in understanding complex neural networks.
- Regulatory discussions are ongoing, with concerns about the adequacy of proposed legislation to address AI safety and governance.
- The future of AI may hinge on balancing rapid technological advancements with ethical considerations and regulatory frameworks.

Questions Answered

What factors will influence the pace of AI development?

The pace of AI development is influenced by financial resources, technological advancements, and regulatory frameworks. Currently, credit availability is a significant bottleneck in AI progress, rather than hardware or power limitations.

How can we improve the efficiency of neural network training?

Focusing on neurons that show strong activations can reduce the complexity of training neural networks, allowing for more efficient guessing and checking processes.

How do AI models compare to human cognitive processes?

AI models, particularly LLMs, operate on a fundamentally different basis than human cognition, relying on a core set of logical operations rather than the complex and varied processes of human thought.

Why do AI startups often secure large funding rounds without proven products?

AI startups frequently raise significant capital before demonstrating revenue or traction due to investor biases and the survival of only those that can secure funding in a competitive landscape.

What parallels exist between innovation in semiconductors and biotechnology?

Both fields face challenges in innovation, but while biotech is currently seen as more complex, there is potential for unexpected breakthroughs in semiconductor technology as well.

What are the implications of recent AI regulatory proposals?

Recent drafts of federal AI regulations, such as those proposed by Anthropic, aim to establish transparency while preempting state laws, but they have faced significant criticism regarding their effectiveness and implications.

© transcribe · For agents Built with care and craft by Gokul Rajaram