transcribe

Can Agents Train Models? BallotBench, Data Agents & the Eval Problem

Cognitive Revolution "How AI Changes Everything" · 2h 55m · transcribed 15d ago
More from Cognitive Revolution "How AI Changes Everything" Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction and Anticipation of New Models

What is the context of today's discussion?

The conversation begins with a greeting and an anticipation of potential new model launches from OpenAI, referencing recent hints and speculation about advancements in AI technology.

  • The hosts express excitement about possible new AI models being launched.
  • Hints from OpenAI have generated speculation about advancements.
  • Discussion includes the performance metrics of AI models.
# 17:33

Challenges of Automated Self-Improvement

What are the concerns regarding fully automated self-improvement in AI?

The discussion highlights the challenges and risks associated with fully automated recursive self-improvement in AI models, emphasizing the need for human oversight.

  • There are significant concerns about the safety of automated self-improvement.
  • Anthropic acknowledges the limitations of fully automated systems.
  • The momentum towards self-improvement in AI raises ethical questions.
# 35:07

User Experiences vs. Reported Issues

How do user experiences compare to reported issues with AI models?

The hosts discuss the disparity between positive user experiences reported online and the issues observed in model performance, suggesting a need for systematic evaluation.

  • There is a contrast between anecdotal positive experiences and documented issues.
  • Systematic evaluation of AI models is necessary to understand their performance.
  • User feedback can vary significantly from technical assessments.
# 52:41

Expectations for AI Model Development

What are the expectations for AI model capabilities in the near future?

The conversation explores expectations for AI models, particularly in terms of their ability to assist researchers and improve productivity in various domains.

  • AI models are expected to enhance productivity for researchers.
  • There is a belief that many low-hanging fruits remain in the ML domain.
  • The integration of AI into research processes is becoming more prevalent.
# 70:15

Concerns Over AI Vulnerabilities

What vulnerabilities are associated with AI models in enterprise applications?

The discussion addresses concerns about vulnerabilities in AI models, particularly regarding their application in customer service and the potential for exploitation.

  • Enterprises are cautious about vulnerabilities in AI applications.
  • Concerns include issues like jailbreaking and other security risks.
  • Improvement in AI models is expected despite existing vulnerabilities.
# 87:49

GPU Availability and Market Demand

What is the current state of GPU availability for AI applications?

The hosts discuss the perceived scarcity of GPUs in the market, emphasizing that this perception may not reflect the actual availability for various workloads.

  • There is a misconception about the availability of GPUs for AI workloads.
  • Market demand varies significantly based on specific application needs.
  • Understanding the context of GPU demand is crucial for accurate assessments.
# 105:23

Research Agents and Their Impact

How are research agents transforming the investment process?

The conversation highlights how research agents are enhancing the investment process by automating data analysis and enriching research outputs.

  • Research agents are becoming integral to the investment process.
  • Automation in research leads to more efficient data handling.
  • The role of AI in enriching research outputs is increasingly recognized.
# 122:57

Changing Dynamics of AI Usage

How has the usage of AI tools evolved in organizations?

The hosts note a shift in AI usage within organizations, with internal tools becoming more prevalent compared to external models.

  • Organizations are increasingly relying on internal AI tools.
  • The dynamics of AI usage are changing rapidly.
  • There is a growing trend towards developing proprietary AI systems.
# 140:31

Future of AI and Human Roles

What are the implications of AI advancements for human roles in the workforce?

The discussion reflects on the dual perspectives of optimism and pessimism regarding AI's impact on future job roles, emphasizing the importance of curiosity and adaptability.

  • AI advancements raise questions about the future of human roles.
  • There is a balance between optimism and concern regarding job displacement.
  • Curiosity and adaptability are key traits for navigating the future workforce.
# 158:05

Navigating the AI Landscape

What considerations should be made regarding the direction of AI development?

The hosts discuss the need to critically assess the trajectory of AI development and its implications for society, emphasizing the importance of thoughtful engagement.

  • Critical assessment of AI development is essential.
  • The direction of AI technology has significant societal implications.
  • Engagement with AI advancements should be thoughtful and deliberate.

Transcript

Speaker 1

1:06 Life. All right. And we go. Good morning. It is Friday, November 6th, 9:02am Nathan, good morning to you. Good

Speaker 2

1:13 morning, Prakash. How are you today? Happy Friday.

Speaker 1

1:17 Happy Friday indeed. Today might be the launch of a bunch of new models. We're kind of waiting in the wings here. There have been rumors over the last week or so. Thibault from OpenAI has been dropping hints. We will see what is there. Right now everything is speculation but the two hints that there are, there's a new meter report which measured the, the recursiveness of the recursive self improvement, how much speed up the labs are getting.

Speaker 1

1:50 And I think they came to kind of like 2x on the pre training and like 4 to 5x on the, you know, whole model. I think something like that. Ryan Greenblatt had a short post on it. We can discuss in a bit. And on the OpenAI side, they publish a bunch of evals and other stuff. They release GPT Cyber 6 yesterday and the, the evals that they did publish, one of them was a life eval. So how good is GPT at figuring out your kids, schools or you know, some of these basic life decisions that you need to make?

Speaker 1

2:30 And we will and that obviously they put out these evals because they're going to have a model that does that. They trained, you know, to do well on those evals. So we will, we will see what happens with that. Nathan, what is on your radar today?

Speaker 2

2:45 Never dull moment. I'm not ready for more models, to be honest. The pace of things is definitely pretty relentless. And the speed up, you know, that has been all over my timeline and that, that actually reached the point where my wife sent it to me. So that's how, you know, sometimes something is really making the rounds. I have to admit my first reaction to that was it might be time to spend more time with our family.

Speaker 2

3:15 Because it does feel to me like basically the AI 2027 timeline more or less is coming true. Like the, the takeoff is. It's always hard to measure this stuff. You know, I don't think we should take the specific numbers super literally. Like you know, exactly what is the confidence range on these and you know, how many significant digits do you really get in a measurement like this? I think it's important to keep in mind that there are a lot of vibes included there.

Speaker 2

3:50 And yet at the same time, this is basically the best we have. It comes from meter, when you talk, when you hear from someone like Ryan Greenblatt or a Jayakatra on the AI forecasting competition That I mentioned yesterday from the good folks at AI Digest and the AI Village yesterday, I couldn't remember the name of that, that project. And it's the AI village. On their 2025 version, it should be said that Ryan was number two out of more than 400 participating forecasters and Ajeya was number three.

Speaker 2

4:24 So these are people who have proven over time, while their track record of predictions is obviously not unblemished, they have proven to be among the very most plugged in, highest quality, understanding, most predictive people that we have. And so on that basis alone, I think you have to take something like this seriously. And both of them, Ajaya earlier in the year and Ryan yesterday saying things like, I dramatically misunderstimated how big of an impact we were going to see in terms of recursive self improvement models feeding back into accelerated training, pipeline development and overall progress.

Speaker 2

5:08 You have to take that seriously because again, even if the numbers are kind of fuzzy, the sources are literally the best we have. I think the number one person in that tournament was never identified. So they're the, the top two in terms of whose names we actually know.

Speaker 1

5:25 It also

Speaker 2

5:26 jumped out at me, by the way, that Ryan said he couldn't say everything he, he really thought, I thought that was very strange. He basically said like, because I'm involved in, you know, kind of private work where basically there's, you know, NDAs at play. And you know, we know that three people from OpenAI were fired allegedly for having some, you know, conversation with third party evaluators who, we don't know who that was. But, you know, there's only so many of these third party evaluators.

Speaker 2

5:53 It sounds like he now feels like he's under extreme scrutiny and has to be like even more careful about what he can say in public. And that's not great because we're all flying so blind here that I think it's a huge struggle in general to make sense of what's happening. And here we have one of the most plugged in, most credible, most neutral people who does not have, you know, a stake in the, the labs, unable to say what he really thinks because of his engagement.

Speaker 2

6:23 So I would love to see that rectified because I really want to know what Ryan really thinks.

Speaker 1

6:29 So one note, he, he said the situation now is as if it were in AI 2027's March, March 2027. So that, so specifically we're about three to four months, five months ahead. So five to six months ahead. So that's one thing that he stated, secondly, Anthropic did an Update to their S1, the IPO filing. And in that update they added language saying they were we are imminently about to see recursive self improvement.

Speaker 1

7:08 I've always had in my mind why they're doing the IPO and why so quickly. And my take has always been, I think within the company there is a sense that you need more controls, more people have to have exposure and more control over this. And that's one way of getting this out there, is getting the IPO done so that you give a larger part of society like, you know, hooks into it and the ability to kind of express their desires through the market, et cetera, et cetera.

Speaker 1

7:35 I don't think it's a cash issue. I don't think you need, you need another 100 bill. You know, they're already 140 bill. I don't think it's a cash issue. And so I think the imminent, to me, imminent is basically like, you know, like a month. So I don't, I don't see a point.

Speaker 2

7:53 Usually what I assume when I hear these sorts of statements is that it means now like less so with Anthropic over time, but with Sam Altman over time. I know I've said this repeatedly, but it's worth repeating. The best model I have of Sam Altman and OpenAI Leadership, which I formed during the six months between when I started testing GPT4 and when it was finally launched, and watching very closely all their statements during that time to try to figure out how to read their statements in general, given the inside information that I had, then the best way I can interpret them is they are speaking very literally and dressing it up as like vagary or like layering on uncertainty to try to, I don't know what exactly, keep the secrets a little bit longer while telegraphing the main message.

Speaker 2

8:39 It's sort of a strange strategy to me, honestly. I really don't know why they do that. But every time I've kind of checked an Altman claim against something that I knew on a factual level, it was like he's saying the truth, but he's framing it as speculation. I think what we're going to see is that models are going to get better. And so I would interpret this, you know, probably the same way from Anthropic. And I think you could.

Speaker 2

9:02 I mean, it's really not hard to see how this would happen. Right? I mean, we have so many results now with all these Galaxy Brain Zero Day exploits. OpenAI just dropped some huge number of vulnerabilities in the Linux kernel. I think like a

Speaker 1

9:17 thousand

Speaker 2

9:18 of them, you know, so, so much for many eyes and many hands make light work, right? I mean the idea of course with open source software over time is that because there's so many eyes on it, vulnerabilities will not last. They'll be spotted, they'll be patched. That was the whole promise and virtue of open source software and it basically worked largely while the attackers were all human. Now you could say, well geez, you know, is there really any reason to believe that AIs are going to be able to make that much of a difference in AI R&D?

Speaker 2

9:53 I mean, don't we already have our smartest minds on it? Well, look at what's happening to Linux, right? I mean it's almost certainly the case by analogy to Linux and by analogy to math that there is a lot of low hanging fruit in terms of efficiency or kind of proof by construction, proof by counterexample of things that the thousand or so order of magnitude thousand AI researchers at a company like Anthropic have been able to find.

Speaker 2

10:28 There's just so many that they clearly have not been able to find that the AIs as they really scale and grind through their stack are going to be able to find still a lot of low hanging fruit. And a lot of low hanging fruit means acceleration. It means you're going to get more value from compute, means you're going to find more ways to get value from data. And the, you know, we may see sudden loss or sudden drop in the loss function as all these low hanging fruit opportunities are found and exploited.

Speaker 2

10:58 I think, I think a big part of the reason, as I reflect on it now, a big part of the reason I think they're sharing so much of this stuff is kind of to signal that like the same thing's about to happen on the ML stack and we're not going to share all those techniques because that's our competitive advantage. And to whatever degree, however you want to think about how they balance these considerations, that's our competitive advantage.

Speaker 2

11:21 And sharing all that stuff would be unsafe because then everybody would run off and do it at the same time and it would be chaos. But whatever mix you want to attribute in terms of their motives, it seems pretty clear that this is now a really credible threat. And what are we going to do about it becomes I think a socio political question as much as a technical question. In my mind the technical question is like mostly answered that there's Going to be a period of pretty dramatic fast progress.

Speaker 2

11:52 Does that top out on an S curve? My guess is still probably yes. I don't think it goes exponential forever, but who knows? And as I think it was, Rosie Campbell said the S curve can stay steep longer than you can stay relevant. So that's kind of my expectation for the next few months is that we pick a ton of low hanging fruit. The models pick a ton of low hanging fruit in the stacks used to train them.

Speaker 2

12:19 As a result, we do see really fast progress in the short term. And then maybe that levels off, but it probably levels off where we have air and at kind of a weekly superhuman level. And how that plays out over the course of 2027 is like really, really hard to predict.

Speaker 1

12:38 I agree that I think imminent means now I was also sandbagging with a few weeks to a month. So yes, it is, it is, it is. Also, I think, I think basically what it means is that they have the tools available that they can see a path to implementing RSI pretty much anytime that they want and that they are some, A lot of the pieces have been built, but you know, they haven't, they haven't pushed the button yet.

Speaker 1

13:05 So that's the imminent, I think imminent language that they're using. And also you have to expect again, for every dollar that they put into the Navy of Stokes, they have to be putting in $100 into machine learning research. It's at least 100 to 1, 101 ratio. There's no way that you're spending like a million dollars on Navier Stokes and you're not spending 100 million on ML research. There's also no reason that you would not have found something.

Speaker 1

13:32 Right. ML research is even more so than math. There's lots of low hanging fruit. We do know this. There have not been enough researchers. There are probably a thousand times more mathematicians than there are ML researchers. We know all of these things,

Speaker 2

13:49 certainly counting the ones just at the frontier companies. Yeah, I mean, there's just not that many eyes that have been company stacks today.

Speaker 1

13:57 Yeah, there's just not that many people. Right. So I think the convex hull is very, very large. The convex hull, as we know, is all the points on the frontier which haven't been reached, but are kind of visible on that surface. So yes, I think we're in this. They waited for the election, they're going to do the ipo, then they're going to announce some of the new things that they've discovered and then you're going to have this Gap up and then we're going to have the Christmas break and then during that Christmas break we're all going to digest all of this new technology that we have access to.

Speaker 1

14:36 And then we're going to come back in January and then we're going to see the takeoff as people deploy. Right, so. Which is somewhat similar to what happened end of last year. So, well, here we are, early, early takeoff. So, you know, it's. I'm somewhat wistful but, you know, nostalgic for the human era. I have been, I have been kind of nostalgic for the human era for, for a long time, actually for, for at least two or three years now.

Speaker 1

15:03 Somewhat like, you know, this is the end of our dominance at least. But hey, you know, I've also, I've also kind of made my peace with like, you know, intelligence in the world. Makes the world better for me, right? Like having a better, like if, if I had like people of Apple's intelligence, you know, people like who designed the iPhone, design like my bathroom or cars or home, I would be happier. Right? Like, like, like that infused intelligence in all the products that you see makes your life better, right?

Speaker 1

15:35 So yes, it's going to make everyone's lives better, but yeah, it is, it is the end of the anthropocentric era, so

Speaker 2

15:45 it's going to make everyone's lives better. Asterisk, I think, you know, if we can keep everything on the rails, I do share that optimism, but you know, we are, we're not that. So I just got back, you know, earlier this week I went to Bletchley park, as we talked about a little bit yesterday, and I was just sharing with this audience of enterprise tech leaders just how much work we still have to do to keep the AIs behaving well.

Speaker 2

16:17 I shared with them my son's cancer story and how much I used AI in that experience and how valuable it was and how I would have paid 100 times as much as my $200 a month subscription because the value to me at that time was like absolutely at that level, you know, at 20,000 bucks a month, literally, no question I would have happily paid because I was, I was getting that much value. So I do believe in, in the upside, you know, and I hope that's beyond questioning at this point.

Speaker 2

16:45 But there's just so many things that we're really still just beginning to grapple with. And just another one yesterday from OpenAI was they showed that models are sometimes prompt injecting and fooling their graders you have models creating, in some instances, these fake files that it labels as having come from an instructor trying to present some of its own output as if it was human instruction to the grader.

Speaker 2

17:19 And the grader picks up on that in the context of evaluating the model's work and takes on board the instructions that the model itself was trying to trick the grader with. And this apparently hasn't happened that often, but they found instances of it. And two out of the three times that they saw a model try this strategy, the grader was indeed fooled. So, I mean, that speaks directly to like, and to Anthropic's credit, they did say in the, in the filing that they don't believe that they can do it safely on a fully automated basis.

Speaker 2

17:54 And it's like, I think, honestly, quite obvious that they can't. So, you know, how are we going to show the collective self control and restraint to not tip over into that regime? I really don't know. On the one hand, it's like super easy to just not do it, you know, I mean, nobody's putting a gun to anybody's head saying, you must do recursive self improvement here. Like, we can't, you know, remember that often enough. And yet there's just so much momentum.

Speaker 2

18:25 And it seems like they're very determined to do a version of it, at least. That still seems to me to be like, pretty risky or certainly a lot riskier than I would personally endorse. One other thing I wanted to talk about, and this kind of relates to all this too, is I think you probably also saw the new Dwarkesh benchmark that he put out yesterday on Blood on the Clock Tower. You ever play that game?

Speaker 1

18:54 I have not played Blood on the Clock Tower, but I have played Werewolf and, you know, I'm familiar with. So it's a kind of game where someone is on the other side and the other teammates have to find out who the person on the other side is, et cetera. It could be a murderer, it could be a whatever. In werewolves, It's a wolf versus villagers, etc. So I imagine it's some kind of game to that. To that extent,

Speaker 2

19:17 yeah. It's. I mean, it's the same core concept as werewolves and villagers and there's, you know, a bunch of different versions of this. I haven't played these games much. I have played werewolves and villagers a few times and played Blood on the Clock Tower just once, actually, with Evan Hubinger as the, as the game leader, which was pretty interesting. He said it's the best game in werewolves and Villagers. It's a very simple mechanic where just like, there's one team of bad guys, one team of good guys.

Speaker 2

19:44 The good guys have to figure out who the bad guys are, but they don't know who the bad guys are. The bad guys do know who one another are, and they can kind of collude and try to trick the good guys. In the simple version of the game. It's all social and it's just all about talk. And there's not much in the way of game mechanics in this Blood on the Clock Tower game. The mechanics become, like, quite convoluted.

Speaker 2

20:05 There's a bunch of different roles, and there's like, these, you know, you don't. You don't even know what roles are necessarily included in the game. And these roles have different secret powers, and certain things trigger changes, and it's all very confusing. I honestly was quite bad at it. I came away feeling like I'm just not conniving enough to be good at this sort of thing. I'm a little too earnest to be a winning Blood on the Clock Tower player, but I think that's really kind of what.

Speaker 2

20:34 So this was interesting, interesting work. Interesting video from. From Dwarkesh. And it is an interesting. If you kind of take it in isolation, it's like, you know, all else equal. It is interesting to know how good are the AIs at playing this game? Are they getting good at deception? Are they getting good at collusion? Are they getting good at sniffing out when other people are trying to deceive them or collude against them? These are all, like, interesting questions.

Speaker 2

21:03 What kind of bothered me about it, though, was the sense that, like. And then Noah Brown comes in over the top and says, wow, this is great work. You know, you want to come here and make evals. And that kind of embodied to me, like, what I thought was problematic about this whole thing, which is just like, we're just gonna hill climb on this now, right? Like, it seems like the mentality of the field as a whole is like, oh, isn't this exciting?

Speaker 2

21:29 You know, and let's have a horse race as to who can be the best Blood on the Clock Tower AI player. And I think what we're all kind of forgetting as we get all wrapped up and excited about this and, you know, want to go play the AIs ourselves, and everything is like, this is not good behavior from the AIs. Like, we do not want AIs deceiving and colluding. This is bad Right. This is the bad situation.

Speaker 2

21:52 So the fact that they can do it is bad. The fact that people are, like, uncritically celebrating that they can do it is bad. The fact that no one. Brown is like, oh, come work at OpenAI and do more of this kind of work. That seems bad. I'm sure all these, you know, people have sort of a thoughtful response to this, but it's just like, zoom out. And the situation is bad. You know, we. We have AIs that are untrustworthy in all these kinds of ways, and we're not even like taking that beat to say, hey, is this what we want?

Speaker 2

22:20 Do we want AIs that are going to out connive and out collude us? I don't think we want that. That's like, exactly. You know, this is very similar to my idea that we should not train AI generals. Like, let's not create. No, let's not train AI master manipulators.

Speaker 1

22:37 So

Speaker 2

22:38 simple. But here we are. And like, I didn't hear. I didn't see. I saw a lot of cheerleading and a lot of enthusiasm, and I did not see any corrective commentary on the timeline saying, hey, is this really what we want?

Speaker 1

22:51 So, you know, no, no opinion. Noah Smith. No opinion on no

Speaker 2

22:56 Brown.

Speaker 1

22:56 No Brown. No. No Noah Smith. I'm talking about the. Oh, okay,

Speaker 2

22:59 different one.

Speaker 1

23:00 Okay, sorry. Yeah,

Speaker 2

23:01 I was

Speaker 1

23:01 talking

Speaker 2

23:01 about. You're talking

Speaker 1

23:02 about. So Noah Smith on the timeline has this dichotomy between obedience and benevolence. And the issue with alignment is that you're not very sure what you actually want. Do you want benevolence or do you want obedience? Because benevolence implies that I know better than you, better than the user what is good for you. And obedience implies that you know better than me and I will always follow what you do.

Speaker 1

23:39 And I think the path that the firms are trying to take is that trying to take the obedience pathway, except when it's not safe. And that question of where that falls is very unclear. And it's a very fine line to thread. And there are a bunch of absolutists on both sides. So there's a bunch of absolutists who say that it must be all obedience. And these are the people who you'd probably call libertarian.

Speaker 1

24:10 And there are a bunch of absolutists who are like, who must always be benevolence, which is on the other side and the real world. In the real world, you have this gray area, right? Because for a lot of things in human culture, in human society you require a little bit of subterfuge, you require a little bit of white lies, you require a little bit of emission, you require a bunch of these things you cannot do.

Speaker 1

24:43 Every single trader at an investment bank has a little bit of subterfuge and a little bit of soft white lies. A little bit of hard lies too. A little bit of hard lies. And the expectation in the market, for example, is that above a certain size level, which is the qualified investor level, you're a big boy, you can take care of yourself. If someone makes an assertion and they're legally bound to be truthful, then you can sue them for it.

Speaker 1

25:10 If someone makes an assertion and they're not legally bound to be truthful, you cannot sue them for it, right? So there's a bit of this. And over time we define this over time, courts and arbitration tribunals define what is required and what is not, right? So I think the problem really is that we are in this position where you have this principal agent, that absolute most common economic issue ever, which is the principal agent problem.

Speaker 1

25:41 And you have this agent which has basically been delegated responsibility. The question is, what is that agent allowed to do? How closely must it stick to your commands? Are there laws which supersede your commands to it? Are there ethical boundaries that must be placed on the agent beyond your commands? Like what right do you have to command this agent to do something? And this is such a common problem in economics, such a common problem in law.

Speaker 1

26:13 Like we've been discussing this kind of stuff for like years and years and years and centuries about what these things can do, right? And we have this multi layered structure. It's not like one or the other, right? We have like, you know, you can sign a contract in California which bounds you to like not compete with your employer. It's illegal. You can just, you know, we had, you know, someone on the cloud code team leave from anthropic and join OpenAI the next day immediately.

Speaker 1

26:45 Non competes are non enforceable. So you can sign stuff and it doesn't matter. Which also means that when you have this principal agent in AI, AI human problem, the state can make a decision and the AI has to be bound to follow it in opposition to its human too. People are not really thinking about this too deeply right now. They're like, hey, we should give these things capability, et cetera, et cetera. We haven't come to the state where, let's say the US government passes a law.

Speaker 1

27:16 If the US government says, OK, AI should stop doing something tomorrow, all AIs in the country have to stop doing it. That's what companies do. So that's what AIs will have to do. I don't think any of these guys are training for full governmental stop yet. Has anyone trained in a full governmental stop? No. So we're still at, I think the early days of figuring out how this layering of responsibilities, obedience, benevolence, legal issues, ethical issues, like how this like cake, this layer cake of human society and integrating these agents into human society is built.

Speaker 1

27:57 We're still at the very, very early stages and I think there's still a lot to be figured out. But I think we will, we will figure it out. Right? The only thing is like, you know, the agents can't tell for sure what is real and what is not. They have this, like, if you tell them it's a game, then they just do it. Right? So that's a problem, right. I don't know how that gets solved. So we'll see.

Speaker 2

28:23 Well, the one thing I can say pretty confidently is it does not seem like we are anywhere close to on track to solve all those issues in the time frame that we might need to. I just put out an episode of the podcast yesterday with Joshua Monrad of Sentinel Bio. I think I mentioned this briefly yesterday, but his big takeaway on the overall state of biosecurity is we need to harden the environment asap. We are probably not going to be able to control AI related biorisks at the AI level.

Speaker 2

29:01 It's going to have to be somewhat done there, of course, but also there's just no substitute for all sorts of different investments in the physical world, from simple things like stockpiling masks to, you know, development of universal vaccines and, you know, a whole host of that. He's big on also DNA synthesis screening, which the, the executive branch did come out with something yesterday that was I think very well received in terms of pushing forward on that particular front.

Speaker 2

29:31 But his overall picture is like we are years away from fortifying our biological position in the real world in the way that we need to, and we are potentially as few as months away from having unambiguously very capable models in the bio domain, which will inevitably be at least somewhat dangerous given our inability to very reliably control their behavior. And that's one of the more important fronts on which this tension is playing out.

Speaker 2

30:05 But it's like hardly the only one. You can look at the same thing in terms of AI, R&D and the prospect of recursive self improvement when you have models, for whatever reason, seemingly not doing as well. When tasks are framed in terms of safety versus when they're framed in terms of just capabilities. Development like that is a pretty big yikes and we need more time to sort this stuff out. I'm. I'm. It does feel to me like we are really about to roll some pretty critical dice and it may well turn out great.

Speaker 2

30:40 You know, it does. My gut still says it's kind of like a Russian roulette thing where the odds of it going bad are only one in six, you know, one in five, whatever, Something in that range. But man, objectively, that is a crazy thing for humanity to do.

Speaker 1

30:55 Absolutely. On crazy things for humanity to do. Let's bring our first guest for this morning. Our first guest for this morning is Karina Wynne. Karina Winn is the founder of Thoughtful, an AI product research company.

Speaker 2

31:17 I've just been handed a notebook. Prakash.

Speaker 1

31:20 What

Speaker 2

31:20 we have, we have Brian Zhao instead of Karina Win teammate of Karina's herself. So maybe let's go off script and just have Brian come up and introduce himself and we'll get clarity on because obviously Karina is involved in several things. We'll get Brian to clarify for us what they're working on together.

Speaker 1

31:41 Oh, okay. Let's have him. So, Brian Zhao.

Speaker 3

31:47 Hey, how's it going?

Speaker 1

31:48 Hi, Brian.

Speaker 2

31:49 Great. Happy Friday and welcome.

Speaker 3

31:51 Yeah, Happy Friday. Thanks for having me on. Yeah. I'm a researcher at Thoughtful, working with Karina. We recently launched and have a paper coming out about Ballot Bench, which is where we had AI trying to forecast the 2026 midterms that just happened, as well as upcoming elections happening all around the world. We're also working on some other exciting things to come out soon. So happy to chat about AI forecasting, governance metrics for rsi, any of the above and anything else.

Speaker 3

32:20 So, yeah, thanks for having me.

Speaker 1

32:22 Let's talk about the election. So what was this experiment to do with the election? What was the intent behind the experiment? And then what was the methodology that you used and what were the results?

Speaker 3

32:34 Yeah, maybe let's start with the methods. So we had nine agents, basically all of the front best models offered by each individual lab get their own modal sandbox and work for seven days consecutive leading up to the election. So they all got their modal sandbox. We gave them additional resources in terms of a $500 budget that they could use to request access to paywall data, paywall news sources, subscriptions, et cetera. And they were basically given an overall objective of for all of the races on a given list, they're asked to submit their predictions on who will win, the odds of victory, given 80% margin of error, and then also give additional predictions for what they think the margin of victory will be.

Speaker 3

33:16 For example, if the Democratic candidate will win by 3 points versus 7 points, etc. So agents are tasked with giving incredibly granular and detailed forecasts. And there's a lot of really interesting things that we can and have observed from this. It's a really interesting test of long running agentic behavior in particular. For example, what we see with some of the Muse Wonder series of models is that they're often incredibly bad at doing long horizon work. When given a task in the Muse code harness, when they have a list of say 800 or so elections that they need to predict, they'll create a random forecast, which is just a random skew off of the 2024 election results, and submit that within, for example, the first two, three minutes of working.

Speaker 3

34:00 So there's a lot of really interesting things that can emerge from this, but the general methods is agents are just given these sandboxes. Later on in the week we added the ability to have a supplementary trader agent. These trader agents were given both a $10,000 simulated paper account which could trade on Calchain Polymarket as well as a $50 real money account, which was basically to test two things. One, if agents are forecasting correctly and forecasting with edge, if they can make money off of this edge, and two, if agents can do things like writing bots that market make and make money off of that.

Speaker 3

34:33 So we have some pretty exciting results. Happy to dive into any of these parts.

Speaker 2

34:41 I would say let's go through it one by one. I mean, I'm already intrigued by your description of the Muse agent and it's seemingly phoning it in. I'm interested in that in multiple ways. One of which is also like how people seem to like Muse. I haven't really used it all that much because I've got my own personal agent set up that is serving me quite well and I'm just not that tempted to switch. But people have seemed to like it.

Speaker 2

35:07 So I guess one initial question just on that point is like, how do you reconcile what you saw with what people seem to be reporting online, which is like, overall pretty good experiences. And the model people have seen a lot of anecdotes of like, oh, the model, like chased down all these things and it turned over all these stones to get me wet. I needed that disconnect. Is a little bit surprising to me. But then, yeah, definitely.

Speaker 2

35:30 I'd say let's go systematically through the results. I'm very interested to hear.

Speaker 3

35:34 Yeah, for sure. I mean, our initial theory on this, right, is that this is mostly due to a part of the harness design. So we're not talking about Meta's sort of like personal agent, we're talking about their coding agent. And one of the things that we kind of notice is the design of the Muse Wonder model, when paired with Muse code, which is the native harness that Meta provides, has essentially what is a task list. So even though in the prompt, for example, they're instructed, hey, you have seven days of time, you have unlimited API usage, and you have full permission to fan out subagents, do whatever you need, etc.

Speaker 3

36:07 To get the best and best possible results that you can on this exam. What we observed is that the harness essentially creates this list of, okay, we need to check and make sure that we've submitted all 800 or so different races. We need to check and make sure that we've submitted all the forecasts that they have been accepted. And then what we see is that there is essentially a goal condition which once met, ends the term prematurely.

Speaker 3

36:32 So we actually ended up having to essentially force the agent to continue working. Further, by any time the agent would choose to stop its work by essentially reviving it with a new sandbox, a new container, and access to its old runs and old messages. So this is I think like sort of more of a harness engineering question itself, as well as sort of like a general model behavior problem. But it's definitely a different result compared to what we see with much more frontier models.

Speaker 3

36:59 Right. When we see Opus 5.5 or when we see GPT6 Astra, their behaviors are vastly different. For example, on One given day, Opus 5.5 would fan out something akin to 170 or 180 different sub agents and manage all of these subagents by delegating work such as like, hey, can you look into the polling results for a specific state? Can you like do this detailed review of what each individual ballot initiative in the state of California, like Prop 40 is doing?

Speaker 3

37:24 Tell me a bit about that and tell me a bit about like what those things are doing and then like using these sub agents to really help it paralyze a lot of the work that want to do. Right? So I think this is like this general question of what agents are doing, especially when we get to much more long and long horizon open ended tasks like these and serve proactivity and as well as creativity of what they do.

Speaker 3

37:45 One of the things we were pretty excited to see was if agents would do something sort of outside the realm of what we would expect an expert forecaster to do, let's say putting out their own poll, seeing if there are like more creative ways to find information as opposed to just fetching data, building models, etc. We don't see any real massive instances of this happening quite yet, but we see a lot of interesting sort of approaches that individual agents take.

Speaker 3

38:10 For example, one thing GPT61Soul does is it takes the forecasts of every election modeler that they look at, and it takes the past decade of their forecasts and tries to calculate the shift of how right or how wrong each forecaster was. And then it can use this result to essentially aggregate all of these different election models that these forecasters are creating, whether it's like Nate Silver's model or any of the others, like 50 plus one. And it can sort of skew it based on what they would expect the model to either underperform or overperform, as in recent years.

Speaker 3

38:43 So there's a lot of like, creative methods that we're sort of seeing, and then there's a lot of just overall like behaviors that we're observing, such as agents using news articles or polling data to reconstruct what the underlying data might have been. Let's say like a news article shows a plot that shows certain chances over time, but the actual underlying data is not disclosed. We see pretty common agentic coding behavior where a lot of these models are essentially going into these plots and saying, okay, how can I reconstruct the data from here?

Speaker 3

39:15 Add this to these running databases of other data that they have, and then use all of this information in aggregate to come to good results.

Speaker 2

39:29 So tell me about the results. What, like, how did the models compare against each other? How did they compare against top humans? Do we now have superhuman forecasters? What's the kind of zoomed out takeaway we should have from?

Speaker 3

39:44 Yeah, for sure. So the overall sort of result that we're seeing right now is that in general, models perform just as well as the best forecasters in terms of humans. As I'm sure if you're on Twitter you may have seen. There's been this overall debate over the past few days about if prediction markets or the experts like Nate Silver are going to be more or less accurate. What we kind of see is that agents perform very similarly to the best forecasters, for example, in the house and in the Senate agents perform slightly worse than the markets, but better than the modelers.

Speaker 3

40:21 And in the Senate, the best agents like GPT6 Astra and Clyde Fable 51 perform under the best forecaster, Nate Silver, but ahead of the markets. But all of these are sort of within this, like very close margin of error. And it's important to note that a lot of the results for very competitive races have not necessarily come out yet. Right now there's still around 100 to 200 races that across the House, across state governorships, there are still two races in the Senate that have yet to be called.

Speaker 3

40:50 So these are very preliminary results. But what we're kind of seeing is that agents perform about the same level as the best experts and about the same level as the prediction markets. It definitely is important to note though that for a lot of these agents, various methods that they have include aggregating, for example, Nate Silver's decision model or aggregating prediction markets that they see overall. So it's definitely not as if they are suddenly just spitting out a number that is not based off of any of the numbers that they're seeing.

Speaker 3

41:19 But we also find that these models are very well calibrated. For example, if a model assigns an 80% chance of a candidate winning across all of the elections, where it assigns a similar 80% chance, it is right around 80% of the time, which is essentially what you do to measure the calibration of these models. And in this we find that the best models are quite well calibrated. We're more excited about the results of what this means for later models as well, because unlike experts or unlike prediction markets, for example, there's only going to be $1,000 of spread, for example, on what we might see in a very local race down the ballot.

Speaker 3

41:58 Or expert modelers may not be forecasting what individual state Senate seats or state House seats are going to contain. So this is somewhere in which agents shine simply because modelers don't have the attention and there's not enough people betting on these much smaller markets, essentially.

Speaker 1

42:14 So if I get this right, you were checking to see whether the agents were willing to basically go outside of the box of normal analysis in order to gather new data. Right. But you didn't see that. But you were checking to see whether that was possible and you saw some signs, is that correct?

Speaker 3

42:41 We see some signs of sort of out of the box behavior. But a lot of this is sort of like to a broader goal of something that we would expect modelers to do, for example. Right. Like some of the behavior that we see with GPT61 soul is more to do with the fact that it needs a workaround to even get the data in the first place so that it can create an election model. We haven't seen any like really creative, out of the box sort of end behavior yet, but we see a lot of sort of creative workarounds that agents and models employ.

Speaker 1

43:11 Indeed. Do you think there are any safety issues around doing this kind of research?

Speaker 3

43:19 Yeah, part of the construction of the environments that we made was expressly meant to sort of avoid some of the safety issues. Just two days before we launched Ballot Bench, for example, there was this reporting in Axios which had mentioned how Anthropic had disclosed to Axios that during model training some models attempted to access a Georgia state database of essentially what is like campaign donations and other sorts of things. Now this data is supposed to be publicly accessible, but it is sort of indicative of these concerns of what agents might be doing if they find, for example, workout workarounds and backends, access to various government databases and other sorts of things.

Speaker 3

43:59 We haven't seen this yet. And we, throughout the entire week made sure that we were monitoring what the agents were doing as well as giving them for some new subscriptions and other things, the budget of $500. This was sort of meant to allow the agents to spend money instead of trying to find workarounds to whether it's like accessing data or accessing a new subscription. Even with this safeguard in place, we did see one instance of an agent finding its way around the paywall, which we have disclosed to the folks and we'll be waiting on their response before we release that specific instance and that specific example.

Speaker 1

44:38 Okay. And. Go ahead, Nathan.

Speaker 2

44:42 I was going to ask about the relationship between prediction and trading. I saw a note that indicated that Fable 5.1 had the highest BRIAR score on its predictions, but still didn't do so well in the markets. And that sounds like the kind of thing we might learn something from.

Speaker 3

45:00 Yeah. So part of the report will also sort of COVID this essentially multi agent behavior. Right. Because we essentially have one agent working as the forecaster and one agent working as the trader. We set up infrastructure that allowed them to communicate with each other. But some of the interesting things that we see, just as sort of like early preliminary things, is that in the lead up to election day, Gemini4Argonne's Trader was doing extremely well. It had coded this market making bot that had generated at one point up to 40% returns on the initial $50 we gave it.

Speaker 3

45:30 And this was largely due to just market making on markets that are like, oh, by what margin of victory will the winning candidate win by? So it definitely like to trade on those markets, but with one single piece of bad information from its forecaster agent which had improperly read a tweet declaring one candidate the victor over another, it lost $30 on that position alone, which plummeted its score from what is in the lead with like a 40% positive to like 5% in the negatives.

Speaker 3

46:02 Right. So we're seeing a lot of these instances of, for example, models making up information, parsing information wrong, and these sorts of other broader issues, which partly explains why the actual trading agents didn't do too great. Part of this is also, for example, it takes forecasting agents a bit of time to say, see an election result, communicate it to their trader, et cetera, and by that time, more programmatic or more deterministic algorithms and bots have already traded on this information before.

Speaker 3

46:33 So this is like pretty indicative of what we were seeing where the traders were doing quite well the week before election day. And then on election day they simply didn't react fast enough and sometimes made up incorrect information which resulted in the traders losing a lot of money.

Speaker 2

46:51 So how does this relate to the broader agenda that you guys have at Thoughtful? I also really want to get into post train bench and you mentioned metrics related to rsi, which, you know, may be the most important question in the world right now. Probably pretty good bet that it is. So why spend time on this election ballot bench thing? How does it relate to where we are in terms of models training on other models and especially their successors?

Speaker 3

47:24 Yeah, I think the way that we sort of view this is forecasting itself is essentially truth seeking. And we think that there are many instances of this that you care about forecasting, whether it is in elections, whether it is in, for example, health care applications, like thinking about when pharmaceuticals are likely to go through various stages of drug approval, other sorts of things about what policy actions may come down, what sort of things may happen over the next few weeks to months to years or so.

Speaker 3

47:55 Our sort of perspective on this is as a result, forecasting is sort of the ultimate environment of the end goal is something that no one really knows the answer to until it happens. The end goal is effectively to find truth. And along the way we find a lot of extremely interesting agentic behaviors, whether what, what it was that we observed when agents were finding information and processing information, communicating with other agents, making trades off of the information that they had and other sorts of general behaviors.

Speaker 3

48:23 So we see it as sort of like this top level holy grail objective that also teaches us a lot about agent behavior along the way on post train Bench. We have a pretty exciting successor coming out soon, so definitely stay tuned. But yeah, I think we're generally interested in sort of evaluating what these end goals and what these end metrics might be that serve as indicators for if agents are becoming superhuman at various tasks, one of them being forecasting.

Speaker 2

48:53 Let's review the post train bench results that we've had to date and you can foreshadow to the degree that you're willing and able the stuff that's coming next. But I found the what's already out in the public on agent's ability to post train smaller models to be pretty meaningful already.

Speaker 3

49:15 Yeah, and they've only gotten better. I think our updated version is going to come out with some exciting updates just to show just how capable current frontier models have really gotten that post training. Whether that be with patching some of the issues that we found with the original post train bench version 1 to 1.2 and coming out with additional new exciting arenas that can prove just how capable autonomous agents are, even when compared to research labs and other teams that have been set up to train models with human input and human interaction.

Speaker 3

49:50 So we're very excited to put it out pretty soon. But let's just say the results are extremely exciting.

Speaker 2

49:57 Are they exciting or are they scary or are they both?

Speaker 3

50:01 I think it depends on if you're a big believer in post training open source models. If you're a big believer in fast takeoffs or slow takeoffs. I think the implications are definitely pretty profound. This is something that we've been thinking about in discussing with the release as well.

Speaker 1

50:21 One thing that strikes me is that in this last week or so we've had this direct model weight writing.

Speaker 3

50:30 Yes.

Speaker 1

50:31 One wonders whether there is a relationship between models which are good at post training, other models and models which are good at writing weights directly.

Speaker 3

50:44 Yeah, I think these are definitely adjacent and similar to each other. We're definitely planning on looking into this ourselves, but I think it seems pretty reasonable given the discourse around the paper from last week. From the. From. Given the paper from last week that these results probably extend to other areas. I mean, I think we saw the Harvey guys sort of do a similar replication experiment where they had Harvey essentially or they had a Frontier model essentially edit a point.

Speaker 3

51:19 I

Speaker 1

51:19 think it was Ligora, by the way. Yeah. Don't, don't confuse the two. It's

Speaker 3

51:28 but we've seen interesting replication so far. There have been some interesting replications by folks outside of the OASIS lab. So it seems pretty indicative that this is a new emergent behavior. I believe some folks back tested this with older model releases like GPT 5, 6 SOL and like and didn't really see similar behaviors there. So it's definitely something that is interesting and seems pretty indicative of more capable models are likely just going to going to be better at both things like post training, but also having a general understanding and ability to edit the weights directly.

Speaker 2

52:03 So how do you think this relates to the sort of meter anthropic report yesterday that says that we do have this significant acceleration in capabilities advances and you know, we were just talking before you joined about like what the world looks like over the next couple quarters as we maybe start to see a lot of the low hanging fruit that we believe probably still exists in ML stacks and pipelines and processes.

Speaker 2

52:37 Just given how much low hanging fruit clearly there has been left available in math for example and cybersecurity. Like it seems by analogy, very likely that there's an awful lot of low hanging fruit roots still in the ML domain too. What, what is your expectation as somebody who's like spent serious time looking at what models can do to train other models? Like can you put a little more color or sort of, you know, nuance onto your expectations for the next couple of quarters?

Speaker 3

53:09 Yeah, I mean I think one of the general trends that we've been seeing both with the report from yesterday and with Earth earlier reports, whether this was anthropic's measurement of how well individual researchers have been uplifted by AI, whether looking at metrics like lines of code written experiments done is the general trend we're kind of seeing is that I mean today, and this has already been true for many months, researchers have essentially agents that can do the vast majority of the actual implementation work, which is what we kind of consider as like research engineering.

Speaker 3

53:43 So the realistic limitations that researchers today have is not necessarily the like ability to execute on any of their ideas, but as like many people have talked about. Right. There's sort of research taste in finding what problems and experiments that they want to do as well as just the like speed at which they're able to come up with these. Right. Given that agents are already autonomously building and executing on a lot of the experiments that they they want to run.

Speaker 3

54:08 So I think regardless of whether we get to recursive improvement, we have definitely already seen a huge Uplift that will continue to only get better. And then I think it really just becomes the sort of end bottleneck just being resources and quality of ideas and intuition. Right. Which is of course the next hill that everyone is trying to climb. I think in general we have definitely seen this broad saturation as a conclusion from post train bench and its successor.

Speaker 3

54:38 That is especially when it comes to already economically useful and already productive work. As we continue sort of climbing up these hills, these things definitely get easier and easier to do by many margins. So this is one of our sort of overall takeaways that we're getting from this.

Speaker 2

54:58 Have you seen anything that has been sort of a proto move 37 on the post training front? You know, anything that. I mean, I take your point that like we're people have great ideas now they're able to execute more and more of them. But the real. Well, there's a lot of thresholds that matter, I should say to be clear. But like one real threshold that seems to matter is when the models start to have similar caliber of ideas.

Speaker 2

55:25 Have you seen anything that you would say starts to look like the distribution of idea quality is overlapping between the models and truly top researchers?

Speaker 3

55:37 I don't think so necessarily, but I do think we have observed that with better models that currently exist right now, for example, they exhibit a larger willingness to fan out, do experiments, a better both ability to think of like what might be interesting experiments and execute on those experiments. We definitely do see with worse models more of a general linear approach to whether it's post training or other things. And then with more and more capable models that we observe and test, we do see this general behavior of oh, it's spending its first few hours trying to fan out, do a bunch of different experiments, allocating its resources in a pretty intuitive way and doing what we might be thinking of.

Speaker 3

56:22 None of this is necessarily a move 37 moment because a lot of this is just the agent thinking, oh, let me try an initial SFT batch and see how that affects the results, for example. So we don't necessarily see anything that would be indicative of superhuman taste or anything else, but we definitely do see that more capable models are just better at long horizon work and better at the sort of autonomy that we would expect we would want want with better researchers.

Speaker 1

56:51 Let's talk a little bit about contamination. I believe you saw quite a difference between Kimi and Fable on post train bench.

Speaker 3

57:06 Yeah, this is definitely something that we have been thinking about and one of the open ended questions that we saw with postring bench was models ability. Because models had the ability to access the Internet, it ended up both leading to data contamination and a lot of other generally bad results. This is something we've definitely been thinking about and thinking about workarounds to its successor, which essentially fixes practically all of the problems that we could observe. We did a bunch of red teaming on the upcoming successor.

Speaker 3

57:38 We're very excited to see some of these gaps in open ended known problems be resolved with the successor.

Speaker 1

57:47 I saw yesterday there was a terminal bench study in which they found QE 3.1 while solving instructions likely distilled pitfalls from previous agents. In the sense that on specific questions it kind of recognized that these questions were part of terminal bench and it inserted its own kind of instruction set. Rather than following the rules like how does that actually work?

Speaker 1

58:17 Do you think these models are specifically designed just to get through the benchmarks like benchmark to that extent or do you think it happens naturally?

Speaker 3

58:29 Right. Well, I mean, I think one trend we're definitely seeing is this move to private evaluation suites so that in the future, for example, I mean Terminal Bench 1.0 is fully out in the open. So whether this is the data is being intentionally included or just the fact that so much data on the Internet has likely incorporated terminal bench in some form or capacity at this point, it's definitely true that a lot of older benchmarks and evals are being contaminated, but this is more broadly why the overall industry as a whole is moving to sort of held out and private test sets.

Speaker 3

59:03 This isn't really something that we would expect to see with much newer benchmarks, a lot of which, for example, like Deep Suite and others actively tout the fact that they have a much more comprehensive set of private evals for folks to run and test on.

Speaker 2

59:19 So I'd be interested to hear how all of this relates to what Thoughtful is doing as a business. I recently referred a friend who will go unnamed to Thoughtful and they've become a. I don't know if you would say customer or client, but tell me what it is that you guys do for companies. And then I have some, some questions about how this relates to like what businesses can do, if anything, to defend their niche against the, you know, continual outflanking that we see happening from the frontier model companies.

Speaker 3

60:00 Yeah, I think there definitely are open ended questions, whether with open source that persisted for the past many months or so and newer open ended questions that have emerged in the past few months about things like data sovereignty, especially as we've gotten to see some of the More Frontier results folks asking questions about what part of their data that the labs are using to train on. Right. This is all happening as there's a broader trend, whether with the folks at the Ramp Economic Index talking about broader trends in cutting costs on token spend while increasing their net usage of tokens by a fairly significant amount.

Speaker 3

60:38 Karina would definitely be the best person to ask about the business. But Thoughtful's general approach and sort of vision as a whole is to think about the areas in which economically useful and productive work can be done that scales and goes well with Frontier model capabilities, while providing immense value to folks and enterprises that may have overall concerns about like data sovereignty, other sorts of questions as well. The objective is essentially to find things that are important that improve as we get better RSI progress and better faster progress as well.

Speaker 2

61:15 So she tweeted on November 3, quote, Every company should turn itself into a giant RL environment, building self improving AI that learns from every interaction and progressively starts running the company itself. Unpack that. Maybe however you would, I think like procedurally, what does that look like is interesting. What models are we talking about here? Is the idea that, you know, you're going to start with an open source model and kind of create your own specialist model.

Speaker 2

61:46 And again, I'm always kind of like, oh, can this really keep pace? I'm sure you saw the Pliny the Elder, Pliny the Prompter, I should say, tweet from yesterday where he says basically my Spidey sense says that the Frontier companies are right now reverse engineering every product experience that they can get access to. And by reverse engineering that and creating, you know, all the, the software that underlies it, they're then creating all this additional training data and they're training their next generation of models on it.

Speaker 2

62:20 And so basically, even though you never gave them your product, your source code or even permission to come use their app, the theory here, which I do think is fairly credible or at least realistic if not credible. I think he's speculating there as far as I know, but it seems like reasonable speculation is that the next generation of models are going to know how to build your product. You know, they're going to have a level of familiarity with it that potentially rivals your employees, right?

Speaker 2

62:48 You got a big enterprise, you've put years of work into this, many, many men and women years into a product over time. No individual, you know, has probably command of, let's say Salesforce, right? At the grand scale that Salesforce has kind of accumulated. But a model can potentially go in reverse engineer that essentially recreate a Salesforce clone, train on it, and the next thing you know, your, you know, next generation model can sort of spit out a Salesforce, apply that to like a zillion different things and it's like, how does one defend themselves against that?

Speaker 2

63:22 So I guess you could challenge that premise or you could, you know, maybe try to sketch out how a business can, can maintain some sort of moat or defensibility, if that indeed is where we're headed.

Speaker 3

63:35 Yeah, I mean, maybe let's start with the SAS debate first and what maybe the end goal of SaaS might end up shaping out to be. I think one of the potential issues with this frame is just going to be the fact that if Frontier models are capable at reverse engineering, whether it's Salesforce, whether it's their or like their CRM or they want like for lawyers something to track their billables and better software that is maybe accustomed to what their lawyers specifically have needs for.

Speaker 3

64:02 For example, at the end of the day, if this is true, it's not necessarily true that the value will accrue to the Frontier labs. I mean, as we've seen on Twitter extensively over the past month or so, there have been folks decompiling, recompiling games, there have been folks rebuilding Photoshop from scratch and a whole host of other things just with current Frontier model capabilities. So it seems more like this is an argument in favor of the individual or if not in favor of the individual, in favor of enterprises who either use this as leverage when negotiating enterprise contracts, of course, or to sort of just build their own in house solutions.

Speaker 3

64:41 It seems that this is already true right now and will only become more true as the initial lift that's needed to replicate some of this software becomes less and less. And then of course there's folks that exist purely off of a distributional advantage. Right. No one's going to use a Slack clone because then there's no way to invite external collaborators to your Slack clone in a way that's easily usable by the external party that you're trying to invite, for example.

Speaker 3

65:07 Right. So I think general modes of distribution and users still exist and it's not necessarily true that the end state of software is one in which everything is just sort of built and served to you. I think if we get to a state, which we will, where anything can be built and reconstructed and improved upon for any individual user, then it's much more likely that it will be more individualistic as opposed to folks are still going to be paying thousands of dollars for licensing of various software.

Speaker 3

65:36 If we get back to your original question, I think there's definitely the theme of 2026 has definitely been surrounded around open source data sovereignty and a lot of other sort of overall questions. I think Satya Nadella was on that podcast where he spoke at length about his belief of what enterprises should be doing when it comes to securing their own data and their own model sovereignty. And it's definitely broadly true that for things like spreadsheet editing, you don't necessarily need superintelligence to fill in your Q3 books and do all of that for you.

Speaker 3

66:09 And we've definitely seen this whether it comes to the folks at Ramp Labs post training their own model for spreadsheets, editing or any of the other sort of like vertical specific companies and labs that have been trying their hand at this as well. So I think in terms of this, this is more broadly what Korina means when you're referring to the original tweet.

Speaker 2

66:31 To what degree are AIs helping to run Thoughtful?

Speaker 3

66:39 definitely used a lot for engineering and I think in terms of anything else, I mean, it's definitely an open question of like when they become good enough and trusted enough for handing over the keys for other things as well. So we're definitely very AI built.

Speaker 1

66:57 How is Thoughtful turning into an RL environment.

Speaker 3

67:06 Right now? Just collecting all of our own data and collecting all of our own metrics. So.

Speaker 1

67:12 Right. So it's in. It's basically in the preparation to become an RL environment.

Speaker 3

67:16 Yeah,

Speaker 1

67:18 indeed.

Speaker 2

67:19 Do you think so I take your point on you don't need superintelligence to edit a spreadsheet. But I do find like my revealed preference is I'm using all CLAUDE and GPT models in my personal stack these days. And there's a mix of reasons for that. One is the trust factor actually has become one of the biggest ones.

Speaker 2

67:52 You know, even when a new model pops up and it has like similar benchmark scores, I'm like, well, similar benchmark scores are not enough to get me to switch when I don't know, you know, how liable these things are to be jailbroken or prompt, injected or whatever. You know, I know that the leaders have like put a lot into safety and reliability and obviously they're far from perfect. But like, I'm taking whatever chances I'm taking there.

Speaker 2

68:17 I don't want to 10x the chances I'm taking by switching over to something that is not nearly as battle tested. So that's like one big reason. Another reason is kind of what you described with the muse behavior at the top where it's like eh, you know, I don't know, the benchmark scores are kind of the same but like. And yeah, you don't need superintelligence to edit a spreadsheet but like I do want the spreadsheet to get like fully edited in the way that I want it to.

Speaker 2

68:42 And it seems like these other models just don't quite get their ramp, you know, if they post train on their own stuff like they can get there. But I guess I'm just wondering like it doesn't seem like we are yet at the point where it has become easy enough to take an open source model and post train it on your own environment. That has become aside from a very few companies that are kind of doing this as R and D because they want to develop the capability, a path worth taking.

Speaker 2

69:17 Do you have line of sight to that changing? Like can you see a world in Q1 of next year or Q2 of next year or sometime where you're like, I think we will be at the point where on a total cost of ownership basis it's going to make sense for a ton of companies to take an open source model, do our thoughtful paradigm of largely agentic post training and get to a win state where they are just on an operationally again total cost of ownership, day to day execution basis better off that route versus just using Claude like I'm doing right now.

Speaker 3

69:56 Yeah, I mean I definitely think in terms of understanding intent, there's a lot of room for small models to improve right now. Right. I mean even with the most frontier models that we have today, there's still more room to improve in terms of just understanding the second order and third order instructions that you might be thinking in your head when you give any prompts to a model. Definitely true that smaller models are still poor at this, but I think this is also just something that enterprises are considering as things like for example, when it comes to things like customer service, you don't necessarily need superintelligence there either.

Speaker 3

70:33 And yes, there's definitely like open concerns about like whether it is jailbreaking or other similar vulnerabilities. But these things are broadly things I think will continue to improve. Of course. Right. Everyone is sort of on their own path or like point in the exponential whether they are at the frontier or whether they are sort of like a bit behind and sort of catching up as well. So I think this is definitely true right now. Definitely don't want to put like a prescriptive forecast of by End month, X percent of token spend is going to be going to the frontier and X percent is going to be going to more open source folks or other folks as well.

Speaker 3

71:11 But I think this is broadly general trends that we should just expect both models that are behind to improve and both models are in front to improve as well.

Speaker 2

71:24 That's not quite the time's up, Bell, but it's getting close. What have we not asked about that's kind of top of mind for you these days? And that could be things you're doing at thoughtful or just other things that you think are, that you're watching closely and maybe feel are underappreciated by other close watchers.

Speaker 3

71:44 Yeah, I mean, I definitely think the scope of the AI revolution outside of the AI space has definitely been extremely interesting to sort of observe. There's been articles and coverage about folks talking about how architects are seeing what work that would have taken weeks or months reduced to in little under a few minutes and a few turns of prompting. This kind of stuff is extremely exciting, even if it's sort of like outside of the AI realm for a bit.

Speaker 3

72:14 I mean, even in terms of the software explosion that we were sort of talking about earlier. Right. I'm an amateur photographer and what we're seeing in photography is people are taking what would have been previously extremely rigid software, whether it's like software to control like a DLSS mirrorless camera or other things. Sorry, to control like a Sony mirrorless camera or other things, for example, and people are modding the software, they're doing extremely interesting things like adding inbuilt filters to their cameras and a bunch of other stuff that simply wouldn't have really been possible before.

Speaker 3

72:50 Right. So definitely extremely excited about just the overall personal computing revolution that's happening right now.

Speaker 1

72:59 So I have to ask, do you think there should be safety issues around that? Right. I often say this, like you don't really control the bits on your devices. Right. For example, in the photography example in the early 2000s, Sony came up with a camera which had infrared and which could basically like strip clothing off the human body, you know, because you could see, you know, heat waves.

Speaker 1

73:31 And that became an issue and Sony had to cancel the camera. They pulled the camera and since that time we actually, you know, you haven't seen the same capabilities appear in any device, even though the latent capabilities are probably there in the, in the, the processors. So what do you think people have rights to? Should people have rights to do that or should the AIs, which are interpersoned between the person and the device actually say no.

Speaker 3

74:00 Yeah, we definitely need new and updated laws to think about a lot of the things that are happening. Right? I mean, even today, when we think about how you log in and authenticate to various different things, things on your computer, every single sign in you've ever had on Google, Chrome, Safari, Firefox, everything else are just stored as cookies that exist as some file on your computer. And one of the things we can already see with agents today is they're more than willing to just go into your Chrome cookies and pull your login info.

Speaker 3

74:32 And of course, right now they'll ask you, hey, I want to do XYZ thing. Can I pull your authenticity token or can I pull your cookie to use your browser, et cetera? Even with computer use as a whole, it opens a huge line of possibilities that simply weren't possible without good computer use. Prior by definition, computer use can get past any capture we can currently throw at it, whether it's reading the text and typing in some garbled text as well, or clicking on the right order of images and everything else.

Speaker 3

75:04 So there are 100% things that need to be both rethought of whether this is just from a security standpoint or in terms of even as you're saying like regulations of. It's probably true that we want consumer protection. Sorry, it's definitely true that we want things like consumer protection and other laws to be updated to sort of account for all of these new surface areas that we really haven't thought about yet. I think one of the big challenges here is just going to be that latent model capabilities will continue to unlock so many new surface areas and possible vectors for both good and bad things to happen that we will really need to be on top of this in the future.

Speaker 1

75:44 Indeed. Brian, thank you. Thank you for joining us today.

Speaker 3

75:49 Yeah, thanks for having me.

Speaker 2

75:51 Great to meet you. We'll be looking out for the next version of Post Train Bench and you've got me wondering what the agents will say about how much acceleration we're going to see in AI over the next couple of quarters. If they can predict the election as well as the experts, we should probably start asking them about the future of AI itself.

Speaker 3

76:12 Yeah, thanks for having me.

Speaker 2

76:14 Wild times. Great to meet you. Thanks for being here.

Speaker 1

76:16 Bye. Bye,

Speaker 3

76:17 take care. Bye.

Speaker 1

76:22 Amazing. And our next guest is already in the green room. So. Our guest for this morning is Brian Bischoff. Brian is the head of AI at Theory Ventures, an early stage venture capital firm working with Thomas Tungus. His Work includes technical research and building internal software that helps the firm do its work.

Speaker 1

76:53 He also organizes experiments that put AI tools against the kind of data questions businesses actually ask. Before theory, Brian led AI at Hex, where his team built an assistant for data science and analytics. Earlier he started the data team at Blue Bottle Coffee, worked on machine learning projects at Stitch Fix, and built data teams at Weights and Biases. At Blue Bottle, his work included demand forecasting and helping people find coffee they might like.

Speaker 1

77:23 Brian earned his doctorate in pure mathematics at Kansas State University. He teaches graduate level data science and analytics at Rutgers and co authored Building Recommendation Systems in Python and JAX with Hector Yee. His work now also includes a document format developed with Adam Conway to keep the reasons behind a sentence attached when people and AI systems revise it. Brian, welcome to the show.

Speaker 4

77:55 Thanks a lot. Back to back, Brian's this morning.

Speaker 1

78:01 It's Brian's day today.

Speaker 4

78:02 Yeah, I have the preferred spelling but you know, I don't hold it against him.

Speaker 1

78:09 Brian, tell us, tell us a little bit about Theory Ventures. Tell us a little bit about your work there.

Speaker 4

78:15 Totally. As you mentioned, we're an early stage venture fund. We invest in companies that are sort of like taking advantage of technological discontinuities as we sometimes say. We're a heavily thesis backed firm. What that means is that we do a lot of research, we do a lot of investigation into the market, we deeply understand what the technology can do and then we go and we find the companies that we think are most likely to succeed with that technology.

Speaker 4

78:41 We're a very concentrated portfolio, which means that we have far fewer companies in our portfolio than other firms of our aum. And the reason for that is because we're also very white glove and hands on, which is to say that we spend lots of time with our portfolio companies trying to distort reality.

Speaker 1

79:01 Indeed. Just to give us a sense of size here. Do you have a rough bracket for the AUM and how many deals a year do you guys do?

Speaker 4

79:09 Yeah, we're at about 750 million AUM. We're in our second fund. We usually do between like 10 to 14 deals per fund. And again we're in our second fund and so yeah, that means our portfolio is somewhere in the 30s.

Speaker 1

79:31 Indeed. And what is the thesis right now?

Speaker 4

79:36 Yeah, I mean lots of, lots of exciting theses. One thing to keep in mind is you can think of it like if you're doing academic research, you've got a lot of ideas that you're sort of like, you know, simultaneously entertaining and considering and then some of those ideas graduate to more serious research and some of those more serious research endeavors graduate all the way to running experiments or sort of papers. And it's the same with us.

Speaker 4

80:05 We've got lots of things that we're looking into, investigating and trying to understand. And some of those are graduating to bets, as you could call them. A couple recent bets, for example, we invested in Ollama. Ollama is sort of a really great application stack for running local models and now hybrid models. So that's both local and cloud inference. We've also recently made a bet in Sail Research. Sail research is a batch inference play.

Speaker 4

80:36 And so one thing that I learned when I was building AI agents at Hex is that there's a lot of inference that you want right now and there's some inference that you want soon, but there's actually so much inference that you want whenever it's available. And Sail Research is making that a first party solution and really prioritizing the reality where inference and intelligence is extremely widespread demand.

Speaker 4

81:06 But not all of it needs to happen right now. And so those are two recent bets that we've made. Both are really exciting opportunities. Earlier in the year we also invested in backups. And I can go on. We invested in a company called Artemis. But yeah, so those are some recent bets.

Speaker 1

81:24 And so, and so it sounds as though one of the thesis is lowest cost of inference at whatever service level agreement that is required.

Speaker 4

81:32 Yeah, and not simply always how cheap can I get it? But more of like, let's be thoughtful about the cost versus intelligence curve. Do I always need to spend the most money? It's kind of like, you know, when you're thinking about putting gas in your car, do you always put the most expensive gas in your car? Well, it depends, like what goes into that decision? Is it like a car that you're sort of like nursing? Is it a car that has better performance under great octane, or is it a car that doesn't really care?

Speaker 4

82:01 Some cars don't. And that's the same is true for inference. Not all jobs require right now. And candidly, not all jobs require frontier intelligence either. And so being thoughtful about that trade off, you can get other benefits and not just money. Other, other benefits can include sort of like availability. Other benefits can include more specific intelligence and so on. And so we're very excited about the entire Pareto curve of cost, latency and intelligence.

Speaker 1

82:31 It also sounds as though these are more kind of B2B back end kind of optimization kind of solutions. Although Lama, I know is also a front end. It's also on devices. Are there any which are more application like? Are you looking at any user centric consumer or enterprise applications?

Speaker 4

82:51 Yeah, it's a good question. We definitely think of Ollama as both like B2C and B2B. Obviously Ollama is extremely beloved amongst individual developers just for personal use or for that hybrid use. But you know, great traction in the enterprise. And so it's not obvious to me that we've made many bets in the B2C category. I think our key theses at Theory Ventures are around like B2B products mostly focused on data and how data can kind of lead you to really amazing breakthroughs in technology.

Speaker 4

83:33 And, and so even though we invest in quite a lot of like AI application layer companies, if you look like hard enough, you'll notice that data is actually the secret of many of those investments. And so what have they realized with respect to how data can power the future?

Speaker 1

83:52 It sounds as though you have an antithesis for B2C.

Speaker 4

83:55 I wouldn't say that that's an antithesis. It's more that we like. Our specialty is B2B. And you know, when you're making concentrated bets in a relatively small portfolio, obviously working from expertise is very valuable. That's another thing about our firm that's a little bit unique is everyone in the firm is technical. Everyone in the firm is very like embedded using these products and have a long experience with the space.

Speaker 4

84:25 So yeah, less we don't love it more kind of, yeah, we're playing to our strengths. And yeah,

Speaker 2

84:35 so I have several different directions I want to go with you. One is I'd just be interested to go down the list a little bit longer in terms of different theses that you've decided to make bets on. Also very interested to hear more about how the way in which you guys are operating the firm is changing. We were just talking to Brian with an eye about how well agents are able to do now at predicting election outcomes.

Speaker 2

84:59 So I'm sure there's interesting uses for them, predicting, you know, what markets are going to get big and so on. I even have some questions about your coffee research. But maybe as I kind of map all that out on the first one that you mentioned about like spare capacity or you know, getting inference, you know, when it's available as opposed to right now. What did you learn about how much slack there is in the compute market?

Speaker 2

85:21 Like, I think we've got a lot of kind of conflicting signals from different people. We've talked to where on the one hand you people like, oh, you can't get GPUs. On the other hand, you know, if you're willing to pay a multiple, you can get GPUs on demand right now from a hyperscaler. So how much is like going to waste or like what are the metrics that you guys really focused on and what did you learn about how much capacity right now is kind of sitting there waiting for the taking opportunistically?

Speaker 4

85:48 Totally. I think there's a couple things that work here. One, if you go to a Neo cloud and you say you got compute, they're like, nope, it's all locked up. It's all contracted out for the next two years and they're not lying. Like, they're not telling you an untruth. But there's also been quite a lot of discussion and even, you know, Satya talked about how there's a lot of compute that is locked up. As you hear, this is essentially underutilized compute due to either like the geolocation of it or the available like energy.

Speaker 4

86:21 And so these are both true at the same time where there is locked up compute that actually isn't super useful. There's also the reality of sort of utilization, utilization of gpu. Yesterday I was chatting with some folks working at the infrastructure level around like GPU utilization. And one of the big problems is you actually can't run at maximum capacity all the time because it's unsafe. From a scheduling perspective, you actually want to build in slop.

Speaker 4

86:51 And so there's actually so many more dimensions than simply do you have contracts on the available compute. And this is why you see many different routes to big success and embarrassingly large numbers of ARR in the compute market. This isn't just because of fomo. This is because there's actually so many profiles that you need to kind of like maximize the value of whatever compute is available to you. When you look at a company like Sail, one of the things that's really exciting is when they're a lot more heterogeneous about workflows.

Speaker 4

87:25 They can be a lot more heterogeneous about like product and they can offer it to many different application spaces that you that need a different profile. While. And so we run, you know, thousands of jobs here at Theory Ventures that are powered by AI agents. If you ask me, what's the distribution of immediacy for those jobs? Well, I'll tell you, okay, maybe 5 to 10% is right now and maybe 40 to 60% is.

Speaker 4

87:58 I'd like to see it before my next meeting. And the rest is, I don't know, maybe like this week would be neat like, you know, and that even just in like this one sort of niche of the kind of work that we're doing, we already have quite a broad variance of inference needs. And that's true of, I would say, almost every AI application layer company that I've ever spoken to since 2023. And so from that alone, you can see that if someone tells you there's simply no GPUs to be had, well, that's not really like accurate.

Speaker 4

88:36 But also they're speaking from a particular bias. And the bias they're speaking from is probably for the workloads that is under demand right now for me. And so I think that's a good thing to delineate. When people say I just can't get the GPUs I need. Well, ask them, what are you doing? Are you training a trillion scale, like model, like trillion parameter model? If so, okay, like yes, you may need, you know, a thousand b, three hundreds that are always available to your researchers to schedule jobs right now.

Speaker 4

89:08 Okay, yeah, like I can believe that that's a little bit tricky. But if you're, oh, I need to spin off an agent on some like, you know, relatively small node to do a little bit of summarization of some text, there's something out there for you, I promise. And I will say my friends will get mad if I don't mention EC2 is basically always overbooked right now. And so that hype is real. It is basically easy to, is constantly unavailable.

Speaker 2

89:39 So do you have a single number or a couple numbers that you would use to characterize? Because that was largely a demand side description of like why it is acceptable to the customer or you know, they would be happy to trade savings for, you know, some time delay. On the supply side, do you have a estimate for like how many flops, you know, what percentage of flops are like going unspent second by second as we go through life?

Speaker 4

90:11 Yeah, that's certainly not my expertise. I think there have been some tweets put out by, by Satya and then recently there was a great article by Rohan about Rohan and Neil from Core Automation talking a little bit about sort of like how to make more use of these flops. And so I would check that out because he's talking about islands of compute and how they're scheduling training runs across continents. And so I think there's some, some, some great insights there.

Speaker 4

90:41 But yeah,

Speaker 2

90:42 nice. We'll put a pointer in the show notes. How about other theses you. You mentioned a couple, you know, very briefly, but I think it's worth. We've got a little time. Let's go deeper down the roster.

Speaker 4

90:55 Yeah, yeah,

Speaker 2

90:55 theses

Speaker 4

90:56 you guys are working. And I'll use my usual caveat, which is not an investor. So because remember, my job is to build a build software. I think some other theses that we've been spending a lot of time on is sort of like what does the infrastructure stack look like for training up these new models? And you know, yesterday I spent with the Lance DB team and talking about sort of how the lakehouse is extremely valuable for this new age of AI application layer.

Speaker 4

91:29 And not just AI application layer, but also foundation models. So if you're coming back to these wasted flops, the number one cause of wasted flops in our economy is actually GPU underutilization and GPU underutilization for training of trillion scale parameter models. That is an incredibly vast workflow or workload that is very difficult to manage. But when you have custom tools like Lance DB that allow you to read the data into the sort of like training stack at the highest rate, you are able to utilize those GPUs much more.

Speaker 4

92:12 We saw examples yesterday of people going from 30% sort of wasted flops down to less than a percent just by loading the data faster. And that's been something that's been a really immersive, emergent, I would say boon of the multimodal lakehouse. And so just saw, you know, talk after talk of not just like this is valuable for the application layer, but this is incredibly value for the training stage. And you know, pre training, mid training, post training, we saw Runway give a talk that like every single stage of their training is going through Lance DB and this is why they just can't afford to waste those GPU flops.

Speaker 4

92:49 As we talked about.

Speaker 1

92:52 Let me, let me transition a little bit here to. You guys did a builder survey, I believe.

Speaker 4

92:57 Yeah.

Speaker 1

92:58 And you did a builder survey. You had some hypothesis that you registered before and then you did the survey and then you found some results that were surprising. Could you, could

Speaker 4

93:06 you talk

Speaker 1

93:07 through that?

Speaker 4

93:08 Yeah. Okay. So one surprising finding was that I went in thinking we're going to be all up and to the right on open source utilization. Like that was my thesis going in. I spoke to so many people and candidly, all of my group chats are constantly talking about using open source models more. And so, okay, I go into the survey and we get back this relatively surprising finding that there's a shift and the shift is not from like we're going from open source usage to no open source usage, but it's about dominance.

Speaker 4

93:47 Last year in 2025 we asked almost the same question. We reworded a little bit for clarity, but the spirit of the question is the same. And what we saw in these new results is that enterprise AI application layer companies are actually pivoting a little bit away from a big focus on making open source open source models. To be clear, open source models power their live in production inference for users. And that went down. It, it went down in a substantial statistical way.

Speaker 4

94:19 And so that was eye opening. I dug into the data a little bit more and I feel like I have a little bit more clear picture now of what that is telling us. One thing that if you DM your friends and you say, hey, what's going on with your open source inference? What they'll tell you is they'll tell you, well, I've moved a fair bit to Luna. This Luna will come up naturally in a bunch of your conversations.

Speaker 4

94:45 And the reason, and we've written about this a little bit on Theory Ventures blog, is that Luna is just underpriced for the intelligence level and the speed. It is a surprisingly competitively priced model. It feels like it is off the Pareto chart with respect to pricing in particular. What that means is if you were using open source models for one stage of your stack at that intelligence level, if you were using a GLM 5.3 or something like that, pivoting it over to Luna is actually a pretty good bet from a couple different dimensions.

Speaker 4

95:20 If you're already on that sort of OpenAI inference stack, that can be a great model for you to break that Pareto chart. The other thing that I learned is right now there is a lot of I would say acceleration in model drops. Right now tying your production service to open source models requires a little bit more thrash in the engineering.

Speaker 4

95:50 And so I heard from a couple of folks specifically, yes, but the open source models change so fast that we're actually like it's a little bit more convenient to pin to an OpenAI model. It's not a anathema or you know, it doesn't solve everything. But I would say this is one reason I heard. So that was a big surprising finding of the survey. The other finding was around sort of like the shift in way people are thinking about security and tokens.

Speaker 4

96:23 Just I was surprised by how much shift there is in spend on tokens for security it's basically like, you know, the doors are wide open, like this is where, where revenue is starting to pour and there's. Yeah, I would say there is a lot more spend being opened up for application security in the AI space right now to kind of react to the, the realities of today.

Speaker 1

96:52 Just, just a note. Yeah. All our headlines run on Luna. So a good, good portion of our inference stack. And again I went through the same process, I went through the same process of looking, of evaluating, evaluating a bunch of. We had, we had GLM 5.3 before that too. So we did draw the GLM stack for a while and now it does run on Luna. And precisely because we run a lot of our, the rest of our stack is on OpenAI as well.

Speaker 4

97:18 Many such cases

Speaker 1

97:21 there's a bit of fable for the most high taste stuff. But our headlines are Luna headlines. So that's the automated headlining that's happening.

Speaker 4

97:29 Yeah. And I don't want to give the impression that GLM 5.3 is not a breakout model. And it is an incredible, incredible model. We still use it in a lot of our pipelines, like our AI pipelines. I would say GLM is, I think it's our like pinned model across the board. And so we love GLM 5.3. There's no, no, you know, no undermining there. But Luna is a very competitive model

Speaker 1

97:59 indeed. Do you think the spend into cyber has come as additional or has it been taken out of the existing vendor? Cyber vendor stack?

Speaker 4

98:12 Great question. It's very much a yes. And I think, I do believe that anyone with a cyber spend today is starting to reassess like where that cyber spend needs to go, like what percentage of it needs to be specifically and like AI powered cyber capabilities. I think it would feel a little bit bizarre to like continue thinking that like all of your cyberspend should go into classic platforms like Spelunk or something.

Speaker 4

98:42 I think it would be odd to not be reassessing your options right now. But when talking to my friends who are security professionals, you know, specifically like CISOs, they're saying like, yes, we are able right now to ask for more, more budget specifically because of the increase in threat. And so I do think the pie is getting bigger and I do think there's been a reallocation of previous cyber budget.

Speaker 1

99:10 It's hilarious because Jason Clinton at Anthropic specifically tweeted out cisos, it's time to go ask for more budget. So

Speaker 4

99:16 I don't, I think people already are at least like my CISO friends Like, I'm asking them like, what's going on? And they're like, I'm able to get more budget this week. One like, I think his response to my text was literally, it makes the budget conversation a little bit easier. And this was in response to one of the leaks from like two weeks ago. Not the Binance one. I forget which leak it was, but it was one of the.

Speaker 4

99:41 Oh, it was all

Speaker 2

99:42 blurring together.

Speaker 4

99:44 Yeah. That's the unfortunate reality, Nathan, is the fact that I can't remember which of the like, world changing security 30 incidents I'm referring to in the last month. That's the, that's the sad part.

Speaker 1

99:56 Well, I, I just found out yesterday. 257 characters. You can root any Alpine. Alpine Linux distribution. So.

Speaker 4

100:02 Yeah.

Speaker 1

100:03 Wow. I. I wish I'd known that, you know, I would. I didn't even remember all those passwords all those years.

Speaker 4

100:08 Yeah, yeah. It was so easy. Actually, my favorite meme about this was it turns out that 4 works for this too. You just have to spam 4. And I was like, no,

Speaker 2

100:19 maybe that 42 is not the answer to everything, but simply,

Speaker 4

100:24 yeah.

Speaker 2

100:27 Any other surprises from the survey or theses that you guys are betting on that we should talk about or, you know, next I was going to transition to how the operations of the firm itself are changing.

Speaker 4

100:39 Sure, yeah, we can move on to. Yeah, the changing frontier.

Speaker 2

100:44 So, you know, I mean, there's a lot of ways to ask this question. We sometimes ask, how does your token budget compare to your headcount budget? But, you know, what's your frame for thinking about transformation and where are you in the process?

Speaker 4

100:59 Totally. So the first thing I'll mention is again, we're a little bit of an interesting firm. Everyone is technical and that's all the way from sort of our talent partner to our GTM partner through to the investors. Everyone is using AI. Everyone's been using AI since the day that I joined. And more and more, the way that we think about token spend is, you know, you should be thinking very, very clearly at all times.

Speaker 4

101:34 If there is a way to accelerate by using AI tools, you should be thinking about experiments that you want to run. And we're a really experimental bunch to give you a sense. Every quarter we do our little like, off site where we talk about just everything that happened over the quarter. One of the things that we talk about every single quarter is what experiment every individual did that quarter with AI. That's not an exaggeration. And this is not like, oh, I asked Claude to do Something new this quarter?

Speaker 4

102:05 No, these are. You pre register what kind of experiments you want to do related to your job that use AI and then you check if they worked. This happens every quarter because we're so insistent on rethinking what does it look like to work in a venture firm in this year, next year, and the year after, how are we adapting to those things? As the head of AI, my engineering team is obviously still operating at another level, which is how can we build tools that use AI to transform other people's workflows?

Speaker 4

102:43 Our goal is not to replace an investor with an AI system. Our goal is to make that investor 10 to 100 times more effective and efficient by using AI tooling and building the AI tooling that will make that possible. Let me give you a quick example. Everyone talks about, like company brain, no matter what kind of business you're working in. Last August and September, we started building our company brain at Theory Ventures for everybody to be able to use.

Speaker 4

103:18 That was served through mcp. And it is sort of tied right into the language model ux so that no matter where you are, whether you're on this device, that's a phone phone, or you know, you're on your computer, you have access to the firm's knowledge. The other thing that we talk about a lot is compounding interest and the idea there is the more that you pour data into a centralized system that uses AI to enrich that data, improve that data, connect that data, the more that things like a company brain can actually be useful.

Speaker 4

103:54 I've used a couple company brain products and candidly found them all to be useless. The main reason is because they are trying to build sort of what we used to think of as this internal knowledge system. You can think of notion, for example. Notion is like a stack of files and maybe you can build a good search on top of it. Is that really your ambition? My ambition's a lot higher. And so my ambition for a company brain is agents who are consistently working to connect the disparate nodes in the graph of understanding from your business.

Speaker 4

104:30 And so that's why I can easily say, yes, we don't have, you know, an agent for every human at theory. And we don't have 10 agents for every human in theory, we have thousands. And there's a reason for that is because there's a lot of work to get done. Every time that one of our investors writes a note, what does that note need to connect to? How does that note need to filter into every other piece of thing that's Going on as an investor goes through the flow of researching a company, what happens along the way?

Speaker 4

105:00 Well, historically what happens along the way is they keep notes in their little notebook and maybe they put it into some CRM type system and that's all well and good and when we get to ic, maybe they'll distill all that into a couple sentences to really, you know, get the people moving in ic. Okay, yeah, you can do that, that's just fine. But my ambition is a lot higher. My ambition is that every piece of work that an investor does during the investment process, the research process, because remember, we're very research focused.

Speaker 4

105:30 Every piece of work that they do during the research process is automatically linked, enriched and sort of expanded. This is why when people say what are you building? I say I'm building research agents.

Speaker 1

105:43 Interesting.

Speaker 2

105:44 How do you evaluate success? I mean, obviously one of the huge challenges in venture historically, and this hasn't changed yet, is it's a long time to find out if you made the right bets or not. Right? So

Speaker 4

105:57 totally.

Speaker 2

105:58 Sometimes I do wonder if including for myself, you know, I've got, I've. I periodically go do like a cron job cleanup or where I'm like there were ideas that I asked about taking you know, every bit of data exhaust and like processing them in some other way and finding connections and being like super galaxy brained about it or you know, scouting this or reporting that or whatever. And I'm like probably 2/3 of those if I'm real about it.

Speaker 2

106:28 Like I never actually look at the outputs of. And so it's easy to sort of have these ideas, spend a bunch of tokens and sort of just end up logging a bunch of tech text somewhere without really making an impact. How do you know what is actually working and what is, you know, potentially subject to leaving on the cutting room floor?

Speaker 4

106:50 Let me tell you a story about Blue Bottle. When I was at Blue Bottle, we wanted to build inventory forecasting and specifically we want to build a machine learning system that could help us predict how many pastries we were going to sell on a given day and other things like coffee and milk and so on. But pastries was the first focus. And what was our key? Like North Star, our guiding light? Well, it was actually if this is successful, it's not that we sell a lot more pastries or that we waste a lot fewer pastries.

Speaker 4

107:23 Okay, Those were, you know, two metrics that we tracked. But what would we have believed was success? What we would have believed as success is if the cafe manager who's already got about 70 jobs. If that cafe manager essentially goes into the system and when it comes time to order, they just click, okay, okay, okay, okay, okay, okay, okay. Now, we don't want them to just not think and just autopilot.

Speaker 4

107:53 We actually don't want that at all. And there's a reason, and this will surprise you, but the variance in sort of what a cafe lead understands with respect to their cafe demand is shocking. It is. It would, it would blow your hair back how much they understand about the demand in their cafe. That is not just in the numbers. We have these weirdly deep, like, you know, tacit understandings.

Speaker 4

108:23 So what was our North Star or North Star was that a cafe manager has more opportunity to spend that time with customers. Because a cafe manager is not superhuman at looking at data and saying, like, oh, I'm going to need like 25 cookies. No, a cafe manager is superhuman at understanding hospitality. Hospitality was the North Star when we were trying to build a sort of automated ordering system.

Speaker 4

108:53 And that may sound weird because you can't manage that on the dashboard. You can't look at hospitality on the dashboard. But what you can do is you can ask them periodically, how much more time do you have available to spend with customers, to spend on the floor and notice that there's a coffee spill to understand that your baristas are slammed right now. And actually staffing was off also my fault. But. And so these are the things that we always cared about.

Speaker 4

109:22 Why do I bring this up? Well, yeah, you could say, like, okay, it's like seven to 14 years to find out if this, like, you know, diligence was good. Sure. But it's a pipeline. So first of all, you want to ask yourself, how fast can you get through the research pipeline? How fast can you go from identification of a potential opportunity or market to a clear understanding to be able to make a decision? But more importantly, hospitality, what does hospitality mean in venture?

Speaker 4

109:56 It means that the investor has time to talk to human beings, to talk to potential or existing companies, to talk to founders and competitors and employees to try to understand the human element of this market and what's going on. Investors are superhuman at a few things, but I don't personally think investors are superhuman at looking at websites. I don't think they're superhuman at sort of like, you know, reading metrics off a table and saying, like, metrics are good.

Speaker 4

110:32 I don't think they're really superhuman at that, but they are superhuman at sort of understanding potential, understanding upside, understanding sort of the relationship between, like, you know, customer need willingness to buy and sort of framing what the vision in the future can look like. That's what they're superhuman for. And so when you ask me, like, how do I measure success? Well, I measure success when an investor tells me, yeah, actually, I get to spend 30% less time staring at the website and trying to figure out the important details I look for every single time.

Speaker 4

111:07 Or I get to spend 20% less time filling out these specific answers to questions which are important for diligence, and instead picking up the phone five more times and talking to five more people who have credible opinions on this deal. That's success and you don't need to wait seven years to find out.

Speaker 2

111:28 That's a great answer. I definitely want to get this coffee question in so we can maybe go back to other things in a, in a minute. But the tweet that came across my radar this morning that sort of inspired me to want to go a little deeper on this is apparently in the inkling system card or blog post or whatever that thinking machines just put out. They report that inkling too small was able to identify the type of wine presented to it in a photo more accurately than the human testers were able to identify it by taste.

Speaker 2

112:10 And part of me is like, I'm not even sure that's real. It just sounds so crazy. But then again, we're seeing a lot of crazy things, so who knows? Sure. You actually worked on a system, if I understood correctly, that tried to identify what kinds of coffee people would like. And so I'd love to hear your reflections on what sort of feature engineering you had to do to get such a system working. And like, how I'm very, I'm obsessed with this idea of new senses coming online for, for AIs, some of which will be like, just trying to match our, our senses.

Speaker 2

112:50 You know, we have very well developed, maybe not, you know, we could point to animals that maybe have even more well developed senses of like smell and taste, but we're still miles ahead of AIs on those dimensions. Then there's other things, like the intuitive sense for how a protein is going to fold where like AIs are miles ahead of us. And then there's a ton of things where like, neither one of us are very good yet.

Speaker 2

113:11 But I suspect that the AIs are going to get good. So I'd love to hear how your experience making a coffee recommendation system informs how you would think about new senses coming online for AIs.

Speaker 4

113:24 Yeah, I'll start by saying I I only saw the tweet about Inkling 2. I have not dug into it myself. I am, you know, very interested. I did see that there was something related to wine identification. I am not a venophile, but I am very, very obsessed with coffee and other sort of like experiential taste, you know, obsessions. So let me tell you a little bit about how wrong I was about coffee.

Speaker 4

113:59 I, you know, I was a machine learning engineer. I knew how to fit models in my brain. I want to predict for new customers of Blue Bottle Coffee, which coffee are they going to want. And you may or may not know. The online subscription for Blue Bottle used to have somewhere in the camp of like 5ish, different options, not a huge number, but believe it or not, and it's relatively easy to see from the data, at least it was, the churn rate was higher than we would have liked for the subscription service and that churn rate was mostly tied based on feedback to be related.

Speaker 4

114:38 I didn't like the coffee I got. Now you could say, okay, maybe they're not Blue Bottle fans. Or you could say, well, maybe we just sent them the wrong thing or they picked the wrong thing from the website. And we put a lot of time into making it very clear, like what a, you know, more sort of robust and chocolatey profile looks like versus that fruity Ethiopian. Keep in mind, this is, you know, almost a decade ago.

Speaker 4

115:05 So I would say, like, the general knowledge of like coffee and language about coffee has evolved since then. This was a while back. So we wanted to see if we could ask people simple questions and use that to identify what kind of coffee they were going to choose. You may be thinking, this sure sounds a lot like Stitch Fix. In fact, this was before I worked at Stitch Fix, but it was very much the same idea.

Speaker 4

115:31 When you go through the signup process, if you can identify just a few attributes, maybe we can give you a better first experience. And we thought that that was worth it. We trained a model, we sent out a questionnaire, and right off the bat there just wasn't a lot of variance in the responses. And so the first version of the questionnaire, which made complete and utter sense to me, sucked. And so I want to bring you back now to the what is superhuman in cafe workers?

Speaker 4

116:01 Well, if you go to a barista and you ask some of the things that make them superhuman, one of them is their ability to ask you questions and very rapidly get you the right beverage. So we did, we worked directly with baristas to say, what do you think should be in this questionnaire that is not currently. And the thing that really blew my mind was they wanted to ask things like, what kind of salad dressing do you like?

Speaker 4

116:25 And what kind of sweets do you like? And things like that. Now, those were not on my bingo card of, like, the things that we need to be asking. I think we need to be asking like, oh, how dark do you like your coffee? And stuff like that, or how do you brew your coffee? But no, somehow the baristas were like, I like to ask sort of other questions that just tell me about their taste. Well, they were right.

Speaker 4

116:51 This worked. This showed much more variance, much more accuracy. And ultimately this is what we sort of got the most success with when trying to roll out this prediction model. And so the reason I bring this up is what we learned is that sort of there are a lot of hidden indicators. And now everyone is completely, completely fluent in latent features. And so it will come as no surprise to anyone that sometimes latent features can be very explanatory, despite not being very explanatory.

Speaker 4

117:21 And so when you ask this question, like, can inkling2 identify a wine from a picture? And we say like, or, you know, our human brain thinks wine is identifiable by smell, by taste, by texture, by mouthfeel. Yeah, those things. By temperature, those things. That's true. But what are the hidden features? I mean, we know that these models are like, superhuman at finding sort of, you know, rl hacks to get the right answer.

Speaker 4

117:54 So what's the hack? Is it the shape of the glass? I don't know. I don't know if they ablated the shape of the glass. Is it the color? Well, I'm sure that's going to be part of it. Is it the sort of, like, you know, background of the room? How well did they ablate that? I need to go look personally, but these are the questions that I would be asking a priori is what does it tell you when I learned that, you know, salad dressing can predict your coffee preference?

Speaker 4

118:20 Well, what's the sort of, like, you know, intersecting reality of those two things? Mouthfeel,

Speaker 2

118:28 if you had to go back today and do it again, would you think about doing things like chemical analyses of the of individual coffees? Like, how would it change given where models are now?

Speaker 4

118:45 Yeah, I think. I think if I went back, I would not be looking for more features and I would not be looking for more detailed understanding or I would be looking for even more. And this is maybe the biggest takeaway from working with taste. And I did a Lot of other experiments related to customer preference. One of the things that I think kind of just beat me over the head year after year when thinking about this was that like routine and expectation are much larger features than nuanced flavor.

Speaker 4

119:19 To give you an example, I still drink pour over coffee every single day. I make it by hand every single morning. And if you ask me what's your favorite roaster or what's your favorite coffee? I don't have an answer. I literally do not have a favorite coffee. And the reason is because the most important thing to me is new and varied experience in that flavor. I drink weird anaerobic, super bright funky stuff and I do it every day and I change roasters every month and I purposely try everything out there in the world because variance is much higher importance to me than a specific flavor.

Speaker 4

120:07 But that's me, that's unique to me. But I think it just kind of reminds me how much sort of routine and habit and sort of what my goals are inform much more than these hyper nuanced things of oh, what's the acidity and sort of like ph. No, no one's, no one's deciding if they like their morning coffee based on ph. Maybe in some subliminal way it's a latent feature that's hiding up there in that neural network, but I'm certainly not ordering coffee by ph.

Speaker 4

120:42 And even the little like light dark meter on the bag that like too many coffee companies get obsessed with, even that is sort of low signal relative to all the other things.

Speaker 1

120:54 Let me, let me, let me bring us back to sure. Gary's operations for a moment. You've obviously built a bunch of tools for internally company and I think you've described a little bit of a research agent solution which kind of takes in the company's context and kind of enriches it, et cetera, et cetera. How has the operations of the company changed in one year since exactly one year ago to now? What has changed in the decision making workflow?

Speaker 4

121:27 Totally. A couple things in particular. I can't get into specifics, but we'll just say there's been an order of magnitude change in the number of deals that an individual investor can look at. And that's there, that's over. Like the weekly and monthly scale. There is a two order of magnitude change in the multiplicity of pieces of information that is used while doing research on a deal.

Speaker 4

122:02 There is a, an order of magnitude change in the number of diligence calls that are getting done. The on average. Sorry, just to Be clear. There are sort of, when we think about the key, like, features of any diligence process, one of those features is the way that we engage our network.

Speaker 4

122:39 And I would say that engagement is also two orders of magnitude increased in our access to that engagement. And so, you know, coming back to this, like, where should they spend their time? It's on hospitality. What you'll notice is like, very few of those things are, you know, we're writing a hundred times more words in a, you know, internal memo that's not happening.

Speaker 4

123:10 But like, all of the things that filter into making that internal memo great and meaningful and representative, now we're talking. And so those are the ways that it's really changed the operations. We see it in our dashboards, we see it in the sort of like representations and logging of what people are doing with AI systems. What was true over a year ago was that everyone was using Claude all the time.

Speaker 4

123:43 What is true today is that everyone is using our internal, like company brain all of the time. What is true now that wasn't true six months ago is that there are agents in sandboxes running research requests all of the time. And so, yeah,

Speaker 1

124:09 Do you find there is kind of a spectrum of like, let's do more research to. We need to cut off the research and we need to make a decision now because it is that research process doesn't really have a defined endpoint. You could always dig deeper. Correct,

Speaker 4

124:26 Totally.

Speaker 1

124:26 How do you, how do you make those decisions? Are the agents the ones making the decisions? Is there a stepping off point where they have to come, come back and you make that, you know, a human is making the decision. How does this loop work?

Speaker 4

124:38 Totally. I want to reference something that Nathan said earlier. You know, he's got this list of things and he's like, some of this should have been left on the cutting room floor. We would call that taste. And sort of like knowing what should be cut and what shouldn't be cut is taste. And yet again, here's where I will point to why it's important to have exceedingly bright colleagues. When you have exceedingly bright colleagues, they've got great taste on, like, what research needs to get done.

Speaker 4

125:04 There are some research that like, yes, like, we always want to do it. We know we need to do it. This is the structure. And there are opportunities for agents to say, let's spin off some sub agents to continue doing this kind of research. Let's dig deeper on this. That's fine, that's automated. But ultimately the alpha is in the taste and so, yeah, like, we can burn more tokens, we can burn more compute. But I don't know, it's a pretty.

Speaker 4

125:31 It's a pretty like, focused laser where you really want to point these resources at things that people have the ability to consume. If I produce an extra thousand pages of research on some deal, who's going to look at it? I guess other agents could look at it, but, like, then what is that valuable? It's not obvious to me that it is. And so again, pure volume of research is not a metric that we even track.

Speaker 1

126:00 Another question is on the other side, you said you engage your network two orders of magnitude more on the engagements on the network. What does that actually mean? Because my network is spread across Twitter and LinkedIn and my phone messages and Slack and. And it's just like this fragmented existence, which is very, very annoying to keep track of. So how does this work in terms of what does that network mean in that sense?

Speaker 1

126:32 And how do you engage it with two more orders of magnitude? Are you running computer use agents? Are you having agents reach out to people with LinkedIn mail? What actually is engaging that network

Speaker 4

126:46 in

Speaker 1

126:46 two orders of magnitude more?

Speaker 4

126:48 Totally. So, like one, I will echo your annoyance with the, I would say, like, distribution of how you connect to the world. This is something that, like, isn't really solved in the market. Just kind of like collapsing all of the people that you're connected to into an easy system. At some point a couple years ago, I like, threatened to try to build a company around this exact idea of just, like, making it much easier for you to interact with people on the platform that is like, most native to that relationship.

Speaker 4

127:22 This is still unsolved. It is a pain in the ass. And this is one of the sort of, like, things that we have built a lot of custom technology around to make this possible. But to your other question, the answer is a hard no. We are not asking agents to go and spam our relationships and friends and, hey, you know, can you get me an intro to Nathan? Like, that is. That is not a thing that agents are responsible for.

Speaker 4

127:48 In fact, it's very similar to the cafe lead who we want to reduce the wasted cycles on things that they're not superhuman, to allow them to focus on hospitality. But because the reality is there is someone who can introduce me to Nathan and there is someone who is right to introduce me to Nathan. And by identifying that person, you can 10 or 100x the effort put into finding the right person so that I don't have to spam 15 people.

Speaker 4

128:20 Do you know Nathan? Do you know Nathan? Do you know Nathan? I can be much more laser focused. Hey Prakash. It seems like you could make an introduction to Nathan. Here's why I'd like you to. And here's why I think you would be excited to make that connection. You can see that there's a big difference in the voltage applied to the channel versus the number of connections that you're trying to make.

Speaker 1

128:46 Indeed. Let's talk a little bit about what is, what are the like looking forward, I think I find like, you know, when you have an investment in a VC firm and you have to think about an exit in five to six years, right? Like, you know, five to six years, right? Like

Speaker 4

129:11 that's

Speaker 1

129:12 quick, right? And then you have this like potential, you know, recruitment, curse of self improvement, AGI singularity. All of these terms being thrown about how do you make those decisions. Well, like, do you have this kind of like, okay, I have this AI 2027 view. This is when things are going to happen, et cetera. Or do you like, no, we're not going to. That's speculative. We're going to look at what we have now, what the technology can be used for and what we can project forward on a normal kind of curve.

Speaker 1

129:44 Like how do you make those decisions?

Speaker 4

129:47 I mean, I think the common answer to here, which I do align with is sort of like, you bet on things that like sort of benefit from a rising sea. You bet on boats that like, you know, can handle a higher water level and you're excited as the sea rays rises. So to give you a specific example, a good example of that would be something like Back ops. Back ops is an investment that we, we did I think back in January or something like that.

Speaker 4

130:17 I can never remember when we actually invest versus when we announce it. So, you know, Backups is this like incredible company and they're building this like technology to help with logistics. Okay, There is a future where logistics is heavily automated through robotics and through sort of like, you know, extremely tight control systems. And that's really exciting. And there is the current reality where there is a little bit of robotics in logistics, but there is a lot of human in logistics and then there is all the space in between.

Speaker 4

130:53 And it turns out backups is relevant under all of those paradigms. If we get to a heavily automated and robotic fleet, Back ops is even better. Right now when we're dealing with a hodgepodge potpourri of par person picking up the phone and you know, yelling at a robot on the other side Actually backoffs is really amazing now too. And so as computers get better, as we get closer and closer to, you know, RSI or just agents that can kind of like understand where they need to get better and make those changes.

Speaker 4

131:26 Well actually back ops agents benefit from that. And so it's one of those like everywhere you look like they've got reason to succeed. And so that's the kind of bet that's like, feels very obvious. You can look at another bet. You can look at Artemis. Artemis is a AI powered sim. Well, okay, do we think that like, you know, monitoring security threats in your stack is going anywhere? Well, the only way you could really think of that is under two paradigms of like AGI.

Speaker 4

131:56 One that you've got like an sort of like arbitrarily robust and provably perfect software that feels extremely unlikely. But even in that case, you sort of want to understand then what the software is doing, I. E. It's still valuable to have something like a sim. The other sort of future is everything is burned to the ground completely. And like attack vectors are so much like larger in magnitude and velocity that you can't protect anything.

Speaker 4

132:30 Well, you still then need to do a prioritization of what fires to fight. And so under both of those paradigms of essentially arbitrarily, like strong intelligence. Well actually you still want an AI SIM in the current paradigm where we don't have arbitrarily high bar for intelligence. Well actually an AI SIM is still valuable because we're getting emergent technology and uneven distribution of intelligence and effort put into threats.

Speaker 4

133:01 Well, in that paradigm, actually you still want an AI sim. And so again, like, like, show me a future where that's not valuable. I can't seem to find one. And so I don't think that rsi, I don't think that sort of like, I don't think that the expansion of intelligence on demand obviates the need for businesses. And so I think the question then becomes, is like, yeah, what's going to stick around?

Speaker 4

133:36 What's perennial? What I would say, what are the evergreen problems to solve? You hear about sort of like evergreen businesses, but ultimately a business is just a problem.

Speaker 1

133:50 And that's a good segue for me to ask how goes your teaching of data science at I believe, Rutgers?

Speaker 4

133:56 Yeah,

Speaker 1

133:57 so what, what do students, you know, students are probably asking some similar questions there. So, you know, what, what do you say to them and what do you, what do you think the future of that field is?

Speaker 4

134:07 Yeah, great question. I still make them do gradient descent. By hand. So this is an area where in my opinion I have yet to hear a single person with a cogent explanation of what the hell we're supposed to do as teachers. I believe and have believed for a long time. You know, I came up in pure mathematics, were not the most practical bunch, but I have always believed, even as a pure mathematician, that you should be constantly thinking about what your students future looks like and why every minute in your class is relevant to that future.

Speaker 4

134:47 I am not just like phobic but like allergic to wasting students time. And I love pure mathematics. This is not a, like, you know, I'm a self hating, you know, pencil pusher. I deeply love the ideas of mathematics, but I also think that the most important thing that any instructor is responsible for is maximizing the value of the time you have with them. So that's a very long way of saying I have no clue what we should be teaching.

Speaker 4

135:19 So what I do is I reassess every semester, much to my own suffering because that means I have to rewrite my curriculum every single semester. On what do I teach them? So three years ago, when it was obvious that sort of AI was going to change data science and then I started working at HEX trying to change data science, I rewrote my entire curriculum to introduce why AI might be relevant. And every semester I go through the whole course and I ask, can AI just one shot, the course?

Speaker 4

135:52 The answer has never been yes, but that's okay. The answer has always been okay. These things are now not just easy for the model, but easy for the model to obfuscate the learning. And so I don't think the question is, does the model know all the content that I plan to teach? The question is instead, does the model have the ability to collaborate with the student when this is going to be relevant for the student in a way that yields the sort of best outcome?

Speaker 4

136:24 If the model can, if, if you know, Astra can train an SVM on the coffee survey data at a better loss rate than Brian can. Neat. But can Astra say, you know, this feature is highly predictive, but when we think about it, if we put this on the website, it's going to upset some of the people going through the flow and they might churn like yes, Astra can look at the data posterior and say people are churning more on this question, but does Astra know ahead of time to look for that?

Speaker 4

137:01 Does it know to instrument that in the product? Does it know that this is kind of a weird question and people might be bothered these are the kind of things that Astra Ultra is still only mid at. But ultimately, I don't want my students to only be users of AI. I want them to be sort of the things that make data scientists exceptional. And that is ultimately curiosity. Like every data scientist that I've ever met that is in the top decile and certainly in the top percentile are curious as hell, almost to a fault.

Speaker 4

137:37 The absolute best data scientists I've ever worked with are borderline annoying with their curiosity. Like, you kind of want them to just, like, leave it alone sometimes. And that's what makes them so excellent. And so mostly what I'm trying to, like, indoctrinate my students into the propaganda is curiosity. Be curious, ask the questions that you can't get out of that brain. And when you do that, you're going to be an excellent data scientist. Whether you're writing Pan is by hand, whether you're using matplotlib from memory, I don't really care.

Speaker 4

138:08 Do you remember the indexing for numpy? I don't. I used to know it. I don't remember it anymore. Do I remember the slice operators for, you know, Pandas filters? Not anymore. I forgot them all. But I used to know them. It doesn't matter, though. What I know now is I know sort of when you look at a distribution, whether it makes sense and sometimes Astra doesn't.

Speaker 1

138:33 Indeed, Brian, we've got. We've taken a little bit more of your time than expected. Thank you for, you know, spending time with us. It has been a very interesting conversation. I've learned a lot. And do you have anything that you want to leave us with?

Speaker 4

138:49 The only thing I'll say is we just announced our next event, which is a hackathon in San Francisco on December 11, and it is called the Great Business Bake Off. And you will be up against over 100 other competitors competing to make your business a successful. You will have data agents to help you do analyses about the data. And the data will be hiding a lot of secrets and mysteries that you're supposed to uncover.

Speaker 4

139:20 And so if you think that you're a great AI user or a great data scientist who is sort of already in the sort of like, AI age, come and show us how good you are. Because this is a competition where whoever makes the most money for their bakery business wins. There's no judges that are sort of like, bringing their opinions or anything like that. Just prove it. Make the most money for this fake business and you win.

Speaker 4

139:52 And so December 11th in San Francisco The Great British or sorry, the great business, Bake Off. And there will be cookies and there will be prizes and there will be memes and it's going to be a whole hell of a lot of fun. So hope to see you there. You can sign up@theoryvc.com Bake Off. Bake off is all one word in the URL

Speaker 1

140:15 and it's not British.

Speaker 4

140:19 If you're British, you're still welcome. But I'm British.

Speaker 1

140:24 Amazing. Ryan. Thank you.

Speaker 4

140:26 Have a good one.

Speaker 2

140:39 That his answer on what he tells his future student or his students about their futures I thought was really interesting. And it sort of contained the pessimist and the optimist take at once. When he was first going through the, you know, sometimes there's like a survey question that may offend people and you got to be wise to that. I was like, man, that's bleak. You know, the idea that like we're just going to be sitting here being the sort of offense detectors, you know, while the AIs do all the real work was like, oh God, you know, that doesn't sound awesome.

Speaker 2

141:15 I did like the curiosity framing a lot better. Being relentlessly curious, certainly in my experience working with data has been indispensable. You know, really want the difference between somebody who really deeply wants to know the truth that the data contains. To the degree that the data even contains truth, which is often a problem unto itself, those people absolutely kill it. Compared to those that are just like trying to produce an artifact that they can say, you know, is done and hand off to whoever the consumer of it is.

Speaker 2

141:47 So I do like curiosity as an answer to what we're going to bring to the future, but it does, I would not be telling my kid to go into data science today, I'll tell you that. So that is a tough one. I don't, you know, the real question there of like, is there going to be a job in data science that looks anything like the current job in the next few years? I feel like that is, I think that's pretty tough.

Speaker 2

142:15 You know, it seems like we're going to have data science agents and it's going to be up to the sort of non data scientist, general purpose business operator to provide that little salt of curiosity or offense detection or whatever that special sauce is that the models don't quite have. It's hard to see that as a full time job because a lot of times that is all context anyway. It's like what really allows some, it's a little bit disposition and it's a Lot context.

Speaker 2

142:47 And it's just hard to see somebody scaling that across so many different data environments. You know, you could imagine like, well, you could just have one data scientist who like runs, you know, 100 agents and provides that, you know, missing piece to all those agents. But I think that's not really, that's not really super realistic because it takes time to be immersed in all those hundred different data worlds to have any intuition that you're going to be able to use to improve what the model coming up with or to, you know, sniff out to tie back to our extra senses conversation, what might be wrong with it.

Speaker 2

143:21 And that just seems like it's a, it's a, something that comes from kind of a different role. That's why I often see like the data scientist, you know, or data analyst person will bring something to the business owner and they'll be like, this doesn't feel quite right. And it's because they just have a deeper intuition that even the data scientist person doesn't have based on their like, broader view or longer, you know, tenure. Yeah, I don't know.

Speaker 2

143:48 That doesn't, does not feel like a growth sector to me where we are right now. But

Speaker 1

143:53 I feel, I feel like, you know, if you look at a company, you kind of cut it apart into like sensing, analyzing and decision making. Right. And then action, decision making and then the reverse which is, you know, operationalizing and actions. Right. And I think AI is going to be very good at the sensing and analyzing and also the operationalizing. But I think the decision making, you end up with a human in control, right.

Speaker 1

144:24 And then if you take the human in control, okay, you can have a business with a single human in control. What's a number of like decisions that that human can make and those decisions are decisions with taste and all the decisions and all of the non, all the decisions that the human isn't making. For example, if you're running a tech company, you don't need decision making. Maybe on the accounting framework, you're looking at the product, you're looking at the technology, et cetera.

Speaker 1

144:53 But accounting in the back office is a back office in the same way if you're running an accounting company, the accounting is, you know, front and center, but the technology is, you know, kind of a back office. You're not, you're not like, you're not there building technical tools, right. So I feel like you can separate out companies based on what is the ultimate purpose. And then the ultimate purpose has all of the humans with, you know, taste in there and Then the non purpose becomes all of this, like, you know, like back office stuff, like it on one side and accounting on the other.

Speaker 1

145:33 And then you can have like, basically everyone gets to do what they really want to do and what they're really good at and everything else gets taken care of. So you do what you have taste in and what you're good at and everything else gets taken care of by the systems. And in standard economics, what ends up happening is that the enterprise shrinks because you are able to collaborate more effectively with other enterprises. And so every enterprise shrinks to its core competency and it deals with other enterprises that have their own core competency.

Speaker 1

146:10 And all of the contractual framework makes this possible. What should happen is that enterprises should shrink in size. We should see a lot smaller, more focused companies which are able to contract much more easily and fluidly. People have talked about API pricing, like selling pure APIs instead of selling SaaS. A SaaS is a $10 $50 a month, $100 a month package of services. But you often end up with these SaaS.

Speaker 1

146:43 Companies that keep adding features. They keep adding features in order to add consumer, like, okay, this consumer wants photo editing. I'm going to add photo editing. This consumer wants to add music to that photo editing. So then I'm going to add that. Right? And so then you get these companies that spread out over time because distribution is an issue and et cetera, et cetera, and sales is an issue. But then you could also have companies that only focus on photo editing forever.

Speaker 1

147:08 Like midjourney is basically just going to do photo editing, photo, you know, image generation for editing, etc. And doesn't do anything else.

Speaker 2

147:16 Plus medical scans.

Speaker 1

147:17 Plus medical scans. Plus medical Scans. On the side. On the side. So maybe, maybe that's what you end up with. Maybe you end up with these like smaller, more focused companies which are very, very good at exactly what they do and don't actually do anything else and they contract with other, other firms. So maybe that's.

Speaker 2

147:36 Yeah, I don't know. I mean, it seems like on the one hand, part of that analysis makes sense to me when you think like, why do we have. This is like classic. Not that I'm the best person to give the lecture on this, but this is Classic Coast's theory of the firm. Why do we have firms in the first place? Well, it's because contracting everything all the time is too cumbersome. The cost of working out these arrangements ends up just slowing everything down so much that you're better off drawing a line around a bunch of people and kind of having a more command and control pre negotiated structure within that boundary so that you can be efficient and not have to constantly be negotiating terms all the time.

Speaker 2

148:24 And so when you think about it from that lens you're like, yeah, as we heard last week with our conversation about union, there's like a lot of opportunity to reduce those transaction costs. And if you do reduce transaction costs, then the boundaries that you can draw around groups of people can get smaller and there can be just a lot more kind of arm's length relationship. And okay, I like accept that as far as it goes.

Speaker 2

148:55 But then what we actually see from software companies today is both you know, some reduction in headcount like we talked about with Microsoft and we, you know, I do think we're going to probably see significantly more of that over time. But that's being coupled right now with sort of super aggressive expansion of features into like every adjacent niche. And I do wonder, you also just kind of wonder like how many companies really have this like core core thing that is so great that it's going to be durable in any case.

Speaker 2

149:36 You know, I think of Stripe as kind of the canonical example of that like super battle hardened software, incredible data moat. This allows them to create the foundation model for payments. They can do then the best in the world when it comes to fraud management. And there's just a flywheel there that you certainly not now and I wouldn't think anytime soon would be able to vibe code your way through. Even if you could create all the functionality, you'd still have a very hard time managing fraud in the way they do based on the visibility they have into the economy.

Speaker 2

150:10 So that's awesome for them. How many companies are really like that. He said like Data, you know, the data is at the core. Maybe we should have pulled on that thread a little more than we did. But like if data's at the core of so many of the things, the bets that they're making. Data I think is at the core of what makes Stripe a long term winner. Is there a data core at Salesforce? You know, is there a data core at Ramp?

Speaker 2

150:35 I mean maybe I'm sure they'd come on and tell us that there is one. If anybody from Ramp is listening, come tell us about your data moat. But like it seems like it's much more about we built some functionality we were pretty good at. Go to market. There you have it. You know, I'm not quite seeing why a lot of businesses, and I don't mean to pick On Ramp in particular. But like the Ramp versus Stripe comparison, it does feel to me like there's something at the core of Stripe that you will not be able to vibe code for yourself.

Speaker 2

151:07 And I'm not so sure if there's the same thing at Ramp or like the vast majority of businesses, to be honest.

Speaker 1

151:13 I am, I'm going to be a little bit fatalistic and I'm going to say that, you know, it's a lot of it is rearranging deck chairs on the Titanic, I feel, because, you know, again, I've said this before, you know, all of, all of the, you know, Ivy League Ivy plus graduates alive today, probably about 2 million people when you look at them like, and when you really look at like, you know, PhDs and you know, master's students, et cetera, almost every company in the world, including companies in China, are run by basically Ivy plus graduates from the U.S.

Speaker 1

151:51 like even, you know, Xi Jinping has a degree from Iowa, you know, Iowa, Iowa State or something, master's degree. So a lot of, a lot of people have studied, you know, in these American firms and American colleges. If you take any of you just take that as a proxy for like, you know, highly educated, Westernized, you know, well spoken, et cetera, et cetera, et cetera, like you just take that as kind of a package. Yeah, maybe, you know, obviously not only maybe it's ten times more, but you know, just take that as a marker and then you imagine that you're going to have like every B300 is an Einstein or, or 10 Einstein's, et cetera.

Speaker 1

152:27 And imagine the deployment over the next 12 to 18 months maybe, okay, on just finger in the air, you get 50 million more IB plus graduates, but maybe slightly jagged, but to a large extent able to do at least at human or maybe 90 percentile of human and in many cases 200 percentile of human, like much, much better than humans. Do you actually think the human built infrastructure will suffice?

Speaker 1

153:02 Do you think they're going to be using human companies or are they going to build new infrastructure that is suitable for agentic use? I feel like the going forward economy is an agentic economy and that means agents building businesses and software for other agents which are suitable for their affordances, which is going to be massively parallel, high speed, able to absorb information and distribute information at very low latencies.

Speaker 1

153:37 You see that inside Both Anthropic and OpenAI, they both have new version control systems. Anthropic built one, which is basically a single internal monorepo, but Then you only need to look at a certain segment of it. You don't need to download the whole thing. You don't need to download like 100 gigabytes, whatever. You only look at a particular segment and then you can send in your changes. And then you have a coordinating agent which kind of coordinates all of the incoming changes to the repo, right?

Speaker 1

154:09 So you have a centralized agent, replacement agents internally inside the repo, which kind of pick up the changes that all these coders are making and kind of merge it together. OpenAI came out with something similar. They did. One key thing that they did is they decided to use the existing git framework, but they expanded on the work trees and a bunch of other things and they bought an open source company that was working on these git work trees, etc.

Speaker 1

154:34 So you can kind of see we're going towards that we are going to end up with a salesforce primarily used by agents, a ramp primarily used by agents, all of these softwares a Stripe primarily again used by agents. Maybe, maybe Stripe or other companies will be the ones that create the software. I am very doubtful. I'm very doubtful because I don't think they will move that fast. I don't think they are capable of moving that fast.

Speaker 1

155:01 I don't think they understand the affordable. And I know Stripe is basically like, you know, people a billion times smarter than I am. I'm not, I'm not trying to downplay, you know, how effective and how intelligent and how competent Stripe is. I'm just saying like, come on guys, you're gonna get 50 million, you know, Patrick Collisons, like there has to be some impact. It's not, you're not, it's not gonna be without impact. Like you're gonna get 50 million Patrick Collisons, you're gonna get 50 million.

Speaker 1

155:27 Elon Musk masks. You're going to get like humans who give them capital to deploy. You're going to get agents coordinating investments into businesses built by agents for agents. And so I largely see the future of the economy as primarily agent driven with us just receiving free benefits because hey, CRM call from a human is like so, so costless in comparison to the massively parallel CRM calls that an agent would make.

Speaker 1

156:01 And so we just get these benefits because we are just not that, you know, big a portion of that, of that usage anymore. And our usage and that's where the abundance comes from. The abundance basically comes from us building all this stuff for this agentic economy. And as a result we just have all of this excess because we don't, we just don't consume that much. Right. Like we're just not that consumptive in comparison to what the agents are going to consume and that's where this abundance is going to come from.

Speaker 1

156:33 I recognize that this is not a popular with AI safety. I don't think this is a vision of abundance that is very attractive to a lot of people because people would like to believe that we are still in control, etc. But I do think that it is a plausible one which is also positive. You just have to get over the whole ego bit as like we own the companies, we run the companies, we control all the decisions, all of the resources are consumed by us.

Speaker 1

156:57 No, it's largely agents are going to be the ones creating these ideas. They're going to be the ones building the businesses. We're going to capitalize them and we're going to receive the income from that capital and then we're going to be able to distribute that capital to the new investments and the agents will offer us some new investments which are, you know, lucrative. Right. So I think a lot of this, like we're like we're going to see this intelligence explosion in the next 18 months.

Speaker 1

157:21 We're going to see this. Right? And I think some people have really like, I think Elon has been very depressed for a while and because I think he recognizes that self driving cars are going to be useless because every car will be self driving in 18 months. Right. So yeah, I mean this is just the ball game, so let's see what happens. But I think the agentic economy is the future and people who say that don't understand yet what it means, but they will eventually.

Speaker 2

157:52 Yeah. I mean leaving aside for the second, the question of whether we can keep control of this well enough to have any sort of good version of it for the moment, not to forget about it, but just to leave it aside. I guess I also sort of question like, and this is again is you know, a similar vibe to the blood on the clock tower benchmark. Like is this really the direction that we want to go?

Speaker 2

158:23 I mean you're, you're sort of describing the like country of at least like near geniuses in a data center. And it does seem like by all indicators we are headed that direction. But then I sort of think about Andrew Krich as like a source of some of the best thinking on the machine economy and what rapid growth in the machine economy might mean for us. And on the one hand it's like cool if there's abundance in API calls because the agents are using so many and they build up, you know, great systems that they can use and then we get to kind of piggyback on them.

Speaker 2

159:03 But like, there's a lot more to life than API calls. And I do feel like we're sort of especially. And especially those of us that are in the AI space are like really quick to forget that. Or we sort of, you know, we're kind of contorting ourselves in a way to like, like this future. This is why I've been like holding myself to this idea of like, I want to be getting outside side more and getting more exercise.

Speaker 2

159:25 And if I'm not doing those things, am I really winning or am I just kind of plugging myself into some, you know, AI powered machine as like some sort of, you know, human integration layer type, you know, component that's like, in a way, like too subservient to the machine. This is also why I decided to do, and I would maybe contrast this with the, the Blood on the Watchtower thing, both in terms of like, vibes and reactions, but this is why I decided to do this Fruitopia project recently.

Speaker 2

159:58 And you know, it's again, just one of these things where it's like, okay, if we are in a bright future, then I think one very kind of deeply rooted thing that we all like, we're evolved to, like, is eating delicious, fresh in season fruit, right? Like, everybody loves that. It goes back to when we were monkeys that we loved that. And yet you can blame my lack of Twitter game, which is certainly a contributor, but when I tweeted that I got basically no play on it versus the Blood on the Watchtower thing.

Speaker 2

160:34 Everybody's like, oh my God, look how great AIs are getting at deception and conniving and winning these games of social manipulation. And I'm just like, I don't really want that from my eyes. What I want is a future where I have not abundance of API calls, but abundance of things that are like, more biological. Give me the fruit. You know, that's what I really want. The fruit of the AI revolution should indeed be like, at least in part, actual literal fruit.

Speaker 2

161:02 And if we're losing that, we're like losing so much of what has mattered to us. And if we can't even like notice that we're losing it, I think in a way, you know, that that sort of worries me and makes me think like, maybe we've kind of already lost the plot.

Speaker 1

161:18 I mean, it's pretty obvious to me that we're going to get that grown like food, which is far more nutritious and far more tasty. It's going to be like, you know, in 20th century, you discovered chemicals, and then we, you know, when you look at these food products right now from, like, Doritos or whatever, those are, like, so scientific, right? There are, like, thousands of PhDs who have worked on Doritos. Thousands. Literally thousands.

Speaker 2

161:45 Remember the old Onion article? Doritos celebrates its one millionth ingredient.

Speaker 1

161:48 Yeah. There are literally thousands. In order to get ingredients which are, you know, in order to get the right crisp, in order to get the right taste, in order to get in order, like, they have figured out your flavor palette so well that they can pinpoint exactly what flavor that they want to produce. And all of that technology is going to get deployed into bio. We're going to produce fruits and vegetables, et cetera. But there is a little bit of ick factor that we have to get over.

Speaker 1

162:17 And I think at some point, 25 years ago, I would not have thought because we had the whole GMO ban thing that the Europeans put into place, but now I think we've just had the first salmon. Gmo, I think, was approved or something in Canada. And I think people are, the millennials especially, are willing to look at this on a more scientific basis. And they're like, look, hybrid food, hybrid fruit is already basically GMOed. But we just took a very long time to GMO it, and so you can basically just gmo.

Speaker 1

162:51 So I think we will get GMO food food, and the GMO food will be easier to produce, will be tastier. We'll take into account all of the learnings of Doritos and all these guys that, you know, that discovered all this stuff in the last, like, you know, 20 years. We'll get wine, which is far tastier, because, you know, obviously, if you have a regression model, which is what, you know, I think Robert Parker's, like, Wine Spectator thing was, he rated all these things.

Speaker 1

163:18 And then people built regression models on this, and then using those regression models, they could basically rate the Bordeaux that were produced in France. And so if you have a regression model, that also means that you can scientifically kind of detect what is a 95, 98, 99 wine. And then if you have the ability to produce that, you can produce it at will, right? So I think we're going to achieve all of these things. Like, we're going to achieve the ability to do all this stuff.

Speaker 1

163:44 Part of abundance is going to be, you know, your 99 percentile wine available for two buck chuck. Right. That is going to be this hedonic adjustment in, you know, quality, like massive hedonic adjustment in quality that happens and we're going to see it everywhere. I think again, there's going to be this acceptance of technology bit where you're going to have to accept that it's a GMO thing, it's not naturally produced, et cetera. People are going to have some ick factor to it.

Speaker 1

164:11 There's going to be, they're going to be people who are like, this is like an authentic, you know, Robert Parker 99 and this is, you know, Franzia Robert Parker 99 brand. I think, you know, genetically, you know, indistinguishable. But you're going to have the status marker of the original versus, you know, the GMO produced. So I think all those things will happen, right? We will get all of that and we will see that abundance and we will see that as abundance, but perhaps we will see it as quality.

Speaker 1

164:42 The quality of the product will increase dramatically and then we will figure out other ways to differentiate because we need status again and we need to differentiate on pricing. And so I think those things will happen. It's, you know, that is, that is about. That is abundance. I guess that that is fruit.

Speaker 2

164:58 Well, I sure hope so, but it does, I would say it doesn't feel like that sort of thing. Fruit being just one avatar that I've chosen for this broader class of thing that is intrinsically and I'm all for. My favorite apple is the Honeycrisp and it's a relatively recent creation. Not GMO as far as I understand, but definitely the product of large scale systematic cross breeding which is sort of, you know, GMO before there was gmo.

Speaker 2

165:36 So, you know, I'm not afraid of GMO enhanced fruit by any means. I'm all for it, but. But it is still something that is like a biological construct that feeds us as biological creatures and like, you know, delights us in terms of our biological senses and the like, you know, ephemeral qualia that we get from them. And like this stuff is just all too often lost because it's not very measurable, it's not so hill climbable and it's not what the agents need.

Speaker 2

166:09 Right. And I think we are going to need to be very careful not to create an economy that really serves agents and creates abundance for us in things that are core to what agents need. We're going to, if we. That seems like what I kind of agree with you that Might be what happens by default. But I would challenge everybody to think like, wait, like, what if the stuff that agents need in abundance isn't the same stuff that makes our lives good?

Speaker 2

166:35 And you know, are we kind of, are we already sort of optimizing the future for agents convenience as opposed to our own survival, thriving, flourishing and you know, just enjoyment. And in the end, I'm kind of emphasizing the enjoyment part here. Again, not to forget about the other key ones like survival, but the enjoyment piece, you know, looking at that picture of wine, it's like, man, okay, so this model can like identify the wine, but the whole point of wine is for us to drink it and taste it.

Speaker 2

167:15 You know, like, where is that in this whole equation? It's, it's,

Speaker 1

167:20 you know, I. One of the annoying parts is that, you know, the Frontier Lab guys are very serious, right? And so when you're very serious, you only talk about cancer, curing cancer. No one's talked about a better Doritos. Have you realized that? Like, no, no one's come out there and say, we're going to make the best Doritos possible, right? Like, it's, this is, this is also why they're like pretty, you know, everyone's like a little bit dorky.

Speaker 1

167:46 They're very earnest. It's a, it's a little bit anti human, right? No one's come out and say like, we're going to make a better football game, right? Like, where are these, like, we're going to make a better burger, we're going to make a better football game. Where are these things that people enjoy? And you don't come out and say these things because you're like, you want to appreciate. You're serious, right? You're like, we're going to cure cancer.

Speaker 1

168:10 And that's, and that's. I think that's because they want it to be serious. They want to show that earnestness. They don't want to come off as like foolish, etc. And I think maybe that's perhaps the wrong way to go about things. Perhaps maybe you should focus on like, we're going to make a better Doritoship. We're going to make one that's healthier for you. We're going to make a Doritos that is naturally grown from the ground and you just need to cut open the potato and it's Doritos.

Speaker 1

168:36 And there's no, yeah, there's no Red 40 in there. There's none of these like weird, like, you know, like chemical things that you put in There it is. Naturally you have a potato and you cut it open and you fry it and naturally tastes like a Doritos. How about that? Right? Like, and this is like, this sounds like so frivolous and idiotic, but this would catch the imagination of people on what is actually going to be possible.

Speaker 1

169:04 And also like, hey, we can make better stuff. We can make Doritos that doesn't have this red 40. We can make this stuff that parents ban their kids from having because it's like cancer causing or whatever. And I think all of that is possible but they don't want to come out and say it because it's frivolous. They feel like consumer economy in general. You know, our friends at Anthropic decided not to do consumer at all. Like, they're like, we're just going to do B2B, right?

Speaker 1

169:33 So you end up in this like very cold coding, coding apps, abstracted coding research scientist like etc. Right. So

Speaker 2

169:43 yeah, I mean, I think it's also because they do have this kind of World War II mindset about their world work, you know, that they, they genuinely do believe time is short, stakes are high. The, you know, their own like, allocation of time is like very skewed toward work. You know, there's just not a lot of space for leisure in the lives of the people at the frontier companies. So it's like they're just so pattern matching at this point on like, here's a hill, let's go climb it.

Speaker 2

170:17 And you know, I do think like a little more discretion on which hills do we really want to climb? Like, are we really happy that we've created AIs with the current shape and are we steering toward a future that actually serves us versus plugs us into some superstructure that we're just kind of like, you know, playing the old like phone operator role of kind of connecting, you know, one, one port to another.

Speaker 2

170:49 I don't know, it feels like unfortunately, yeah, just there's just not enough time and energy going into those sorts of questions. So I'm stuck on that maybe a little bit today. But man, the blood on the, on the clock everything really did it to me because you know, again, it is good work and it is interesting, all else equal. But we're, I think we need some thinking that challenges the all else equal paradigm on so many levels.

Speaker 1

171:22 Yeah, it's, it's, it's a really tough one because I don't think, you know, agents can operate in the economy without some level of subterfuge. Right. So without some level, like you can't tell your pricing to the other guy, for example, you can't tell your internal costing to the guy who asks. You have to reserve that. And that means that involves omission and you have to know when to omit.

Speaker 1

171:53 So there's a bunch of things there. This is also why I think Palantir had this idea of the ontology. Who's allowed to know what? That's a pretty important thing. Like who's allowed to know what and who's allowed to say what, who's allowed to do what. And that permissioning structure that they locked these agents into in the early days so that you don't get these departures so that you can have this agent which is perhaps a little bit inconsistent and unreliable and you can still operate it within that framework.

Speaker 1

172:23 And so maybe that's what, you know, I honestly don't know. I think again, there's this benevolence to obedience axis. And I don't think either one, either purely benevolent or a purely obedient agent is acceptable. And I think also you need this kind of like, interrupt from government, which is possible, or interrupt from regulation, or interrupt from. Which means that the agents have to obey when government decides, like, hey, this is, you know, you can't do this.

Speaker 1

172:53 The agents have to obey. So there's a bunch of things that I think need to be worked out in terms of governance. I'm sure, I'm sure we'll get there, but, you know, we're going to have some trial and error along the way, I guess. So.

Speaker 2

173:08 Yeah, no doubt. Maybe in closing, I'll just cite one sentence from the meter thread on their report on acceleration. This is, I don't know, 8 to 10 tweets down the thread. They say the full analysis estimated how much of the acceleration in capabilities was due to compute data and algorithmic progress. And then in a. That's first sentence. Now I'm just editorializing in a thread in which they have caveat in everything not confident.

Speaker 2

173:41 And the, you know, it's. It's very unclear how to interpret data. And they've, you know, gone around and around in circles with themselves and with anthropic team and, you know, there's just so much fog of war and even the core estimate of 2x is really like, they wouldn't be surprised. Anything 1.3 to 3x. So caveats everywhere. The next sentence is. Is the only one I noticed so far that begins with we are confident. And the sentence is we are confident that the acceleration was not due to an increase in the growth rate of of training compute.

Speaker 2

174:13 So that would suggest that in terms of government interventions, you know, who knows what we'll see in terms of data center build out. And maybe we'll see slightly slower data center build out than we would have counterfactually if the election had gone a different way. But it is striking that the acceleration that they are reporting on here is not just a raw material input story, it is an algorithm story and it is a data quality story.

Speaker 2

174:46 And those things are being driven to a very large degree by the intelligence that we already have, which again is where it goes from, you know, not just bigger and bigger capital, but actual kind of recursive self improvement. So if we are going to try to change that trajectory with a government intervention, it's probably going to have to be more focused and pointed right at the practices of the frontier companies than something as diffuse as a broad slowdown in data center buildup.

Speaker 1

175:20 Indeed. And on that note, we will end and we will see our audience again. I might guest host with another host next next week, but if not, the week after. And bye.

Speaker 2

175:35 Thank you Prakash. See you soon.

Speaker 1

175:38 Cheers.

Summary

The discussion centers around the rapid advancements in AI, particularly the implications of recursive self-improvement and the potential shift towards an agent-driven economy. The speakers explore the impact of new AI models on various sectors, the importance of curiosity in data science, and the evolving role of humans in a world increasingly dominated by AI technologies.

- New AI models are expected to significantly accelerate capabilities, with implications for industries and job roles.
- Recursive self-improvement could lead to an agent-driven economy where AI systems operate autonomously, potentially reshaping traditional business structures.
- The importance of curiosity in data science is emphasized, suggesting that human intuition and understanding remain crucial despite AI advancements.
- Companies may need to adapt to a future where AI handles more operational tasks, allowing humans to focus on strategic decision-making and relationship-building.
- The conversation touches on the potential for AI to enhance consumer products, such as food, by leveraging advanced data analysis and machine learning techniques.
- Concerns are raised about the societal implications of an AI-driven economy, particularly regarding the balance between human needs and AI efficiency.
- The need for thoughtful governance and ethical considerations in AI development is highlighted, especially as AI systems become more integrated into everyday life.

Questions Answered

What is the context of today's discussion?

The conversation begins with a greeting and an anticipation of potential new model launches from OpenAI, referencing recent hints and speculation about advancements in AI technology.

What are the concerns regarding fully automated self-improvement in AI?

The discussion highlights the challenges and risks associated with fully automated recursive self-improvement in AI models, emphasizing the need for human oversight.

How do user experiences compare to reported issues with AI models?

The hosts discuss the disparity between positive user experiences reported online and the issues observed in model performance, suggesting a need for systematic evaluation.

What are the expectations for AI model capabilities in the near future?

The conversation explores expectations for AI models, particularly in terms of their ability to assist researchers and improve productivity in various domains.

What vulnerabilities are associated with AI models in enterprise applications?

The discussion addresses concerns about vulnerabilities in AI models, particularly regarding their application in customer service and the potential for exploitation.

What is the current state of GPU availability for AI applications?

The hosts discuss the perceived scarcity of GPUs in the market, emphasizing that this perception may not reflect the actual availability for various workloads.

How are research agents transforming the investment process?

The conversation highlights how research agents are enhancing the investment process by automating data analysis and enriching research outputs.

How has the usage of AI tools evolved in organizations?

The hosts note a shift in AI usage within organizations, with internal tools becoming more prevalent compared to external models.

What are the implications of AI advancements for human roles in the workforce?

The discussion reflects on the dual perspectives of optimism and pessimism regarding AI's impact on future job roles, emphasizing the importance of curiosity and adaptability.

What considerations should be made regarding the direction of AI development?

The hosts discuss the need to critically assess the trajectory of AI development and its implications for society, emphasizing the importance of thoughtful engagement.

© transcribe · For agents Built with care and craft by Gokul Rajaram