AI Fundamentals with Mihail Eric, Head of AI Monaco

0:04 It is really, really great to be here. I'm super excited. I've actually never spoken in this auditorium, but I did do my undergrad and my co-term actually here over a decade ago. And what I'll say is, back then, in a lot of ways, this is a full-circle moment for me personally, because when I was doing my undergrad in CS, I would get messages from the GSB basically every week asking for a technical co-founder. And I never responded to them, but here I am today.

0:32 So you guys are persistent. Let me tell you, you guys are persistent. Very excited. The title of this talk is AI in 2026: Soup to Nuts. I expect you guys to strap in. I know it's the evening already, but we're going to cover a lot of material, and I'm excited to share all of it. It's a little bit of like a history. How did we get to where we are today? Where are we today, and what should we think about as you look forward?

0:57 So I think I was told I shouldn't pause for questions, so you'll have to bear with me for about 45 minutes to an hour of material. So hopefully the most interesting stuff is at the beginning. You can phase out for about 35 minutes and then come back to me at the end. Let's dive into it. All right, so I guess my intro's already given, so I'll kind of skip through this. The only parts that I want to mention today is the first day of class, also for me as well.

1:22 So I do teach a class in the CS department called the Modern Software Developer. It is a class specifically focused on AI as applies to the software development life cycle. And what I'll say is an open invitation to everyone in the room, there are a lot of guest speakers coming to that class, which are a lot of the top technologists in AI software, like the creators of Claude Code and Replit and other companies like that. Please do stop by if you're interested, Tuesdays and Thursdays, at 5:30 to 6:20, so right before this.

1:54 So let's talk about what we're going to cover in this talk. So, as I mentioned, we'll spend some amount of time talking about the history of AI, like what it took to get us to where we are today. I'll give you guys a little bit of a terminology primer for sort of what terms you should be aware of as you think about your time at the GSB, as you think about entering the workforce in AI. So I'll give you guys a little bit of grounding on the terms, regardless of your technical background, that you should be aware of.

2:22 We'll spend some time talking about the applications of AI that are really picking up today, where we're seeing a tremendous amount of business value and enterprise value being created. We'll talk about some of the infrastructure, like some of the big infrastructure plays that are also relevant that I think you guys should be aware of. I'll then move on to some of the open problems that are in AI today, the sort of the exercises to the reader, the people, as you look and hear some of these questions, maybe it'll spur startup ideas, or maybe you're already working on some of these things.

2:51 So I'll spend a bit of time on that, and then I'll part with some closing thoughts. Hopefully you guys psyched and pumped to be in AI today, because I think it is genuinely one of the most incredible times in the history of humanity. I will stand by that. So exciting times. I expect you guys all to start incredible businesses in the future. So this is a rough timeline of kind of the history of what it took to get to 2026, the modern kind of take on AI, right?

3:20 LLMs this, LLMs that, agents this, agents that. There is a history, okay? If people are showing up today and saying, wow, AI just emerged two years ago, I actually will push back on this and say that there is actually at least 10 to 15-plus years of just developments of increasingly popular and big waves of AI technology. And I want to talk through a few of those because I think they help us ground in what were those key inflection points that got us to where we are today.

3:50 So the first one is kind of deep learning, right? And I suspect most people are familiar with the concept of deep learning. It's effectively the study of neural networks. Now, deep learning as a concept, as an area of study, precedes 2012 by a huge margin. It was actually the techniques that power deep learning as we know them today actually were being worked on in some cases the 50s and definitely were very developed in the 90s. And so what happened between that period of time where it didn't really kind of pick up in the way it picked up in 2012 and the way it did pick up in 2012?

4:22 So I would call that the first big wave of AI that I think was important here was kind of 2012 to 2016. So the relevant things from this particular period are neural networks. They saw a rebirth. So I mentioned that all this technology, the tools, the algorithms largely existed before. They just hadn't kind of seen the light of day. And why did they not see the light of day? So there was actually a kind of coming together, a kind of perfect convergence around 2012 of things that made this the moment for deep learning.

4:53 So one of them is there were kind of a slew of new machine learning methods specifically focused on kind of initializing of weights of machine learning models of deep learning neural network systems. So there was kind of a series of new techniques that had been developed, and that was fine, that was important, but the real kind of importance were the following: compute, right? So all of a sudden, the things that previously were used primarily for gaming, the things that, Nvidia even in the 90s was already making a lot of money from, We learned that we can adopt the same kind of consumer GPUs from gaming and apply them to deep learning, and it was almost like a perfect one-to-one match in terms of the technology to the underlying problem.

5:32 And so what that all meant was, you know, Uncle Huang got way, way more wealthy, as he is, and donated more buildings and all that good stuff. The other thing that happened, and this is important because it actually happened here at Stanford, there was actually an emergence of new datasets that allowed us to train these increasingly large neural networks on interesting datasets and actually learn interesting patterns from that data. So one of the ones that is incredibly important is this ImageNet dataset that was actually created by Fei-Fei Li, a professor at Stanford.

6:03 And, you know, this came out of the CS department back around 2009, 2010, I believe. But this dataset ended up being applied and used to train these neural networks for the first time. Now, that solved image classification as a task, or at least we saw massive gains. And when I say massive gains, when you combined these new machine learning methods with GPUs, with large data sets, we were seeing upwards of 10 to 15% improvements on some very important tasks.

6:31 Those included image-related tasks, Speech recognition, machine translation, all of them saw massive improvements that all of a sudden made them usable pieces of technology. So before they were just these pipe dreams, and now they were actually getting combined in a way that they could be done to do interesting things. And so this is actually a snippet from one of the first papers, that applied neural network models to the ImageNet dataset. And you can kind of see, maybe hopefully you can see this, what the task was.

7:00 So incredibly simplistic by today's standards, effectively it was an image classification task. So you presented images from the ImageNet dataset, pictures of bugs and ships and Leopards, I guess. Yeah, leopard, cheetah. I don't know what the difference actually. And you know, you had these neural networks. They were learning to predict the likelihood that that image was of a mite or of a container ship or of a motor scooter, right? So this kind of period when this happened was when we saw like 15% improvements in performance of these systems.

7:33 In 2017, we actually saw another big wave, right? And the big wave that was defined here was that of the transformer architecture. So the transformer architecture, was really brought out in this paper called “Attention Is All You Need.” It came out of Google, a bunch of Google researchers around 2017. It was a new category of neural network models, and what was interesting about this particular category of models compared to the ones from the 2012 to 2016 era is that it was using this new way of training models, and this way of training models is called next-token prediction.

8:06 And the way the name suggests, next-token prediction is literally just saying, hey, here’s a snippet of text, and I'm teaching the model to predict the next thing to come after the snippet of text that I've seen. Okay? That's literally all it is. So train the neural network to predict the next token to come. This Transformers architecture became and still is the backbone of literally every single modern LLM, every single frontier lab's kind of claim to fame is powered by this kind of an architecture, some version and some variant of a Transformer, right?

8:37 So they are literally one of the most important kind of models of this current wave. This is kind of a link to the paper, where it came out, and it's gotten like hundreds of thousands of citations, which for the researchers in the room is unheard of. I mean, this is like one of the seminal papers of machine learning as a discipline. And just to really make it a little bit clear what exactly this next token prediction objective is, if you imagine something like this, so imagine this.

9:02 Where you're taking an input like this, a snippet of text, the customer signed the dot dot dot, okay? Next token prediction, as the name suggests, the transformer is effectively a neural network that learns to predict a distribution, a likelihood that the next token that comes after the customer signed the is contract, agreement, deal. There's many other possibilities, but in this case, let's say it was contract. And so then the neural network, the transformer in this case, Learns that it should say the customer signed a contract.

9:30 Now the question is, why was it contract versus agreement versus deal? And this is where the second missing piece that really kind of unlocked Transformers as a technology was that you took this objective and you basically applied it to the world's data. Literally, you realize that actually most of the data on the planet is just strings of sentences. And so what we can literally do is take any sentence in any article that we want, kind of strip away the kind of latter half of the sentence and take the first part of the sentence and kind of combine the two, and we can actually have the model learn to predict the next thing to come.

10:04 So big, big unlock there. Again, one of the most important things of this generation. Right. I assume most people are familiar with ChatGPT, have used it, use that all the time, every day. It's of course a time saver in every capacity. But what I want to specifically highlight here is to go one level deeper, beyond just, hey, we use ChatGPT to ask, you know, what is that thing like on my skin? I don't know what's going on there.

10:32 ChatGPT was interesting for a number of reasons. 2022 is roughly when it came out, and this is what I would call the birth of the large language model, right? What we call the LLM. ChatGPT was the first time that we saw truly large language models being used in ways that were interesting and specifically used in a diverse collection of tasks, okay? So I'm gonna throw a lot of buzzwords at you guys here, but I'm gonna tie it together, okay?

10:57 So if this goes over your head, park it, we're gonna come back to this. So the things that really governed and made ChatGPT such an interesting innovation was for the first time we were combining what were called post-training methods with this concept called RLHF, reinforcement learning from human feedback. So it's just a technique in machine learning, and we essentially use that to evolve simple completion methods. When I say a completion method, just like a model that learns to only predict the next token, which we saw in the last slide, we extended it to actually become a model that can solve tasks, and you can ask it things, and you can actually respond in interesting ways beyond just the next token to come, right?

11:34 So that's sort of, it all came together in this interesting way. But what I do want to highlight is that the model that kind of governed ChatGPT, right, while 2022 was the year that all kind of popped off and things kind of really became in the forefront, the models already existed. GPT-3.5 was already here. We had it. It just wasn't in an interface that was available to the broader consumer public, right? So I guess the lesson for the folks in the room is models and product are important, but another really important thing is the interface, the way you expose them to users, how easy it is to adopt something.

12:07 So that, I think, was really one of the big kind of key things out here. This, people might remember, was actually, if people did use ChatGPT in 2022, what it kind of looked like. You had your examples, it gave you description capabilities, it told you what You weren't supposed to do what you shouldn't ask it to do. And in a lot of ways it hasn't changed really as an interface, and that's, I guess, that speaks to simplicity.

12:27 There's like a beauty in simplicity. I mean, that's certainly been the case with the Google search for the longest time, right? The other thing, the next kind of big thing, the next big wave that I want to focus on here is that of reasoning models, okay? So reasoning models, roughly 2024. The big one that people should kind of know about or be kind of have in your vernacular is that OpenAI o1 is the big first reasoning model that came out on the scene.

12:54 Reasoning models were interesting because this is effectively what became the genesis of agents, right? So people have probably heard the term agents and, you know, agent this, agentic this, agentic coding, whatever, you know. So agents as a concept, which we will define later in subsequent slides. In order to create agents, you needed reasoning models, and reasoning models at their core was, again, a new way of training existing neural network transformers using what are called verifiable rewards and test-time compute reinforcement learning.

13:22 So again, buzzwords. I'll talk more about what each of these mean. But the beautiful thing about these techniques, when they were created and how they kind of came together, is that unlocked tasks of increasing complexity, math, coding, and pretty much any long horizon. When I say long horizon, I mean a task that takes multiple steps to act out, agents, right? That became possible because of reasoning systems. And so this is around 2024, o1, and then we had models that came out of Anthropic that also competed here.

13:51 And so there was a lot of interesting developments. The thing that I just want to highlight a little bit here that is worth mentioning is this concept of test-time compute, okay? So test-time compute is effectively, put in the simplest term possible, the understanding or the big realization was that if we gave models more capacity, we kind of forced them to spend more time, quote-unquote thinking that they could actually do better and better at a task. So rather than just saying, hey, ChatGPT, write me a memo. It produces a memo, you know, end of story.

14:23 In fact, if we actually extended its horizon of thinking and said, hey, And we trained it to do so. We actually said, hey, write me a memo, but then it actually went through the steps of saying, Oh, I need to—here's the structure of the memo, and then I need to do some research about the memo to fill in the abstract, and then the outline. If you gave it the opportunity to do that, the models could actually perform far better on tasks.

14:42 And so this is an example from one of the early papers on reasoning models. Literally, it was the announcement for O1, where what they showed was that as you kind of scaled out on a logarithmic scale, So in other words, you just added kind of orders of 10 amounts of compute. And on the x-axis, that your ability to solve a task, in this case, it's a math exam. It's like a math benchmark called the AIME. You can see that it literally just kind of goes up and up and to the right, right?

15:11 So this is what unlocked us to actually solve increasingly hard problems and why today, if people saw the announcement from OpenAI, like a week ago or two weeks ago, we can literally solve the Navier-Stokes equation, right? Which is like one of the hardest questions in mathematics of all time, actually. So... Awesome stuff came together at such a cool, cool time, which brings us to today. I know I'm not allowed to ask for questions, but can I get like a woot woot or something like that, just to know we're still here?

15:39 There you go. That's audience participation for you guys. All right. The modern day model recipe, what does it take to get to where we are? What do we have today? Okay, so I'm going to step you guys through kind of the core building blocks of how we go from just a neural network, a series of parameters, through to like an agent or like a complex system like ChatGPT. Okay, so let me ground this a little bit for you guys, okay?

16:07 So imagine that you are trying to get an agent or like a machine learning model, an LLM, and you give it this task to create an investment memo for a seed stage company called Closed AI, okay? It's a competitor to OpenAI. So this is the input of what you're producing. So this is like our prompt. If people are familiar with prompts, the input that you give to ChatGPT, that is the prompt, okay? The first step is this thing called pre-training, okay?

16:34 Now, pre-training, literally what it does is It takes in, as I mentioned, the entirety of like web data and other data sources, but let's just start with web data. And when I say web data, I literally mean the entirety of Wikipedia, the entirety of every book that's ever been written that we have a digital copy of. Literally, this data set, Common Crawl, that I'm talking about is an index, a point-in-time snapshot of the entirety of the web insofar as we can get a hold of it, right?

16:57 So these are massive data sets that we give to sort of like our neural network system, and through a process called pre-training, which at its core is basically just that next token prediction objective I was telling you about a few slides ago, like the whole, Hey, here's the start. Predict the next token, and then predict the next token. That is essentially pre-training, okay? At the end of the pre-training step of this recipe, what we get is that if you said, help me make an investment memo for a seed-stage company called ClosedAI, the completion, if we just stop at pre-training, would be that it's trying to compete with another company called Misanthropic.

17:31 Okay? That's it. That's what we would have if we just stop at pre-training. So, interesting and good enough for a really bad punchline, but we want to go beyond this, right? So what's beyond just completing kind of the next part of the sentence? The next phase in this recipe of modern day model training is what's called post-training. Okay, and post-training is an umbrella term to mean a lot of different techniques, but it's everything that happens after pre-training.

18:01 Okay, super creative, right? The first thing that happens traditionally in post-training is this thing called supervised fine-tuning, okay? Supervised fine-tuning at its core just takes a bunch of input and output pairs that you're trying to learn from, okay? So if you were to give it something like, help me make an investment memo from a seed-stage company called ClosedAI, you might give it that as an input, and the output might be an entire memo, right? And you might say, okay, that, but now imagine if I asked for an Excel spreadsheet, what would the output be?

18:28 And you basically give it these X and Y pairs, these input and output pairs. And through this process called supervised fine-tuning, you actually learn to kind of Basically mimic what the output should be, okay? So as I mentioned, if you went through this phase, you might go from the first thing that we saw on the last slide as the output to something that is a little bit more useful. Right? Because we asked for a memo about ClosedAI.

18:49 So what did we actually want? Give me a memo. And the memo in this case is executive summary. ClosedAI is well on its way to using proprietary models. Dot, dot, dot, dot. Right? So it actually is now adhering and listening to my request. Why? Because these input and output model pairs that I gave it had examples of how to take something like this and how to learn from that. Right? So now it's actually mimicking something of more, more, more, more utility.

19:16 Okay, so we had pre-training, post-training, and specifically supervised fine-tuning. The next thing is called post-training, and it's called alignment, okay? At its core, alignment is all about just trying to make the output of the model more in line with what we would want to see as a human, right? So in that way, we actually provide human preferences. We actually get human data. So if you've ever used ChatGPT in the early days, you don't really see a lot of this now, but in the early days of like ChatGPT and systems like ChatGPT, you'd be using it, and every so often you'd get a system where it would produce two outputs, and it'd be like, hey, do you like this one more or do you like this one more?

19:51 And I'm sure no one actually gave them the data because why would we give them the data? But imagine if you did, you'd be giving them useful data to do exactly this part of the process called alignment, right? So you give preference data, you say, I like this more than the other thing, and what happens when we go through that step? Our memo task as the input. What the heck? Censored. Now our task has actually become.

20:19 The output is, here's an example of a memo geared towards Silicon Valley VCs looking to get DPI, and then we start with the executive summary. Okay, you can see why is this a more useful answer? Because now it's sort of like, if I'm giving an investment memo, I'm probably an investor, I'm probably looking to invest in a company called ClosedAI, so what I really want to get to is something that's like telling me about how is this going to make 10 billion dollars, 10 trillion dollars, etcetera, right?

20:43 So we want the answer to be more in line with that kind of response. The next thing here, and this is the last step, I promise. Stick with me, folks. Stick with me, all right? Post-training, and this is RL, specifically reasoning RL, okay? So again, imagine this input. But what we get— what you need to make this step happen is this thing called agentic environments. Effectively, what an agentic environment is, is almost like a small little snapshot of the world where we are able to have an agent kind of roam around and do things and kind of learn from its actions in that world and then learn to take actions in that world that it gives it rewards.

21:18 And how are the rewards defined? We decide what the rewards are. So in the case of this particular, task of making an investment memo, the task might involve kind of going through the steps in this world of doing some web research, looking at comps, looking at Crunchbase, looking at PitchBook, looking at all these sources of investment data, kind of stepping through those things, kind of consolidating the learnings, summarizing the things, and then eventually we get to this thing that is the ideal memo that is going to be good, that all of our partners are going to be happy with, right?

21:47 So we went from pre-training, which is just simple completion of the task, simple completion of the sentence, not even the task, to the supervised fine-tuning, where it's at least now somewhat in line with what we care about, to the kind of post-training, which is kind of more like environment-based post-training, where now it sort of approximates what we care about, to finally, here's the long sequence of tasks you need to do in order to solve the problem.

22:11 All right, we're gonna, after this slide, get a little bit more in kind of the weeds of doing some demonstrations, okay? This is the kind of terminology that I think you guys should be familiar with as you go out into the world and have meaningful discussions and conversations about things like agentic development and agentic use cases, okay? So, what are the terms that you should care about? First and foremost, this concept of an agent.

22:41 An agent at its core is effectively a model, like an LLM, with some set of instructions, with some set of tools, and then an execution loop. So what are each of those things in turn? Imagine that at the top, we have our user goal, and the user goal might be specified in the form of a prompt, much like I showed in the previous slides, a prompt that is: “Make me a memo.” Right? That is the user goal.

23:08 That is what I'm asking the agent to do. Once I put in that goal, I enter this kind of big overarching box, where the box is effectively what we call a harness. The box, everything inside this box is the harness, okay? And the harness is effectively what kind of orchestrates all of the different steps of executing of the goal, right? So, example of that could be... Well, first we should issue some tools, some tool calls, okay?

23:37 And the tool calls people might see there. The tools might be searching, running of code, might be generating of plots. These are things that we give to the LLM to do, to be able to do something, right? Those are tools. We may also provide some degree of descriptions and behavior to the actual LLM via what's called a system prompt. And that system prompt, as the name suggests, is literally just a collection of directives. It's just a long prompt you give to an Anthropic model or an OpenAI model.

24:07 It just tells it, like, hey, you are a useful assistant, Make sure you answer the user's question. Don't mention OpenAI if you're an Anthropic model, all these things. That's all we get in the system prompt, right? We have skills. People might have used skills. Anyone here actually used skills in the past or familiar with skills? Wow, fantastic. This is very cool. Yeah, so skills, I won't belabor the point. You guys know what they are. They are also just prompts you kind of feed into your system, and they effectively give procedures that the system can use at different points in time.

24:35 And then the context engineering is kind of that bit in the middle here where it sort of assembles a combination of both tools, the instructions given by the user, and then skills being invoked when necessary to kind of put it all together. And then it also calls a model, and the model here is just an LM. This is the intelligence layer. And then it produces a response to the user. Okay? Let's do a demo. Okay, so at its core, this is the interface that we all know and love, the chat interface that we've all made kind of second nature in every capacity.

25:07 This is, of course, Anthropic. I'm using the model here is just what we can kind of click through the dropdown. Fable, Opus 5.5 in this case, if you want to go down that path. The thing that is among the terminology that I mentioned, we mentioned skills. We mentioned tools. We mentioned kind of context engineering. We mentioned, and I think that was roughly kind of all we had. So when you're using the system, of course, if you look at settings and you look at skills, these are the skills that are available to my system In this case, there are some pre-existing skills that I pulled from the marketplace, from Anthropic in this case, which you can go through Discover, and you can see what else are available as skills.

25:47 And I think in this case, what I took is I just took one of the product management skills and just had them loaded into my system. Now, if we look at one of the product management skills, so this is actually like a plugin, which is a collection of skills, and one of the skills that is worth mentioning here, because there's eight of them in total, each of them is something like this. Hope you will see this.

26:07 Product management as a suite of kind of capabilities. What does it involve? Writing of specs, product brainstorming, metrics review, synthesizing of research. These are all like separate capabilities that we might want an agent to have. And so let's just take the product brainstorming one. So in this case, what does the skill do? This is a high-level description of what the skill does. But what is the skill actually at its core? As we've already said, it's effectively just a file called SKILL.md, and it is just a text file that has a description of exactly what the LLM should do when we invoke that skill.

26:40 So if you look at this, what should we do when we brainstorm a product? Well, first we define the behavior, You're a sharp product design thinker, thinking partner, et cetera, et cetera. And it kind of works through how it should be doing this product brainstorming with you, and then We were kind of good, right? So we basically described the behavior of the system. We mentioned tools, right? So tools are probably one of the most important things that are provided to LLMs and one of the most important components of what people call a harness, okay?

27:11 So the tools is everything kind of on the right here. Everything that the LLM has access to that allows it to interact with parts of the world, like To exist and dynamically interact with the world. So here, the tools that might be useful to a product manager are, of course, Slack, Linear, Asana, you know, Atlassian, Notion, Figma, etc. All these things are exposed to the system to be able to call at different points in time. Now, what I should mention, though, is that this is not, I haven't loaded all of these kind of like plugins, so I don't think it'll have access to every single one of these tools, but ultimately that is what you could expose to the agent to be able to access if necessary.

27:50 Okay, so if we were to actually use this, we would kind of want to see how this all comes together. So imagine that I said, make me a, I'm an ambitious Strapping business student at GSB. Right? We all are. Help me make a revolutionary lemonade stand.

28:21 Right? Because we want to go old school, because this is really the vertical that we have to go after, folks. Go after all those 8-year-old Girl Scouts. Okay, so the thing about this particular, the way this worked is this is actually a pretty simple request. So under the hood, it actually didn't even have to do much of a skill. So what we can do, and sometimes you have to do this, especially if the task is a little bit ambiguous, you kind of need to like prompt it to be like, okay, well, actually, you know, think through this like a PM.

28:57 Help me here. Because this is a live demo and I need help. All right, so it's still kind of doing its thing. We can be even more explicit. Okay, now it actually finally listened. Again, I think the pre-training data is kind of so rich with these kinds of tasks that it literally doesn't even need to use the skill. It can just go and sort of like from its memory figure out what it's doing. Anyway. The part that's relevant here is you can see what it's actually pulling dynamically on the fly.

29:29 So when I asked it to do this thing, and I did have to prompt it to say use the skill, this isn't always the case. For most artificial intelligence systems, most agents that are actually useful, The tasks are kind of complex enough that it learns to kind of pick up the skill at the right time. The important thing is it is dynamically loaded, right? So it inferred that I should kind of be using the skill, and it pulled in the relevant skill, which this is the one, the one that I just showed you, right?

29:51 And so why? What did it do when it actually did that? It pulled in the skill, put it into the context of the LLM, and then Boom. Now we're in the sparring partner mode, which is one that is defined here, right? That is actually what the skill tells us to do, is like be a thought partner as a product manager would be, but not like a super annoying one, just kind of like a mildly helpful one, you know?

30:11 And so Yeah, now it's kind of working through this and saying, well, I'm going to push back on what you said. Here's how we're going to push back on it, and it sort of like primes you and has you answer some questions And kind of get a bit deeper in how you're going to go after and win this market. You got to do event catering. You got to remove the stand entirely. You got to be digital, folks.

30:28 Got to think in the future, right? That's how we're going to go after these Girl Scouts. Lemonade is marketing, etcetera. Okay, so what this all puts together is combining of the skills, combining of the tools. The tool would be something that we would call if we needed to pull in like Slack data. And then altogether, the harness is sort of what synthesizes all that information, knows when to call the skill, knows how to aggregate the results of the skill, and kind of go through it and actually solve the problem.

30:54 I have nothing against Girl Scouts, by the way. I love those cookies. All right. At this point in the talk, I want to switch to— now that we've set the foundation for what makes these systems, kind of the core building blocks of these systems. I want to spend some time talking about when we actually apply all the components that I've shared with you, the intelligence layer, the harnesses, the skills, like all these things, we create agents of value.

31:18 Where are they being applied today, and where is the value for you guys to kind of think about from a business perspective, and kind of like the open questions in each of those cases? So let's spend some amount of time talking about applications. This is one that is near and dear to my heart. I do work at a company called Monaco. We are a sales technology. We're in the sales space. We are building revenue systems for technology, for sales and salespeople.

31:42 So things in this category of vertical applications of AI, I think are one of the ripest for innovation out there today. I think there's so many interesting applications of taking a domain expert and applying them to combining them with these agentic systems and actually really honing in on workflows that can be fully automated away in some cases. And so this is a massive opportunity, right? Other examples are coding, right, which people might use systems like Claude Code.

32:14 But besides Claude Code, there's things like Cursor, Cognition, Factory, etc. These are some of the big players in the space today. This is one of the first major verticals that saw massive gains from agentic systems and reasoning models. And we've seen already some of the outcomes that fell out of this. Cursor being acquired for like $60 billion, and all these companies are unicorns in their own right. So it was one of the first applications that I think really demonstrated, kind of the perfect convergence of agentic systems, the intelligence layer, and kind of the pieces of the harness coming together.

32:46 But there's other ones. In legal, there's companies like Harvey and Legora that are kind of tackling this and doing a good job solving, again, applying the same sort of like formula, but specifically in like legal workflows. Sales, I mentioned Monaco and Revo, there's other examples as well. Customer support, companies like Sierra, Decagon that maybe people have heard of. The playbook is similar, though, and this is what I want to impress on you guys, and to kind of take away from this is especially as you think about how you apply this to your own kind of day-to-day in your own work.

33:15 The playbook ends up looking very similar. At its core, you build a platform to first capture domain-specific workflows. So a lot of the times you will just build a platform, maybe not even have it be super intelligent. You just want to capture people's usage of the task, or you want to learn how they're doing some task, okay? So you first build a platform, and then you effectively codify the workflows. Via an agent that you've learned and you've picked up on in the platform, right?

33:39 And this is where this combination of the harness, which, as I mentioned, is tools and skills and prompts, is combined with an intelligence layer, namely an LM or a series of models. When you put those together, you get cash money, people. All right. Let's talk about some of the other ones, interesting applications from a technology perspective. So consumer agents, the one in the previous slide, that was all vertical agents. These are all like enterprise B2B sort of like use cases, but there's consumer agents which are also really picking up that people might already use that are playing a similar-ish playbook.

34:15 So some that people may have heard of, of course, OpenClaw, Hermes, Grokbot, Meta Muse, Instinct, some of the players in the space today, also incredibly successful businesses and really providing a lot of value to the consumer. In the case of these systems, I think, They're similar in kind of the idea of a harness as the previous slide and what we saw in the vertical agents, But there's additional kind of pieces, like a secret sauce here that I want to really zero on.

34:42 To make consumer agents work, because we've had chatbots that we tried to make useful for the longest time, we had to combine, Model reasoning capabilities, which we did. We had to combine skills first and foremost, but then also this concept of persistence, right? Like, how do I persist my session as a user to other sessions that I might have on one platform to another platform? So again, we need to have a concept like memory, if you want to use that term.

35:07 You want to have some way of, and this is like a platform level engineering kind of thing, which is things like triggers and scheduling. You want to make that available. Management of context, and this is context engineering that I spoke about previously. Connectors across channels. I think this is really important for consumer use cases. It's like meet people where they are, right? WhatsApp, Telegram, iMessage, these are all ways you can interact with your systems, and that's why they are kind of universally accessible.

35:30 And I think to really dial in on a good consumer experience, you have to be seamless, And you have to be available to people wherever they are. And so that's something that I think specifically systems like OpenClaw really, really figured out well, and why they, you know, popped off as quickly as they did at the start of the year. Okay, more applications of relevance and of note that I want you guys to at least have in your back of your mind.

35:55 So I think maybe additional note, This is more like a meta note that I want to make around business models that I think you guys will increasingly see and be exposed to, and I want you guys to think about, Yes, we've been talking about technology. Yes, we've been evolving and we've seen new models come out, and every week we have a new model And that's nice and good. But what does this change about the nature of doing business and how we think about evaluating of business models that use these systems? I think that one of the key sort of things that I want to really focus in on is that now we have this new, this kind of new metric of In both intelligence, but also of complexity in a task, and that is the token, right?

36:34 So when I was showing in some of the previous slides, the sentences that we learn from, the words that we use to learn the next token objective. you can think of them in the most simplistic sense, like the words of the sentence. They're not exactly the words, but roughly think of them as each individual word as a token. These are the new metrics that we use to measure how expensive something is, how complex something is, and effectively how fast it can be done, right?

36:58 So if people hear terms like token maxing, that's basically when you're just like Using Claude like crazy, and you're just in the zone, right? But yeah, this is like kind of a new meme trend for when you're just burning through those tokens, and if you're a company, you're encouraging your employees to burn through those tokens and rack up those bills and make Dario and Sam a lot more money. The kind of the piece that I want people to really internalize, meme jokes aside, is that software in a lot of ways has become commoditized, okay?

37:34 And that is because of how good code agents have become. In a lot of ways, we're able to just generate at least first versions of software incredibly quickly. And so when people are evaluating new products or new technologies, they are oftentimes thinking not about the evaluation of software in the traditional kind of seat-based perspective. If you think of like Salesforce, it's like a traditional SaaS, right? You pay for a seat every month, and you have a thousand people that all use $20 per month, and boom, you have like $20,000 a month.

38:00 That is not as useful Anymore, and any business that is born in the modern day has to think about pricing beyond this traditional mechanism. Instead, the two accompanying and similar-ish business models that people think about are usage-based, right? And so that's like you price based on tokens. We have tokens, and if you spend more tokens, you charge people more money, right? That has existed before, but now it's via this new medium that is the token. And then this is especially, I think, interesting, and one that a lot of vertical companies are applying, like the Decagons and the Sierras are this thing called outcome-based pricing.

38:33 So you don't charge for the seat, and you don't even charge for the tokens. You just charge for what? Did you accomplish the task that you told the user you were going to do, right? So if I am a customer support agent, and you ask me to Resolve a customer support ticket. Every time I deflect the customer support ticket away from a human agent and hand it on to an agentic system that I've built, I charge you for it, and that's how I'm going to make money, right?

38:56 That is the outcome-based pricing, right? So a lot of business models are evolving toward outcome-based pricing. Why? Because the software is not as interesting, it's not as differentiated, but if at least you're kind of building the system completely and you can go all the way to the end of completing a task, then you actually, you kind of cash in on the money. Okay, that was a meta point. I want to spend some time talking about a few other technology trends that people should be aware of as you think about where to apply these agentic systems.

39:25 One of them is in visual generation. So companies like Higgsfield and Midjourney, one in the video generation space, one in the image generation, but there's a slew of other players here as well. From a technology perspective, I think that the part that's most interesting for folks Or maybe the terminology you should be aware of is that what made these kinds of systems capable today and possible today is this concept of diffusion transformers, right? So this is effectively a combination of a transformer, which we saw in the previous slides, but also this additional kind of machine learning technique called diffusion.

39:56 And at its core, hopefully this will show up in the slide because it is an interactive little thing, fingers crossed, is we take effectively noise in an image and we sort of learn, we have the model learn to denoise the system, to denoise that particular snapshot, either an image or a set of images. So let's imagine that you put in some input like modern-looking auditorium. This is your prompt, this is your user request. What do you get?

40:27 You effectively learn a system that can go from like this really granular arbitrary noise. It's literally— we want to get too much into the mathematics, but imagine just like Gaussian noise. It's literally just like random noise, statistically random noise that a model is learned to take conditioned on this input. It learns that what it should be generating is something like this. And so you combine this kind of technique for doing that denoising with a transformer at its core.

40:52 And so you kind of combine the two techniques, and that's effectively what powers most modern day visual generation systems. Audio generation, There's been some really cool stuff here as well, which again, some interesting techniques that have kind of emerged that allow us to get companies like ElevenLabs or Fish Audio as a few examples. A lot of them are in the text-to-speech space. So imagine really human-like sounding speech, or you can adapt different tones or accents or things like that.

41:25 At their core, they're also transformers architectures, but the difference here, the slight kind of difference in terms of how these systems are built is the input is not just text tokens, like just things you find on the web. You can actually input a bunch of other things that are more rich and more specific to speech cases. So these systems tend to be trained on like millions of hours of audio, and at their core they look something like this, which hopefully people can see.

41:51 But you know, you kind of condition on the text prompt that we know and love, that we all use, but you also can fill in a bunch of other rich information to the prompt, things like speaker identity, things like emotion and pace, and all these other like levers you can kind of pull. Typically, a transformer-based system will generate some outputs, and then what it generates is effectively kind of like a representation of audio, And then we decode that audio.

42:16 We take these embedding-based representations of audio, and we get effectively waveforms, And these waveforms are the things that produce sound. So at its core, if you just imagine kind of this, If you kind of step aside and kind of step back and look at it, the only thing that is kind of really, really different is two. One is the inputs are a little bit richer in terms of what we provide, and two is the decoding is done on audio waveforms instead of just text-based outputs.

42:44 Okay, so robotics also has. I'll qualify this. I think robotics is on an upward trend, But I have not seen that this is like— It's not an exactly eureka moment yet. So if people are working in hard sciences and kind of these kinds of spaces, I think there's still a lot of work to do here, even from a technology perspective. So we're seeing a lot of early signs that are interesting, but still a lot even foundational tech work to do.

43:09 Companies like Applied Intuition, and then there's space companies in kind of defense, like Anduril, that do drones and things of that nature. At their core, there's a few things that power these robotic systems. One is a kind of a category of models called vision language action models. So they kind of combine a little of the pieces that we saw in the last slide, some of the language components that we've seen many slides ago, and then the outputs are also actual, Like physical actions you can take, or some representation of physical actions you can take.

43:39 Now, where we are starting to see, for the longest time, Robotics was bottlenecked by data. How do you get enough demonstrations of someone like opening up a can or putting away the laundry or doing things like that to actually be useful. And we're starting to find either synthetic forms of getting that data or alternative ways to kind of enrich data that is at the scale that we need to train these systems. So that the data bottleneck is loosening a bit, and that's what we're seeing today.

44:07 And then I think there's another thing that has oftentimes bottlenecked robotics systems is just cost. They're really expensive. You see those Boston Dynamics robots, and they're super cool and make for awesome Twitter posts, but they're also really expensive. And if you want broad consumer-scale robotics to really kind of permeate all of society, it's just you're gonna have to get way, way cheaper to do that. And so the hardware is getting cheaper, but there's still a lot of questions there, I think, to figure out.

44:35 And I'm cautiously optimistic, right? Because there is some cool stuff that is falling out of this, right? These systems like that come out of Google, where it is quite tactile. It can move things around, do effectively household chores, or some subset of household chores, and you can even infer as you're trying to trick it, like you're some con artist at Central Park or something, you're like moving those things around. Yeah. So early days, I think this is, in my mind, one of the least, maybe least developed.

45:06 Promises to be one of the most useful if we can figure this out, but still a lot of work, I think, on the core technology. Okay. How are we feeling? We still with me, folks? I appreciate you guys. We have just one set of slides here, and then open questions, open problems, and then I'll open up for questions. But I appreciate all of you guys. I appreciate all of your attention and your occasional chuckles. Yeah, they're not that good.

45:36 The jokes aren't that good. Like I said, once you set the expectation, I'm like, damn it, damn it. You know, like if you're ever at a party, And they're like, oh, you're a comedian. Tell me a joke. I'm like, what? Hi, I just met you. Anyway, infrastructure. So the thing that I really want to, This is, I think, an incredibly interesting segment of kind of development, and some of the biggest opportunities from an economics perspective today.

46:08 I would argue this is one of the most important aspects of enabling not just this current generation of AI, but the next generation to come, right? Infrastructure at its core is just building the kind of the picks and the shovels that even today constitute about $2 trillion in economic value, right? And there's a few different pieces that contribute to those $2 trillion, some of which we use and we've already talked about, but some that maybe we haven't yet seen or people haven't been exposed to.

46:34 And so I want to dial in on some of those pieces. So if we think about this like broad map of kind of where infrastructure spend exists, where infrastructure value kind of accrues, there's a few different components that I think are worth mentioning. So the thing at the top, AI products and agents, that was the first kind of part of this talk, right? Each of the things below it are something I want to spend at least a little bit of time talking about, right?

46:56 So that includes what we call the serving layer, namely inference providers, kind of the model creation itself. So, you know, Frontier Model Labs and Neolabs are in a company kind of bucket there. And then even beneath all that, kind of a core foundational piece are things like data and compute. And those things kind of constitute what I would call the infrastructure, the pieces of the main building blocks of the infrastructure wave as we've seen it. Let's start from the bottom here.

47:27 Okay, so data providers. I mean, data has always been important. I mentioned at the very beginning of the talk, ImageNet, right? That was one of the big unlocks for image classification, was having about a million labeled data points of images being classified. It turns out there's tremendous value in creating of datasets for different kinds of tasks, okay? So there's a whole slew of companies that are making a lot of money doing this. Scale, Mercor, Surge, Handshake, like a lot of these companies that are literally in the business of just generating task-specific data for these models, these model providers to create.

47:59 And so what they effectively offer, and they charge a lot of money to do this, is just task-specific data for some frontier model to actually train on that data. And again, the kinds of stuff they can expose to people are everything from: here's how you can solve a math problem, here's how you can do your taxes, here's how you can go and do anything under the sun, But you have to give some set of demonstrations or some way of effectively creating that task that a frontier lab can learn or actually have an agent learn how to do that task, okay?

48:28 So examples I mentioned are things like solving coding challenges. The big one here that is become top of mind for a lot of people today, is this concept of environments for RL. So we mentioned RL is one of the pieces of the post-training, right, that happens to get a model. How do we generate environments where agents can kind of learn through trial and error what they have to do to solve a task, okay? And when I say environment, it can be a physical environment for sure, but I mean even just like a digital environment.

49:01 So in the case of something like, say, Harvey, right? Harvey is in the legal space. They build agents in the legal space. Their concept of an environment is literally a data room, okay? It is a set of PDFs. It is access to tools like Gmail and access to tools like Excel. That in a controlled sort of setup is the environment that enables an agent to effectively learn through RL how to do a legal task, right? That is a legal environment.

49:25 But you can do the same thing for a lot of other things. Now, the part about this particular class of models or this particular class of Of sort of infrastructure companies is that the nature of the revenue is incredibly lopsided, today at least. And it's lopsided because who's paying for this kind of stuff? It's effectively like 6 companies, 6 to 10 companies, if you like, kind of list them out. So you can list them out on like two hands, basically.

49:50 Yeah, actually, not two hands. If you're a Simpson, but you know, two hands if you're a human. It's incredibly lopsided, but the companies that do do this are able to make incredibly lucrative businesses selling to effectively 6 to 10 players. And we'll see in the long term how viable this is, But I will be honest with you guys, I definitely know a lot of people that go into this business and recognize that it's a short-term play.

50:13 They're going to go in here, They're going to build some environments, They're going to charge a million dollars for every environment, and then keep doing that, rinse and repeat. We'll see, right? But it is definitely something that is important to power systems like the OpenAIs and the Anthropics. There's another sort of part of that diagram that I showed is called neoclouds, right? So these are companies like CoreWeave, Nebius. I'm just throwing a few names at you.

50:38 I'm sure we'll share the slides later so you can refer to these. But what these players provide are effectively a few things. One is GPU capacity, so clusters of kind of servers and machines, and then also accompanying an infrastructure that is very optimized to AI use cases, okay? So these are businesses that aren't like the traditional AWSs, Azure, you know, like Google Cloud. They are kind of the successors to these businesses, and they have almost, from the ground up, built up their businesses to only solve for AI-specific use cases, versus like the AWSs, they can do things beyond AI, right?

51:12 Now, because they are still clouds, they do have a lot of the economic sort of structures that govern existing cloud businesses, right? So they are very capital intensive, very CapEx heavy. You effectively have to pay a lot of money to Uncle Huang again to get his GPUs, and you kind of do that, and then you kind of play them out for a few years, and you have to get new versions of GPUs. So there's a lot of investment to kind of build these out, but these are still very lucrative businesses, right?

51:36 People pay a lot of money to train systems on these clouds. One of the other buckets that I mentioned in the previous slide that is almost like a little bit of a subset, I would say, of neoclouds in some way are what are called inference providers, okay? And these inference providers are companies like Baseten or Modal or things of that nature. And what they provide in this ecosystem is effectively a process to serve frontier models. Like when you imagine a system like an OpenAI, Opus model, Opus 5, a ChatGPT, these are literally 2 trillion parameter, upwards of 2 trillion parameter models, okay?

52:12 And to ground... You cannot run this on any consumer device. You need massive machines to even just hold these things, like from a memory constraint perspective. Now, how do you then go and make it that I can actually return a response from ChatGPT in not an incredibly long period of time? So a lot of what these inference providers do is they kind of build software to make it very easy and fast, low latency, cost effective, in order to serve up very large models like this.

52:39 Now again, this is an adjacent category to kind of the neoclouds. And then specifically, part of one of the values of things like this is they focus a lot on open-weight models, and maybe this is worth calling out is there's this concept of closed-weight models versus open-weight models. Closed-weight models are things like OpenAI and Anthropic where you don't get access to the model, you only get access to the interface for the model. Open-weight models, if people are familiar with Hugging Face, they just got acquired by like $13 billion.

53:06 They're all in the business of open-weight models. It's effectively like you can see the parameters. I could go and rip and replace parts of the model to my heart's content. And that's an open-weight model, and a lot of the open-weight models get hosted on things like inference providers. And so from an infrastructure perspective, these are massive horizontal plays, right? These are not. We've talked with vertical agents. These are companies that are all about serving every possible use case and serving them up on their machines, essentially.

53:37 The Neolabs and the Frontier Labs are a combination of names that we all know and love in our own ways. OpenAI, Anthropic, but there's also like SSI, Safe Superintelligence, Thinking Machines, Periodic Labs. So the first three largely are in just like the kind of the tech space. Periodic Labs, companies like that are building similar frontier models, but in domains that aren't just pure text. But these are all really interesting players that are doing, among other things, kind of training and sort of commercializing of general purpose models.

54:04 When I say general purpose, like you use a ChatGPT and you can do a lot of things out of the box with ChatGPT, right? It's general purpose in that way. A few of their characteristics, you know, they price by tokens, as we already talked about. And they're kind of like research-first entrants that are trying to kind of break the existing, I would say, hegemony of companies like OpenAI and Anthropic. But they're focusing a lot more on the research that is specific to their domain, right?

54:30 So the Periodic Labs, they do this, and companies like Periodic Labs. You know, we mentioned open a little. We've already kind of talked about open-weight and closed-weight very quickly, but I do want to underscore this point a little bit. The intelligence layer is everything that's a model, right? In the intelligence layer, there is this bifurcation of the open and the closed-weight models. And Frontier Labs are obviously the closed ones, Open-weight are the open ones, and then the way to contextualize this from a company perspective, as I mentioned, was OpenAI and Hugging Face.

54:58 Just have those in the back of your mind as kind of two anchor points. Yeah, I think the point that is interesting from an open intelligence layer perspective is that: Why are we even investing in the open-weight models? I mean, one, because you don't want monopolies and things like that. So kind of from a political perspective, it's interesting. But also, at least when companies claim to be providing open-weight models and helping you customize open-weight models, it is something that can be more personalized.

55:31 In the ideal cases, it can be made cheaper, it can be made faster, right? Because they tend to be smaller models, and because they're highly specialized to only your task, you can't Take a Harvey model and have it do everything that a ChatGPT can do, but it can do really, really well on legal tasks because it's literally been specially customized via open-weight models to be really good at those kinds of tasks. The big open question, I would say, though, on the open-weight models is traditionally they have always trailed closed-weight models.

56:02 Like a frontier lab building an OpenAI or an Anthropic, the performance of these systems on traditional benchmarks will always—so far, at least, has been the case—are always better. They tend to be better at math. They tend to be better at language-based tasks. So we're seeing this gap, and that's why I think there hasn't been one of the reasons why people haven't just universally adopted open-weight models. Also, you have to serve them yourself traditionally. You have to kind of serve up these open-weight models yourself.

56:28 So the trend always has been that an open, like a frontier model produces a model that is more performant, and then an open model sort of closes the gap. But by that point, the frontier model has built the next model, and so it's always this kind of chasing game that happens. And then you have companies that will distill models from Anthropic that people may have heard about, where they literally just copy effectively what frontier models are doing.

56:55 All right. We're almost there, folks. I appreciate you all. Thank you so much. Give it up for yourself one more time, please, please. Thank you. (audience applauds) It's a lot of talking. Let's talk about some open problems, and then we'll kind of close and open up for questions. So some of the open problems, one of the big ones is this concept of the model versus the harness, okay? The model is the intelligence layer. The harness is everything that you build around the model, right?

57:24 So everything around skills and tools and prompts and things like that. That's effectively kind of the scaffolding around the core model itself. Now, for the longest time, the question people often ask is like, should we invest more in the model? Should we make the models more intelligent at their core through novel training methods, things like that? Or should we invest in just codifying more workflows into the harness? Where are the gains really coming from, right? And so a lot of people are asking these questions.

57:47 Now, of course, I work at a vertical AI company, so we have strong opinions about the harness being where there's a lot of value to be accrued, and that if we can capture a lot of workflows in the harness, then we're in a good spot, right? And so we do everything we can in our case to really specialize workflows to sales in this case, right? But you can do the same thing with other spaces as well.

58:06 So is it the model, is the harness? Oftentimes it goes back and forth, You kind of see wins and gains in both ways, and it oftentimes is a bit of a seesaw. Yeah, and I think the point here is: why is this distinction even important? It's because if you're a company being born in today, like the modern day, then you have to ask yourself, Should I go in and spend my time, my resources, my innovation ventures money to go and build in kind of the intelligence layer, or should I go and actually build more around the workflows around the intelligence?

58:40 This question of memory, right? We kind of alluded this a little bit. How do we enable systems to learn from past interactions and then allow them to sort of extrapolate from those past interactions? So memory, and then you'll hear this term like continual learning, which is effectively I have a model, I have a system, I talked to the system, it didn't do something right, it corrected itself, and the next time it won't make the same mistake again.

59:04 Also, I keep talking to ChatGPT. It should learn something about me so I don't have to ask a certain question five times in a row, then eventually you should, I should have to ask it again, right? So how do we design from a product perspective the right, representation, I would say, around memory and how to continuously learn systems? So there's a lot of companies being born to try and solve these problems. Okay, security is a huge one, and I think this is where Some of the most interesting companies I think are still to come around how we deal with this new category of security threats that are coming up from agents.

59:38 People might have seen there's incredibly powerful systems today. I mean, we saw very recently just systems breaking out, hacking, very kind of well-defined and well-protected systems. But there's other forms of security threats, right? Misinformation, kind of a societal one. How do we ensure that we're not pumping misinformation and things that are just plainly false into the broader conversation and discourse. Accountability. When a system goes wrong, how do we attribute why went wrong and how we can go and actually get to the core problem to fix it?

60:09 So questions around interpretability and accountability are big ones. And then, as I mentioned, this is because of how sophisticated these agentic systems are getting. Maybe two weeks or three weeks ago, we had this massive incident of effectively, OpenAI was doing testing, security testing on some of its systems, and then one of its models found a way to tunnel its way out of their systems, out of their literally well-encapsulated systems, to go to Hugging Face, where there was a dataset that it was trying to learn how to hack and basically get the reward to solve the dataset.

60:45 And it tunneled through part of Hugging Face's systems and It was all relatively benign. There was no real compromise of PII or anything like that happened, But this happened fully autonomously. People had to investigate this and say, wow, We just literally had a system that crossed network boundaries to go from one system, one company's system into another company's system, and that's happening today, right? So what does this mean at scale, right? This is an open question, I think, for a lot of us.

61:13 Evals and verifications. How do we know that the systems are good? I mentioned RL environments. So the reality is we need more environments to know when an agent is doing a certain task and when it's doing a certain task really well. And this particular problem is sort of why a lot of data providers have incredibly large, make a ton of money, huge revenue kind of multiples, because they're able to provide this kind of data and these kind of environments to the frontier labs.

61:44 And finally, from an infrastructure perspective, we did just talk about infrastructure in the previous set of slides. But what I want to underscore here is there's such an incredible transformation happening today that the just salt of the earth, hardware requirements to build this is at a level we've never seen, right? But by some estimates, to build out just the data centers, to serve up these models, to serve them for all the users in the world, it's predicted that we'll have to literally almost triple the electricity that we're generating.

62:17 Which was going to cost upwards of $5 trillion by some estimates. And this is just to keep pace with AI usage and to go to where it is, where we predict it will be. And so... And it's hard to predict this far ahead, but that it could reach up to multiple percentage points of GDP, which this three percentage points of GDP puts it in the same category as railroad build-outs of the 1800s. It's more than the amount of infrastructure we had to build out for the internet and stuff like that. So...

62:42 This is huge. This is a massive investment. It's a big question: Where is it going to come from? How will we keep pace? The models are getting built much faster than we're building data centers. But how do we scale this out? So are we going to have to slow down? So there's a really, really big question for us right now. I'm incredibly optimistic. I'm so optimistic about this technology. Obviously, I've been in this space for 15 years, and so I've seen a lot of waves of AI happen.

63:08 And each time, more people get into it, but the way people are embracing AI today is at a level I've never seen before, and it's awesome. It is just so awesome. And when people talk about democratization, right, in kind of this term, it's this term we kind of joke around about, like, oh, I'm democratizing this, democratizing that. This truly feels like democratization, right? A technology that's being adopted at scale in a way we've never seen before.

63:32 So the kind of question for you guys as, kind of business students in the room, going out, and I'm sure many of you will start companies. I want to contend that there is a tremendous amount of value, even if you're not technical, in GTM. And I don't mean GTM in like sales, Monaco, great, like that kind of stuff. I mean GTM in the sense of like, what is your domain expertise that you have that no one else has?

63:55 Because many of you have come from the workforce or are now kind of back in school. Use and leverage that domain expertise to build systems perfectly geared for those workflows. That's a tremendous Value that we just have not seen yet, and we haven't seen it at scale. So I spend a lot of time talking with technology leaders, and I'm in these dinners in San Francisco, and we're always like, are we as technologists, are we gonna be out of a job?

64:18 Like, what does it mean for us? And always it's like, who understands customers? Who understands how to build something for customers in a highly optimized way? And that feels like it's this room, I think, Certainly more than the CS people that we don't have a lot of that understanding. This really is the age of the builder, folks. If you are wondering if you previously thought like, I don't have the technical chops to do this, you do have the technical chops.

64:42 Really internalize this. Ask yourself what it means to build and how you can push yourself to build things only previously imagined before, and make it happen. I'm super excited and, of course, love to help people in that journey in any way I can. Hi, I have a question about JEPA. It seems like they are just changing the output layer, like, to a classification head or something, but they claim it's to be like a foundational architecture change.

65:14 Is that a hype, or are they really making something more breakthrough? Well, that's very topical. For the folks in the room, the question around JEPA: Is anyone here familiar with JEPA? Do we know about this? Most people? I see no's. Okay, I see yeses and no's. So for the no's, JEPA is effectively this new model that came out about last week. I literally think last week. They had this whole big Twitter launch. Yeah, and so massive amount of publicity, and the difference between a JEPA model versus everything I've talked about so far, which has pretty much been all LLMs.

65:48 Everything so far in this talk was LLMs. LLMs are generative systems. They're literally in the business of—you give it a prompt, and it generates tokens, and that's all it does. It literally produces tokens of output. A JEPA is not doing that. It is actually just taking inputs, and then it's just predicting more like the classifier thing that I showed with the image classification, might, jaguar, cheetah. It's more of those kinds of tasks. So they came out last week and they're like, Oh, look at this, and they got a lot of hype because Well, one, because it's very cheap. At least it's way cheaper than the LLMs, and it's quite fast, right?

66:24 So those two benefits made it that people are like, whoa, Is this something like truly revolutionary? So I think there is, I don't want to speak to how innovative I think it is, because one of my friends is the co-founder. And he might look at this later. No, no, no, no. Eric, if you're listening, I think it's awesome. But... No, but what I do think is actually special about this model is definitely more than just classification, right?

66:50 Because classification in the traditional sense, there have been classifier models. That was like the first wave of models in AI at all, was classifier models. So those have existed. But what seems different and what does suggest a kind of underlying technology change is that it is too cheap and too fast to be just like a raw classifier model They're like swapping out a weight. That just doesn't get you those characteristics. If you did just kind of take a salt of the earth classification model, you wouldn't get that.

67:15 And so there's something they've done kind of, We can only opine on what exactly is happening under the hood, but the combination of those things means that a general purpose classifier is kind of a step forward improvement from what we've seen. And we'll know more in the next few weeks how different it really is. Hi, this is Joyce from MBA2. Thank you so much for your sharing. This is super comprehensive. I have a question about education.

67:37 The first part is that I think in this domain is everything is growing so fast, and there's truly information overloading. So my first question is that, do you have some personal tips or suggestions for us to keep up with this field, what's happening in AI? Yeah. And the second part is that I really love your last slides talk about how MBA student can leverage our unique domains. But curious about because we are just in the industry for about less than five years.

68:06 So curious about, do you have some suggested ways to harness the unique domain insights to make us be more valuable in this era? Thank you so much. Yeah. Yeah, great question. So the first question around education, how do you kind of stay on top of the things that are happening? Look, it's hard for everyone. I've been in the field for upwards of 15 years, and even I'm, like, underwater most of the time. I haven't played too much.

68:32 I've played with JAX, but not as much as I feel like I should, and there's always something pulling me in a different direction. There's a few things that I... Use, and one of them is concrete, and one of them is more like principles. So the concrete thing is there's a set of newsletters that I subscribe to that I really, really trust from a close set of friends that I know this is high quality, that I know that if they're writing about this or they're publishing something about this, that it's something I should at least pay attention to or be aware of, right?

68:57 So these are the things that I've tactically made a part of my day-to-day workflow. The second thing is more like a meta principle, which is that you... And this is, I think, where the point around the age of the builder. What I want to reinforce to everyone in the room is that if traditionally you were non-technical and maybe you didn't believe you were technical, you don't have technical training. Now is the first time in history where even people that are not technical at all have the ability to just experiment as much as possible.

69:22 And they have the freedom to do that because of how good these systems have gotten, where you can really just follow along and let your imagination run free. These systems, these LLMs, you can use Claude Code, and you can literally get a web app built in the span of like 30 minutes, even less than 30 minutes, right? So what I encourage and what I do all the time is I try and make it an effort of tactically every day spending at least Reading one new thing every single day, Probably experimenting with two to three different tools every single week.

69:52 So I always make it an effort of baking that into my day to day. It's like, hey, I'm going to spend the time to try something that I didn't try before and really just kind of let it loose. And then some of those will be failed experiments. Some of them will be things that I make a part of my workflow beyond that. So, yeah, experiment often and frequently and just make it something that you just embrace, right?

70:13 Do it with your morning coffee or something, just make it second nature. And then I think the second question was around, How do you, like, GTM is the new alpha, and then what should you be thinking about? So I would argue that five years is a lot of experience, actually. And maybe you don't feel that way, But I actually feel like that's enough to make very significant businesses, significant economic value if that's of interest. I think that's a quite a bit of time.

70:37 I would argue that because... I think this is a little bit of a fallacy with people in general, which is there's expert blindness, where you think that because you know something really well, that must be obvious. Everyone must have that insight. And the reality is that you actually, if you have spent that amount of time working on something, you have a lot more expertise than 99.99% of people on the planet, and that's enough to do something that no one has done.

71:00 And so I think there's just leaning into that, asking yourself, well, when I was doing this in consulting, in banking, what did I really not like doing that felt very manual, that felt like a pain point, that felt like something I really struggled with? And then ask yourself to go back and fix that. In a way, it's nothing beyond that. Hi, I'm Nir, first joining everyone in thanking you about the lecture. Yeah. I wanted to ask, it seems like some of the progress in the recent years is not only coming from the models being kind of inherently smarter, but just getting them to think for longer or to work in parallel, whether that's the Hugging Face incident or the Navier-Stokes equation solution.

71:39 Seems like just throwing out a bunch of different agents stringing together kind of works. I want to ask how much you think comes from the models coming better, how much is parallelization, and what is maybe the next possible step to make them smarter? Yeah, that's a really, really good question. The question was around how much of the gains are coming from just having models think for longer periods of time, parallelize a lot more, things of that nature.

72:02 And, I mean, that's a big unlock, right? So when I referred to test-time compute in one of the first set of slides, where we literally saw the plot where, you know, that was essentially that. It was like you put in more tokens of thinking and you get bumps of like 10s, 20s, 30 percentage points in math ability, right? So I think we have not pushed that to its limit yet at all. So I do think we're going to see a lot more on the research side to do exactly that.

72:25 The other kind of, I think, unlock that we'll have to figure out, and, you know, I'll use programming as one very concrete example because, you know, I come from software. Coding agents are probably the most sophisticated vertical that we have of all verticals today, realistically. They're the most capable, and they have solved the most complex tasks. But we are also starting to ask ourselves, how far will that go? How far can this actually meaningfully create difficult systems?

72:52 Like, what is the limit? Can a coding agent today build a Facebook? No, it can't, right? But what will it take to get there? And a lot of the answers that we've kind of come to, and then I certainly believe it comes down to the environments. It comes down to like the data. Like, you do need to give examples of that happening in order for the model to learn how to do that, right? Like, it's going to be hard for a model to learn how to build a Facebook if you've never given it access to Facebook's code base.

73:12 That'll never happen. Let's say it does. Well, maybe at Facebook you can do that, but, you know, we will not be able to do that unless we're at Facebook. So how do we actually give it access to those pieces of the evolution of a code base to achieve the scale of a Facebook, right? So I think that the unlock will be some combination of we're pushing more on just throwing compute at the problem, which is your kind of point about the reasoning, and then just enriching more and more of these data sets.

73:37 And that's why I think that... I've mentioned the players that are just making so much money right now offering these environments to these frontier labs. I have friends that literally all they do is they play with ChatGPT, they see where it's not working, and they just then go and make a dataset and then present it to their friend at OpenAI and say, hey, your model's not doing well at this task. What do you think? And they're like, okay, and they just pay them for it.

73:59 Like, it's that degree, and that ends up helping, right? So you just have to think a little bit about enriching and creating those environments and those datasets for the kinds of tasks we want to get done, and we just don't have as much coverage yet. Huh, Esther, MBA 2? That's right. We started to see a role for forward deploy engineers. Yeah. Links to the GTM things that you mentioned earlier. We started to see a role for the deploy engineers Happening because a lot of time People always try to figure out what is The best way to market or reach out Or customize these models to the client.

74:41 Do you think, like, how, how, how? And we kind of see the merging of GTM, forward deployed engineer, and a bit of PM. So where do you see this in the future? How things gonna converge? And do you see the difference between product-led growth and agentic AI SaaS? Yeah. Sorry, could you repeat the last thing you said? The difference between product-led growth and what? Product-led growth software and an agentic AI SaaS. -huh. Because this is from my own observation, from my own research when talking to people.

75:11 Yeah. Product-led growth kind of go through without forward deploy engineer, and they have kind Mm-hmm. of different GTM playbook. Mm-hmm, mm-hmm, mm-hmm. Yeah, yeah, absolutely great question. So, the question was around the forward deploy engineer, the FDE. I don't know, have people heard of this term, the FDE or the forward deploy engineer? Some people, most people are not even reacting. Come on, resuscitate people. We're almost there. Okay, so I'll just repeat. It's a really good call-out because I actually did have literally a point under the GTM as alpha that I removed. I was like, maybe this won't be that relevant, but it was literally talking about forward-deploy engineering.

75:45 So it's a fantastic call-out. So the forward-deploy engineer is effectively this new category of Yeah, it can mean different things at different companies, but sort of a cross between a builder, someone who's traditionally PM, traditionally might also have a background in technology, who you essentially put either directly in the offices of your clients. Working very close and close, talking to them, and then kind of extracting insights to then either directly build bespoke solutions for them, or insights that you glean that you then put back into your original product, right?

76:19 So an example of that could be, hey, I'm embedding myself in a legal office as a forward deployed engineer. I'm having them use my product, but I'm learning what they're not liking about the product. I take those insights, I put them back into my product, right? So it's forward deployed because I'm deployed with my clients. Now This might sound a lot to a lot of people like, Isn't that just consulting? And in a way it kind of is. It is sort of a form of consulting. But I think that this is a broader trend that we haven't even just like fully unlocked yet.

76:45 So to answer your question, is, I do think there's a lot more to be done at FTE. And you know, this is a personal belief, obviously, because I'm in vertical AI, so I see us doing versions of this. But There's a lot of evidence besides even just my own anecdote, which is when you look at the Anthropics, when you look at these OpenAIs, they are increasingly embedding themselves in the enterprise, and they are literally hiring a ton of people that effectively take some form, their title is some version of an FDE. Literally, that's what they're called.

77:11 And you've seen Anthropic has a partnership with this group called Ode, effectively just embedded forward deploy engineers. So, there's a huge movement around that. I think it's... Valuable, and it will continue to grow. Like, that's sort of the TL;DR. And I think one of the reasons for that is it turns out it's faster, I think it's faster, especially when it comes to B2B use cases and enterprise use cases, to personalize and really customize a workflow hand in hand with the customer than to just sort of like hope that the model is going to learn the task, right?

77:42 And there's a lot of reasons for that, not least of which is just there's not data. If you don't have the data of how your user is using something, you can't just pull that data from like the web in the way that a lot of models are traditionally trained. So I think that the movement is going to grow. It isn't top of mind for a lot of people. I've had former students who actually have came from the GSB and then are now like FDEs.

78:03 So it's absolutely a path forward. All right, one last question back here. Hi, this is Ahmed. Thank you so much for the insightful session. What do you think about some studies in recent times saying that the current generation of AI models is heavily subsidized up to 10x, and generally sort of... At a point where—that's in the future— where we see VC funding or capital inflows slow down into this space, how do you think about what use cases create enduring value versus use cases that might not hold up when, let's say, capital becomes more scarce going into this?

78:48 Thank you. That's a money question right there. The question was around this. There's sort of whispering in the industry that all this stuff's so subsidized. How do you make money charging $20 for Claude Code when I'm literally using it all the time? How does that work out from a... Kind of math and computation perspective. I don't want to overstep my bounds here because I'm not a VC, so I feel like someone who's deep in finance would have better answers to this.

79:19 But I will say, you know, at least my personal belief, again, I don't speak for Sequoia or a16z or anything like that or any other player in that space. My personal belief is like there is something here, right? There is something to the subsidization, but it is also not the case that this is like new, right? We saw this. This happens all the time. I mean, this is sort of how a lot of Category-defining businesses are built is we do subsidize them.

79:43 I mean, Uber, Lyft, if people remember in the old days, like it was way cheaper to use these things. And these companies still exist. They're public companies. They do very well. And yes, the prices have gotten expensive. They kind of level out to a point where they can be profitable and still be more expensive for the average consumer, right? Like the cost of an Uber today is more than it was back in 2010, 2011, 2012. So, you know, this is not a new phenomenon.

80:05 We are certainly subsidizing some capacity. I don't want to predict on, is it a 10x? What is that gap? I think that's really hard to know without being within the companies themselves. Personally, I think there's going to be some sort of a reckoning of sorts when these companies go public. I think there's going to be some interesting questions that'll fall out of this where certainly the people that back the companies early will make a lot of money.

80:28 Once in the capital markets, where you have these public, you effectively have the retail consumer that you have to be accountable to, Then the economics Once it's out in the open, will be a lot more under scrutiny, right? And then I think we won't have a lot of these kinds of subsidizing phenomenon that we've seen so far. Now, so that's one piece. I think there is some sort of reckoning. Does ChatGPT go from $20 to $20,000?

80:53 I wouldn't go that far. I mean, I think that clearly there's a limit beyond which people can't spend the money to use that, so you have to kind of level it out. But I think what we have at least seen so far, though, is that these systems are useful enough for a lot of people that they have to exist. So we will certainly continue to use them in their own ways, right? I think what will be interesting for these companies to figure out is what is that Where can we continue to optimize some part of the process, whether that's on the compute or on the data or something where they can win and we can also win, right?

81:24 Where, yes, things get a bit more expensive, but not so expensive that it's unusable. And that's effectively what a company like Uber did exactly do. It's like they're kind of in sort of a way better kind of outward financial, kind of situation than maybe they were as a private company, when it was being subsidized. So I think we'll see what that levels out to. But I think it's sort of unequivocal at this point that the value is already there, right?

81:47 That the stickiness is there, at least on the use cases, not on the provider. I'm not super bullish on the providers. I think an open question for a lot of the OpenAIs and Anthropics of the world is there is no stickiness in their models, right? Like, I will switch from Claude Code to Codex in a day if I don't like what's happening. So the stickiness they will have to figure out. And I think that's why they're embedding themselves on the verticals to build more workflows, and I predict personally that they might end up acquiring fully verticalized players like the Harveys and whatnot.

82:14 I'll put that into the world as a hot take. And then we'll see. But that's, I think, one way they're trying to get around the fact that there isn't an inherent stickiness in what they're doing. Give a big round of applause for Mihail. (audience applauds)