Agentic AI MOOC | UC Berkeley CS294-196 Fall 2025 | Evolution of System Designs by Yangqing Jia

0:01 And like Dom said, I'm actually one of our own. I graduated. I did my PhD between 2009 and 2013 before AI and deep learning and things were cool or basically hang out in SDH. If some of you have visited Bears, and we didn't really have that fancy building back then, still getting food on Euclid Avenue and things like that. I think it's probably good to basically just talk a little bit about who's this dude? What is his thought process and things like that.

0:30 Back at Berkeley days, and then my first job back at Google, I was a researcher looking into AI like FAIR models and things like that. And one of the things that we did back at Google was GoogLeNet or what we call Inception, as one of the leading image recognition models. And then I realized I wasn't really a good researcher, but I'm a slightly good at writing code. Started writing an AI framework such as Caffe, and later participated in TensorFlow and PyTorch and things like that.

1:01 And one thing that we found was that with all those AI algorithms evolving, we start seeing very much different patterns from the conventional computer science, whether it be microservices, or databases, or data analytics, and things like that. And of course, for quite a lot of the system designs to change for us to build newer, larger, and more real-world scale kind of applications. And so basically it started basically like going from research into engineering, and then after many years started a company called Lepton AI, trying to basically build a cloud kind of infrastructure that fits the AI workload, especially for training and inference and things like that.

1:41 We got acquired by NVIDIA, so basically still technically back in the big company environment. And so I feel that I don't really have very deep knowledge about the state-of-the-art research part of AI, but hopefully, the experience taught me a little bit about stories on the road, what kind of things people are caring about in the industry right now. What kind of cost-benefit ratio analysis and things like that that people are thinking about when they are doing a startup and model and applications and things like that I would like to basically share that with you guys.

2:13 I want to always start with a more not scientific but interesting thought process. We've seen quite a lot of people talking about the term AGI and basically saying, hey, there's this mysterious model that seems to be able to understand everything, and it's getting smarter and smarter. Now, there's no question that they're getting smarter, but are they so mysteriously good, or are they actually being built with some really simple principles that we've learned in CS281 or things like that.

2:49 And so I grew up in China. My home language is Chinese. And so we've all typed on keyboards typed in English. It's very simple. It's what you see is what you get. You type A, and then on the screen you show A. Now, if you have a bit of knowledge about Chinese textbooks-- yeah. It's kind of weird. The mouse isn't really-- I'll just put it over there, or I can minimize it. Let's just put it there.

3:20 It normally wouldn't have too much sense. So basically, if you have seen all those Chinese logos and things like that, you know that Chinese characters don't really show up on the keyboard. It's an indirect typing kind of process where the character is different from the QWERTY keyboard, so what you see isn't really what you got. In the old days, people solved that problem by having literally a Chinese keyboard with individual keys as metal foundries somewhere up there.

3:52 So the way you do it is you move around that keyboard, you find a key. Out of something around 3,000 to 5,000 common characters, you click that thing, and there's a mechanical structure that gets that foundry up, clicks onto that role. And then when you release it, it comes back, and you find the next key. A huge keyboard. Now, over our early computer science days, we've known how to type 26 keys and so on, but imagine how do we deal with that.

4:27 And the other question is, we're typing sentences. Moving these handle around is hugely difficult. How do we actually make sure that things are more efficient when we're actually typing? Of course, today we don't use these kind of keyboards, although Chinese students are still typing on QWERTY keyboard, but this is how things started. With the fundamental question of the same logic as when you are basically saying like, how do you actually seek to the next block of data on your spinning hard drive and things like that, the idea is that the distance between the current character and the next character matters a lot.

5:04 If your last character is on the top, and then you want to type the next character of your phrase down there, then you are of in a bad position because you have to move quite a lot, which is why in the old days, if you use Windows and things like that, in a spinning hard drive, there is things called defragmentation and things like that, making sure that the same file stays in the same-- I forgot even that word-- stays in the nearby part of the hard disk.

5:33 So then people started becoming smart. I mean, that was basically in the 1920s and 1930s. And people were like, how do we actually group things together so that we can minimize the movement of my handle? It went out to libraries. They started counting. They started counting what we call today bigrams and n-grams and things like that. Basically you're saying, OK, given the previous characters or given the-- yeah, it's kind of weird because the mouse is not showing up.

6:05 Should we just keep it over there? yeah, right. Probably doesn't block too much. So then we basically start the counting of those bigrams and more, and then we start to group things together. The words that are basically like the characters that happen together gets grouped together so that things get more efficient. Now, for example, here over there from the top to bottom was what we see is [CHINESE], which is gas, and [CHINESE], which is paint. Here what we're seeing is [CHINESE], which is China, and then [CHINESE], which is the Chinese ethnic group, and things like that.

6:39 And by doing that, people found that the typing speed comparing to a random keyboard got massively improved. Now, that was basically in the old days. We were like, this is simple. You just count, and then you basically do prediction of what you would type next. In today's LLM vernacular, we call this thing with a more fancy word called next token prediction. We have a model. We see what kind of things are in front of it, and then we predict what is the next token.

7:11 There were researches on to basically see whether you can use the history or use the surrounding part or the future to predict the current word. There was a model called BERT a few years ago, which is basically saying given the character before it and after it, how do we predict the current word? And the models that we are using quite extensively today, called GPT models only use the history kind of similar to what we are typing something.

7:39 The difference is that in the old days, we were basically doing all these things manually, so we cannot really go very detailed. We can probably count two or three characters before it. Conventional NLP were actually constrained by these constraints because you can basically look at bigrams, and probably four or five bigrams is probably the upper limit that you can go, because that correlation matrix gets multi-dimensional crazy. Today, we can actually look into what we call the context length.

8:09 The [INAUDIBLE] models have normally millions of tokens context length, so you can look. At a lot of the previous contexts to predict the next token. And as Ilya Sutskever would say that basically, this is a mode of compressing knowledge into predict the next token. And by doing that simple action and producing that sequence, then you will be able to capture a lot of implicit intelligence or things like that. So a lot of these basic ideas come in similar ways, of course, with more sophisticated and powerful models.

8:44 And so I want to basically cover is four topics. One is basically in the space of LLM, what kind of algorithms are there? What kind of models we're looking into. I'm not going to be diving into the exact math because I'm a terrible researcher, so I'm sure that there are way smarter researchers who can basically tell those. I'll just talk about it from a outside-in or application-driven approach. And the second is basically, on top of these models, what kind of applications are out there?

9:13 What kind of applications are doing great? If you want to build an application, what kind of parameters you might think about? And then we dive down into the infrastructural side, which is really my comfort zone, talking about AI Infra and why is it different from the conventional cloud and things like that, because cloud has been around for 20 years and why suddenly things are changing a little bit? And in the end, I want to basically talk a little bit about hardware and stuff like that and see if we can find similar analogies in the conventional hardware land.

9:45 And of course, when we talk about hardware, this hardware is going down, and let's wait. My apologies. We used to joke that the two hardest things in computer science is printers and display. This is one of the hard problems. All right. Seems to have stabilized. And yeah, well, after this, I think, if you like, I can basically talk a little bit about the pain points of doing a startup and things like that, if you like. And of course, one mandatory disclaimer is that since like we are-- I'm currently employed in the company, all these kind of opinions do not really reflect the opinion of the company, and it's more basically me from an individual contributor and a dude with too much code kind of perspective.

10:29 So just similar to what we were just talking about, I mean, there is quite a lot of interesting models that comes up and starting from ChatGPT or even earlier models that you might remember, such as Chinchilla and PaLM from Google and things like that. GPT-3.5 really started this trend of LLM frenzy. And we've all been asking ourselves every six months, basically saying, hey, have we actually plateaued? Have we actually plateaued in the model quality? And it seems that even today, we haven't really seen a plateauing effect, and the two streams of models, both closed-source ones and open-source ones, are continuing to be having a very healthy competition.

11:09 And as of this graph is actually a very nice tradition from 2024. And in the last couple of months, we've seen a really great closed source models such as GPT-5, Grok, Gemini, Kimi coming out. And in the open-source land, we've actually seen a very quick change of thrones, I would say, going from LLaMa models to the models such as DeepSeek, Qwen, or [CHINESE] as in Chinese, and GPT open-source from OpenAI itself as well.

11:40 What we are basically observing is that closed source models continue to ace in the absolute quality. And as we talked to enterprise clients or basically partners out there such as, say, the Salesforce, the Pinterest, and things like that, people basically do feel that closed-source model actually leading in quality and especially the newest capability of doing reasoning and stuff like that. But the gap between open-source models and closed-source models are becoming smaller and smaller, I would say.

12:10 In 2023, maybe the gap is around a year and more than a year. And today, what we're thinking about is something around six months. So basically, when there is new approaches coming out, normally open-source models are able to catch up really quickly. In terms of reasoning, for example, DeepSeek is probably one of the first papers that detailed the capability of test time scaling and that actually educated the whole community to actually catch up pretty quickly. And the next question is people often ask is like, there's quite a bunch of modules, and are we actually in the land of a self-fulfilling prophecy and we're only training modules and not using them?

12:49 a.k.a. Is there a bubble? I tend to basically use some more objective measurements in addition to the news pieces from TechCrunch and other places. Of course, really great news pieces. I tend to use basically more objective usage statistics to look into whether there is a battle or not. And there's a very nice website or web page called OpenRouter, which aggregates a lot of the models into more unified API so that people can easily consume those. And a side effect, or basically a side result of that is basically it will be able to look at the total consumption, at least from that channel OpenRouter, and also the distribution of different kind of models and how popular they are.

13:32 And so starting from 2024 and even earlier, we can see that the consumption of the models are actually surging. So I remember about a year ago at that time, the consumption of something around sorry 50 billion to 60 billion tokens per day. And today, when you look at it, it's something around 4.9 trillion, so about 10x growth in the last year.

14:05 Apparently, we are actually seeing very solid use of these models. An interesting thing is that you can see there is a very big battle actually between the open-source models and closed-source models. So the graph here, that chart here, this shows on September 15, what are the models that are most popular. And you can see there on graph there is Anthropic. There's Gemini. The one so Sonoma Sky Alpha is actually Grok-4 being tested in a semi-secret fashion.

14:35 And the open source models that we mentioned such as DeepSeek, and a little bit down there, Qwen, and things like that are still been doing pretty well. There is also a ranking by application areas in different applications. So this is really a good place if you're interested in diving into more data science and analytics of the dynamics of the different models. So apparently, there's a continued growth of consumption which actually gives us hope that AI this round has actually been very helpful.

15:04 And so if we look at the more researchy side of the world, this is my personal understanding, not completely right, about why there are new things that continuously excites us. And some of the ideas, as I kind of tend to basically tie it back to the other research topics or research ideas in the ML community. So in 2022, basically, we started knowing about GPT. And it's similar to, I would say, Alexander about 13 years ago, because this is really a structural innovation or change that fundamentally improved the quality, in this case, the text understanding, and back in AlexNet days, it freed computer science researchers from using simple, handcrafted features such as SIFT and pyramid kernels to a more principled deep learning approach.

15:55 Now, as we saw ChatGPT taking on very well and people are doing structured training and things like that, and when we believe that the model might be just plateauing, then a mixture of experts came out. The idea is that you can actually use a massively sparse model and sparsely activated model to improve the efficiency and decrease the model size while actually maintaining a pretty good quality. This kind of thing happened in the past as well. And if you think about very early Bayesian learning things, ensemble with very simple classifiers work really well.

16:29 And in the convolutional neural net days, the GoogLeNet paper or the Inception paper adopted a similar approach by having not one big chunky network but sparsely-- well, actually in that case, it is still sparsely parameter but always active different paths in each inception unit to achieve a much better parameter utilization. And you can also derive things from the theory of tensor decomposition and things like that, but that's a bit too theoretical, and I'll refrain from diving deep into that.

17:04 Now, these are all structural changes, and then we feel that structure fine tuning and things like that using a very simple loss function to basically train a model has plateaued. Then in 2024, we found-- and OpenAI's blog post actually set a very interesting theory, which is like you can actually start using test-time scaling or basically have the model mumble longer in some way during testing time, and having it reflect on its intermediate results to get a better result.

17:35 This kind of is similar. I try to basically draw some analogies, similar to the idea of, say, multi-instance learning, or basically fully convolutional neural networks. Back in the days, by doing prediction multiple times across the space domain and try to get a better result. In this case, it's basically doing prediction and learn a much better time domain and accumulate and reflect on the intermediate guesses that it can have and get a better result. Now, there is debates about whether test time scaling is the last leg for us to squeeze out a bit of the intelligence out of an existing model and things like that, and people are a little bit pessimistic about it.

18:16 And I think one thing that really made people much, much more interested in the model innovation is the massive application of reinforcement enforcement learning. DeepSeek was actually one of the early papers talking about that, but we know that OpenAI has been using this to train their models for a long time. I think one thing that really excites me personally is that reinforcement learning allows us to actually define in a more principled way a much more sophisticated loss function, because before that, the idea is we have a bunch of sequences, and then I use the next token as the loss function, the objective to train the models.

18:53 It's not really exactly tied to the end result. Reinforcement learning allows us to basically say a model should basically do some rollout or predictions, and then look at the end of the horizon, and then provide feedback about the quality of the end result. Fundamentally, it's basically saying it's difficult to define a loss function that is actually matching the thing that we want to do, but RL allows us to have a principled way to do it. RL ties back to really old days of robotics and things like that.

19:24 I think if you're in EECS department, I think the prelim examination still requires you to read that big Sutton & Barto book, which is interesting because it has a great comeback over the years. And so basically, if you look at the modules and things like-- well, if you look at the hype cycle of Gartner, which is one thing that or industrial participants like me would normally look at, you can see that a lot of the ideas are very quickly iterating and evolving.

19:51 If you look at module Ops and then RAG, Retrieval Augmented Generation, and things like that has only been out for around a year or two, but it's already been kind like it had its days, and it's coming down. And then right now what we are looking at is prompt engineering, agentic AI, and stuff like that are becoming more and more important. This ties to a very interesting question, which is like these kind of days, how do we actually think about building applications on top of it?

20:18 Because you might have remembered that RAG or basically like search-augmented chatbot and things like that used to be a big thing. It continues to be, but it seems that now everyone is talking about agents. So how do we actually think about applications, and how do we actually build applications and things like that? So back in the Lepton days we were doing AI Infra and unavoidably we'll have to look into what kind of applications people are building so that we can serve them well.

20:45 And one observation that we have is there are normally two types of applications, ones in consumer apps, and one is basically business services or enterprise services and things like that. Today, what we see is a thriving number of consumer applications because the models are indeed very smart and are actually able to foster a lot of innovative creativity, entertainment, and productivity and things like that. Business applications or enterprise applications have a very strong potential to actually make billion dollar businesses, but as we all know, businesses move slowly and things like that, so we are only seeing the emergence of that.

21:25 Let me give you some examples. So basically, we were basically, thinking how hard is it to make an application that's an API, and what is the distance between API and then something that's useful? So this story-- I want to basically tell you this story which literally happened within some of our engineers. And I was basically chatting with a gentleman called Yadong who is in Singapore, over Chinese New year. He was like, I'm so bored. This is Chinese New year, and other than my grandparents or my parents actually forcing me to have a kid, I don't have no other topics to talk about.

21:57 He was like, let me build an app. And so one thing we were thinking about is there are so many blog posts and things like that, and it's really difficult for us to consume the massive amount of information that's feeding into our browsers. And he was like, how easy could it be if we could have a bot that lives next to our browser and helps us basically do digestion and analysis and things like that? So over two days-- basically, he was a very smart front end engineer.

22:22 Over two days he built an app which lives still today. And so I'll show you how it works. So let's say we have Berkeley's home page opened, and I'm too hasty to understand what Berkeley is. So the app I basically say, please summarize for me. And it goes like a person reads the page, and it gives me the summarization and things like that. Says, "Berkeley is a leading public research university known for its groundbreaking research, top ranked academic programs, and vibrant student life that contributes to a better world and things like that.

22:52 And I can basically ask questions like, when is-- sorry-- when is Berkeley founded? Hopefully, it answers me correctly. Was like 1868, if I'm still correctly remembering this. So those kind of things makes it really easy for us to build something that wasn't possible before. And you might have heard about stories like a one-person startup and things like that. This hasn't been a startup yet, but it shows that by combining smart like application knowledge and basically like other general programming skills and a powerful LLM model we'll be able to do something that's way more efficient than we could do before.

23:34 So a small, cute application. And there's quite a lot of much deeper thoughts over there coming along this line of thought. And if you look at basically the consumer app landscape, it shows itself as a very highly fluid and competitive land. And Andreessen Horowitz, every half a year, basically puts out a report called the top 50 general AI web products and other kind of products. And this is, I think, August, maybe September, August 2025.

24:05 And as you can see, basically there is quite a lot of applications out there. Loosely speaking, they get into a few things that you can see. Chat is one of the biggest topics. So ChatGPT, Gemini, DeepSeek, Grok, and things like that. The other one is basically coding is becoming much more prominent and used by people. So you might have heard about Cursor and also Replit and things like that. There's also search that comes quite often. So Perplexity is the most prominent one, also started by Berkeley grad student.

24:37 And all these things are evolving really quickly, and the landscape shifts very quickly. So this, for example, is showing what's different from six months ago and now. About that's 11 that are basically that are just like popped up into the top 50 in as short as six months. And Grok and Quark and folks are actually in top 10. So, in this case, people are constantly asking, what is the mode?

25:11 Because people might think of some models that are performing really well, but because of the underlying LLM model doing really well, then in the end basically very quickly got surpassed by other applications. One of the very prominent example is a company called Jasper AI, which started in the GPT-3.5 days to be an editor/helper or something like that to help you basically do content creation and things. But for better or worse, due to ChatGPT models being too good, it started basically having to pivot into other areas.

25:44 We also see that models such as basically, I think, ElevenLabs having its own model, and then Midjourney having its own model and knowing the customers really well and being able to hold its mode relatively well. But in general, it's like a blue ocean kind of field, but there's quite a lot of competitions over there. And one thing that we are seeing is there's a lot of traffic, but who are willing to pay? And then in all those consumer apps, one trend, a very clear trend that we see is that if the application actually targets prosumers, a.k.a, they're still small consumers, but they are actually embedded in those kinds of things into their daily work and productivity, they seem that people are very much willing to pay.

26:30 And I'm just cherry picking a few succeeding examples. And Cursor, as we all know, is basically embedded into quite a lot of the code and practice that we have today. Runway, as a multimedia creativity and multimedia editing kind of tool, has been liked by a lot of the content creators, such as the TikTok guys and also people who are basically starting shops on Etsy and things like that. There's a very interesting hardware startup called Cloud, which employs a very simple idea.

26:57 I actually have one here. So it basically have a credit card-shaped kind of recording device. Recording device is not really new. It can basically record your conversations and things like that, and then connecting that to voice recognition and ChatGPT to tell you what are the action items, what are the conclusions, what are the main debating points of the recorded meeting, which is liked quite a lot by people who are in the legal industry, in the consulting industry, and the health care industry and things like that.

27:26 And ElevenLabs, as we mentioned, has been awesome in voice models. And all of these guys are actually having multi-hundred million revenues. And the logic behind that is basically like it seems that when things get into productivity, you're very much willing to pay. As a consumer, for example, I play with Midjourney. And then I was like, hmm, do I really want to spend $20, $50 per month just for fun? And it's like, nah, probably not going to do it.

27:54 But if this is basically like coding, or basically, if I'm a content creator, and I was thinking about to make sure that I can actually do my job much better, am I willing to pay $20? And I was like, of course, I do. From a startup guide perspective, if you look at every single person's IT or productivity tool, you will find that it's not a lot of money. So let me count from a startup kind of perspective, when we have one new person joining, one new gentleman or lady joining, how much kind of SaaS tools we would set up for them.

28:27 There's GitHub. That's $20 or $25. There's Google Workspace, $25. Slack, $12. 1Password, $9. GitHub Copilot, that's about $19 a month. A bunch of other things. So it's very easy for us to basically for people who actually thinking about productivity to shell out a couple hundred dollars to support doing a better job. And I think that's basically one of the areas that the smart startups are doing really well over there. So when we think about enterprises, things are in an interesting position where a lot of the enterprise applications are growing much faster than the more conventional ones, but not as fast as the Cursors and the Perplexities on the consumer side.

29:13 Enterprise applications have this unique barrier in the sense that it just needs a lot of work. So let me give you one example. We all do search daily. We use normally Google. And in my own country, Baidu and things like that. But the question is, what search engine do you use at work? Maybe you're doing an internship in a company or things like that. What kind of search engine do you use? Normally, I get two answers out of that.

29:40 One is, well, there isn't a good search engine because all those things like wiki and Jira tickets and things like that just cannot be aggregated. And the other one is basically saying, our company has our own internal search tool, but it sucks because it's just not high quality. There's a startup called Glean, which it took a little bit more than three years to reach $100 million ARR. And quite a lot of our friends in different companies actually, I was chatting with them was like, did you guys use Glean.

30:10 And it was like, it's great. This is the greatest search enterprise search engine that we have experienced. The idea is that it basically connects to the multiple sources such as Google Drive, GitHub, internal wiki, and a bunch of things, Microsoft 365. And it will be able to integrate all those authentications and access control and things like that to basically give every person a search engine that is indexed on the set of information that he or she is actually able to access inside the company.

30:42 Lots of dirty work, but addresses a very big pain point of enterprises. And it's combining with AI, then it's actually doing pretty well. So, I think, basically, when we think about the potential of AI, and answer the question whether there's a bubble or not, it's still a lot of potentials, especially with the more nascent enterprise market. Lean is one, and there are a bunch of others, such as Sierra, who is basically doing, I think, legal AI and things like that.

31:09 And if you go to, say, Sequoia or Andreessen Horowitz reports, you can see quite a lot of potential companies over there. People basically started normally by using closed-source models and build some end-to-end kind of applications. And then as data start accumulating, they started basically building their own models and things like that. There is a constant debate whether the OpenAI and Anthropics are actually going to be taking over the world, but in the industry, we see things happening differently.

31:41 So there's a very interesting app called Duolingo. My kid actually used it to learn English. I don't know why they are actually born here, but they use that to learn English, probably just to play some games. Duolingo was actually able to-- people were like, oh, well, when AI models are coming out, Duolingo is dead. Not so. They're actually living pretty well. And what we found is that Duolingo is actually able to use AI models-- OpenAI and things like that.

32:07 I don't know what exactly they use-- but they're able to use AI models to start creating content and more interactive content much more efficiently. And it creates this very complementary relationship where the better the AI models are combining the deep knowledge of the customers and the language learning expertise that Duolingo has, they create a 1 plus 1 larger than 2 kind of effect. And it basically boosts the quality of the whole product and experience.

32:39 So I think this basically it's going to be continuing, and we will see very healthy synergy between the model builder companies and also the application companies. And of course, all those companies need cloud infrastructure under there to support the massive scaling. Very fresh off the boat news was like this morning-- I didn't know about ahead of time-- this morning, NVIDIA and OpenAI are announcing that they are, I think, investing $100 billion or something like that into computation infrastructure to build and to scale the next generation models and things like that, which is basically the massive scale that we're seeing in InfraSight.

33:16 And so, in general, I think, starting from even before ChatGPT, maybe like 2020 is what we found is that AI infra has become the third pillar, the third pillar in the IT or cloud strategy. No longer has a part or a small part of all those cloud and IT but really becoming its own. And when I say the third pillar, what are the pillars that we're talking about? First of all, what is the need? The need is basically very well summarized by Professor Richard Sutton's blog post in 2019.

33:50 He's basically saying, the biggest lesson that we have learned in the 70 years of AI research is that we can employ a generalized approach, general method in today's context, large language models, deep learning models, and things like that. And we basically deploy a large amount of computation, numerical computation onto a large amount of data, and that seems to be the most effective way. So we need a large amount of computation power. And if you look at the computation history, a lot of the compute idea actually came from numerical computation.

34:23 In the 1970s, we know that scientific compute is a big thing. And on the hills of Berkeley there is a NERSC. It's basically built by the Department of Energy. And if you have taken or if you have taken the introduction to parallel computers taught by Professor James Demmel, and Kate Yelick, you know that a lot of the things that are taught over there is about how do you actually carry out large scale matrix multiplications and things like that.

34:50 And those applications were physics weather simulations and stuff like that on a cluster with CPUs in the back in the days with good old software such as Slurm and things like that chugging along. And then when people want to basically have someone else to basically deal with the computer and things like that, they started the business of virtual private servers so that some companies, data centers and things, can take care of the physical machines, and people build applications on top.

35:16 And that eventually rolled into what we know as cloud today, cloud services, AWS, GCP, and Azure and others. So we call it conventional cloud in the sense that they started in the web services space. And what they need is basically we host a bunch of machines. We do virtualization. On top of that, we put a bunch of microservices or small workloads over there, making it more flexible for people to not worry about machine operations and things like that and serving web traffic.

35:50 Web service cloud grew for quite some years, and then when the idea of Web 2.0, such an old idea, Web 2.0 came, there is a lot of idea about user-generated content and matching user's content and the consumer's interest, TikTok and YouTube and things like that. There's a lot of need for data analytics. We record the user activity data, whether it be e-commerce or content consumption and things like that, and we started building recommendation engines and things like that.

36:24 And that actually required quite a lot of data analytics. So around 2014, there are two interesting companies coming out, and one is called Snowflake building the idea of cloud data warehouse, and the other one, a more open-source side came from Berkeley and which is called Databricks on top of Spark. And that set the idea of data cloud where there is a massive amount of data and massive distributed training-- sorry, distributed computation using SQL and data analytics and things like that.

36:53 And in 2020, we actually started doing AI, we see that a lot of the patterns are kind of different, because today, we actually call for quite a lot of exaflops of float and even lower precision float computations. It has to be carried out with relatively small amount of data, but a lot of compute and a lot of communication, kind of similar to the very old scientific computation days. And so to put things in more abstract or basically more principled terms, there are two parameters that are governing the thought of how we think about infrastructure.

37:27 One is data movement or IO, and the other one is numerical computation or computation. In the old days, in web compute, there's quite a lot of IO. We moved data around web pages, images, and things like that. There's not too much of compute, because I just need to read things off the disk and serve it to the customers. But lots of IO and the smaller compute and the workload looks like microservices. It's embarrassingly parallel. You have a Node.js server.

37:55 You replicate it. There's more traffic. And if you scale it down, if there's no traffic. And then data compute is slightly different in the sense that still there's quite a lot of IO, but the computation is not that embarrassingly parallel. You actually have to use abstractions such as Spark or basically MapReduce and things like that to carry out one SQL as one big distributed system. SQL gave you a very good abstraction, but under the hood, you have to have a lot of complicated distributed scheduling and things like that going on.

38:25 AI is different in the sense that compute is way bigger than IO. We always talk about large training data sets and things like that, but that's normally still about, say, a couple tens of terabytes, or maybe 1 or 2 petabytes or things like that. Computers are a lot. Training one single image model is already 1 exaflop. And we used to joke that if the whole city of London, everyone starts using their pens to do calculations, it takes them 4,000 years for the whole city of London to train one model, And in LLM today it's even bigger.

39:00 And it's very distributed and in the sense that for data infra, you can actually have some machines go down and you just redo the compute of that chunk of compute on the next available MapReduce shard. In AI compute, when you launch a distributed training and if one of the GPU machines go down, then it's the whole training job is done for good. You better basically restart the whole job. There's a bit of tolerance and things like that, but if you look into the modern day AI distributed training is not that easy.

39:28 And that actually really basically gives us the thought, hey, it just looks more like MPI, like the old day Slurm kind of days, like the old day scientific computation days. The conventional cloud value proposition that originates from the web cloud days no longer holds, because conventional cloud-based data sets are going to be giving you an easier way to acquire and install software. You can basically say, I can launch a cloud database without having to install MySQL myself.

39:55 And the other one is basically cloud services that I'll give you a flexible supply chain by having EC2 machines being pulled up and released flexibly all the time. In the AI world, it's different in the sense that if we list out, there's not too much applications in AI, and there's not too much variety. It's pretty much basically AI frameworks. You install back in the days PyTorch and TensorFlow. Nowadays, more popularly, PyTorch, and then a few dependent libraries on top of that, such as Megatron LLM or Lightning AI and things like that.

40:26 So it's simple. That's not too much variety of software. The workload is also quite unified in the sense that there's a huge amount of numerical computation or matrix multiplications and things like that. Not too much compute, storage, network, big data and things like that. It's simpler. Supply chain-wise it's more rigid in the sense that in conventional cloud you have microservices that works really well on virtualization, and you can actually migrate workloads between different machines and things like that.

40:52 It's because it's very nicely laid out. In AI cloud, when you launch a large training job with, say, like a bunch of GPU machines, then that machine physically are just stuck together. With high performance networks and things like that, if one machine goes down, you need to move the whole thing. You shut down the whole thing and you find another machine and you start running things over there. And for better or worse, if you actually have a bunch of CPU machines, during the day, you're probably serving web traffic, and during the, night you can repurpose it to actually do data analytics and things like that.

41:26 In the GPU land, for better or worse, the workload is not that interchangeable because it's really good for numerical computation, and you cannot really run a web service on top of GPUs, so interchangeability becomes harder. And that drives the emergence of what we call the neocloud over the last few years. And there's a very nice analyst website called SemiAnalysis that I highly encourage you guys to read if you're interested in this land. They coined the term neocloud with the idea that there are companies out there who can actually focus on AI compute and AI workloads, aggregating GPUs and not really worry about database services and things like that.

42:08 The most prominent ones are basically CoreWeave, Lambda. There's a European one called Nebius and a bunch of others. They start basically showing their value by providing more AI-centric computation resources that would reduce your costs, improve your availability and things like that, and, of course, the hyperscalers are also getting into this field to create more AI-centric cloud services. The reason is that basically you really cannot just get a bunch of GPUs and start running with it.

42:42 I was chatting with one founder CEO last year, and he was like, man, I'm spending so much time hiring GPUs and basically monitoring GPU health and things like that, which isn't good. There are basically two choices up there. One is basically like you just have an internal team that starts using Excel or whatever to basically manage the cluster. And we did that back in the Berkeley days. I was the sysadmin for our computer vision group and wasn't really the best choice.

43:11 And the Captain Vinyals up there that I joke about, that Oriol Vinyals, who's basically leading Gemini today, and he was also one of the poor sysadmins back days. The other choice is you use Kubernetes. Now, you might have heard quite a lot of things about Kubernetes being the best thing since sliced bread. AI researchers don't seem to think so, mainly because Kubernetes is a very nice abstraction, but normally for SREs, site reliability engineers or operation people.

43:41 And when you actually basically provide the Kubernetes type abstraction to the AI researchers, they're like, no, I don't really know how to think of parts, deployments, parallelism, completion and things like that. As an AI researcher, even if I did a startup, I still don't know about those kind of things. Researchers like, I want a simple mindset where I get full machines. I get MPI set up, and I run this PyTorch job, and I just happily wait for the result. If one of these things go down, please give me another four machines, run the same thing until things go well.

44:19 I don't really want to deal with Kubernetes. This is the fight today, the holy war, I would say, between the Slurm researcher kind of mindset and the Kubernetes operations kind of mindset, which kind of comes very often in the community. Now, the reason is actually-- the fundamental problem of developer efficiency are just in a conflicting state. Developers want things to be really simple and abstracted. Throw a job and get it done. Infra-wise, for better or worse, GPUs do die today actually.

44:50 The charts that you see here is actually from a very nice analysis from Facebook who are running their cluster of-- I forgot how many machines, but probably at a scale of 100,000 or something like that. They find that there's quite a lot of reasons, software bugs, faulty GPUs, memory issues, and things like that, and network issues that might cause the machines to go down. So you do want some sort of abstractions to hide these operational burdens away from the developers and get them a seemingly stable environment for them to carry out their job and not worrying about fixing GPUs 2:00 AM in the night.

45:32 So basically we're thinking about this. I had to basically use Lepton as like a bullshitting marketing architecture. But what we found is that the best practices today for a lot of the startups is four things. One is basically to find a multiple cloud supply chain, because today GPUs are still in short supply, and normally when you want to basically find 2,000 GPUs to train your model, maybe your current provider doesn't have it, but the provider in the next state might have it.

46:01 So you basically want to figure out how to actually smoothly migrate your jobs between different clouds. And of course, since GPUs are so expensive, you want to basically make sure that you are actually actively looking at utilization and things like that and not really just wasting idle GPU hours. Normally, people would start looking for an AI native platform that allows them to basically abstract away the Kubernetes jargon and focus on the training jobs. A few really good examples, and I'll focus on the Berkeley oriented ones.

46:32 So we have Ray as an open-source project a couple of years ago, actually, maybe 10 years ago, starting to basically serve reinforcement learning before it's cool. And today, as a general computation framework, there's a company called Anyscale that basically supports Ray and basically does commercialization. There's a very interesting new project for a couple of years now called SkyPilot that allows people to find efficient computation resources and then basically orchestrate workloads across different clouds and things like that.

47:01 And essentially, I think people can actually piggyback on these kind of frameworks and platforms and then build their own teams on model applications, because there's quite a lot of things to be sorted out. Taking the really famous coding company called Cursor, for example, they will be using standardized SaaS services to basically stabilize their inference workloads and things like that, and focus their effort onto defining or designing and training the next generation coding models.

47:32 That will be one of the key Competitiveness of the company. So those kind of things seems to be a common best practice that we observe in the industry. Lots of the things are different, but in the end, a lot of the conventional wisdom of operating a cluster and operating services start basically coming back. I'd love to basically show this picture because it shows that AI infra is no different when it starts getting into products, basically. And so this is one Saturday afternoon back in the startup days.

48:06 And then one of our clients was basically running a large-scale chatbot that suddenly saw a fluctuation of their services. And when we looked into it, it was a very classical error that happened. One of their engineers accidentally shut off the main model that they're serving on our platform. And so, of course, everything got idle. Well, what do you do? Then you restart it. Now, if you think about it, when you shut off a main service, a lot of the requests start basically aggregating and waiting, and you start a service of multiple replicas.

48:46 What will happen is there's first replica or first machine that pops up and start accepting traffic. All those crazy traffic start getting in there. And what happens? That poor guy just dies. The second guy pops up with, what's going on? And immediately he got killed. And this is the typical thing that often happens in a lot of the AWS like region-- well, if AWS region fails and then it comes back normally, these kind of storms happen as well.

49:14 So we are like, yeah, we know about that. So Saturday afternoon we're basically sitting there chatting together with them was like, OK, let's basically put-- what is that calle?-- a traffic regulator in front of the service, so that we only allow a certain amount of traffic coming in, gradually putting the services up to speed. In 15 minutes, we're actually able to recover the whole services, but we were like, it's no different from conventional cloud services. So when we are actually looking into basically a productization of these AI models, all these kind of conventional wisdom comes and becomes helpful.

49:51 All right. So the last thing that I think is pretty interesting to talk about, especially at Berkeley, is that we see the hardware and software designs to come back to a very interesting old way. So in 1985, there was one of the very famous supercomputers, I guess, that came out, which is called the Cray 2. The computation power of Cray 2 today is not really as big. It was probably half of an iPhone 5. Back in the days, it was one of these really beautiful mainframes.

50:25 The thing that you are seeing is basically the-- the circular thing in the front is the compute units hosting the CPUs and the memory and things like that. The brown thing in the back is the cooling tower. It just deals with cooling and stuff like that. Back in the days when all those kind of mainframes have this beautiful idea that there is one bus, and then you can attach different CPUs, and you can attach different memory and storage on there.

50:49 And this whole thing operates as if it is one single computer. Really nice to do programming, really not that flexible. And then during the cloud days, we were like, that's different. We shouldn't do that. We should actually use small machines and things like that. So the thing I'm showing on top there, on the top right part is a typical server design by the Open Compute Project, OCP. OCP was created by a bunch of industry participants like Facebook and Google and things like that.

51:16 And it basically says I'm actually being able to design really modular small machines. Each machine operates on its own. And then I can basically shovel it in and shovel out individual machines really well. And then it serves microservices and things like that really well. And then AI came, and we feel that we start needing to beef up the machines. Today, one of the most popular servers that we're using to run the AI models is what we call the DGX boxes.

51:47 And both NVIDIA and AMD actually have similar kind of setups for such boxes. It's what we call 4U rack mountable unit which is about that tall. It hosts eight GPUs and two CPUs, and it's a very beefy machine that is able to run like DeepSeek kind of GPU models and things like that. Becoming really bulky but still it's a server. It's a bigger server in the internet days compared to the smaller internet CPU servers. And then I think as of last year, we started designing an NVIDIA system.

52:21 It's called NVL72, which gives rack-level really nicely integrated set of servers that can do large distributed training. So if you look at that rack, what happens is basically the middle part-- as you can see, there's a bit of like pattern difference over there. The middle part are all high bandwidth switches that allows the machines to talk to each other. And then the ones on the top and the ones on the bottom are basically in a service as we normally have.

52:54 If you think about the cloud days, each machine is its own, and machines basically talk to each other by primitives such as RPC or MPI and things like that. But each machine is its own domain. As a machine, you cannot access another machine's memory without asking for permission from it. In the NVL72, and similarly, industry participants have different kind of setups over there as well. The new pattern is that with all those NV switches each machine is actually able to directly access other machine's memory without asking for permission.

53:28 It operates very much like a mainframe with all those CPUs and GPUs being able to access all those memory in that rack. In the HPC world, there is a very interesting project called UPC, Universal Parallel C, by providing an abstraction of multiple machines memory into one so that we do not need to use primitives to ask for permission to access each other's memories and things like that. This is exactly basically like achieving some sort of that design.

54:00 The benefit is if you are training massive models, if you are deploying massive models, now you no longer need to worry about chopping the model into different pieces that can fit into individual machines. You can basically say, hey, put the model into that big box or big rack, and everyone can access everyone's memory and things like that. So a lot of the optimization pieces that we think about, such as disaggregated profiling and decoding, prompt caching, and things like that becomes much easier.

54:31 This is still quite nascent. We haven't really seen like the software side like PyTorch and things like that fully embracing that. But we are seeing really early explorations of that. So VLM project, SLM projects, see Berkeley is great, because all those projects come from Berkeley. LLM and CLMs, we're actually actively looking into those and utilizing the newest hardware innovations which seems to be thinking about ideas back in the mainframe days to actually push the boundary of those.

55:01 I think NVIDIA actually had a pretty nice slide that I want to basically show, the evolution of the thoughts of the scale. We basically started about 10, 15 years ago by just using GPUs as one additional card into my box. And then we start with saying we need more GPUs, and today we need more and more GPUs. And we're thinking about a rack level or basically a data center level scaling of the models for training and inference, which is really nice.

55:28 There's an order of magnitude increase in the computation needed and also the corresponding memory needed. And as one example, back in the days when we're thinking about computer vision models, the state-of-the-art model such as GoogLeNet was about 6 million parameters. And we're like, that's a fairly decent big model. And today, if you think about LLMs, an $8 billion parameter model such as LLaMa 8B is considered small, very small, very small.

55:59 But computer vision models is three orders of magnitude down there, and back in the days we thought it was big. And so size really we do see quite a lot of increase in size, and hence, that actually drives hardware innovations that drives back the software innovation. I kind of want to basically end the talk by a very interesting application that I actually did back in my last AI job as the big data and AI general manager back at Alibaba to showcase how difficult the real-world environment might look like and how AI isn't really able to solve all those problems.

56:36 So you know we were flying into and out of different kind of airports. San Francisco is a beautiful airport. If you go to JFK or LAX, you know that some of the airports aren't the most convenient ones. One of the biggest challenges of all those airport operations is that there's quite a lot of garbage time between a plane landing onto the runway and basically like that plane hits the tarmac. The thought process is if you're in the air, you don't care because you're watching movies and things like that.

57:06 The moment you hit the ground, it's like, oh, I need to get off the plane. Same thing for passengers, same thing for planes, but planes want a faster turnaround time so that they can be useful. And back in the days, this is done by manual ATC personnel and things like that. And if you read the news, you know that since the Reagan days, there is a growing shortage, a continued shortage of ATC personnel. And how do we solve that problem?

57:33 So back in the days, I was actually participating in some of the projects called smart city or smart airports to create a digital twin of this airport by laying out the map of the airport, laying out basically all kinds of operational data of the planes, their flights, and things like that, and using computer vision and sensors to basically lay out the positioning of different airplanes and things to analyze things like where are the hotspots? Are there some sort of taxiways that are actually blocking, unknown to ATC personnels, blocking the traffic efficiency and things like that?

58:06 And to increase-- normally, we claim that it was actually able to increase the efficiency of the airport operations significantly. Personally, off the record, I still find some of the airports that we worked on highly congested, so it may or may not be useful. But if you think about it, these are the real-world challenges that we're facing, not only basically saying, hey, here's the prompt, and please give me a text telling me what I should do. But really to connect data sources that aren't text, and then basically create actions that are not text either.

58:42 There's a lot of new innovations such as physical AI and things like that, robotics and stuff going on. Professor Fei-Fei Li from Stanford, for example, is having a company called World Lab that basically looks into these areas. So I think there's quite a lot of real-world problems that are expanding beyond LLMs, and there is a massive amount of opportunity that we could unleash in this round of AI bubble-- and I don't think it's a bubble-- in this round of AI innovation to create the next, I guess, industrial revolution.

59:14 So I'll stop here. Hopefully, some of the materials will be interesting to you. And happy to take questions, if you have. On a more central kind of world, how does the business model evolve to be competitive? Something like that. So I think basically the conventional model of hyperscalers, me working in one in the past is often basically selling compute, and then sell the operational expertise or basically the operational simplicity to customers.

59:46 And today, with LLMs, I think it becomes even more simple. So hyperscalers, I think, they are having a mindset shift from selling resources to more selling workloads or results or tokens or something like that. They do have a unique advantage in the sense that the pool of compute is big, and they themselves actually consume a large amount of AI compute as well, because internally when they train their models, so basically they run batch compute, things like that, that creates a bunch of compute need that can be basically plugged into the tidal wave of inference needs that unavoidably comes when at a large scale.

60:32 So, I think, basically, like hyperscalers would continue to have a very advantage in increasing the efficiency of the underlying GPU operations. A few hurdles today still haunts the field. For example, today, a lot of the training algorithms aren't really designed to be interruptible. Back in the days, I think, if you read Google's Borg paper, there's a very nice practice inside Google engineering saying that you want to build your workload to be interruptible and idempotent, meaning that you can rerun it.

61:10 AI is such a nascent and hot field that a lot of researchers, me included, are just a bit spoiled because we can just hog resources and the inefficiency we don't care too much about that. But I think as we get more and more mature in the engineering operations and care more about the efficiency and productivity balance, then hyperscalers would actually naturally use their scale as an advantage to continue winning. How do we think about the economy of token cost versus price?

61:41 So a lot of the inference basically, although, they aren't actually charging the price, the cost of serving those tokens is actually still higher than the cost. Or in other words, some of the application companies when they are charging their customers and then when they're paying the token model providers, the fee they pay the model providers are actually higher than the fee they collect from the users. This is indeed like a real question.

62:13 I think there's one gentleman who actually wrote it. The current coding bot models. They're basically saying coding bots when they're selling seats is actually a glorified version of insurance. And insurance basically is built on the assumption that people would consume less or basically saying, if someone sells you car insurance, they assume that you don't get into accidents. You don't want to maximize your accident count. And as a result, the economy analysis seems to be not adding up today.

62:51 I used to have similar thoughts as well. When I was doing the startup, one of the things that we cared quite a lot because we are infra people is we look at cost. And one thing I start realizing is that when the industry is going pretty quickly, there is, I would basically say, unavoidably or actually in a better way, waste or basically a money losing mechanism. Silicon Valley is probably the only place where investors are willing to basically believe in a random dude, the random technical dude coming up and basically say, I have this brilliant idea that's going to be losing money for many years, but it will be creating value, and give them money to do a startup.

63:33 I think the logic is that cost is never going to be a thing in the long term run of things. For example, today GPT-5 or GPT-4.40 and things like that are something around the price of GPT-3.5 a couple of years ago, but the quality is much better. So, I guess, the more slow equivalent would actually apply to the tokenomics as well in the sense that if I am only losing 50% of the money, then next year with the same quality, I will be breaking even as long as customers are happy and we're creating value.

64:09 So I think a lot of the companies in Silicon Valley actually build on this belief that we focus more on creating the value and worry less about burning money. And, of course, this is actually one structural risk that if there is no value, then this whole thing is a house of cards. But it seems that right now we see traction in search productivity, coding, and things like that. So I would be more optimistic on that. Thanks.

64:36 I casually mentioned that RAG is kind of falling out of favor, and there's new cool terms like agentic AI coming in. But it seems that RAG is still very useful because we cannot put all the information into a terribly long context and get an accurate result out. Yeah, I fully agree, actually. This is one of the interesting parts like in the startup land where there are new concepts coming in every day, and then there's a natural tendency, especially on Twitter, for better or worse, to chase the newest concepts.

65:09 Vibe coding, for example. And you're right. Basically, I think what I observe is that RAG or a good search algorithm is actually hugely important for us to get accurate information. It was my anniversary, basically. I was basically asking OpenAI, I was like, could you actually recommend me a few wine selections for celebration? And one thing I found is that it starts using web search and things like that under the hood combined with LLM. So I think RAG is not really fallen out of favor but has been understood or basically adopted in a more implicit way.

65:49 Kind of similar to no one talks about databases today, but we all use databases. So I think it's getting mature. And there are other teams who are basically still in a very vibe kind of state, and RAG is something that's actually been useful. The end result is a bunch of very relevant information. And the source is a bunch of too much information. So there has to be multiple stages. I think right now what seems to be useful is similar to the old recommendation systems as well is you use some weaker models to do a coarse ranking, and then you get a sufficient amount of candidates, and then you do a fine-grained ranking.

66:30 Now, normally coarse ranking is using keywords, vector embedding, and things like that, and then you use LLM and concept window because they already understands things much better. The first step is optimizing on cost, and the second step is optimizing on accuracy. And the balance in between is kind of something that people tune quite a bit in different applications. I think the concrete question was, when we're basically looking at the Berkeley website, and then we use ELMo to summarize it and do question/answering, where does the knowledge come from?

67:00 Is it that the model already learned that Berkeley was founded in 1868, or is it inferences from the context itself? This is kind of a question that bothers me quite a bit as well, mainly because, I think, in theory, the model should be basically only learning general practices or logic from the training data, and it should rely on the facts that are in the text.

67:33 But I mean, the boundary is kind of unclear because there's nowhere in the text that will basically say like a dog is an animal because we think that that's common knowledge. But what is the boundary between common knowledge and specific knowledge? And I think that comes quite a lot with-- that's one of the reasons that hallucination and things are just causing big issues today. And I think that's why RAG like an effective data search and data source is super important, especially in more enterprise and vertical applications, because general search is more about common knowledge.

68:11 But then verticalized search, like enterprise search, is more about summarizing, reading information. And that's why I think tying back to the last question, I think RAG and accurate data processing pipelines is super important. Quite a lot of orchestration abstractions in the general compute days and in AI days as well. Why is it continuing to be an important topic, and what are factors that made it hard? I guess that's the question.

68:42 Let me basically try to refine that question. Basically, there is premise seven or eight years ago about basically an abstraction layer that would allow people to very easily do like a cloud dev environment for multiple cloud and things like that, and why is it still not so? So I think I can basically answer it from two perspectives. One is the implicit difficulty in training, and second is the implicit difficulty in data. In training, in the past, a lot of the cloud practices are basically having the assumption that machines are all abstracted away as replaceable individual virtual machines, and their exact physical location in the racks or in the data center does not matter that much, because the workloads on top of there is not tightly integrated.

69:34 And today, what we found is that for AI training, locality actually matters a lot. When we're actually requesting for machines to train on a distributed training job, we better find these machines that are actually on the same rack attached to the same switch. So the conventional cloud practice of abstracting things away using VM and Kubernetes and things like that actually doesn't really work that well in AI compute. And that changes a lot of the narratives and infrastructure pieces and things like that.

70:03 The other thing is, when we talk about data. Data locality is still quite important because if I want to launch a machine somewhere, it's really easy to just install PyTorch. But if I have a petabyte of data that I need to basically install onto that provider's storage that's connected to that machine, then it becomes very fragmented. So for the cloud providers, they normally have Elastic Block Storage, Elastic File systems.

70:33 They have Object Storage and things like that. In neocloud or conventional Slurm environment, we normally have good old school NFS, Lustre FS, and other type of parallel FS, and providing an abstraction that would span across all these different types of storage, and also create a clean data pipeline to efficiently pull data into that becomes a challenge. And I think SkyPilot basically tries to abstract some of those away, but essentially, the data movement bottleneck still is there, making it not as simple as just launching a microservice.

71:09 And those kind of things makes infra evolution a little bit slow, that I learned in retrospect of during the startup was that-- and by the way, I mean brilliant co-founders and things like that, we've had a really good run. So it's not a failure story. But one thing we found is that as technical people, we've often been thinking about things in an efficiency driven or basically in an inside-out driven kind of approach by basically saying, how do we actually craft a very nice product that people will be able to use.

71:41 That actually led us to build really nice platforms and things like that our customers like. But at the same time, one thing that we've been learning from the last two years is that getting awareness out there is actually super important, and it's at least as important as building a product. And that's why there is all kinds of startup showcases and things like that, and sometimes you feel that all those CEOs are just going out there like clowns, basically, blatantly bluffing way, talking about how excited and how good the products are or things like that.

72:13 Having done a startup I start understanding those kind of things better, because just like the research paper says, attention is all you need, and only by driving attention and grabbing the customers and basically efficiently using the customer feedback to very quickly iterate on the products, because this is way more important than ever before, and then do this flywheel, can a startup be successful? We were joking that if we were to do Lepton around again, not saying-- it's still very successful and happy, but if we were to do it, again, we would be bullshitting more, in a respectful manner.

72:54 Kind of like observing great engineers and great scientists over the years, and one thing I find myself being very useful to brag a little bit is that I learned to speak the language of both sides. And that helped quite a lot, because researchers see me as their friends, because I can actually talk papers, talk about math and things like that, and understand that they do not really want to deal with infra. And engineers see me as a friend, as well, because they see that I've done sysadmin in the past and acknowledged the inherent difficulty of fixing machines.

73:30 And for better or worse, in many places, there is a bit of professional hatred between engineers and researchers because they don't go head to head. And I feel that being able to be in the middle and helping smoothen the message actually could help quite a lot. And then, in the end, people are like, oh, yeah, we're actually talking about the same thing but just with slightly different priorities and things. And a good thing is both researchers and engineers are very mathematics and logics driven.

73:55 So I think it's something that if there is a smooth conversation enabled across the spectrum, things get much better than when it's not. So the question is basically like, in this area where models are becoming more and more expensive to train, if one wants to do a startup, what might be places of its unique value? So I might have an overly biased view of that. My feeling is that basically like training yet another foundational model on test space, but that's unless you have a very strong financial backing and you have a team of really talented researchers.

74:38 And at the same time, I think there's quite a lot of application areas that get to that are yet to be solved, especially in the enterprise service land. And over there, I think large companies and even OpenAI and folks will not have enough focus onto that because they have a wider problem to solve. And as a result, the startup will be able to combine the domain knowledge and the nascent or basically the new techniques like LLM and things and combine it to solve a vertical area really well.

75:06 And that's the place where I think startups will be running much better and much faster than the big companies and create differentiated value. So if I may, I can summarize it as basically what is the balance between scaling and centralization versus more needed edge compute, distributed compute, and efficient compute on the other side? And today, we're talking quite a lot about huge clusters for training and inference. But then there are robotics and, say, mobile phone intelligence and things like that, on the other hand.

75:40 So I think this is basically the balance between exploration and exploitation, in the sense that when we are still shooting for the best model, we don't care too much about cost. We want to let people onto the moon or Mars. So burning a bit more fuel doesn't matter that much. And that's basically the trend that we are seeing in the last two years. There is this path of exploitation more efficient models happening for sure, especially in the robotics land, mainly because when it comes to inference, it indeed is like that.

76:10 This also kind of has-- and this is also kind of similarly happening in the cloud land where the telcos, telecom companies, and also the CDN companies like Cloudflare are basically thinking, how can we actually deploy AI compute capabilities on the edge node on the top of entrance, basically, pop entrances like that? And over there, you may be able to see a more distributed on computational resources closer to basically where the computation is needed.

76:41 And I think that's basically like when that's where people are looking into more efficient models, efficient compute and things like that. Still nascent but there are already a few companies who are trying to do that. So, for example, so someone who is basically [INAUDIBLE] student now teaching at MIT, one of their grad students were actually starting a company called Eigen AI, Eigen AI lab, which is proposing the idea of, if I remember correctly, AEI, artificial efficient intelligence, basically saying we want to basically build models that actually can run on resource-constrained platforms like a robot or a phone.

77:25 I will basically say it hasn't been the main interest in the last two years, because there is so much top line to be reached, but as we look more and more into the application areas, I'm sure that they will become ever more important. It's interesting. Thanks. Last question, basically. So I think the question was, could you share the biggest technical problem that we encountered in the Lepton days?

77:58 Interestingly, I've got really great co-founders, so technically, it wasn't really too much of a trouble. So one of our co-founders is the creator of etcd, which is the fundamental database on Kubernetes, so I relied to him all kinds of Kubernetes infra problems. The biggest slightly related to technology challenge that we had back at Berkeley labs and days was actually supply chain management because there's such a shortage of GPUs, and GPUs are such an arbitraged market. We found ourselves to spend quite a lot of time procuring GPUs and negotiating contracts and negotiating pricing terms.

78:37 Can we do a six-month term? Can we do a three-month term? Can we do a committed spending but with flexible scale up and down over a period of time and things like that? That consumed quite a lot of time of hours. And interestingly, when I was talking to a lot of the fellow startup founders, those were the problems that they were facing as well. So I really hope that basically, in the next months or years, there could be a more fluid marketplace where the startup founders and enterprises can actually go and find efficient computation resources in a multi-cloud or across the globe so that they can focus on building models and applications and rather than basically doing InfraOps, which has been the 30-year-old topic.

79:25 So that was one of the big problems that we encountered.