Transcript
0:00 Hello, how are you? It's really a real privilege and honor to introduce our featured guest today. One of the people who has helped shape modern computing and AI, Jeff Dean. Jeff Jeff joined Google in 1999 and over the next 29 years or 27 helped build many of the foundation of modern computing and AI from map produce to big table to tensorflow to Google brain.
0:54 His contribution has been recognized and the highest levels. He is a elected member of national academy of engineering. He is recipient of both the ITE John von Newman medal and ACM prize in computing. And now Jeff is beginning a new chapter and today is his first day here in Stanford.
1:26 That makes today very special and exciting. So I can't think of a better person to have this conversation with Jeff Men Dong. Don, please. Don is one of the leading researchers and the intersection of AI security trust with computing. Her work and Berkeley has been enormous.
1:59 She is Mac fellow and the elected member of American Academy of Science and Arts or Arsenal Science. she is also a great startup professor. He had four startups that reached one of them reached 100 million revenue. What I have always admired Don is she doesn't just stop doing great research but also really turn that into real impact. Just like Jeff Dawn also is beginning a fascinating new chapter from Berkeley to matter one of the largest organization on AI.
2:42 So we have an interesting contrast today. Don is moving into large platform and the Jeff is leaving a very large platform to start something new. Apparently even among the two best people today on stage cannot agree what is the optimal size of a company doing AI. So I can relate a little bit myself.
3:14 After 30 years working at Microsoft, I joined Zoom. So perhaps the real lesson simple. It's not about the size of the ship. It's about whether there's a new exciting blue ocean worth exploring. And AI today is absolutely that new blue ocean. The last decade has been about making AI more capable.
3:46 Now the next chapter raises the bar for us to have to answer even more important question. What can we actually accomplish with all of this great intelligence? Can AI cure cancer? Can AI make our society better? Those are some of the questions I hope we can explore today. So please join me in welcoming our fireside chat Jeff Dean and our moderator Don. Thank you.
4:26 >> Great and thanks a lot for joining us Jeff. >> Yeah thank you. >> Really excited. I mean the whole entire auditorium everyone is really excited. so before we start so first just some personal notes. So Jeff and I we have known each other for a very long time more than a decade. I wanted to just briefly share actually one of my most memorable moments. interacting talking together with Jeff. So this is long time ago almost a decade ago. so around Illear 2017. actually is one of the top machine learning conference back then was much smaller was actually only like a few hundred people. Now it's like tens of thousands of people. so remember at icleare so my group actually our paper won the best paper awards that year at iclair for a paper on generalization for neuro program synthesis. This is actually way before now LM based code generation. so Jeff thank you so much. So you I don't know whether you remember you actually came to me to congratulate me for the best paper awards and Jeff told me at this clear there are three best paper awards one from me my group and two from Google.
5:55 >> So that was thanks a lot. That was very memorable. That was a very good paper you wrote. So, >> no, thank you. Thank you. >> And also I remember how excited you were at the time then telling me about one of your recent work back then which is actually right a mixture of experts. -huh. >> So, I mean the paper is titled of course back then it's a longer title is outrageously large neuronet networks the sparsely gated mixture of experts layer.
6:27 >> Yeah. >> Yes. So when I when I heard when I was listening to you explain to me about the paper I thought wow it's really fascinating. But however back then I didn't foresee I don't know whether you even you know had could have foreseen how influential the the work has become essentially has underpinned now almost our frontier air models and our architecture. Yeah, I mean I think the in the intuition behind thee work was really that you want to have really really large models that have a lot of capacity to remember lots of things but that you want to make them efficient by only calling on the parts of the model that are most useful and you can have kind of modular architectures where you have lots of different kind of experts and you learn which experts are good at which thing and then when you're in the context of processing a particular request or particular token you can activate just the portion of the model that makes sense for that token, right?
7:31 Like you have in a real brain, you have lots of different pieces of your brain that are good at different things. There's one that's thinking about Shakespeare and sonnetss. There's another that's like active when there's a garbage truck backing up at you while you're driving and those are not active at the same time. So, it makes sense to save energy to be able to have very large capacity and then find the right pieces of the model to activate. So, we were pretty excited about that and we that was one where we sort of knew it was going to be a big deal because we could see like 10x better sort of training compute to quality ratios than if you had dense models which were kind of the main thing people were using at that time. So, you know, one of these ideas where you see 10x improvements, you're like, okay, that's that's pretty significant. It's probably going to be an important idea. So, >> yes. Yeah, it's amazing. I mean even just listening to your explanation now actually it's quite similar even to the explanation back then when you right when you were explain to me so it's amazing the timelessness of the idea >> yes so Jeff you have done so much work amazing work looking back in the past decades essentially over the past you know several decades you've helped shape nearly every major computing platform and also many key advances behind modern AI from map reduce and big table to tensorflow TPUs and Gemini. so yes we have you know really a lot to talk about. So looking back, what has surprised you the most about the trajectory of AI over the past decades? And is there something you think the field is still underestimating today are missing today? Yeah, I mean I think obviously there's been a lot of growing interest in AI starting, you know, we were starting to do work on scaling up deep learning models in maybe 2011 2012 was kind of when we really started that at Google and you know we were starting to see amazing results from training neural networks say that were 50 times bigger than had previously ever been trained and you would get like amazing reductions in you know image classification error rates speech recognition improvements were were kind of like equivalent to the previous 20 years of speech research in terms of lowering error rate by just scaling up a fairly simple deep acoustic speech model. So we sort of saw that trend pretty early and knew this was going to be important for all kinds of different you know things in you know really hard computer science topics like computer vision or language processing or speech recognition.
10:24 and I think one nice thing about kind of the deep learning paradigm is it's really unified a lot of these fields that were previously kind of had separate kind of particular techniques that the fields were working on. and by essentially learning from very raw forms of data, we've been able to have these, you know, kind of paradigm shifting things where all of a sudden you squint at problems differently with this general learning lens. that has been really helpful. And I think over the last you know say 15 years more and more people around the world in more and more kinds of disciplines have really just realized that this is a extremely powerful way to model data to be able to make predictions to be able to you know make important kinds of classifications in you know medicine and science and you know consumer products and chat bots and all the kinds of things you're now seeing. I guess the the level of investment in the the compute hardware and so on is is is quite large. So I think that's a you know a sign that people are confident in the you know the ability of this technology to change the world even more than it already has. so that's that's kind of interesting and exciting. so I guess that would be a summary of of what's been happening.
11:47 >> Yes. Yeah. Thanks. And we'll talk about now you'll be right even furthering the journey with your startup. We'll talk about in a in a few minutes. >> And so again, yeah, you've done so many really foundational seminar work. We don't have time to go through them all. So maybe just a couple of them to share more of your the lessons and in insights that you have learned. so let's take one example, TensorFlow. So, TensorFlow really helped make modern deep learning accessible to millions of researchers and developers around the world. So, when you designed TensorFlow, what principles mattered the most? And looking back, what would you design differently today?
12:35 And what lessons from TensorFlow do you think should guide the next generation of AI frameworks for increasingly agentic AI systems? Yeah, I mean I think we had built a previous system within Google that was not open source called disbelief that had a way of expressing various kinds of neural network computations and then could map those computations onto various kinds of hardware platforms you know CPUs, GPUs and it would distribute computation in a way that was kind of seamless to the user. So you could say I want to train this model. I want to use a 100 computers and you know 1,600 cores or I want to use 500 GPUs or or whatever. And then the the framework underneath would take care of that. And so we wanted to bring that kind of you know the ability for the ML researcher or developer to be a little bit abstracted from exactly how it's mapped onto the compute hardware because that's a really nice elegant abstraction. You can say I just have this beautiful ML model in my head and I don't care how you implement it but please make it run really fast. so that was one of the things we wanted to bring to TensorFlow and then we wanted to generalize the kind of computational abstractions we were providing to be just a combination of a computational graph of various kinds of you know ML oriented operations like matrix multiplies and vector operations and so on. so I think we got that reasonably right and we wanted to open source this so that people all over the world who have machine learning problems could you know start to take advantage of being able to have a common framework to express those ideas. You know it's much nicer when you have say a research paper that comes with a implementation of the ideas in the research paper rather than everyone trying to recreate that paper from you know the rough text description which always leaves a lot of details out. So that was kind of the idea behind Inflow and then we saw a lot of applied uses of AI that we thought having a a framework like that would be really useful for people. things we did wrong I think well first we didn't have kind of this eager execution mode that has been popular in frameworks like PyTorch and Jax and has been added to TensorFlow. so I think that kind of ab improved the abstraction further. and then the other thing we did wrong was in the open source release, we created a subdirectory called contrib and we allowed lots and lots of different external people to contribute all kinds of different sort of helper libraries and things ways of doing things. and I think that just confused the community a lot. like we should have kept the core TensorFlow release without that directory because what ended up happening was there were 10 ways of doing everything depending on which subdirectory of the contrib thing or which particular sub libraries you use and it would have been better to have those be built libraries built on top of the core TensorFlow thing. So I think that that caused a bit of confusion in the TensorFlow user community you know which of these sub contrib things should we use. So, you know, always good to learn lessons from things. So, we would not do that if we were doing it again today. But I think, you know, it really did help lots of people around the world get introduced to machine learning, be able to start doing machine learning on important problems they cared about and I think that's been good.
16:07 >> Yeah. Thank you. Thank you. Absolutely. Yeah. Thanks a lot for sharing this really valuable right lessons and insights and so on. And I still remember before TensorFlow you know a lot of people I mean including my students and other collaborators and so on like really it took so much efforts for people to just even implementing right the basic >> Yeah. And if you want to do anything at scale you have to sort of build a distributed system underneath the covers. Exactly. Exactly.
16:36 >> And if you can sort of avoid having to do that and have a common abstraction that's always a good >> Right. Right. Right. Yes. I really I also really loved the computational graph abstraction and so on was really elegant >> as a compiler person. I like computational graphs. >> Yes. Yes. Absolutely. Yes. yes. Thanks a lot for the great right contribution to the field and so on. So now let's fast forward. So Gemini of course is Google's latest generation on frontier models and you have really taken a significant right have made significant contribution in leading the effort as well. So now looking back on the development of Gemini what surprised you the most and what lessons from building Gemini do you think that you can share and can help shape the next generation of AI systems? Sure. I mean I think Gemini is really the culmination of a few earlier research projects both within legacy deep mind and Google brain and other parts of Google research. And at that time we sort of realized that we were all converging to very similar kinds of directions like training trying to scale up the size of models we trained. we had some efforts independent parallel efforts on how do you make language models also be multimodal so they can understand images and so on and you know I wrote a onepage memo I'm like this is just silly we should just all work together let's combine our people and ideas and compute resources and train one you know model that is multimmodal from the start that brings our best people together from across you know multiple research organizations within Google and I think that was really good. my colleague Oriel Vignyals and I sort of founded the project and were the co- tech leads that that started it and we sort of brought people together. I had worked with oral previously because he was in Google brain and then had to move to London for personal reasons and so ended up moving to Deepmind and but we kept in touch and so we sort of saw a way to bring ourselves back in closer collaboration and you know I think that's been a really successful thing. I think the focus on making the model u multimodal from the beginning has been really really good. You want your the model that you're going to use for everything to understand text and language and code and images and videos and audio and other modalities besides so like we put a little bit of say LAR data in the training data so it at least knows that LAR data is a thing because that's an important use case for you know further training of Gemini models. so that that is important and that's that's been I think a success of a decision we made at the very beginning to to bring that through all the current Gemini models as well. I think we we had a little bit we wanted to make the model good at lots of things and so I think maybe our focus on making it amazing at coding was lagging a little bit and we realized that and are work you know work to catch catch up on that and I think we have good efforts underway but you know I think by focusing on on that you end up with a system that is able to really do a good job of reasoning and you know doing other kinds a task that where it needs to sort of work its way through you know breaking a complicated problem down into you know multiple subpieces and so on and so that's a a good you know if you improve coding you also tend to improve that capability in non-coding things so >> yes yeah yeah that's great yeah I mean Gemini also right started so many of the the the first like advances and and so on yeah was back then also I was really impressed that Gemini was one of the earliest foundation models that like you said actually started with the mult model from the beginning is a big advantage.
20:36 >> Yeah. And we've also worked on generative models. So having the ability to generate images in not just sort of use them as input but also as output and also videos to generate videos to generate audio input and output. So >> great great yeah thanks. So yes so looking back I almost call it's the midest touch. So essentially this is one thing that stands out right about your career is that so many of these ideas from map reduce toe and and so on they actually took years before the broader community realized how important they would become but you had the latest touch and you had the special right like really foresight to actually build and develop these ideas and systems and so on. So I think many people in the audience would love to learn like how how do you do it? How do you distinguish between technologies that are genuinely foundational and those are simply fashionable and what engineering principle that has remained surprisingly timeless throughout your career and also as AI shifts from models to increasingly autonomous agents.
21:54 what new direction and new abstractions do you think we'll need over yes the the next few years and decades for future >> yeah I mean it's a it's a good question I mean I think I've been very lucky maybe as part of it but I also think one of the things I I try to do is keep track of like a lot of different trends or like research topics that the community is exploring you know I often tell students it's better to to skim skim 10 papers than to read one in detail because you kind of then get 10 points in your cloud of like what might be possible. or even skim a 100 abstracts because what you want to be able to do is connect you know important ideas that have not yet been connected.
22:42 And sometimes when you think about a hard problem, if you have some of these like things that are now kind of vaguely starting to be possible in mind, that can help you kind of shape a a full solution to a problem where instead of it seeming like seven unsolvable problems, you can squint at it and say, "Okay, well, these five things, I know there's sort of vague work going on that seems important and might be able to address some of the things." And then there's two sort of additional things that I have no idea how to do, right? And the but if I work hard in them, I can imagine solving those. So that's kind of like the perfect shape of a problem that you want to work on in a reasonably long-term manner like five years or something because I think the five areas that seem kind of not fully solved but kind of are starting to take shape and then the other two areas where you're going to have to sort of really think hard and figure out new new techniques and so on is like to me a really good level of risk to take on Like you don't want to pick problems that are impossible and will take 20 years to solve and you have no idea how to do any of it. But you also don't want to pick like a two-year problem where it's very obvious what you need to do because that's less about, you know, advancing the field in dramatic ways.
24:04 It's more about like practical engineering. And sometimes that's super important to like do the things that are incremental improvements to what we're doing now. But you want to look for the things that are, you know, maybe very different ways of doing things that might be possible or might be possible soon. and I think one of the things that I also tend to use as an engineering toolkit is just being able to do back at the envelope calculations of like, okay, well, if I wanted to do that, you know, how long would it take to process that much data? Or what if I needed to send all that data over this kind of network? Is that even feasible?
24:39 Or would that take a 100 years? If it's 10 seconds, that's very different than 100 years. And being able to sort of mentally think through possible solutions or possible problems and understand the shape of solutions from those first principle kind of engineering, you know, rules of thumb is a really really helpful skill and it's also something you can kind of do in your head and practice on your own. Like you're like, "Oh, I wonder how I would solve that. Okay, I would have this kind of thing. If I did it that way, it would be, you know, this kind of approach and if I did it this way, this kind of approach and this one seems much better than that one because my rules of thumb say this would be 10x better. And I think that's just a good general engineering skill. It mostly comes from practice and repetition and seeing ways of other people solving such problems.
25:29 So, I don't know, it's not like I have any magic answer. and I've done a bunch of things that have not worked out well, too. So, that's also a good good tip. >> >> Try lots of things that might not work. Some of them will. >> No, thank you. Thank you. Yes, this is right. These are great insights. right. So, as to summarize, having a broad overview of the fields and the trends where things are going and also having a good balance not too near-term and not too long-term where we don't even know where to start and also being grounded in first principles.
26:08 and the underlying engineering principles. These are right. Yes. A great combination for people to potentially learn how to get this minus touch. >> yeah, thanks. So, so now let's look forward. so even just in the past couple years Frontier AI has been making amazing advancements. so for example even just so last weekend actually at Berkeley we hosted the agentic AI summit with close to 5,000 inerson attendees and 100,000 joined online and so on and we were really celebrating the huge advancements in agentic AI >> but also at the same time there was a lot of discussions actually with the underlying currents is as well we want the society to benefit from the great capabilities and power of agentic AI but also at the same time we are seeing increasing different types of risks. So for example right as you know so my group for example we have been doing a lot of work in frontier AI in cyber security >> and my group for example we've developed like cyber gym exploit gym these are leading benchmarks evaluation for evaluating frontier capabilities in cyber security that's been used by all frontier labs in their including their system cards to and demonstrated the really fast increase in frontier AI in capabilities in cyber security and many people here may have heard about the recent incidents of open eye hugging face and others. So in this case actually opening eyes agents were trying to solve actually tasks in exploit gym in our benchmark and the agent decided that there may be information from hugging face that could contain you know data that help the agent to solve the task. So the agent decided that they actually exploited several vulnerabilities and had a very sophisticated attack chain and it lasted over four and a half days and the agent eventually by breaking and exploiting vulnerabilities, breaking out the isolation environment and also established some stepping stone through some third party platform that eventually was able to actually compromise hacking face infrastructure and so on. And luckily in this case it's the agent only tried to get data related to cyber explosion and didn't cause other harm. But however in other incidents and also in the future such behavior from agents can of course cause even cause much greater harm and risks. so one question for you is what's your thinking in this space?
29:26 for example also recently we have seen leaders issuing different various open letters. Dennis had a recent call for this new entity for to help with AI governance and Jensen had his call for open open ecosystem, open way models and Mark Zuckerberg called for building AI that benefits everyone and so on. and also recently including me over a thousand leading air researchers from Frontier Labs also signed this letter for pacing the frontier and so on. So I think this is on top of mind for many people. So what's your think and what's your thoughts in this space?
30:12 >> Yeah, I mean obviously when people are building these models they can be used for lots of different things like lots of different kinds of technology. You know, I think the vast majority of the uses of these models is incredibly positive for the world. Like advancing, you know, AI in healthcare and AI in education, you know, being able to make people able to solve problems they couldn't solve on their own will make people able to do more. And I think that's super exciting.
30:43 Obviously, they can also be used to find security vulnerabilities, and that's a double-edged sword. You could use them to patch security vulnerabilities that exist and there are lots of them in the world and you could also as if you're a malicious user you could use them to to exploit them. I mean I don't think that I mean yes these models can now do things that sophisticated human cyber security attackers could also do maybe even beyond but I also think they can find vulnerabilities that sophisticated human you know cyber defense engineers might not be able to find. So there's always this balance of you know you know, people working to defend security system, computer systems, and people working to attack them. And I think they just now have much more sophisticated tools on both sides.
31:40 so, you know, I I'm not a cyber security expert, but I think that it is definitely a a cause for concern. I think in a lot of cases things you don't want models to do maybe have non-technical solutions like you just make it highly illegal for people to break into computer systems. That's already kind of illegal. you could consider, you know, what kinds of u regulations would make sense in that space. and we will as society get through the kinds of things that we don't want these models to do and really promote the kinds of things that we want the models to do.
32:17 Yes. Yeah. Thank you. yes. Agreed to write to hear your thoughts on this. Yes. So the important part is as frontier AI makes these fast advancements. on one hand right we want to we want the society to benefit >> from the the good use use cases. At the same time, it is important for society to develop both technical solutions and also these non-technical solutions to actually help mitigate the broad range of u I actually worked on a paper a couple years ago with a bunch of awesome colleagues including John Hennessy and and Dave Patterson and others about you know seven different areas where we thought AI was going to impact significantly many of which are you know positive like healthcare and education. Some of which are kind of you know like geopolitical risk and computer security risk which are you know less obviously purely positive and then you know areas like job displacement which can be you know a complex economic issue that you know many people at Stanford here are are studying and looking at. So you know I think that paper is is pretty interesting. It's my only paper with its own website shapingai.com because we had a ambitious co-author who set up a website.
33:42 >> Yeah. Great. Great. Thank you. so looking forwards another topic that's generating increasing discussion is recursive self-improvement >> where essentially AI systems they can actually continue to improve itself to learn to improve itself and so on and in particular with this RSS recursive self-improvement AI the the speed of AI advancement could increase further and so on. This is also one of the key reasons that underlines the recent letter open letter that I mentioned this pacing the frontier.
34:20 I'm curious to hear your thoughts. What's your thoughts on recursive self-improvement? What's your timeline? How long do you think that will take? and also as I know this also relates to your your now just started really exciting startup. So love to hear your thoughts on that as well. >> Sure. I mean I think the idea of using machine learning to improve machine learning is not not a new one. you know my colleague and actually co-founder Quacle and others sort of kicked off some of the early work in this space with neural architecture search which is a you know way in which you can have a model generating model that generates machine learning model architectures and then evaluates how well those model architectures work on a variety of metrics like can they learn quickly, what is the compute cost of training them and so on. and then through an iterative reinforcement learning based process, the model generating model can get feedback on which kinds of decisions in the model architecture made sense and worked well and which decisions seem to be bad ones and over time it learns to generate higher and higher qual quality model architectures that can learn more quickly and achieve higher quality. As I remember that's one of the earliest papers where the the paper had a price tag of I think a million dollars.
35:45 >> Yeah. Several million that's list price. It was actually much cheaper than that internally. But yeah, I mean and so it used actually very tiny models to evaluate how effective the architectures were and as you learn which ones seem effective, then it would occasionally do experiments that were scaled up versions of those to evaluate, you know, does that actually hold? and and Quac and others did some later work on the evolved transformer which took kind of the basic ingredients of the transformer architecture and used an evolutionary algorithm to sort of find different ways of assembling it and mutating the components to then come up with a transformer that was like 30% more efficient than the plain vanilla transformer. So these ideas I think are really important and recursive self-improvement is about how can you make the entire set of things that are necessary to go into a model get better in an automated way. And so like often you have teams of people working on evaluating what kind of data is going to be most effective to improve the model's quality or you know what kinds of eval are needed to help evaluate the the model. and what kind of model architecture as I mentioned. so I think you can actually have pretty effective automated loops of all of those things that are all working to improve that particular aspect and put those solutions together into something that can then you know improve the overall model and quality and mix of data and so on. and if you squint at a lot of modern problems in science and engineering, they actually all have this form of there's a sub there's a sub there maybe a large problem that you need to break down into a bunch of sub problems. Then there's a sub problem and you need to sort of figure out what a possible approach to solving that problem is. You need to implement and try that approach. You need to evaluate how well that approach worked. And then you need to iterate and get the feedback about how well that worked back to the process that's going to inform what is the next experiment you do. So science you know this is the basic scientific method. This is the basic way in which you do engineering design is you iterate on your design and you compare various attributes. so the idea behind our new company Discovery Loop which is a public benefit corporation and our you know fairly ambitious mission is we want to automate machine learning science and engineering so that we can improve the rate of discoveries across many different fields and obviously we're going to start with a narrower set of of domains because we think that's important to have a little bit of focus initially but we think there are a lot of reusable pieces of infrastructure and common techniques that are common across domains and also by building models that are really good at understanding many many different domains of science and engineering you can get kind of PhD level expertise in a model across many different domains and so no one human has PhDs in 20 different fields but we think by having that ability you'll be able to really figure out what are the important subpros orchestrate you know, agents and multi-agent systems on solving some of those subpros, recombining the results of the sub problems into solutions for the larger problem and continuously iterating on these sorts of loops. And we think being able to run those iterative loops each iteration, running those more quickly by building the right kinds of tools that can implement an experiment or evaluate an experiment quickly. so that you can say run an iteration of an experiment in a minute or an hour instead of a day or a week.
39:47 and then being able to get feedback from running, you know, thousands and thousands of experiments in parallel and use that feedback to then decide what are the next set of experiments you want to run will lead you to higher quality experiments that you run also. And so increasing both the the speed at which you can run experiments and the quality of those experiments we think is actually going to lead to something pretty pretty amazing. And there's infrastructure you can build that manages like the whole set of experiments you want might want to be running next and tries to evaluate them in terms of like well what do I think the likely expected value of this experiment is and what is the cost of the experiment in terms of compute resources or other things. we think that's a pretty pretty exciting direction and so that's what we're setting out to do. Me and my four co-found or three co-founders, Sanjay Gimawat, Oral Vignials and Qua Clay.
40:41 >> We've and it's super fun because we've worked together for anywhere between 14 and 30 years. and we've collaborated in pair-wise versions on a number of different things including you know a lot of the things you mentioned map produce and big table and spanner and tensorflow and you know a lot of the things like model distillation and architectures and so we think it's just going to be a lot of fun to collaborate as a a group of four and to build an awesome company that is working to advance scientific discovery and as a public benefit corporation. We want to bring those scientific discoveries to as many people in the world as possible. So our goal will be to distribute those widely and we might make decisions that are not in the company's financial interest but are in the broader societal good of getting those discoveries out.
41:35 >> Yeah. Thank you. Thank you so much. I mean that's Yes. Yes. >> That's that's a that's a great vision >> and I've been working there for 12 and a half hours. >> Thank you, Jeff. I was unemployed at midnight for one second. >> So, thank you, Jeff. Thank you, Don. Now, we open the floor for two questions. That one first. >> Please go ahead. >> Thank you very much to our panelists for such an inspiring chat. my name is Joyce from Chong Business School in China. and I'm a shareholder of both Google and Meta. So, I have two questions if I may for one question for each. So the first question for Jeff you know your resignation caused Google to lose 200 billion in market cap on the day. So it's clearly a big deal you know in the eyes of of the world. so my question to you is what do you think you can achieve within a startup that you cannot achieve within Google and and also one thing that puzzles me is with Google's you know compute and all the talent you know what what do you think in your that Google needs to do to catch up with you know for Gemini to catch up with Fable and my question to to Don is why Meta out of all the big tech companies you must have offers from everybody so why meta Thank you.
43:01 >> I'll go first. yeah, I mean I think yeah, I mean I don't attribute things in the stock market to any particular events because it's always hard to >> It was you Jeff. >> you know I think first I have incredible fondness for my time at at Google. you know I've been there 27 years. Amazing colleagues. So they're in good shape. They have a plan for making the Gemini models. Awesome. It's gonna be great. and I'm excited to go off and do this. I think it's going to be really, really fun.
43:39 Sometimes a focused small company with everyone focused on exactly that mission. And I think it's a great mission is going to be amazing. okay so so I answer the question yes so actually many the a number of the leading yeah companies Frontier Labs have reached out to me in the past including Google at the time >> because you do great work.
44:14 >> Thank you. Thank you. Thank you. but I at at the time I wasn't quite ready to right to join like big tech companies and so and also I have been doing my own startup as you know mentioned I actually have done four startups in the past so the most recent one actually was acquired by Meta and at the time deciding whether we wanted to join Meta that was also actually difficult ical decision.
44:46 The startup was doing really well and we have great customers in Fortune 500 companies and and so on. but for I think also has a great platform. I think it's really meta Google like really arguably the the biggest distribution platform for meta with the billions of users and hundreds of millions of businesses on the platform and with the technology that we have been building really as the leading technologies for providing air and security and so on I think can actually have a huge impact with such a big platform. so yes, with that I'm very happy and also it's only been just a few weeks. Really excited to to help >> so we have last questioning this gentleman.
45:38 >> Hey, my name is Jonathan. I'm an engineer. So a big fan and user of your work. I promise I'd be asking this even if you didn't make a giant career switch 12 hours ago. the every first principles analysis of the AI industry would suggest that data is a large moat so you need a large company to win you need a lot for a distributed system you need extreme capital expenditures so large company would win and you and your founder co-founders and individuals like nomir are evidence that excellent engineers can make outsiz impact at a large company and so I have a I have a non- Googlele specific question but I do want to ask for your inside baseball perspective. why does it seem like a lot of AI talent retention and all of the frontier research despite structural reasons for you assuming them to be happening at a large company? why is all of that seem to be still going towards startups? and it this doesn't have to be a Google specific question, >> but why are small companies seeming to win over large companies when every in conventional wisdom would suggest that large companies are absolutely going to dominate this industry?
46:49 Yeah, I mean I think first there's the rise of cloud computing and you know deployments of very large amounts of ML oriented compute in various cloud platforms and so that makes it possible possible for you know a small set of people to you know raise a fair bit of capital and be able to use those platforms without having to build them themselves right and so that I think enables people who have you know a dream or a vision or a a direction they think is really interesting to explore to do that in a smaller setting than a company that has to sort of deploy all the infrastructure and we can rely on amazing companies like Google you know or other cloud providers to do a lot of the heavy lifting to build out the infrastructure that is needed in order to support you know people's you know idea of what a company should be and We're super excited as a small company about working in a really focused way on science and engineering auto automation because we think that's just a a really ripe area for for something and we could have done that within Google probably but I think the the mission of you know 10 people in what we hope will be office space somewhere in Palto just focused exclusively on that is just cuts through a lot of things that you know are slight distractions in a large organization, but large organizations are awesome in in many ways and I have incredible friendships and you know benefit that have benefited from all the things that large companies provide in Google setting over many years. So it's a little nerve-wracking to go out without that kind of support. but at the same time it's also exciting.
48:39 >> Thank you. >> Okay. Thank you. >> All right. I'm sorry we are out of time. I know there are a lot of questions, but I hope we can use AI to guess what the Jeff and the Dawn will be thinking. Thank you. Thank you.