🔬 Google's AI Scientist Started as an Attempt to Automate Kaggle — John Platt, Google Fellow

0:00 Are you talking about introducing explicit priors that you know based upon some human intuition or maybe in this case LOM intuition >> when you talk about multiple hypothesis testing right there's predictive models and there's descriptive models a predictive model is like let's say you just have a you have some inputs and you have some outputs and you just I just want to build a piece of code that tries to just have the lowest error rate on some data set statistical model a descriptive model is actually what science is trying to get to which is okay it should be able to extrapolate because it has sort of the physics or the actual some description of reality that's captured within it and then you can use it to extrapolate yes Newton thought of apples and gravity but gravity isn't actually about apple right if you take the 17th century machine learning model like oh apples will fall but how about planets you know I don't know I have no data about planets so who knows what they do right the distinction between those is a little bit blurry right because when a physicist or scientist comes, they use their intuition or maybe even more than intuition. Like essentially there's maybe a a solid pile of facts that they know about the world and then they make sure that whatever model they build is sort of consistent with what's known.

1:17 My co-host R.J. it's a pleasure to have John Platt, you know, with us today. John is a Google fellow and head of applied science at Google research. He has really a fun like background. He I guess you described yourself when we were talking a few minutes ago as a mega nerd. >> Oh, gig nerd. >> Giga nerd. Giga nerd. He's excited in absolutely everything. And it really it really shows Yeah. You was correct me if I'm wrong about any of this stuff, but so you started college at 14 and started your PhD at 18 at Caltech.

1:47 You're you were advised or co-advised by John Hopfield, right? >> Oh, yeah. Yeah. >> Yeah. Yeah. Who just won a Nobel Prize in you know two or three years ago. >> Yes. So John created several responsible for several textbook algorithms. one known as plat scaling another one sequential minimal optimization which is the textbook algorithm for training SVMs. even today it's still if you use sklearn it's there. John is discovered and named two asteroids has a Oscar for technical developments from 2006.

2:18 so if you've ever watched a a Pixar movie you've seen John's algorithms and work. John has an airdose bacon number of six or three three and three from either side and I'm going to skip over like 20 years of your career but then jumping to Google at you know working at Google sciences you worked on fusion quantum computing climate modeling and many other topics is that more or less right >> that that's right yeah >> okay okay cool did I miss anything important for today >> no I mean I've also done you know lots of applied math and signal processing and all sorts of fun things that.

2:52 >> Yeah. Yeah. I think you also your your Wikipedia has a fun story about patents and the iPhone too. >> The iPod. >> iPod. Yeah. Yeah. >> Yeah. Welcome. >> Thank you. Thank you for having me. >> Can you tell us about the ERA is the I think the way that the acronym is pronounced. >> and and I know that there's a lot of different semi-related stuff out there both within and outside of Google. So what can you tell us a little bit about the details of ERA and and what makes it special?

3:25 >> Well, we've been doing u sort of AI for science in Google research for more than 10 years now. And around 10 years ago it was very much using I don't know what you call it now maybe classical classical machine learning models you know things like you know convolutional nets or whatever to and they were specific models to build to you know solve specific science problems but about 2 years ago we got very excited about these more general LMS that have popped up in the last few years and we were wondering what can be done with them and of course a lot of people have been playing and trying to figure out what the right what the right thing to do is. And we kind of stumbled into this mapping. in other words, we found that many different scientific problems can be mapped into something we call scorable tasks. So you can often phrase a scientific problem as a gosh, I really would like to have a piece of code that, you know, maximizes some score. And it's surprising the number of different sort of scientific problems you can make a lot of progress on by mapping into that framework. Well, one thing is a lot of scientists spend a lot of time sort of building models. They might be statistical models or they might be you know physically based models. and if it's a statistical model like in machine learning your scoring function is well I have some data set and I'd like to have the fit the model and the data set go up. and we can talk about overfitting in a minute, but u >> that was one of our questions.

4:56 >> and then that's actually very that's actually very interesting. So machine learning is kind of a subset of this sort of scorable task, right? But you could do other kind of things like especially Michael Brener who's the lead author on the ERA paper. He's very very skilled because he he he likes to to sort of knock out a scientific paper in an evening now with a tool. so there's something in applied math called asmtoic expansions and which is you're asking how does how does an ordinary differential or partial differential equation but say ordinary differential equation behave there's some parameter has an epsilon in it and you're trying to say how does it behave as epsilon goes to zero and you could turn it turns out you can turn that into a empirical task by essentially asking that it it proposes some solutions that are asmmpyically correct and you check to see if the asmtoic solution is correct for like epsilon equals 1 e minus 4 or something and then you you check that fit but then then you ask you ask Gemini which is the core AI underneath it to to do the mathematical reasoning try to solve the problem while while also maximizing the fit to the to the data. So you can actually there's a lot of sort of tricks you can do because it's not that the underlying thing that's that's altering the code or the underlying thing that's sort of making the decisions is not a random process. It's an AI itself that is smart and knows about things and knows a lot about the world. You can get a lot of solve a lot of interesting problems because that sort of core inner loop is an AI that has huge amounts of prior knowledge. So that's sort of the the trick. So, we we we've been running around trying to map lots of scientific problems into scorable tasks and trying to solve them and it's it's really been kind of fun and I'm happy to talk about the ones that I've been involved in at least.

6:47 >> Yeah, I would love to hear about some of the more So, it's a stat statistical one is what everyone listening will probably know about. What you just mentioned makes sense. What are some of the other interesting ones? >> let's see that that are not statistical. Let's see, cuz we have interesting ones like one that we just put a paper up on archive is or actually I think it might be on GitHub. It's you often run into this in remote sensing because there's always a trade-off. There's satellites flying above the Earth and there's a trade-off between how frequently they can revisit a spot on the Earth, what their spatial resolution is, how big the pixels are, and their spectral resolution, so how many bands they have. And ideally you'd like to have monitoring of the earth that's constant and you know a frame every five minutes at at hyperspectral resolution at at whatever 10 centimeters. You can't get that. But for for example to monitor CO2 the the atmospheric concentration of CO2 you can take data from one satellite that's for example it's OCO2 or O3 OC3 is actually attached to u the international space station but so it gets you like a little strip of CO2 measurements that are highly accurate and pretty high resolution. you can actually try to do there you could from because a lot of it is in the infrared weather satellites like goes has some infrared bands and it and it takes a picture every essentially 5 minutes but the pixels are very large and it doesn't have such great spectral resolution in terms of it wasn't designed to find CO2. So you just ask one to estimate the other and you shovel other data in like what's the current weather what what's sort of the long-term you know albido and so you so came up with this very nice this very nice model that can sort can do almost like super resolution an informed super resolution of of one satellite to another. So that's like one example.

8:47 >> Yeah. Yeah. Okay. So any any scientific problem that you can map into this framework. So the the input is to to error is sort of this mapping and the output is code is that >> well sort of I mean the input is you the way we've got it set up in in the the product is you just start talking right and so because a lot of times it's nonobvious to how to do this mapping although you know experts like Michael Brener know how to do it so we actually wrote an agent that actually helps you it sort of talks to you to try to help you define ine what your scorable task should be. So there's so already there's sort of a instance of Gemini sitting there trying to help you write a code. So that's actually sort of almost like an intermediate result. you start talking to it about your problem and it tries to produce essentially a a a Python notebook underneath that that has a that has a score essentially a function with a with a scorable which essentially produces a score and then it starts to mutate that notebook in a clever way because again it's Gemini and it will try to sort of keep proposing code that tries to maximize the the score. So what is different about this than just a generally a a general agentic system that can sort of optimize notebooks?

10:06 >> right now it's a it's essentially its own in modern 2026 parliament we actually worked on this in 24 and 25 but in the modern modern parlates it's kind of a specialized harness that runs an algorithm which in for era was Monte Carlo research. So essentially it's keeping hundreds or thousands of possible instances of notebooks and then it selects one and I can explain how it selects one and it decides well how what what can I do? Gemini asked itself what can I do to make that notebook be better and then it will make a new one and test it and then put it back into the candidate pool. So you can imagine the candidate pool is actually tree structured because it's you know every candidate possibly has some children and what you do is you pick based on something called the it's actually a fairly standard algorithm in from reinforcement learning called upper confidencebound UCB. So you essentially pick it's an optimistic algorithm. So it tries to estimate say what's the what's the 95th percentile outcome of mutation. and it tries to estimate that and it picks the one with the highest bound, the highest optimistic bound. So in other words, it doesn't always pick the best performing notebook. It tries to predict like what's the current performance plus two sigma of of its guest. And so it it's always trying so it hunts around.

11:27 >> So So high recall basically >> high recall it's trying to make its bet so that it it most efficiently tries to make progress which isn't always greedily doing the best candidate. Sometimes it's the fifth best. we're also play we've also played around where where it kind of recombines it sort of takes ideas from two candidates and and smashes them together and tries to make a third candidate out of that. >> How does it seed the initial candidate pool?

11:57 >> Well, that's the amazing thing is that underneath Gemini is actually good at writing code. I mean you just ask it write me a thing because it you have you have a textual description of the problem. It isn't just oh here's your scoring function start. You say a textual description of the function and you might give it in fact we have under some things like here are five papers that people tried to solve this problem with and it's kind of smart. It actually goes and reads the papers and will actually take a first stab at code. It might not be great or it might, you know, sometimes it has bugs and it returns essentially minus infinity, but it will then try to mutate the code and say, "Oh, it'll try to make it be better." So, >> it it's pretty cool. You don't actually have to give it I mean, you can if you want give it a some starter code, but you don't have to.

12:46 >> How many agents are you spinning up? I guess maybe not agents or how many different you know, tree branches are you spinning up at each iteration? >> Oh, at at every iteration. Well, there's a trade-off. You'd like to do a lot of parallel work, but if you do too much parallel work, you can't learn from previous things. So, right now, we use about the default is is 10 par. So, so you you try to grow 10 leaves at a time.

13:10 >> Okay. >> that seems to be about the right trade-off. >> When you say you can't learn from previous iterations, that means that the orchestrator or is there some sort of Yeah. what like what's the that you said that there is some step which is able to like recombine or make decisions beyond just like the score I mean so yeah I guess maybe one of the questions is as a human when you are doing some sort of ML project you don't just look at like oh there's this one metric that we're trying to optimize often times there's like orthogonal metrics sometimes even insights such as like just watching you know training curves can sometimes give you intuition about what's going on or like looking into specific examples Does it do any sort of introspection like this? Is there >> Well, it has it has the history of but I meant why you can't do too many things in parallel is is if you if you have 10 parallel searches at once, then what the the number one can't actually see what numbers two through 10 are doing. So if you do a thousand at once, then you're using a huge amount of computation without a lot of cross learning. Whereas once you finish a little batch, you get the history. It it it sort of obviously you have to prune it so it doesn't blow up the context, but but you get the history of of what it was thinking about as it was kind of writing the code and the results of the code. So So it it can learn from its previous attempts.

14:35 >> Okay. And does it learn across? >> Oh yes. Yes. Essentially it's essentially it's it's like one essentially shared context. Yes. So it is it is sort of thinking as it goes along. It's not It's not like it's a thousand different completely independent branches. >> You're really pushing Gemini's like long context abilities. >> That's right. And you have to do you have to do the right management and stuff. >> Yeah. Yeah. Yeah. Okay. Oh, that's that's cool. I mean, going back to R.J.'s question, the like key point here being the first the first key goal is to I guess identify what specific score that you are trying to optimize, right? Sometimes that is, I agree, like kind of the hardest part of the problem.

15:16 And so I find it interesting that I'm not sure I'd always trust my agent to do that part. That part seems like the more human task in the loop. >> It is. And often you have to be careful. In fact, a lot of what you do, it's kind of it's very meta. I guess everything everything we do is very sort of high level. You have to make sure that the there's no one common thing is you come up with a scoring function or the agent does or you do it together and then the iteration finds a a way to cheat or hole like oh no I didn't mean that and so you have to go through and often sort of play and have a loop around it where you kind of iterate like no no I didn't mean that or you have to tell it in its instructions okay you know don't don't do this so yeah there's often iterations and you so you you even with agent help you don't necessarily get the the right scoring function from day one and in fact it's really neat because I mean in the old days i.e. 2024 or something you know a lot of a lot of grad students would spend a lot of time like doing scientific software and it's just so much effort to write code at all that you kind of try maybe a few things or a few things that are very related and then you sort of stop because you have to write your paper you have to do your next experiment. This thing is kind of underneath kind of relentless because it keeps trying and keeps trying, keeps trying and so the people who use it are now spending all their time almost at the right level almost at the scientific creativity level. What does it mean to have a cost function? You know what I mean? And so that's almost like the essence of the scientific problem.

16:51 You're not so much now in the details of oh oh I have to import this CSV file or I have to get this database to work or whatever. It's you're now sort of thinking almost like deeply philosophically about your actual scientific problem, not down in the grungy goop of of of worrying about, you know, databases. In fact, one one cool thing the agent can do is actually suggest data data sets to you like, oh, have you thought about maybe pulling in this data set and doing a join? And so it'll it'll make suggestions about like, you know, data sets you can join with, which is kind of cool. Going back to what you said a second ago. in terms of agents love to hack things and the reward hack, >> are there do you have any fun stories or interesting stories about, you know, where things were comedically run off the rails?

17:38 >> Boy, I'm blanking. I know other other folks have run into it. I don't know if it's I don't know if I have enough details to sort of say what to sort of express the comedy of, but it it it does it you you you you kind of get >> surprised. Yeah. I don't know if I have any really concrete sorry, I'm I'm blanking. >> No, it's fine. Yeah. I always like to think of machine learning as like a it's it's sort of like the old genie stories before monkey Paul like >> careful what you wish for because you you're you're going to get it. Yeah, >> that's right. And and you have that and that happens very much with this. So, you have to be careful. But on the other hand, it has some knowledge. The nice thing is that sort of Gemini knows a lot about many things sort of more than any one person can do. So it at least knows especially if you point papers point you know like here here are five papers that tried to do this in some way. So to some extent it it does have that genie feel but to some extent it also sort of does sane things. This is why remember the whole idea of sort of evolutionary coding. It's been around since the 70s. everyone's loved to do that. like, oh, let's mutate list code or whatever to to do things.

18:52 But the reason why it just hasn't taken off is that random mutation in code space is pretty much worthless. I mean, just like well, just like DNA, it's sort of like, you know, most most things are harmful. So here it's like, oh, no, no, we can actually find it sort of knows sort of underneath and knows interesting gradients to try, which is why the thing works that that the underlying loop itself is an AI. So yes, it can it can maybe overfit and have funny sort of genie problems like you allude to, but it also has some amount of sanity because it sort of >> it knows about the world and it it it has world knowledge in it. So >> the paper though you were doing Gemini 2.5 and I think you know Gemini has advanced quite a bit. Do you have metrics or have you you know this is a tool that you're continuously using and it sounds like you're improving and I'm wondering like do you internally have you seen like almost like a phase transition in how effective this tooling has been? Has it how dramatic has the improvement been over the last like I guess year or two?

19:51 >> Oh well I mean year or two. Yeah. Amazing. In other words, every even every half version of of I mean essentially I think it would have been impossible under Gemini 2.0. >> Yeah, I think so. In other words, it would it wouldn't have worked. >> So, you started at 2.5 and that was like just the >> Oh, no. We've been trying to experiment with these things actually for a while. Okay. And things just weren't working and then they started to work and then now they're they're just amazing. So, the progress on Gemini major versions has just been stunningly amazing.

20:21 >> Yeah. Yeah. I think this is an experience a lot of people have been having where things were just seemed unimpossible whatever are suddenly becoming magically useful like really quickly. >> And so so if people are I even say this to scientists cuz there's some people like oh I tried whatever 2.0 go and I didn't like oh yeah that that was a long time that was a year ago that was like a long that was eternity ago right yes and in fact all the all the we even have one of the preprints where we've sort of combined >> era with anti-gravity and that you know the whole that whole harness of anti-gravity is pretty amazing too and it's that that's the one where you can sort of pull in lots of papers and it can write lots of code for you and >> so yeah >> is that publicly available or is that >> the the anti-gravity Yeah. Yeah.

21:08 >> Well, anti-gravity is certainly publicly. >> Oh, sorry. Sorry. Yeah. the the era plus anti-gravity. >> not yet. >> Okay. Okay. Not yet. Okay. >> I find this area really fascinating cuz like you said, it's been there's been some form of code mutation out there since the the dawn of computer science basically. the canonical problem is sort of the overfitting or multiple hypothesis testing problem I think which is maybe a little bit better match to the problem where you're basically my hypothesis now that this algorithm work now my hypothesis and so that you run the risk that sort of it has exponentially exploded right because now suddenly I have like these it's like hyper hyperparameters that I'm optimizing and so that you have this explosion of state space that you're exploring and so that it seems much easier to to sort of overfit to a problem. What are your thoughts about that? Because on the other hand, empirically my experience, I even tried the you know sort of open- source version of ERA. I kind of strapped it into Claude and it's running right now so I can't tell you how well it's working.

22:12 >> Okay, I'm curious. >> Yeah, I'll I'll let you know. but I'm just curious to know this is a question that's been on my mind about just general AI for science and what so what are your experiences with this sort of on the the sort of on the ground? I guess there's two questions sort of embedded in your question I think right because when you talk about multiple hypothesis testing right there's predictive models and there's descriptive models right I you know can you tell I've been doing machine learning for a long time in statistics >> that's actually a really good point though do you mind explaining that expanding that I'm not sure that's something that everyone would audience would be familiar with like >> especially in in in modern days I think people are trying to sort of obscure the two >> if you started with LLM I'm not sure that distinction would be meaningful That's right. So, a a predictive model is like let's say you just have a you have some inputs and you have some outputs >> and you just I just want to build a piece of code that tries to just have the lowest error rate on some data set.

23:10 That's just a statistical model, right? A descriptive model is actually what science is trying to get to, which is okay, it should be able to extrapolate because it has sort of the physics or the actual some description of reality that's captured within it. And then you can use it to extrapolate. because it's sort of like you know yes Newton thought apples and gravity but gravity isn't actually about apple right if he had just fit if he had taken as a machine you know the 17th century machine learning model like oh apples will fall but how about planets you know I don't know I have no data about planets so who knows what they do right >> so so when you say extrapol I'm curious okay so when you say extrapolative so there's different ways I could think about this one of them is you said a model of physics or model of the world are you talking about introducing explicit priors that you know based upon some human intuition or maybe in this case LLM intuition or are you talking about this is the the physics is actually learned by the model or the >> it's more >> the underlying process of the world is under the model >> the distinction between those is a little bit blurry right because when a physicist or scientist comes they use their intuition and they or about or or maybe even more than intuition like essentially there's maybe a a solid pile of facts that they know about the world and then they make sure that whatever model they build is sort of consistent with what's known. currently in era it is it's sort of it is LM intuition essentially that's what I was trying to say about having a good gradient underneath that that especially if you point it at existing papers it will try to build models that are kind of sane underneath because if again especially if you give it guidance like oh be sure to incorporate this and this or look at these papers you get these things so it will there is a you can introduce a bias towards certain model choices and it will have a bias because it it just its own little sort of world knowledge is accumulated inside of inside of itself in in in sort of pre-training.

25:15 >> Can you give us examples of what that might look like? Is it modeling something in a way where it's actually there's different for example if you're doing something with partial differential equations there's these you know formalisms people have like u neural operators for example or where you can embed a you can encode a differential equation in some sense or I think that's called physics informed neural networks or something is like one for fluid and kind of PTE type modeling systems or for let's say molecular systems there's often times this idea about equivariance I you know the model is like picking up on these like you know tricks which have been developed in the literature or are they adding some weights for some physical prior or something yeah kind of what does that look like when it when it introduces like a physics >> when it introduces some sort of you know knowledge like that >> I don't know if I have enough data to sort of say oh you know 73% of the time it does this >> but especially when you point it at existing papers it will try to in fact it will do very well at adapting the the the methods that are described in the papers for the problem. in fact we'll do an amazing job. You can actually often just recreate or reverse engineer paper that that's again what Michael Burner actually likes to do this he'll he'll say oh that sounds like an interesting paper. Oh we did this actually for the we had this thing.

26:37 It was actually kind of a hack. I suggested this to Michael where there's this one MIT professor who he came up with a some code to do essentially if you have a rooftop if you have a rooftop with a fixed area and you want to sort of maximize the amount of solar power you capture over a day sort of solar energy you can build up which of course captures more sunlight and you can sort of build you can have it design a widget you know with involving mirrors or struts or solar panels to sort of stock at whatever, you know, angles or or sizes you like and usually with with a with a maximum height and then try to figure let it sort of explore that design space. And I believe Michael, we can ask him. I believe it actually just I don't think he actually installed the simulator. I think I think the code just just reproduced the code >> because it has like a coding agent inside of it. So just reproduced the code from the paper and sort of figured it all out. So yes, it's it's very good especially if given a pointer to what other people have done. It's very good at kind of like oh I haven't seen it try equivariant modeling that that can get very hairy if you know about k clips gordon coefficients it's it's pretty fun. Yes. So I don't know if it'll do the true equariant stuff but it it actually knows a lot. I remember actually when Gemini 2.5 came out, I know this is not exactly about era, but I I remember sort of thinking, "Oh, this is a new world." When like the day 2.5 came out, cuz I said, "Hey, Gemini 2.5, can you write me some boosted decision tree code?" And it did.

28:15 >> And it worked. >> It just did. Yes. And I said, you know, yeah, this is the this is a new world. So yes, I think for to loop back to your question, I think yes, if you give it sort of guidance about, oh, you know, it's important to put this kind of thing in, it will. And so it it it won't necessarily, at least not that we've seen, discover completely new like if you didn't know about KP Gordon, go fish, I don't know, you know what I mean? didn't know about something, it won't completely discover a new kind of physical model from scratch, but it will certainly if you tell it about interesting constraints about the world that are known, it will certainly evolve. I don't know if I answer your question.

28:54 >> So, so there's still room for humans for the next year or two. >> Oh, in fact, there's going back I think there's totally room for humans because I don't know. I mean, we have co-scientist that tries to help you come up with sort of hypothesis generation, but really I still haven't seen sort of the creativity and the philosophy and the sort of the the sort of the careful rigor. You totally need the humans. I I don't see humans going away. they can make strange suggestions and I've used co-scientist actually for an interesting problem in geochemistry and I learned about a new kind of ion I didn't realize happened in magma but I but and so it'll tell you interesting things and you'll learn stuff but I don't think it sort of substitutes for human creativity >> you know going back we were just talking about two years 2 point you know two you had Gemini 20 to 2.5 and this was like you already saw a leap and now it's been another year or two and now we're at 3.5 and you know there's you're saying this is working much better. I mean whenever you look at a graph you know you can if something looks like an exponential it can either you can either be in a sigmoid or you can be at the beginning of a takeoff right I guess every exponential turns into a sigmoid eventually >> every exponential turns into >> but the question is like where are we on that >> I mean I I guess I'm a big believer in sort of the whole jagged >> yeah the jagged frontier and so certainly at least what I see I mean I don't know what's going to happen in a couple years but yes there there's some big spikes out in jaggedness in terms of coding ability and just gathering knowledge and and finding related things and and that's huge and wonderful which I think is great for scientists. So far, it's kind of less in terms of rigor and we can talk about things like the international methad, but and math in general, but but but in terms of sort of philosophy and creativity, I think it's still kind of not and I I maybe it'll maybe everything will people some people are saying everything's going to inflate and pass, but I'm still seeing a lot of very strong jaggedness. So I I can see well maybe it'll get to be extra good at coding and extra good at at fitting models and extra good at making suggestions and things but I don't know.

31:04 So far no so far you need the humans. >> Yeah I want to get back to the question about the multiple hypothesis testing. >> Sorry totally love the the tangent. multiple hypothesis testing is when you make have a descriptive model and you're saying this is the way the world works and you have a bunch of data and you take a billion darts and you throw and so you have to be careful. There's some something called false discovery rate right and so the question is >> is this finding descriptive models or is this finding sort of predictive models and fundamentally the scientist is there to make sure that whatever is saying is descriptive. We haven't been able to make a system so far out of these pieces that that can really sort of discover completely new physics or completely new science, but this is sort of a power tool to help you discover completely new science. So, so I maybe I'm trying to unask your question about I think this gets at the heart of it.

32:03 Yes. >> Yes. But then you're saying what about just pure over Okay, let's set aside. It's not it's not trying to figure out a descriptive model of the world. That's still up to the scientist. But what about just plain old overfitting? >> Yeah. >> Yes. You have to be very careful because it's a power tool. It can I shouldn't probably say is it can slice your fingers off. You know what I mean? You have to be very careful and you have to be very rigorous. In fact, now you have to be more careful more rigorous to not fool yourself. You really really need to be just excruciatingly careful about having, you know, very hidden hold out sets you don't look at. you you have to be just super super rigorous to make sure that you don't completely because it is a total power tool.

32:44 >> So the question to how do I not slice my fingers off is you need to use the same techniques but be very careful with them. >> Yes, >> that's a very clear answer that get >> Yeah. Okay. Yeah. I actually don't think I've heard any guests say that. Yeah. It's like it is I think a very important skill, maybe one of the most important skills in the new people talk a lot about taste. >> Yeah. But and maybe this is variation of taste but >> the taste is like the other side right now.

33:11 >> It's like the rigor. It's like the Yes. yes. In fact, if anything >> Yeah. taste or rigor, which one's more important? >> Well, I don't know. I think people at least the way I am viewing it is is I mean the aspect of that research researchers are software developers which there's a lot of overlap in a lot of fields. I'm seeing that software engineers are it's almost like obviously there's a lot of concern like oh no what am I going to do you know coding seems to be you know getting automatic so I think I think there sort of both I think there's a lot of people get pulled into well I I'll be the creative source so I'll I'll try to figure out new science I'll try to figure out new products I'll try to sort of really re be very very creative and I again I'm strong believer that I don't think that's going to go away there's also people sort of pull towards rigor like, oh, I want to make sure this doesn't that this doesn't crash. I want to make sure this scales. I want to make sure this isn't wrong. I think you need both.

34:08 And I think you need people who who are really good at both. But they don't necessarily have to be the same people. But yes, I think you need I think this is even broader than science just as sort of software engineering evolves. Yeah, it'll be, you know, the people who will bring the creativity and the people who will bring the rigor and I think those will be sort of anchors. There's some other things that I've seen are related work out there. There's a a really cool leaderboard for, you know, like claw leaderboard, agent leaderboard for scientific problems from Stamford. I don't know if you're familiar with it.

34:44 It seems like a really interesting idea to me to have, you know, sort of different agents kind of competing on the leader. So, it seems like if you squint a little bit, what ERA is doing is is kind of a leaderboard, but it's internal and it's recombining ideas whereas what are your thoughts about this and do you is that like a thing that you guys are working on and is there problems with that or advantages to that? Ironically, you know, the whole era project actually started because people may not realize Kaggle is actually part of Google.

35:19 >> Oh, yeah. >> and so it was called the auto kaggle problem. So it was actually like that's what it was is let's try to let's try to u u have a system that can sort of you know win at Kaggle competitions. >> So that's sort of why it sort of has the shape. That's sort of how the project started. And it goes back to sort of overfitting, right? If you've ever actually competed in a Kaggle competition.

35:42 >> I have done Kaggle competitions and or I've done one. It is a really interesting phenomenon because there's this overfitting is like rampant. Yeah. Yeah. And it's really impressive how people can overfit to certain data sets in a way that is Yeah. >> Or we even we had a we have a fun project I can talk about more if you that tries to mitigate contrails gen contrails. I'm happy to talk about that if you want. and we had a contrail cattle competition and people actually beat us, but they found that we had a half pixel error in our labels and they and they cuz it had to do with the center versus the lower left like where is 00? Is it in the lower left of the of the pixel or is it in the center? You know what I mean?

36:27 So, they found that and exploited that and and squeezed whatever a little bit extra stuff. because it turns out when you make artificial data and you rotate it, you have to make sure that that you take into account that half pixel offset. So they so yes people or people themselves will be act like these LLMs and and try to sort of reward hack upon these things. Oh, it sort of goes back to what is it? Goodart's law. You know, good hearts law. Any maybe quote, let's say any metric that becomes a target is no longer good as a metric. Yeah.

37:00 >> and so that's the I mean it's good and it's just you have to be you have to be very very careful and you have to again you have to have like layers of rigor like okay but we'll do this and we'll optimize for this but you have to realize okay that's just now goodart's law applies and you have to be careful and so that's a lot of reasons why the whole AI field has been kind of constantly exhausting these these things because again goodart's law applies individually to every to every leaderboard you make. So again, it's sort of you just have to step back and be very very careful. I I maybe that's not I don't no that that's really useful to my thinking. This is you know as we've had guests on it's been a recurrent theme of how do you manage all this the complexity that's introduced by LLMs and atte science.

37:49 >> I think my followup question was about overfitting in Kaggle. yeah, it is. If you you had an auto autocaggle problem and then the question is given autocaggle, how often was it successful? I mean, I assume you probably just ran this on like all of your Kaggle competitions or something. >> Well, we tried it on various like what they call playground competitions and it did very very well in the playground competitions. we've entered into different competitions.

38:15 some of them it turns out there's in the last few years just the the number of leaderboards and competitions whatnot have just exploded far beyond Kaggle. So we've done very well and some of them like one thing we're super proud of is the whole the CDC set up this competition where you try to predict next week's the number of COVID and flu cases that will happen in every state and territory in the US. and you try to predict a week in advance and ERA did super well on that.

38:46 >> It's funny because in some s Google invented the concept of using data to track disease progression with Google flu. So it's kind of funny that you were sort of going full circle 20 years later or something like that. >> So that did very well. other ones where we've entered we weren't quite as good often because people are very again you know you have to sometime sometimes how well you do in these competitions is a measure of how much sort of TLC you put into it and how much you're willing to squeeze the last 0.001 O1 and so so it was I mean ERA did well. You got you pretty close, but you but we didn't we didn't close the jump in the last whatever 30 places or whatever because no one was there to shave the last, you know, 01 off the thing. Yeah.

39:33 >> Is it very iterative like you get to it, you know, it sort of I saw the charts in the paper and you know, you sort of get these step changes as it discovers something and then and then flat and then so is it very much human in a loop like okay you've stalled on the problem like try this kind of thing. Okay. >> Yes. At the outer loop which is I think that's the almost like the more fun creative part. so yes. Oh here have look at this paper. Oh, you're doing something bad or you know what I mean?

40:02 So, it's almost like having a a hyper eager grad student or something who doesn't sleep. and you sort of tell it things and you and you and you sort of guide it around. >> How often does someone intervene versus like what is the outer loop look like? it might run for a few hours and and come back and give you some examples and then you would you know you can as as far as you like you can you can sort of keep trying and keep poking at it and >> so that's it's very much designed to be human in the loop then.

40:34 >> Yes. Yes. >> Interesting because a lot of the other tools I've tried tend to be very the one shot. >> Well I guess it depends on your definition right? I mean it's obviously you talk to it and you start it and it'll go for some number of hours and then come back and then but of course then you say but then that's where the human creativity kicks in and then you're sort of doing the outer loop where you sort of every you know depends if you want to sleep but you know every few hours you you go and you give it another try and you >> what kind of budget are you giving this thing >> like you you blew through a million dollars accidentally kind of thing. I don't actually know because we're using sort of you know internal calls to to to Gemini. So actually I I don't actually know.

41:17 >> So but look there's token budget but then there's also like I'm solving a problem that is computationally expensive. >> Oh yes that also it essentially underneath it cuz cuz the the scoring function itself might have you know Monte Carlo estimation or or whatever. Yes. So you actually end up you can actually end up using a lot of compute to just just even do or simulation like if you have a simulator inside has to run a simulation. so yeah you can you can you can spend a fair amount of just CPU or GPU. So my little experiment with ERA ERA and Claude is is to build a a neural network for some classification problems and and so they obviously like if you have enough data then you know larger networks work better but they're more expensive to train and you get to you start to run into a question of how do I manage my budget if I have a fixed budget so that I'm spending my my my my dollars on the most effective solutions. That's right. And I think that's still something we need to figure out. But it's of course it's no different than if you have a grad student and they're trying to train a very very large neural network or a very very large data set. They themselves have to there's some thing like oh is there a scaling law? Can I extrapolate? So it's not I guess you it's an the same problem but maybe more urgent because it it just it runs into this problem because it's so relentless.

42:40 It runs into the problem much quicker than a grad student could. One of the things that you optimized was contrails. can you talk a little bit about that? >> well let me maybe spend a minute or two talking about the contrails problem. >> Yes, please. >> For context, contrails, not chemtrails, which is a conspiracy theory. >> yes. although you should also dislike contrails, but maybe not for the same reason. So, contrails are if you've ever seen those white clouds formed behind jets, those are called condensation trails or contrails. And it turns out they add at least according to the estimates that people have about 1% of all anthropogenic global warming is caused by contrails. Why is that? I can just talk about the maybe the physics of that. So, it turns out that there's actually two counterveailing effects. Contrails are are well sometimes if you've ever seen them they they streak and then they kind of go away. Those don't really do anything but sometimes they last for a long time. You'll just see in the cl in the sky just like almost like a waffle of of of of just persistent contrails they're called. And there's two effects that they have. Those are thin white clouds. So they reflect sunlight but that only of course happens during the day. It turns out all all objects emit something called black body radiation. And the Earth does at whatever the temperature is about 300 Kelvin. It it's in the far infrared around 10 microns. And at those wavelengths, contrails have very low albido. They're almost essentially black. And so they'll absorb a little bit of the outgoing infrared radiation and then remit it both directions. So essentially they'll reflect some of the outgoing heat. So it'll trap heat like a blanket.

44:22 >> and so because that happens 24 hours a day, they tend to be warming. And it turns out it's a surprising again there's some uncertainty about it but you know contrail cirrus cirrus that sort of comes from from from contrails might might cover especially in places like Europe which has a lot of airline traffic a few% of the actual sky is covered by sort of additional contrails which so it adds in those places like Europe it adds about one watt per square meter of of forcing locally at least which means that Just to give you a sense, all of anthropogenic warming sort of average across the whole globe is about three watts per square meter. So in places of high airplane traffic, it can be a lot of warming locally. so what can you do? Well, it turns out contrails are caused by areas in the atmosphere that are ice super saturated. They're a little bit like rock candy. So like when you have rock candy, you get a water solution that has too much sugar in it. And any little you know little bit of sugar in it will just crystallize all the the the the sugar out. Just like in this contrail there these regions they tend to be kind of pancake shaped only a few hundred meters tall. And if you fly through it the jet exhaust has a little bit of moisture in it which will turn into droplets and then freeze. And then for every gram, if you're in this bad region, for every gram of water, ice or soot you put out, it's about 10 kg of water gets sucked out. So there's this enormous 10,000 to1 curing ratio. so it's a big problem. So what you can do is you can figure out where these regions, they're invisible, of course, these regions of ice saturated are, and then tell the plane to go underneath, and you only have to drop essentially what they call two flight levels. So it it actually it costs a little bit of fuel but not very much to kind of avoid these sort of bad regions. So we built a system that sort of looks at satellite images and tries to detect where contrails are. So we have essentially a continuous monitoring system and then try to build a model of of because it turns out the weather models are not quite accurate enough to to find these places of ice super saturation. So we build a custom model again like a convolutional net or a UNET or something to essentially to predict where they're going to happen so that then you we give maps to a flight planning software so they can dodge it and and inexpensively reduce the the the climate impact of of aviation by a lot. What's the physics behind why you can predict that? Right.

47:02 Is it just I see it in the satellite and then tomorrow I think it'll be there because planes go to the same place or >> Oh, no. It's because you're trying to detect these regions of ice super saturation because they're very very persistent. Essentially they're >> Oh, they're persistent. >> Oh, yeah. Yeah. I mean, no one knows exactly, but they could last for days. Essentially, they're caused by they think sort of warm moist air being injected just at the boundary of the tropopause between the just at the bottom of the stratosphere. And then when humidity gets up there, it sort of sticks there for a long time and then gradually dissipates.

47:34 >> Got it. So, so it's there's just a sort of >> they're like bad spots in the atmosphere you don't want to fly through. >> Right. Okay. And so once you've establish that, it's probably good for a couple days at least. >> well, you have to keep predicting where that is. >> Yeah. And the the models you're using are you mentioned like CNN's or something like that. >> That's right. And we haven't replaced those with era level models yet. But there was a very interesting problem that came up which is you sort of want to know well just how much warming did this contrail make and how much did it add to global warming. because for example you might want to find the biggest ones because there's some fuel cost and maybe it costs a bit of money for the air airplanes to avoid it. So you say well gee I'd like to kind of know how much it did. But that's actually what they call a counterfactual problem like okay you made a contrail and certain amount of infrared radiation happened. So we can measure that if you're careful but would have happened if there hadn't been a contrail there.

48:33 That's a very difficult thing to estimate because you can't access the universe where the >> didn't happen. So you have to make these things called counterfactual models. And there's those are if I don't know if your listeners know counterfactual models are actually pretty tricky to to fit and make. and we were there was remember there were two there were two the reflecting the the sunlight and then there's the infrared. It turns out the measuring what the effect of reflecting sunlight is actually more difficult and we were actually stuck on it for 2 years. We we had a model that worked okay a counterfactual model for the outgoing longwave radiation but not for the reflected sunlight. era actually helped us find a a model that sort of searched all the confounders and sort of figured out like oh how how can we estimate it because we again we even had like test test code on on sort of artificial because you can kind of inject artificial make artificial data sets where there sort of injected contrails and sort of figure out oh well we know how much it was because we we injected it and so again our own attempts didn't even pass our own tests but era this thing actually did and and sort of unstuck this problem.

49:43 So, yeah, we're in the middle of writing up a paper we have a paper about the outgoing long range radiation, but we have a paper that it's not submitted yet, but we've talked about it at at EGU, I think, where we actually solve this problem. So and and the the models that error comes up with are they just like a big monstrosity of of of code or are they like pretty basic and is just you needed the intuition to develop that?

50:12 >> Yes, it's actually in this particular case it was actually more of the latter that it essentially sort of helped identify what the it was a very simple model with some just a some number of confounders that we just hadn't we just hadn't tried that combination before and it worked very very well. So yeah, it actually sort of came up with the the and it was seen in in retrospect. So that was I think a a big win.

50:32 >> Yeah, that's interesting. I know you've done a lot of work in climate. What other other stuff have you done? >> I think I talked about this right. The the I talked about the CO2 thing. That was pretty fun because it's this it's still quite a the reason why estimating CO2 in the atmosphere is an interesting problem is we actually don't know what the carbon flux is into the bio in and out of the biosphere. I mean we do we know that the biosphere captures right we emit a bunch of CO2 out into the atmosphere and some of it gets absorbed into the ocean with sort of mostly in organic chemistry some some phytolanton and a lot of it gets absorbed on land but but the error bars about what happens is are moderately large and the error bars 50 years from now are very large like the models in 2100 we don't know how how the biosphere will react to the everinccreasing temperatures and CO2. So we don't actually know how much the CO2 will absorbed and the the the error bars are 300 ppm of CO2 just from the uncertainty of how what gets absorbed. And just to point out, you know, right now there's about what 440 450 ppm. So it's huge. I mean it could be seriously amazingly awful or not great but you know it the the 300 ppm is like enormous uncertainty. So it would be really nice to figure out you know can we can we reduce that. So this is like the CO2 concentration is like one step towards that solving that >> and I know that Google has made some really big improvements and climate modeling and and weather prediction as well, right? is I was at Nurups this year, this last Nurups did and and and I stopped by the climate track and you know I maybe only had a chance to listen to two talks or something but I was it really blew my mind the sort of step change I think that's happened in the past I don't know what it is maybe 10 years or whatever in terms of climate modeling and I know a lot of that happened at Google can you talk a little bit about what has happened in Google and other places that has made that allowed that really big transition in in climate and weather modeling.

52:51 >> okay, so let me People often sort of collapse climate and weather together. Well, because they're fundamentally the same physics. >> Yeah. >> Although at least for the atmospheric physics, they're obviously when you start having ice and land, you know, climate is long-term weather. And so the complexity of a full earth system model, which is a climate model, is much much bigger than an atmospheric model. like you have to actually measure what's the water flux and the CO2 flux in it out of the land or what will happen with ice and so there has been a step change with weather models right sorry I want to make this so weather is up to 15 days okay approximately because you know weather itself or the atmosphere appears to be chaotic >> I'm I'm hoping I don't know if I should explain chaos essentially it's the butterfly effect right that small perturbations like a butterfly flaps its wings and the weather will be completely different in you know 2 or 3 weeks. So weather is trying to predict the actual trajectory of the atmosphere over say 2 weeks and that's now that has been a huge step change and that's because that's been a lot of not even the new LLM stuff that was based on you know the 2018 era machine learning stuff and just a large amount of data and a large amount of compute. So, it's been there's been a lot of sort of very clever work and a lot of it from Google making new u new weather models and it's been it's been great and in fact we had a a really neat breakthrough cuz now we can apparently predict tracks of cyclones, tropical cyclones much more accurately many days in advance. And so places like Jamaica had got hammered by a terrible hurricane. And a lot of the classic models didn't actually predict it partially because it's often that the especially the intensification. It's all being driven by the what the surface temperature is cuz hurricanes are people might not realize are heat engines essentially.

54:53 They they convert sort of heat in the ocean to big atmospheric motions. So weather has been great. climate is much more difficult because you don't actually care about you're not trying to predict whether it's going to rain in Seattle in 2070. You're trying to get kind of like averages and what makes it difficult is that it's what they call non-stationary. So that the in fact literally the the the it's like the underlying physics or the underlying like you know plants are behaving differently and and ice behaves differently and and and so it's a it's very very difficult to use sort of classical ML on sort of true climate models and so that's why sort of the whole discussion you guys had about you know what we're talking about about descriptive models and multiple hypothesis testing that is incredibly severe in climate because we don't we have no data from 30 years from out and we don't want to wait 30 or 50 years to find out whether we were right or that we overfit. So whatever things we do, you have to be kind of careful and try to peel off sub problems and and the problem of of exactly how do you inject how do you build a big model that can predict into the future but is still constrained by what we know. It's a fascinating problem. I think it's still unsolved but it's a it's a great problem to have because of again these uncertainties. We really would like to know what will happen in in 60 years to the climate. So it's still it's a thing.

56:17 It's a very very interesting problem to work on but so far AI has not revolutionized it because it's very very resistant again because of this data problem. It's it's a low data problem. Does the butterfly effect the chaotic nature of weather? Does that also impact climate or is the time scale so large that you have a closed system for which you know maybe it's oscillating between poles or whatever but it's it it's sort of when you look at it that time scale it's more stationary. It's unfortunately non-stationary in a different way. But the original sort of whole chaos thing was well I maybe people came up with it but in meteorology it was back to a person named Lorent who had this sort of very model very simple model OD. So the difference between climate and weather is is weather is where are you on the attractor >> and climate is about the statistics itself of the attractor. The problem with with climate is that we're altering it. So the attractor itself is changing is moving and there could be and there could be everyone talks about tipping points that means that the attractor suddenly changes and the trouble is that's very very very difficult to predict. So even the attractor's shape changes quickly >> or could and the trouble is when you run a simulator >> you don't know like is it did it go unstable because my model's not great or is it an actual physical instability >> and it's extremely difficult to tell the difference. What do you do? Especially when I mean to me it strikes me that not only do you have not have future data, you really don't have much past data.

57:52 You can do some measurements and ice cores and lots of stuff to try to do that. But there was nobody with an instrument 100 years ago. >> So if you have annual data or whatever, maybe you have if you're lucky 50 data points in any one location, right? So whatever we do has to be very constrained by what we know. But it's just very diff I'm just telling you sort of the horns the dilemma people on. People make these, in fact, people in applied science in general, I would say climate is the most extreme, make these things called process models where what you do, and I've seen the code. oh well, you know, I'm going to be reductionist and I'm going to sort of take the horrible complicated climate thing and sort of boil it down to a thousand pieces and then I'm going to, you know, find the expert who wrote a paper about, you know, piece number 763 and he fit a cubic to some data like has like for example one thing that's very mysterious which is related to contrails is how does ice behave in clouds? It turns out you might again get everything is complicated once you dig into it.

58:54 But it turns out that like when you make a contrail, how long does it last? Well, it depends on because the way contrails can evaporate is ice starts to to accumulate as I said and then the ice crystals get big and then they fall. But of course, how quickly they fall depends on their shape which is not known. And how much does the the the contrail mix from the from the moist inside the contrail out? Again, people have approximations, but they don't know. And so the the uncertainty is very much confound. And it's not just oh John who cares about contrails. It turns out that the actual physics of of microysics of ice has very strong implications about what climate models do. And it's sort of we just don't know.

59:41 So I'm trying to say it's very gnarly and it's not a solved problem. My hope is that with tools maybe not like today's era but maybe tomorrow's era because it remember it can as I was saying before it not only can fit data it can read papers right and the question is it can read a lot more papers than we can so maybe maybe we can integrate all the data that or all the knowledge that people have carefully evaluated much more than any one person writing a piece of code and fit data I mean that would be that would be utterly glorious. we don't have that today, but that that's sort of one of the hopes that I have even for a a more amazing tool in the future is something that really can write code in a sane way, even much more sane because it'll be constrained by all the scientific knowledge that we've accumulated so far.

60:33 That would be amazing. We don't have that today. >> I mean, that strikes me as being very similar to biology. >> Oh, yes. Oh, boy. Right. If you've ever played by biology or even looked at biology, there's so many exceptions and so many hacks in the biological systems. Yes. Oh boy. So, yeah. it would be amazing if we could have a thing that could really integrate all known scientific knowledge with data and try to synthesize sort of new models and and new things. I think what you're saying is that AI can be an unlock here to some extent because the the models are so peacemeal. They necessarily peace meal. And so that being able to assemble the jigsaw puzzle, not to mix metaphors, but to assemble like a really this jigsaw puzzle having just scale and capacity actually helps a lot.

61:28 >> That's right. The one things that these AIs have is somehow you know humans even I'm pretty well right I think but it's just difficult for me to kind of integrate across the n squ different my n is the number of papers I've read in the life it's pretty big and it's just difficult for me to even do that n squ thing but somehow there's just so much data in those billions and billions of parameters and you can also give it access to read pdfs that it can somehow start to pull things together that people wouldn't do it. So that's again I'm starting to see little indications of that inside of ERA. I'm not claiming that's what ERA does today. But yeah, that that's sort of my hope of where this is going to go.

62:08 >> I think I've heard a lot of people suggest something like the the route to intelligence is to combine LLMs with some form of search. So it's actually amazingly like ERA doing that, right? Something which you know is a maybe a a very a very strong database lookup with a a good search algorithm is one way. And of course, I mean, there was the whole I mean, people still do, I guess, the whole rag thing, of course. and if you think about it, Google itself, you know, the 10 blue links things, >> it was a or is a a form of AI even before we had lenses, right? Because it was like you can you could cast yourself back to whatever 2010 or 2015. You could ask Google about literally anything and it will tell you stuff, right? And so, >> surprisingly well, actually.

62:51 >> Yeah, surprisingly well. because somebody on the internet has written about it probably. So if you can match that >> in fact that was one of the reasons why I wanted to come to Google is just that was such an was such an amazing thing right >> I wonder how much of our audience did a search pre Google and just know how bad that experience was. Yeah, I remember 1998 I think I think when Google like I was using Alt Vista and like I I don't know Google had just gotten released and I used it and I just start sorry a digital I just I just dropped just like a like hot potato or something and started immediately using Google. No, it's so that is sort of a form of AI and so yes it might be yeah it could be that just having access to all of that and sort of keeping it in mind at the same time >> that model of sort of scientific discovery in as much as it pans out is kind of comforting too because it is reductionist so that you can look at the individual parts and understand them. So it found the exact things to assemble, but they're all actually maybe fundamentally things that people have invented or or it's done iterations on and and so all those little pieces are individually understandable and then you can also put them together into a coherent picture. I think for a lot of problems like biology or climate science, I don't think we would trust the answer unless it was in that shape.

64:15 >> Yeah. >> Because if there was some giant blackbox model that said, "Oh, this is how a cell works." It's like, do I believe it? I mean, I don't know if I believe it because I can't examine it. So, >> but I mean to to argue against it though, like if it if it works really well, >> but you'd have to gather you'd have to I mean, you have to test it obviously, but a but a statistical model, you have again it has to extrapolate.

64:36 >> Yeah. >> Yeah. And it has to extrapolate to the extreme or the sort of the the black swan events. >> That That's right. And so it's very hard. This is why things like self-driving cars are very very it's a very difficult problem. It's all corner cases. >> Yeah, >> it's kind of amazing how well they've done. >> That's an interesting point thinking about like when you know coming from the world of physics where a model was usually a single equation or a small number of equations which uniquely define a system and everything about it and you just crank you just find a solution to the system and you you now know everything you need to know. And I think something like alpha fold was kind of a shift for a lot of people where before they thought oh protein folding is you know problem where you just if we find the right force field and we have the right computational engine we will solve protein folding >> and the thought of even really solving it in a datadriven way was only appeared a few years before you know alphafold alpha fold one came out and it's interesting that I It's sort of forced the new AI modeling has I think forced people to re evaluate almost what is science because alphafold is and similar models are incredibly powerful. There's a lot of things that they've opened up as tools but at their core they often times don't give intuition in nearly the same way that let's say the physi that most physicists historically would have wanted. And so I guess it's sort of there's this old saying all models are wrong some are useful. Yes, >> box said that.

66:10 >> Yeah. Yeah. When do you find the datadriven models to be sufficient and when do you want sort of like something which is interpretable that humans can actually understand? >> I think it boils down to almost like the difference between weather and climate. if you're in a datariich regime >> >> like weather or even proteins because of PTB u you can feel oh yes in other words I' I've got enough data to kind of cover and so a statistical model like alpha fold do and so in fact a lot of people happily use alpha I you know I think it's really revolutionized my understanding I'm not a biochemist but people seem to love it and the and the and one amazing thing they did is they exhaustively they just just ran it on all PDB and and published it which is just really really cool.

66:57 >> It's like six billion protein producers. So the vast majority are actually quite accurate. Yeah. >> Yeah. So that's just amazing. So but it feels closed if you know what I mean. >> But when it's like climate and it's open and it's non-stationary or you have to make these big extrapolations, you have to be much more cautious. or maybe biology again may and there's probably there may be parts of biology like oh there was this virtual cell challenge from the Ark Institute >> that had a funny result >> I I I know that people were there may have been some overfitting for at least that would say more sorry >> or I guess maybe at a high level I think the simple baselines for >> work very very well things yeah yeah just like >> one of one of the classic things in whenever you do biology is just always start with a simple baseline maybe this is probably just good ML in general analy yeah understand your simplest case and in biology there are many problems where they're extremely resistant to anything beyond the simple baseline even if you have a lot of data >> that that's right and in fact I tell people the same thing I said always just just fit linear regression just fit >> just do it >> just do it just do linear >> or SVMs I mean SVMs are are just a different >> different form of >> linear regression yes so when do you need the more process modely thing I think it's just when you >> when you have I mean sort of climate is on one end and I don't know weather maybe on the other end maybe that may be too extreme but I think just where are you on the on the data richness thing when do you when can you feel like oh no I really have a closed I really have a closed problem and I think I can actually cover it >> yeah closed problem that you data fully covers is yeah I think that that makes a lot of sense in what I've seen as well >> yeah I think one thing I'm kind of curious about is when you're working on climate modeling, what are the you you talked about contrails, you've talked about I guess CO2 predictions, what are the the the broad things you're trying to accomplish? So, one of them is I guess making up inter like making interventions and the other one might be making predictions for things like insurance or like how do you help like adjust for some sort of climate change or like Yeah. What are the like principal goals I guess for you specifically or the community at large?

69:16 I think you you know just like any community there's probably many different goals. For me I'm or and my team we're very very interested in interventions. so like which ones are possible at at at relative cost. I mean contrails was kind of amazing because it turns out that the intervention is quite low cost and we and also one amazing thing about contrails is they are local unlike things like CO2 because so if a country decides to fix contrails over itself it actually improves its I mean it it has global effects but it mostly improves the the climate a little bit over themselves. So they they like that.

69:53 >> I I guess though if you are in a cold climate and you want to warm it up this is now your own you could own >> Yeah. It it turns out that it's a little bit asymmetric. The warming is constant essentially and global. The cooling only happens when you're sort of at a good the sun is at a good angle over you. So it's very rare that the uncertainty there are contrails that where are uncertainty bounds in terms of the warming. There many many contrails mostly at night of course where the where where it's it's like largely warming and we're we're very sure in terms of like two sigma there's very not very many contrails where you say oh I know for sure that it's cooling and I want more of them like like so only over sort of the poles in polar summer do you know that the contrails are cooling and therefore if you got rid of them they would warm up but there essentially no flights over Antarctica and not that many over over the >> polesm in the summer. So >> So no one's gonna no one who lives in a cold climate is going to use this maliciously.

70:55 >> Well yes in that in that well they wouldn't know for sure whether it was warming or cooling and so they would do stuff and they so mostly we just sort of ignore we don't recommend that people fly those. >> You also brought up a interesting point about the economics. I mean a lot of there I think was a lot of resistance historically about certain you know to climate certain climate change interventions which have in some sense the market has just taken over like at this point unambiguously like renewables and batteries are just almost universally unamiguously just better than alternatives >> for for for non-mobile I mean you mobile I mean that's a really good point yeah yeah like planes we we do not have a solution to >> correct I mean there are some battery powered planes, but they're very small and have to go very limited range.

71:43 >> They probably will never actually be >> it's hard to imagine the physics would be very very unless we came up with something like nuclear batteries would be kind of amazing but I don't we don't know how to do that >> or even if we did I think the the risk of like people would be too afraid of a nuclear battery going wrong or something. >> Oh yeah. Yeah. Since we don't know what they are, we don't know what the risk. I guess we don't have the risk. Yeah.

72:03 >> Yeah. So we don't know. So yeah, that's that's the problem with I I talk about I have given talks about climate change and I talk about the p the pie chart of badness, pie chart of sadness, >> which is there's no one silver bullet for climate change, right? There's so many different things that contribute greenhouse gases just from across our economy. So they sort of all have to be fixed or many many of them have to be fixed. So there's no one single thing. I mean I've worked on fusion. Fusion is cool and it might actually knock a lot of them out if it's cheap enough, which we don't know cuz we don't know if it'll work yet.

72:34 >> I mean, fusion is one of those interesting things where the joke was always fusion is 30 years away, but I think it's actually now less than 30 years away, maybe. >> Yeah. No, I think there's a there's a def definite probability that that someone will make commercially relevant fusion even by the end of this decade. So, I think it's like three years away, not not 30 years away. >> This is very real. Interestingly enough, I think a lot of that I'm going to now just stump or advertise some of our other episodes, but a lot of it actually comes down to material science.

73:05 interesting enough in that >> Well, I'm Oh, super. Sure. Sure. Sure. Well, yes. Sorry, we could talk about fusion if you >> Yeah. Yeah. There was actually two two things. One is fusion. The other is better control systems, which I think is actually >> Yes. And in fact Google DeepMine has been working on control systems for Tokamax to make sure they don't essentially go unstable and go disrupt. Yeah, that's right. >> Disruptions are are quite interesting to themselves.

73:31 >> Yes. >> Yeah. It's basically the entire the all energy in the the tokamac columnates into one little beam and then >> and it hits your vacuum chamber and you're very very sad. >> Very sad. >> Yeah. Yeah. I think people believe that eater could be could turned on after $30 billion disrupt and then basically have a 30 billion $30 billion brick or something. >> Oh yeah, I guess I guess you could try to patch it. I remember >> working we we again before LM we worked with a fusion company called TAE and I was in their control room and yes it was kind of sad. You have to be very careful. we were making systems to recommend new experiments and they were very very skeptical and jaundice which way they should because because I've even been there even under human control it's like they were doing some experiment then you hear this big bang and it was like oh no and then it's like you know then the apparatus is down for two weeks as they patch some >> you were there during a disruption >> oh no this is this is sorry they have filled reverse configuration >> oh okay which has its own I mean things you know there's some arc So, so what is that? Sorry, I'm not familiar.

74:37 >> Oh, oh, what's the field reverse configuration? Well, it turns out toamax are not although they're perhaps the most studied form of plasma. There's many different kinds of architectures, essentially ways to try to stabilize and and compress plasmas. there was a a shape essentially, it's essentially a self-contained football plasma called a field reversive configuration where essentially the magnetic field inside and outside are opposite. So, they're separated by something called a separatrix. And that is sort of in theory unstable but in practice stable.

75:07 like for example when you run magnet hydrodnamics MHD code it it's unstable under that assumption but that's that's an assumption that's not the way the real world works. And so yeah it was kind of disfavored for many years but TA and other people I think Helion have have F FRC's because they are actually relatively robust. you can actually knock them against walls and they'll they'll still they stay stable and yes, but you can still get discharges and things that punch holes in your vacuum chamber which is kind of unfortunate.

75:39 >> For clarification, so you have these fusion reactors. They are or trying to be reactors maybe and apparatuses. Appares and you create a plasma. The plasma is magnetically charged >> or confined. Yes. >> Or confined. So it's confined by a magnetic field. So you have some sort of magnetic system that is tunable by a computer and then the computer tries to kind of maintain the confinement.

76:11 >> Well F FRC's kind of once you make them they're sort of sustained. There's different ways of of trying to make sure you Okay, so all of fusion boils down to something called the loss in criteria. There's essentially and it's it explains why fusion is hard. Essentially, you can just very easily in on the back of an envelope just show that the density, the temperature, and essentially the energy loss, it's called the confinement time. It's one over the amount of time it takes for the energy to decay away 1 over e in a plasma.

76:39 So the the product of those three numbers has to be bigger than some constant and then you can get fusion and if you don't then you don't. And that's explains the the fact that it's a product of three numbers explains why fusion is so hard because every approach has an Achilles heel where one of those numbers is not very big and then they try to desperately make that be higher. >> and every approach is different. Every approach to fusion is kind of different and a lot of you have to be a bit skeptical when there there's all these sort of breathless news things about fusion because it'll say you know now confinement time is start like oh stable for x minutes or whatever and it's talking about like one of the three numbers but you have to have all three numbers before it you can get fusion. I I think it the whole field is is making a lot of progress and it's very exciting but you do have to you have to be a little bit cautious about the breathless news articles that only talk about one number. So what is the computational part of that?

77:32 >> Oh, it unfortunately for better for worse it depends on the approach. So for tokamax as Brendan said it's that there's this it's mostly stable except that there's occasionally this instability that takes all the energy and max smacks it into one place and so you have to sort of keep everything sort of under control. So it's a control system. F FRC's themselves have very simple instabilities. So, for example, they have what they call a Z instability. So, it's fine. It's stable.

78:01 It'll just wobble. It'll literally wobble back and forth, but you just make what they call a PID controller that just keeps the football in the center of the reactor and and things are fine. >> And it does that by adjusting the magnetic field. >> Yeah. It sort of it actually adjusts, I think, the electric field. It sort of by sort of knocks it back and forth. The issue that people have is really depends on which which sort of plasma architecture they're deciding to use.

78:23 Climate is, you know, sort of political because of economics basically probably mostly maybe other stuff, but the the the economics of it, you know, you have to persuade people to to somehow spend more or you have to have a solution that it has like this happy coincidence where it's both economically better and and better for the climate. That that's hard. >> Yeah. In many cases, it's not. I mean many cases it has hasn't been earned but >> well you think about like okay predicting even weather right you can prep and you could see how that could be economically beneficial so what kind of work are you doing with interventions and how does that kind of interact with economics like it sounds like the contrails one I did it in a an analysis and said actually this is great because it it's very low economic impact but high value >> that's right so if you try I think there's there's sort of energy intervention. So you have to be you have to sort of compete with existing forms of energy. and that that's not trivial unless there's a co- benefit or there's some there's some sort of clever you know just co- benefit like this again is highly speculative. It wasn't our work. There was a startup that was I don't know if you saw the news. It was last year, I think, where someone figured out if you inject mercury into a a fusion reactor that the neutron flux can actually transmute the mercury into gold and then you can then you can sell the gold.

79:53 which I thought was very clever. It might not work, but >> you know, as a physicist, the one thing I want out of a fusion reactor is is helium, but that's a different story. Sorry. >> Oh, you Oh, helium 3. Well, I mean, helium 4 is kind of boring, although it's it is getting because the strategic reserve has been shut down. There's less of it. Yes. >> and of course I want helium 3. Well, not even just a fuse, just to make dilution refrigerators for >> quantis or so. Yeah. So much so much technology we think about actually just goes out the window if we run out of helium.

80:23 >> That's true. >> No one's thinking about Sorry, that's like a complete aside. >> yes. The fact that that the US had a helium strategic helium was to reserve was for our very important blimp fleet. >> Yeah. >> But but they kept it for decades anyway. So that was nice. But then we stopped. We got rid of it all. It all went up in the air. >> Yes. In in balloons and stuff >> or out of natural gas wells.

80:43 >> Yeah. >> sorry, now we're talking about helium. >> Yeah. No. So, so interventions. What are some of the most exciting interesting ones? >> Well, I'm very excited by fusion. I mean, I don't know if that's intervention. that's sort of a a source of energy cuz if we can make if we can make it work and we can make it be sort of low enough capital cost that that will actually help a lot because at least the current models are renewables are great ideally you'd like to electrify everything right which has problems because like you can't electrify flights but you can try to electrify a lot of stuff you know they're EVs you'd have to figure out how to electrify things like cement or steel Those are hard, especially things like making steel want reduction power anyway to essentially you're adding carbon and you're reducing iron ore. so there's a lot of sort of things that are difficult about electrifying everything.

81:36 But if you could electrify everything then then the amount of electricity required would grow by a factor of five and you could try to grow renewables renewables plus battery. and trying to squeeze all of it out, it starts getting ever more expensive because you just need ever more you need like a huge number of batteries to sort of cover the the last, you know, few percent or even, you know, 10 or 20%. So, we do need some sort of power that that can cover the last 20%, something that's, you know, base load. So, fusion might be a a thing for that. so, that's super exciting. Again, there there's no one sort of silver bullet that can sort of cover all the cases. So I'm happy to sort of talk about any specific case, but it's sort of like >> the world the world is a very complicated place and the the global economy is a very complicated place. So it's super hard to sort of talk about sort of interventions in general.

82:27 >> Yeah. >> Maybe instead of interventions, one thing I'm curious about is how does this make affect decisions and to for example like what do we build? How do we build? I think you're from LA, right? Or at least you >> Well, I spent 11 years. Okay. So yeah, you spent a lot of your life in LA. I mean, LA just basically large parts of it just burned down. and maybe probably close to where you used to live. so the the, you know, this is something that I think a lot of people kind of saw coming. maybe partially due to regulatory issues, but partially due to other issues, you know, and we were completely unprepared and it seems like there is a lack of preparation about what to do next or to sort of adjust for this. And I mean have you worked on basically predicting like new risk assessments or you know suggestions of like what do we actually change to maybe harden you know society even that for what's coming regardless of whether or not we're actually do something to make solve the underlying problem.

83:29 >> That's right. So in fact there's a big effort at at Google into something called career crisis resilience. And so we had a very fun project called fire fires. I don't know if you know about this. so it turns out that for wildfires a lot of these wildfires you could c if you if you only caught them early enough u it's very easy to put out a a wildfire the size of this room but even if it's like an acre it gets much much harder. And so and of course under certain circumstances they can grow you know exponentially from from the size of this room up to an acre. So that might be hard to catch but they often sort of start small and spend a while. So we figured out that oh if you had a global constellation of low earth orbit satellites that could detect in the midwave IR that which goes back to the black body essentially that's the temperature of fire. they stand the fire stand out in the midwave IR. And so, we designed, a sensor that if you built, it depends on exactly what their orbits, but, you know, roughly 50 to 80 of them, you could actually find, fires about the size of this room, about 5 m on a mount, maybe it's a bit larger than this room, 5 m on a side. and anywhere on the planet and again depending on how many satellites you had within within like 15 to 20 minutes. You'd have to put you know a fair number up like you know 80 to get them within 15 minutes and then you could actually intervene. You could decide not to if you wanted to you know have the fire burn fuel and you thought it was safe but if it was going to blow up to something unsafe. So you worked with a now a nonprofit called Earth Fire Alliance that we're part of and so they're starting to we've launched one satellite which is a prototype. We've worked with a company named Muon Space to actually sort of make the satellite.

85:20 So that's cool. We have wildfire boundary detection and we propagate that information out through Google. So we can actually sort of figure out from existing satellites and existing data feeds where the boundaries of fires are and then we sort of tell people through through their Android phones or through search about fires. we've worked with the US Forest Service on making new models for how fires propagate because again that goes back to these process-based models from the 70s by a person named Rothermell. So we've actually made a little neural network proxy model based on a new essentially to sort of be able to run it very very quickly. so we worked with the forest service on that. so yeah. Yeah, we're very very interested in trying to minimize because it turns out people might not realize the World Health Organization estimates that there are 300,000 excess deaths a year across the world from wildfire smoke.

86:14 >> Yeah. I mean I remember was it if it's been a few years since we had a really bad fire season. Maybe what four or five years ago there was this cloud of smoke which like crossed all of northern US and Canada and caused a lot of respiratory issues I think. >> Yeah. >> Yeah. And it's very hard to track. I mean, you have to get these you have to get the estimate these excess deaths from statistical means.

86:35 >> But yeah, it's a very serious public health problem and also just very scary and burns people's houses down and it's terrible. >> My view is that climate change is sort of like a serious disease. Do do you treat the symptoms? I.e. do you adapt or do you try to attack the underlying thing? And the answer is well if it's serious enough both, >> right? And so yes, so we take sort of adaptation especially around climate resilience very seriously at Google and we try to give peopleformational tools to sort of help. That's part of the reason why we're working on weather and then sort of cyclone prediction and things. So it all actually hangs together. So it's more than just you're right. It's more than just interventions. It's climate resilience too.

87:13 >> Yeah. Having lived through four or five fire seasons in the on the West Coast, they're they can be quite quite nasty. And it used to not be I mean I have a cabin up in the Sierra Nevada mountains and yeah it used to be oh you know summertime it's nice and then now it's like well not every year but yeah there's like you know winter, spring, summer and smoke. >> Yeah. Fire season you're just >> Yeah.

87:36 >> I want to stay stay to the west of the fire line and >> Yes. And Yes. So that is another thing that I'm interested in and that Google's also very interested in is is climate resilience. Are these IR sensors small enough that they could hitch a ride in like a micro satellite grid? Like would it make sense to would it be almost cheaper just to hire I mean to to pay someone who's watching a constellation?

88:00 >> Oh, they're not that small. The thing is you need refrigeration because >> Oh, okay. Okay. >> because it's midwave IR. So, you have to you have to keep it cool. >> So, these would have to be their own special >> satellites. They're not super large. >> Okay. They're not like the, you know, the the satellites in in geocynchronous orbit are these giant monsters because of all the optics and who knows what. But yeah, >> and they have, you know, they're basically just eye sensors with a resolution of 5x5. no, no, that's the other cute thing is the resolution is about 50 by 50 m, but you can use super resolution because it's essentially multisspectral and you sort of know where fires are and you have so yeah, there's a a fair sprinkling of AI on top of them to to reach that 5x5 meter.

88:44 >> Yeah. Yeah. >> When you also have you have a convolution over the what you're reading out, right? As well >> the I forget the frame rate. the satellite is moving. I don't remember what the point spread function is. I'm sorry. I don't but you're right. They do move, but I don't remember how fast they I don't remember how fast they this also has something called a broom sensor. So, there's this funny thing of trying to you kind of spread out the spectrum one way and there's also it's it's a somewhat complicated thing. It isn't just a like a a Polaroid. It's it's a complicated sensor.

89:19 >> Yeah. Sort of switching gears a little bit. you know, you've been at the intersection of AI and science for quite some time. I think you've sort of wound your way into and out of it back and forth. >> How do you see the field has evolved? Because I feel like it's evolving very quickly now. And like what are the sort of lessons that you've you've learned that you think the community has learned and how do you think this should change if you are a young scientist or young practitioner? How should this change what how you should approach you know the future? Well, I think there has been a phase change in the last 12 to 18 months. So, I mean a lot of what we used to do as I as I said was build these specialized models to solve individual problems. And if you think that's your job, it's kind of fun. You you find a problem, you solve it. You find another problem, you solve it.

90:11 >> But now we have these much more general AI things. And I think the whole AI for science community is kind of still feeling around. There's such the fact that they're working is so new that collectively we're not sure like what's the best thing to do or maybe there's no one best maybe there's a tool chain and I think I think we're all trying to figure out like what should we do >> and so there's a question of what should young scientists do? I think it would be I you know I I I have a son who's who just turned 21 and he's really into both sort of AI and coding and chemistry and I look at him I think he's doing the right thing because he's both learning a lot and trying to be a domain expert about RNA but he's also sort of using vibe coding and using all the tools. I think that's I think that's the right answer is is is because everyone's figuring it out still be a deep domain. I don't I don't think domain expertise is going away because it goes back to a lot of people who said it goes back to taste and trying to figure out how people get taste without doing all the the grunt work. That's an interesting open question but develop domain expertise but also try and play I would say with all the different tools that are available because it's not like oh yes we know what's going to happen and the smart old people are knowing what's like no we're experimenting too and so yeah so I would say definitely develop domain expertise and try to use these tools and try to solve big hard scientific problems as best you can there's still the huge open issue about what do you actually do about physical lab work that is not going away because you know experiments are the the ground truth and >> and a bottleneck >> and a bottleneck. I mean people are talking about a lab in the loop but that's still very very very open because how do you how you no one has I think as far as I know a general lab that does everything. There's a lot of very specific you know labs that are controllable.

92:13 >> so I think it's just we've gone through this phase change. it seems super exciting. Again, I would advise people to play with whatever tools are available and to develop sort of deep domain expertise and and taste to the extent you can. And I would advise people also not not to be scared and like try stuff you know you know I'm always happy we have student researchers at Google and they come and they do sort of wild and crazy things and that's always just delightful. So yeah, people should be trying sort of wild and crazy things and see what happens. Yeah, this may be a question without an answer, but when I think about how I developed expertise and how a lot of people developed expertise, it was by starting with a, you know, a a simple defined problem and then hammering it and then in that process of exploration, you learn more and you know some ways you go broader, some ways you go deeper, but you still the process of just banging your head against the problem which now would be instantly solvable teaches you the skills you need to solve harder problems. What advice would you give to your son for that?

93:16 >> You know, I don't know. Maybe it's a bit like hiking, which is you Yes. I mean, you obviously can't drive everywhere. Or you could you could drive up the mountain. >> Yeah. >> Or you could hike up the mountain. And maybe it's okay, even fun to occasionally hike up the mountain even if you can drive up the mountain. >> yeah. I mean, >> in the old days, I'm going to sound like a real old man. In the old days, >> in the old days of six months ago, >> well, no, no, I was even thinking of in the old days of the 80s and 90s like >> you know a lot of people take it like oh there's open source packages there's there's learn there's there's all sorts of things we didn't have that I had to write my own numerical library I had to write my own machine learning I've written boosting probably rewritten it four times four different languages and and so now I know boosting you know it's like and so maybe not taking the totally easy I mean obviously I mean there's this trade-off like oh but I want to be as efficient and and productive as possible. Yes. But you also have to develop the muscles. So it's a little bit maybe like being an athlete. Like there are places there are times when you're actually doing exploit when you're trying to run as fast as you can. And then there's also training time and so maybe people just have to train.

94:17 >> And it's entirely plausible that if you spend time actually hammering away and doing the hard work, even if it goes slower there, that pays dividends into your larger you know productivity long term. Like even if locally that one moment you are not being maximally productive by not exploiting an LLM that feeds into something. >> I hope so. I hope the thing I don't know is I hope people in their careers it's hard because right the the the whole world seems to want to optimize everything and your fault. No, I don't. It's No, I don't. It's my fault, but it's just sort of the the the the you know, and sometimes you have to set aside time. Like at Google, especially in my group, we have this concept of 20% time, which I still I very very strongly in my own group try to protect. It's like you can do whatever you if you want to learn stuff, if you want to try stuff, you don't even have to tell me. You don't even have to tell me. In fact, I probably shouldn't tell. you know, just do stuff for for Exactly. for learning and also because that's where the sort of creative juices are. I don't want to so occupy people's time where they have nothing like where they they can't feel like they can play or learn or try new crazy things. So I know 20% time is unusual and there just seems to be this strong impetus in the world to just like like I said optimize and squeeze everything out but you do lose something when you hyperop you sort of overfit.

95:39 >> You overit overfit to productivity as you're so >> that's right. So I I know that might be my advice might be swimming upstream against perhaps cultural norms. >> The thing that always comes up for me here is that I there's a the problem is not stationary. There's a new skill set that will be the the right skill set for the future, right? And that the question in my mind is always just tangling. Okay. Is this a skill that is an enduring skill that Yes. that or or or like maybe it it wasn't enduring yesterday, but today it will be enduring because the like I've seen that you know like for just as an obvious one you become sort of like a manager when you're when you're using LL skills transfer. Well, >> some of them don't, but a lot of them do. And so that as a manager, you lose track of of the details of what's going on and you trust your people or agents or whatever to have that managed so that they can report up to you and answer you know sort of the highle questions and and get the judgment about the little things correctly and and so that is that what we've come to is that we're just like middle managers now. No, I I mean I don't know. Again, I I I have a little management work, but you can't I don't you don't want to be an empty suit.

97:07 >> In other words, because the things might get it wrong, especially LM that are sort of really weird. They don't make the same kind of mistakes that humans make. >> And so you you can't, you know, fully trust them. You have to be rigorous and like, you know, poke at it and make sure. Although, you should be poking at software that you write yourself, too. I mean, you shouldn't trust yourself. That's that's one thing I've learned.

97:27 What did Fman say? You know, you absolutely can't f fool yourself and you're >> you're the easiest person to fool. >> So, so that might be I don't know if I I can quantify what's enduring, but somehow fundamentalness I mean really learning a domain that is about the world for example. So this is why I like well biology or physical sciences I think those will those are enduring sort of fundamental things. math is very enduring, but even things like rigor and checking and that sort of thing, which goes back to maybe management that that you want to really make sure that the LMS are producing the right things or they haven't cheated in some way, but again, you should be doing that to yourself, too.

98:10 >> Yeah. >> So, I think there's some enduring and also just the enduring value of sort of creativity and thinking out of the box. And so, I think those are enduring. I don't know. There's some there's something very fundamental about all of those. >> So, you mentioned Fineman. if you don't mind me changing gears. >> I took a class from Fineman. Yeah. Not just any class. >> Yes. The the the his sort of physics of computation class. He did it with in fact Hopfield and and and Carver me. that was fun. At the time I don't think any maybe Fman knew. I felt like none of us knew what even the problem was. I mean I guess Fman was trying to say oh let's do quantum simulation which I guess is it turned out to be the right answer but yes at the time it was well I guess it was cool first of all it was DARPA funded it so you were supposed to go once a week to get a prime rib dinner but I just went every week anyway for the students.

99:07 >> Yeah. So ju just for a little bit more context, this class was basically the class right after Feman and I forget who else proposed the concept of a quantum computer without really knowing what it was, but knowing that there was some sort of >> well there was plenty of there was plenty of room at the bottom essay which I think was in the very early this I took it in '82. Okay. >> Oh, that was when he taught it.

99:30 >> And he did say there was plenty of room at the bottom. But the way the class was structured, it was it was like a guest lecture on Tuesday and then Fman would stand up on Thursday and explain why that was all wrong. which was pretty fun. And then we encountered something which other people have also encountered something which called the Fman effect. Maybe he was so charismatic or something. He would explain things and he would say, "Yes, yes, I understand. Yeah. And then you walk out to think, "No, no, I I did not."

100:01 >> Yeah. So, it was kind of fun. But a lot of the guest people, I mean, maybe the sort of showed the chaos. I mean, a lot of people, I think it was Danny Hillis came, there was all these interesting guests. So, I would say in the union of all of them, I think showed the sort of mass confusion of what was going on because there was a lot of like, oh, should we make computers reversible because we have to make sure, you know, can they even be reversible?

100:22 Can can the sort of the bottom limit of of the heat per operation be zero or is there some sort of thermodynamic limit that was like a big deal. I don't think that's I don't think that's a big deal now. >> Yeah. But >> wait. So so you're talking about the land hour limit, right? or not what's on them. >> well there at the time it was all this this question about like can you have rever like billyard ball computers and can they be reversible and things so and also sort of like what kind of computing that's why Danny Hillis came like you know I think that was in the era of the connection machine and what the original thinking machines not mad but the original one in the 80s >> so he came >> what was the state of like general computation in 1982 like I mean I mean at this point >> to rounding error we had zero I mean we had I remember when I sorry I got there I was Carver me's CIS admin and we had a vax 11 750 that maybe did a myip 1 million 1 million operations and the whole research group shared an 80 megabyte disc drive that was the size of a dishwasher. It was it was very exciting. so what about complexity theory? What what was it? Because I know that there's a lot of interest in quantum complexity and how it relates to gravity right now. And I wondered I don't have a mind in my mind about when complexity >> we we could try to look it up. I don't think there's of course the whole there's a whole hierarchy of you're right quantum complexity classes.

101:43 >> I think that was developed after that because this was in 1982. In fact I I'm not even sure there was a gate. We never talked about the gate model. There was no gate model of quantum >> I mean quantum gate. Yeah. Yeah. Yeah. >> Yeah. Yeah. I forget. I'm sorry. I >> think a lot of those Yeah. I don't think those came out until like the the what 90s or something. Yeah. I mean I I remember reading Neielson and Chong the classic quantum forum information >> yeah quantum information textbook which which brings up a lot of those points now I think about it I that textbook was written in what the late 90s or early 2000s that's right >> and I think that was the first textbook which like put down kind of the general knowledge of the the the field but I could be wrong I guess I I think now that when I was young I don't think I took it just for granted that this is this is the book that everyone >> Yeah but this is a Well, you remember in the in the early ' 80s, I mean there was a whole bunch of excitement around neural networks and it's really interesting cuz again we really did not know what we were doing. No one knew what they were doing.

102:41 >> There was like an interesting like neural networks then SVNs then neural networks. Oh no no no this is before that because in the 80s everyone said everyone said this is has the capability of revolutionizing computing >> but what does well I mean there was a tremendous excitement around hfield networks in fact nurips came out of because there was a workshop at snowbird that was nominally private but everyone tried to crash and so they they spun up nurse and so that was >> snowbird is a is a skiing trip with a with a computation conference attached >> that's right that's I learned to ski because I didn't know how to ski and then I kept going. Anyway, >> that workshop came out of the Santa Barbara workshop in 1985 which came out of some local things at Keltech which is called hopfests.

103:25 >> So, so yeah people thought oh wow something involved in fact ironically there was like yeah something about associative memory and if you actually dig down into what transformers are they are associative memory. So in fact there was even a paper called you know hopfield networks are all you need. so >> yeah I forgot that title was a reference to that paper >> to the era to the era so it's actually it's like it's like the whole thing has sort of come full circle and in fact I think we did we have collectively as a as a field revolutionized computer science but it was the the sort of the hopes and dreams completely outstripped the capabilities because again we effectively had zero compute >> yeah I think there's the the history of machine learning is different paradigms as like compute versus memory scaling versus data become available in different levels.

104:20 >> That's right. And the fact that I think people don't realize that the reason why neural networks one is they're the one compute limited thing although again with we with now that we have transformers things are getting memory limited again but but and the fact that they ride on top of blaz and so the fact that blaz was being optimized by things like GPUs I mean I don't actually know if in fact I'm pretty sure the brains don't work by matrix multiply but but it was just that that did that the algorithms co-eolved with the with the hardware >> and And that's that's why we're here.

104:52 Who knows? I mean, if we'd gone down some other path where people really cared about some other compute making who knows what what architecture we have ended up with. I don't know. I mean, didn't a lot of this start with like I think people were hacking PlayStations to trade to train neural networks or something or to do I guess maybe it was even before that for like super >> There's a there's a funny story by a friend of mine from grad school Brian Kotenzaro at video where he Brian sorry if I get this wrong but he came to Nvidia and and was having a lot of trouble getting traction and and and he basically they were they were doubled and tripled down on on gaming and he basically one day had a meeting with Jensen and convinced him let's you know like this is we have all these people using CUDA for blaz basically and and for deep learning in particular and convinced Jensen and they he said it was like a 15-minute conversation and they pivoted the whole company the next day or whatever. Okay, this I've never worked in I mean I have friends Dave Kirk who's the first chief scientist and and Bill Deli were both friends of mine from grad school. So yes, but I don't know the details of of way. Now remember that people like in in Hinton's group even in the around 2010 they were using GPUs to do the deep learning but but even in as even in I would say 2007208 they weren't that much faster than CPUs.

106:21 Again they they they only started really exceeding in fact I think this not a coincidence in the era of you know the original imageet and some of the speech some of the speech recognition stuff. So that's I think that's not a coincidence that they really researched when when when GPUs passed CPUs. So there's that that's not a coincidence either. >> Yeah. I really want to know how does one get the opportunity to name an asteroid. >> Oh well again is Caltech there was there's were some wonderful people Jean and Carolyn Shoemaker and they were teaching a class in in planetary science. I like planetary science and so yes as part of that class they took us they they took us through the sort of asteroid discovery thing. Now this is in the 80s so now there's all sorts of amazing amazing systems. Well there was for a while there something called linear which automated the thing but this was before anything was automated.

107:16 So yeah, it was just part of a class and what you would do is you go the steps even though we did it out of order in the class but the steps we used to do this is 40 years ago is you would go to Palomar there would be a fast telescope which literally now a museum piece that's in their visitor center but at the time it was a real thing you would put a piece of film in it you would take a picture and then you'd wait for a few more minutes and take a same picture of the same point of sky you'd put in a stereoscope like back at Caltech you would see if anything you would look around you would see if anything floated Because it's this it would move by a tiny bit. So it would pop up and you can see it was because it's different in different eyes. Then you would be able to see it as a something that was projected in a different place.

107:56 >> That's right. So you would pop out at you literally. And then you would go to a measuring microscope and you would take measurements of known star references and where this floater is. And so if you got accurate enough measurement, then you would send it off to a person, I don't think it's him anymore, his name Brian Marsen at the Minor Planet Center in in Arizona. And then he had a big software system to do to sort of piece together these are called apparitions. And if you found and if you happen to have done an observation, which was the final apparition, which allowed his software to connect it into one big orbit, then you would get discovery rights and you could name the the asteroid. But now it's like amazing. There's there's this observatory now in Chile called the Vera Rubin Observatory and there's this amazing telescope called the Simony Survey Telescope. Essentially automates this process. Essentially can just take many frames of the sky. It's an utterly stunning instrument. And so they discovered 11,000 asteroids in 6 weeks.

108:54 >> Oh wow. >> Yeah. So it's now >> How many of them are there? I mean at least >> Well, it depends I guess on the cut off but yeah. So, but there probably millions of of asteroids. >> So, there's still a lot of opportunity. >> It's true. But I don't even know if they bother name. >> So, I don't know. Maybe you don't need I don't know. But people are doing occult. Sorry, I can I'll talk about this forever. There's there's some there's stuff called occultation. Like one of my asteroids I I was talking I was sending email to a amateur. I if an asteroid happens to pass in front of a star, it dips. You like when they discover exoplanets. But if you're super lucky, it'll dip and then it will dip again because there's a moon. And so one of my asteroids this amateur found a little moon around it. So So that was I think last year >> around the asteroid.

109:41 >> Yeah. >> Interesting. >> So that's pretty cool. So there's still room for for >> your asteroid has a moon to it. >> Yes. In fact, apparently it's actually actually there's a little bit of a story there because it's not really my asteroid that I found two of them. I named one of them after my dad. I waffled. So Carolyn named it after a professor at University of Washington and I said, "Oh, but I wanted to name it." And so I whed it and she was nice and so she gave me one of hers. And so then they named that one after my mom and that's the one that has the moon.

110:08 >> Oh, okay. >> So So the asteroid named after my mom has a moon. >> How big is it? >> It's like the main body they think is about four kilometers across. I'm trying to get the units right. And the moon is about 1 km. So it's actually it's not it's a pretty big binary. They're not quite twins, but yeah. >> Did other people in that class >> discover? I think there was one other person. It's a little bit of a crapshoot. Yeah. And the fact that I found two was quite unusual.

110:36 >> Was there a reason why you was it just pure luck or just staying there all day? Do I try? I know. I I tried to be careful, but >> Yeah. How does one become get an Oscar? >> Yeah. Academy Award. >> well, again, maybe I was at the right place. I my thesis advisor is named Albar and I was a student intern actually in 1986 at a place called Shumber which is a boil discovery place but they had an AI lab and so they were we were all doing sort of computer graphics research and we were all thinking you know there at the time this was revolutionary I realize it's now considered incredibly boring like you could actually use like physics simulators to to make computer graphics movies and at the time I was like wow that's really cool so I said you know you could use the theory of elast elicity to make floppy things and and so I said I sort of like here's the theory of elasticity and I wrote it elastic simulator and I made you know fabric and you know stretchy things and and stuff.

111:29 So they said oh wow that's cool and so the sort of descendants of that became a lot of the physics simulators that people used in in you know Pixar and their various movies. So yes I tell I tell interns sort of half jokingly well if you do a really good job as an intern you can get you get awarded. So, so how like how long was that between the time you did that work and then >> 20 years?

111:52 >> It was 20 years, >> which is actually not atypical, right? Cuz they want to when they give you an Academy Award, they want to make sure like, oh yeah, it's sort of well used and everyone uses it and stuff. So, yeah, but but of course the obviously like in that 20 years like, well, everyone does it. It's obvious, but you know, >> this was specific work done for a specific movie or something. >> No, it was like a it was like a paper in cigarette.

112:10 >> It was it was it was a paper and then Pixar just became like I think somewhat important core to like a lot of their >> Yes. And and in fact some of my friends in fact a lot of my friends did did a lot of these simulators. So yeah >> I'm curious about since you have been quantum computing adjacent and worked on quantum computing directly at Google applied sciences for a while. I think you're not currently on that but I'm curious to see what >> I'm still dabbling in you're still dab.

112:35 Yeah. So where do you see the trajectory of quantum computing over the over the years going? Because this is one of those things which I guess kind of like fusion was sort of had a a sense at first like being very exciting and then seeming like it wasn't going anywhere for a long time and then maybe now we're seeing hints again of it being exciting and I I'm or maybe that's like my sort of quantum adjacent.

112:58 >> I think a lot of people have gone through that. I tend to average I tend to average things out over the I guess I'm old enough now where I sort of average things out over the decades and it's just progress. >> But I mean like see going from you know starting from fman just trying to figure out even what this means as a concept all the way up to now where you know I there's this recent Willow result of quantum error correction which actually seems genuinely >> the right achievable with like the right scaling laws and stuff. I mean does this is this like a do you still think that or I mean would you say that quantum is 20 years away or I mean till like some simple but practical algorithm which actually succeeds or I mean is it do you see this accelerating or do you think this is still going to be something because I mean there's a lot of areas where we we saw oh this is very you know this advanced very quickly and it was very unexpected and I'm wondering if this is a thing that you think will be quick or will not be or is this I don't know >> I think sort of in between it's not a purely software thing because we need to build systems that are large enough and stable enough to be able to you know be a quantum computer instead of a quantum apparatus we're in the what they call the NISK era that was a >> intermediate scale quantum which was I think coined by John Prescll it's not a very good term sorry but I guess that's the term we have and there are you know the quantum team made this wonderful paper which I think I'm co-author on one of many for something called the quantum echoes algorithm which could be applicable now which fundamentally it's it's it's in the style of of Fineman's proposal which essentially it's well the technical term is you could try to fit a Hamiltonian to observe data like an NMR what that means is yeah you have a physical model that's parameterized and you use the you use the quantum computer to kind of adjust the parameters and try to figure out inside of a loop to what the right so you could for example decode NMR parameters. So that is a a practical thing that's used. Now the question is is it going to be it will be big enough to make breakthroughs that's still TBD. the quantum team at Google is has been very executing against amazingly well against a a roadmap that that Nevin laid out a few years ago and they're just continuing to march down this thing where they're scaling up and when they hit their last milestone they should be able to have a a quantum computer that does amazing things and they're continuing to march it along. So yeah, it's it's I think on the scale of a few to several years. I I don't I haven't sort of kept up on exactly what date they're saying, so you should ask hard exactly when that's going to happen. But yeah, they're they're so they're marching along. So So the main the interesting question is will the superconducting computing be the winner or will one of the other sort of alternate technologies and that's still TBD. I I still think superconducting is is a very promising thing because it is it is very scalable. so yeah, no, I don't think it's 30 years away. I don't think it's tomorrow. I don't think there's going to be I don't think unless well, even if there's a sudden hardware breakthrough. These are very finicky things. So it's like someone might have a brilliant idea, but it will still be a while before cuz fundamentally, at least in the NISK era or for a while, these are fundamentally analog computers and so that they tend to be very very very finicky. So I wouldn't expect all of a sudden, you know, something some some phase change happens.

116:30 No one has figured out how to scale up cubits in a way that interact in just to kind of arbitrary size or still on the order of 100 200 cubits I think or >> you mean in terms of okay so >> in terms of >> well it's a little like yeah yeah exactly so for superconducting cubits yeah people I mean although people are working on it the ratio of the number of physical cubits to logical cubits is still relatively large for the error rates that you need and so maybe there'll be a breakthrough there I don't know or there other things that might have you don't need that ratio to be so high because a lot of it has to do with the the 2D connectivity of the of the of the chips that you lay out your cubits on a 2D chips and for things like neutral atoms they in theory in theory can connect anything to anything else but of course in practice we don't really know and we don't actually know what the limitations of neutral at least I don't know the limitations of what neutral is maybe maybe the super experts know >> super connecting cubits in particular have this sort of tension where you you want to have you know nice clean resonators which you know you you get a clean resonator by decoupling from the environment and then you get good interactions by coupling resonators which involves covering to the environment. So it seems like >> true but at least but but there yes but there's like the whole environment and then there's like little tiny holes that that you want to go through to your neighbors.

117:53 >> This is not like a fundamental uncertainty relationship or something. It's like this is a technological limitation that difficulty or something. >> Yeah, the hardware team in in and Google quantum is is very very skilled. They're very very skilled. So, yeah, they're they're really good at at making these designs and making these things actually work. So, I I I find them impressive. >> Yeah. >> It'd be exciting to see that advance. >> Yeah. Yeah. Yeah. >> Yeah.

118:16 >> I guess we'll just have to hold a breath and wait. >> Just wait. Yeah. I guess I'm just very patient, so I just start working. So 20 years ahead of >> that's what Dave Bacon who runs the the the software team in Google quantum always he he teases me like John like oh no I can't work in quantum computing was like 10 years ago because it's going to be you're always 20 years ahead of time and so I have to wait for 20 years like okay well you know 10 years have gone by like you >> halfway there >> halfway there I don't I don't take that as a an absolute yeah but I also started working on fusion 10 years ago we'll Yeah. Or >> may maybe you're actually causal like you start working on something and that reality just you know catches up catches up.

118:58 >> I guess so. Maybe. Who knows? I don't know. >> I I started working I loved convolutional nets in the early '90s. >> Yan Lukun claimed I checks out. >> I coined the term convolutional net. As far as I can tell that that may be true. I because everyone called it Lynette because it was a very specific thing. They all worked for Yan. I said well I don't work for Yan. I don't want to call it Lynette. So I called it a convolutional net. So I don't know. That was a more generic term. It's a good term.

119:21 >> Yeah. >> Before you go, is there anything you want the audience to take away? Any messages you want to deliver? >> I think error is an example of it. But I I'm really amazingly excited about the potential of AI for science. I think it's I think it's going to be an amazing power tool for scientists. And I think scientists won't be replaced. I think the in fact I'm hoping that they'll spend all their time on again the creative stuff, on the rigorous stuff, on the philosophy stuff. And so I I think it's gonna I think it's going to be way cool.

119:51 >> I I hope so, too. >> Yeah. No, yeah. My intuition is that a lot of a lot of I've heard a lot of people say that that I hope they're right. I hope it's not just a sort of coping with a reality that's uncomfortable. The other question we almost forgot to ask. If you could remove a bottleneck in your industry, which whatever you however you want to define that by fiat then like magic. >> Yeah. By by magic, what would that be?

120:21 >> If I could get a magic wish, I would say someone please make the the everything lab that you could like send JSON blob to and it will do any experiment at all. >> Okay. Automated, >> but it have to be anything. So essentially, so I guess we have to solve the sort of AI complete robotics problem, I guess. But if we did, then that would be stunning because right now things like ERA, it's all computational.

120:43 So someone has to gather the data. >> So yeah, if we could just break that. Oh. Oh, that would be so amazing. That would be so utterly amazing. >> Yeah. great. So I really appreciate you taking the time to to to see us. and I think you you like kind of flew in and and adjusted your schedule a little bit to >> Yeah, I was I was sort of flying over San Francisco to get home and so I said I landed in San Francisco.

121:08 >> So you you made a big effort to be here. We really appreciate that. It was really fun to talk to you. >> Mhm. >> Yeah. Listen, it's been a blast. Yeah. >> Okay, cool. Thank you for having me. >> Yeah, you're welcome. thing.