MIT 6.S191: AI for Science

0:12 It is my great pleasure and honor to introduce our first guest speaker for introduction to deep learning 2026 and he's a very special one. Chris Bishop is a technical fellow at Microsoft. This is the highest technical distinction that one can receive. And the founding um the founder of Microsoft research AI for science which is a global organization overseeing a portfolio of work at the intersection of machine learning, AI and the natural sciences. And Chris comes from a background deep in physics and machine learning and has brought that to a really unique perspective in terms of thinking about bringing AI and deep learning to real world problems in these domains. And he's also done great contributions for the community in terms of deep learning education publishing seinal books on machine learning and deep learning in particular. So please join me in giving him a big round of applause and welcome to Dr. Chris Bishop.

1:21 >> No. >> Thank you, Alberta. I'd like to begin by asking you to think for a moment about what you consider to be the greatest scientific discovery ever made. Now, I haven't given you a metric, so this is really a matter of opinion. I'm sure there will be many different opinions here, but in my view, the most extraordinary thing that science has ever uncovered is the fact that our world is described by mathematics. It's not obvious, right? Making marks on a piece of paper allows you to predict the future. It's extraordinary.

1:55 And it's not just mathematics. It's very simple mathematics. Even if you're not familiar with these equations, um I think you'll agree these are very simple equations. And yet these equations, perhaps a little bit of relativity uh sprinkled on top really describes everything that we experience in our everyday lives. Everything we can experience without using specialist equipment. So that's extraordinary. But these aren't just rough and ready models of the world. They have extraordinary precision. So the electron, which is one of the building blocks of matter, is like a little magnet. And we can use theory to calculate the strength of that magnet. And this is what we come up with. And the error bars at the end are because we neglect some terms in a kind of series expansion. And then we can go and do an experiment and we can measure uh that magnetic moment. And this is the number we come up with. Again, there are some experimental error bars. And you'll see that these agree to 13 significant figures. That's just staggering precision.

2:58 So we live in a world that's described by mathematics where areas like the earth's atmosphere, materials, new drugs, we have the equations that describe them very precisely. So we ought to be able to just design a new drug on a computer and we're done. But we can't. And the reason is rather ironic. Those equations are very simple to write down, but solving them turns out to be extremely difficult. So we could say that the underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry and that includes biochemistry which is just complex chemistry of course are thus completely known. The difficulty is only that the exact application of these laws uh leads to equations much too complicated to be soluble. Now those aren't my words. Those were said by the theoretical physicist Paul Durat nearly a hundred years ago.

3:55 So, we're in this fascinating but very frustrating situation that for a century now we have these beautiful simple equations that describe the world with exquisite precision and yet we're frustrated because we can't solve them in order to go and tackle lots of real world applications. So the question is how can we best use the knowledge of those equations in order to accelerate scientific discovery and to address that I just want to step back for a moment and think about the scientific discovery process itself.

4:27 So we have a scientist and of course today a scientist would generally be a team of scientists or a team of people from many different disciplines working together but let's just uh call that a scientist. The scientist formulates hypothesis. Uh those hypotheses are then tested against experiment. This is really the foundation of the scientific method gathering evidence from the real world. So those observations from the experiment can then uh inform the scientist and the scientists can go and modify their hypothesis. For example, if you're looking for a drug, you're searching the vast space of potential drug molecules. You dream up by whatever mechanism some candidates you synthesize in the laboratory, test them against proteins. Some of them bind, some of them don't. And you use that information to refine your list of uh drug candidates.

5:14 So that fundamental iterative loop of science of course has been with us for a long time. Um and it doesn't operate in isolation. Uh the scientist of course talks to other scientists, reads the literature. So this is part of a whole scientific ecosystem. One development that I think is very important happened really in the 1960s with the development of digital computers and for the first time this really allowed us to bring those equations in into play because we can solve the equations in uh for other than toy problems by using digital computation. It's very very expensive typically to do that but it gives us a different kind of oracle since we know that the equations are accurate. Instead of doing an experiment we can do a simulation. Of course, there are pros and cons to experiment and simulation, and they're sort of complimentary, but it adds another uh pathway, if you like, to the scientific process.

6:05 Now, we're in the early dawn, I suppose, of a of a remarkable AI revolution, which is touching many things, including, of course, scientific discovery. And so AI is really impacting every aspect of this process from the formulation of hypotheses. How could we design a million molecules rather than half a dozen? Something that a human alone couldn't do. Um analyzing data from experiments and helping us interpret them. Um driving automated wet labs. There are several companies I think in even in the Boston area looking at um automated wet labs and how can AI help to automate and accelerate the experimental process. and of course helping the scientist in their uh day-to-day work addressing that vast literature of millions of papers. A human being can't read the papers but an AI can read the papers and and and that's incredibly valuable. Um and so AI is impacting this whole process. This is enormously exciting and enormously important.

7:01 Microsoft is very involved in this. We have a platform called the discovery platform which is an agentic AI platform built on Frontier LLM that's helping to accelerate this whole discovery process. That's not what I'm going to talk about today. I'm going to talk about a different and very exciting way of using those fundamental equations to accelerate scientific discovery and it's what we'll call uh AI emulator. So the AI emulator provides a third pathway, a third kind of oracle to use within this scientific discovery loop.

7:35 And I'm so excited about this that um four years ago I made a pitch to the chief technology officer of Microsoft to sort of give up my lab director job and form a new team called AI for science. And this is this is what we're focused on in the AI for science team and is very complimentary to what we're doing in Microsoft with the discovery platform. So to understand what an emulator is, I want to again just step back a minute and look at a very important theorem at the heart of machine learning.

8:05 which has this wonderful name of the no free lunch theorem. So the no free lunch theorem essentially says you cannot learn purely from data. You can only learn from data in the context of some kind of assumption. Um sometimes called prior knowledge, sometimes called inductive bias. It's phrased in different ways from different perspectives, but it all comes down to the same thing. You make a prediction by combining data with some assumptions. Now you're all familiar with large language models. And in large language models, we use data, lots of data. It's very, very data intensive. In fact, we're using data at an unprecedented scale. And it's that scale to extremely large internet scale data sets that's allowed large language models to achieve their extraordinary capabilities.

8:53 The assumptions, the inductive bias are fairly lightweight. We have uh attention. you know, prediction for the next token depends um to greater or lesser degrees on previous tokens and that waiting is itself data dependence. That's the attention mechanism that's at the heart of the transformer. But it's very very general, very lightweight inductive bias. In science, we're in the opposite regime. In science, data is scarce. It's expensive. It can take a student a year to figure out the 3D structure of one protein. That's one data point. Um and yet we have uniquely in science this very rich inductive bias. We know the equations that describe the world. So how can we combine that expensive data with this rich inductive bias?

9:40 So let me just click down a level and say a little bit more about what I mean by inductive bias. So we have these equations. If you're not familiar this is Shreddinger's equation. We won't be doing any maths in the talk, don't worry. But this equation describes the properties of matter at the atomic level. Um describes uh it's an equation for an object called pi the wave function. I'll say a little bit more about the wave function uh a little bit later but a a relatively simple equation that certainly should be part of our inductive bias. That's part of our prior knowledge.

10:13 We have other things too. Um invariance and equivarian. So here's a molecule. If I calculate and measure the molecule is just in a vacuum, it's not interacting with anything and it has some energy. I could calculate the energy. If I rotate the molecule, the energy must be unchanged because space is isotropic. So that's called an invariant property. Now, if the molecule is represented by the coordinates of all the nuclei of the atoms in three dimensions, when it's rotated, all those numbers will change in some complicated looking way. That's what we feed into the neuronet. But the output must remain the same. So that would be called invariance. If the molecule has a magnetic moment, then when the molecule rotates, the magnetic moment must rotate with the molecule and that's called equivariance. So again, that's a property that we want our solution to satisfy.

11:02 And in fact, that idea is really the foundation for a whole sub field of of deep learning called geometrical deep learning. Very beautiful and fascinating field. Um, we have conservation laws. Energy is conserved, mass is conserved, momentum, electric charge and so on. exact conservation laws of the physical world and again would like to respect those. So how are we going to exploit these inductive biases? We could um and and a lot of work has gone into this. We could design specialist neural network architectures that sort of bake them in automatically instead of having a general feed forward you know multi-layer perceptron. we structure the network uh in in a particular way um we can make sure that no matter what the values of the parameters and the weights it will always satisfy some conservation property or some equivarian property that's a very beautiful and very elegant uh again subfield um the equations so we can always write an equation as something equals zero so the something squared can be a term in the loss function we ought to drive that term to zero then the equation is satisfied and there are lots and lots of other ideas that people have explored.

12:10 They work. They can be quite good. But there's a little catch. And the catch is something called uh this become known as the bitter lesson. Um this was a blog written yes 2019 by Rich Sutton. So Rich is one of the pioneers of uh reinforcement learning. Um if you haven't read this, every single one of you by the end of today should take five minutes and go and read this blog. And if you work in the field, I suggest that on the first day of every month, you go and reread this blog. Could save you a lot of pain.

12:43 What it essentially says is that over the years, many people have improved the performance of machine learning systems by baking in additional prior knowledge. But they've always lost out to somebody else who simply had more data. Because the speed of computers has been growing exponentially, we've been able to train bigger models with more data. And so provided you have provided data is available, an important caveat, you can just use more compute to scale the model. And the more data eventually overtakes the clever prior knowledge.

13:19 So here's the dilemma. We have these beautiful equations. We're not going to just ignore them. Um and yet we know that data eventually um wins out. So how can we reconcile these? So the way we reconcile them is this concept of a deep learning emulator. And you can think of this as a sort of shift of mindset. So there's shredding's equation and we would think of shredding's equation as an equation that we could solve let's say to predict the property of a molecule. We're trying to design a new drug. We want molecules to have certain properties. In principle we can solve shredding's equation but it's very very expensive. In fact, the cost of solving Shreddinger's equation grows exponentially with the number of electrons. So, we're really forced to look at relatively small systems.

14:05 But instead of solving Shredding's equation a fresh every time we want to solve a new problem, let's do something different. Let's ahead of time solve Shreing's equation lots and lots of times for lots of different systems. Say a whole bunch of different molecules. That'll be expensive, but we kind of do that once. and that generates a data set and we'll use that as a training data set for machine learning. Then in the second phase we will train a neural network trained on that synthetic data that again will be computationally expensive nothing like the cost of training a frontier LLM because remember the data is expensive to calculate the data sets are not huge so the models are not huge so this this second stage um is not too expensive but it but it still costs something and once we've trained that neural network we'll call that a deep learning emulator and we can think of that as an emulator ator of the software that would solve the equation directly. So think about solving the equation directly we'll call that a simulator.

15:07 It produces data and the data is used to train the emulator. And if we get this right, once the emulator is trained, it should produce the same answer that the simulator would produce, but it will be very very much faster because solving those differential equations is an iterative numerical process that is computationally very intensive. Whereas once the network is trained, it's a feed forward network and it can just it's just a function and we can calculate that extremely quickly. Now, if we're going to do this once off, it wouldn't make sense. But if we're going to use that emulator thousands or millions of times, we're going to amatize the cost of all the data generation and the training um over those over those future uses.

15:53 So what are the benefits of using emulators? So remember we're training on synthetic data generated by simulation. And so by simulation I mean you know conventional software the kind that's been around for many decades that iteratively and directly solves the differential equations slow and expensive but we use that synthetic data to train the emulator. Now that data is very richly labeled you know it's very easy to collect images but if you want to use those images for training a neural net you may have to label them and labeling them is the expensive part. cutting somebody going through and saying this is a bicycle, this is a tree and so on. The data here is automatically labeled. Whatever quantity we want to predict once we've solved equation and got the wave function, we can just calculate the properties, let's say, of the molecule in in in that case. So the data is perfectly labeled. We don't have to worry about personally identifiable information. You know, if you deal with a medical domain and you're dealing with um patients medical records, you obviously have to be extremely careful about privacy and consent and so on.

16:56 Here we're just dealing with the solution of differential equations. And this is a very important point. The quantity of data is is unlimited in a practical sense. We can always generate more data as long as we've got the compute to do so. So people who training frontier LLMs complain that they run out of data. Well, we'll never run out of data here because we can always generate more data. And the scale and complexity of the problems we're trying to solve is such that more data will always be useful. It will always improve the results and the predictions we can obtain because we're attacking such a vast problem. So from a practical point of view, we can always generate more data just by using compute.

17:39 And the other thing to appreciate is the compute that I'm talking about here in data generation is not the extremely expensive huge clusters of highly interconnected top-end GPUs used to train frontier models. This could be single GPUs. It could even be CPUs. We can also exploit underutilized compute. In a company like Microsoft, we have huge compute resources all around the globe. Some of those compute resources get more intensively used in a region during the daytime than at the night time. There's always some underutilized compute. And because the process of generating these training points is uh massively parallel. Um we can launch many many jobs. So maybe some of them get preempted because a customer come comes along and obviously they get priority for the compute. But we can very efficiently soak up a lot of underutilized or unutilized compute in the in the training process.

18:33 And then most important of all, really the fundamental reason we're doing this is that trained emulator is much faster than the simulator. And typically, I would say more than three orders of magnitude. Usually, the first thing you try, you're about a few thousand times faster than the original simulator. Of course, it depends a lot on the on the actual application domain, but that's a good rule of thumb. You're looking at factors of several thousand acceleration and sometimes much more. And that that's huge. I mean the factor of a thousand is is quite transformational. Now I'll show you a concrete example uh a little bit later. In fact I'll show I'll show it to you now. In fact um so let me make this concrete by giving you a sort of an everyday example which is weather forecasting. So if you go to your phone and you look up the weather forecast, it'll tell you what the weather in Boston is going to be for the next few days. Where does that come from?

19:25 That forecast comes from something called numerical weather prediction. So numerical weather prediction involves solving some relatively simple looking differential equations a bit like Shredding's equation. They're partial differential equations over a number of variables. They're not the fundamental equations of physics but they're easily derived from the fundamental equations of physics and they describe the way um air flows and the effects of corololis force and pressure gradients and condensation in the air and so on.

19:57 We need a supercomput because we have to solve this with quite fine time steps in order to get good accuracy. We need quite a a rich spatial grid in in three dimensions. Um and we have to solve it quickly because we have to solve it faster than the actual weather. We to get a forecast you have to overtake the reality. So you have to solve it quite a bit faster than the real weather is evolving um in order to get a forecast.

20:20 So national centers around the world um you know the UK Met Office or Nur in the United States will have supercomputers and they'll run these launch these big jobs every six hours or or whatever um initialized with the current state of the weather and then trying frantically to predict accurately what the weather's going to do a few days ahead. Now this is a great target for an emulator because for several decades now many organizations around the world have been using supercomputers to simulate the Earth's atmosphere. So there's a huge amount of simulated data available uh much much of which is in the public domain and readily available and can be used to train emulators and quite a number of teams have trained emulators from various different uh groups in industry and academia and it's been very successful. Um I'll focus a little bit on on what we've done. We've done um something a little bit different. We've taken it a step further and rather than just produce an emulator of a particular simulator, we've trained an emulator on very diverse data. So data from many different simulators, different length scales, different different resolutions, different time scales and so on in order to force the model to learn uh more fundamentally about the dynamics of the earth's system, the atmosphere and the oceans.

21:33 Now we call this a foundation model because what we can then do is fine-tune it for particular applications. Um and and there are quite a few of these, but I'll just mention one of them. So one problem is to predict the flow of pollutants particularly nitrogen oxides uh through the atmosphere. Now data uh sensors for pollution are relatively sparse compared to weather sensors. I mean this building probably has a little weather station on the roof. There are weather sensors um all over the place. So data for weather is extremely dense and very very rich. Data for pollution um is is much sparer. If we just use the pollution data to predict pollution flows, we don't get particularly good results. But instead, if we use an earth system model, a foundation model, trade all that weather forecasting data as a foundation model and then fine-tune it, we get much more accurate prediction. So, this is the state-of-the-art for predicting uh pollution flow through the atmosphere.

22:26 So, that's one example, one practical example of um an emulator. And because it's so much faster than running simulators, it's several thousand times faster. So, for example, we can produce the UK 8day weather forecast in 1 minute on a single A100 GPU. It's very disruptive. It's not just, oh, it's faster. That's good. Um, it's it's actually very disruptive. It changes the way we think about weather forecasting. So, it's it's it's very exciting.

22:57 But it's not just weather. This is very very broadly applicable. For the rest of this talk, I'm going to focus on the molecular scale, predicting the properties of molecules and materials. That's the focus of of my team. This is still a vast space. There's still a lot to do here. Um many many different uh um applications. You can see some of them listed here. But but let's just dive into this and see how we apply this in the molecular scale. So let me just give you another example here. This is a model we built called Matagen. Um I think you'll all have come across diffusion models for generating images where you um take images and you add noise and then you train a neural net to dn noiseise the image successively. This is the same idea but now for crystals.

23:41 So crystals specified by saying um what is the the unit cell right is a periodic system. So you specify the coordinates the the axes of the unit cell the types of atoms and how many there are and where they're located. Um and so we can take known stable crystal structures, add noise, um and then learn to den noiseise them. So this is the a little video of the denoising process. So starting from random and then it's dinoising to choose the types of atoms, the uh geometry of the unit cell and the locations of the atoms within the unit cell.

24:16 Um and when it's finished, what we should have here is a stable configuration of atoms, a stable material. So if you just put atoms down at random and sort of let go, they'll all rearrange themselves. They wouldn't like it. Um what what we're trying to do with Mataggen is to generate new stable crystal structures. Now we don't just want to generate crystals at random. Of course, what we want to do is generate crystals that have desired interesting properties. So Matagen is a conditional diffusion model. So it can take as input properties of materials, electronic properties, magnetic properties, mechanical properties and so on. And so what you see here are examples of different crystals generated by mataggen having different uh desired um physical characteristics.

25:02 So if you think about the molecular pro the problem of designing new materials and new molecules as a search problem the space that we're searching is vast. If we just consider small organic molecules, the number this this space has been quite well characterized, the number of small organic molecules with drug-like characteristics is about 10 to the^ of 60. It's like the number of atoms in the solar system. It's a colossal space. So finding a new drug is searching that vast space to find a molecule with a long laundry list of desired properties. It's a huge search problem. materials probably an even bigger space. When you're into proteins, it's a, you know, colossal combinatoric space. The space that we're trying to search in finding new molecules and materials is is truly vast. Something like Mataggen, it doesn't immediately in one shot produce the miracle material.

25:55 But what it does do is help you focus on a region of chemical space where you're more likely to find uh examples which have the desired properties. So if you then subsequently generate those potential materials and run a screening using an an emulator that calculates the property. We have an emulator called matter sim that does that. That emulator that screening is thousands of times faster. But you've got another huge acceleration by looking at good candidates to start with. You're not generating crystals at random and then testing to see if it's a magnet. You're saying uh please generate magnets that have relatively low rare earth metal content or whatever it might be. and then you're looking in the right region of chemical space. So that's really a further acceleration.

26:40 So I'll talk a bit more that's crystals. Let me talk about molecules. Um before I do though again let's just you know click down a level and say what is it we're really trying to do when we when we're looking for a new material or new molecule. What we're really doing is solving a fundamental tradeoff between three different things. So these things are accuracy, speed, and generality. You could have any two of those, no problem at all, but you need all three.

27:10 And they trade off against each other. And so it's finding interesting regions of that space with better trade-offs than we have before. That's sort of fundamentally what my my team does in effect. Um speed here is computation. It's cost generality because we wanted to generalize. We obviously don't want to make predictions for things we already know about. We want to generalize at least to other um parts of chemical space that are broadly similar. Um and we need accuracy that's sufficient to be of practical value. Let me say a little bit more about accuracy.

27:42 There's a particular threshold of accuracy that we strive for. It's called chemical accuracy. Um its value is about a kilo calorie per mole. And it roughly speaking chemical accuracy is the accuracy you would need to achieve such that your computational model is as good as experiment. If you can get to chemical accuracy, you're kind of in the ballpark where you could imagine doing far fewer laboratory experiments and now doing computational experiments instead. But it's a very high bar to achieve. And one way to see that is is with an analogy. Um if you were asked to weigh the captain of the boat, you could do that by weighing the boat with the captain and weighing the boat without the captain and then subtracting them.

28:23 Now, mathematically that's exact, but in practice the boat um is is much heavier than the captain. And so to get a reasonably accurate value for the weight of the captain, you would have to measure the boat plus captain with very high precision. And it's like that in chemistry. The energy changes involved in a chemical reaction are of course everything. They're crucial and yet they're a small fraction of the total energy of the system. And so when you calculate the energy of the system, you need very high precision. That's chemical accuracy. So chemical activity is a very high bar. Um that's that makes this a very tough problem.

29:01 So I'm going to take a moment now to um and I'm told this is a fairly intense course. So we're going to be fairly intense for the next few minutes and I'm going to take you on a little journey of a story which I think is absolutely fascinating and it's one of the the core projects in my team. Um we're just going to look a little bit further at Shreddinger's equation, what it means to solve it and why it's hard. So Shreinger's equation is equation for this object called the wave function denoted s and it's a function that is in very high dimensional space. So so x1 to xn are the coordinates in space of the electrons the electron positions in space and s is a function jointly across that entire space. So it's um it's a very large space. Here's the caffeine molecule. It has um two dozen atoms. Has about 100 electrons.

29:48 So for the caffeine molecule, Z is a function of a 300dimensional space. Now uh for the experts in the audience, the following would be a slight oversimplification. It's not incorrect. It's just a simplification. Um but imagine we're going to try and solve that by dividing that threedimensional space up into a grid. Now if it's in one dimension, you know, we could divide it into a thousand points. That's no problem. In two dimensions, the grid would be a th00and by a thousand. So that's a million points. three dimensions that would be a billion and so on. The number of grid points is growing exponentially with the dimensionality of the space and so it is when we solve Shreing's equation the cost grows exponentially with the number of electrons and that means that this tiny molecule which is a lot smaller than many drug molecules is already out of reach. Nobody's been able to calculate directly the properties of the caffeine molecule by direct solution of shreddingers equation and we want to solve much much bigger systems than this. So that's the challenge that really Durac was referring to nearly 100 years ago.

30:48 But something really interesting happened in 1965. Walter Con discovered that instead of calculating the wave function, we actually don't need the wave function at all. Instead, we can calculate a much simpler quantity which is just the electron density. The density of electron density of electronic charge over space. So it's a function just of three dimensions. And whereas it's exponentially costly to compute the wave function and the wave function tells us everything we need to know. Um instead we can compute the density uh which only has cubic cost in the number of electrons and all the things we want to know. We can calculate them from the electron density without ever needing the wave function.

31:35 Now, we're all familiar in computer science with problems that are exponentially or combinatorically hard and then we have some approximation scheme that allows us to solve it in in realistic time and often those work very well in practice um even though potentially they're approximate. What's extraordinary here is that this is exact. This is an exact transformation of an exponential problem into a cubic problem. And that's so extraordinary that in 1998 Walter Comb was given the Nobel Prize in chemistry for the discovery of density functional theory.

32:05 And density functional theory has become one of the most highly cited um topics in the whole of science. Here you can see the exponential growth in the number of papers as digital computers became available and it became more and more practical to use density functional theory. You can see this has become hugely important in many different fields. Roughly speaking, this is the only practical method we have of solving Shreddinger's equation for for realistic systems. And if you look at um national supercomputers, this is from a national supercomputer center in the US. And you look at their scientific simulation workloads, you find quite often that density functional theory accounts for maybe a third of the entire workload of that supercomput when it comes to scientific simulation. So it's hugely um hugely important in practice and and quite an extraordinary result.

32:56 Um and again if you like a little little cartoon analogy to help you remember this after the talk here's one way to think about density functional theory. So it's an analogy um in three-dimensional space we have this plastic molecular model. So think of that as like the wave function. It's something very complicated. It's living this three-dimensional space. Normally we would calculate the wave function then calculate everything from the wave function. But instead, we don't need to work with the wave function. We can work with this lower dimensional projection, like a kind of shadow in this two-dimensional space. And that's what density function theory is. And by the way, this isn't AI generated. Somebody had actually plugged these bits of plastic together and took a photograph.

33:31 So, congratulations to them. Now, at this point, you're probably, you know, you're probably a bit suspicious, right? I mean, we've got this exponentially hard problem. Somebody comes along and now I can solve it all in cubic time. Uh, and it's exact. It sort of sounds a bit too good to be true and in a way it is a little too good to be true. There is a catch. So Con wrote down a set of equations um the con sham equations which are iterative equations. Instead of solving Shredding's equation you solve the cone sharm equations. Um they have cubic cost cubic in the number of electrons but they contain a term it's a very small term called the exchange correlation functional. It's a contribution to the energy but it's critical because as as you as we saw we have to have very high precision. Although that's a relatively small term, it's essential that we get this uh exchange correlation functional term um to be accurate. And con showed that it's universal. In other words, there is a single exchange correlation functional that describes the whole of molecular space. All the crystals, proteins, molecules, everything, one universal exchange correlation functional. He wasn't however able to write down an explicit analytical form.

34:43 So it's sort of a mystery. It's it's an unknown. So what do people do in practice? Well, over the last 60 years, people have invented handcrafted um approximations to the exchange correlation functional. Today there are about 800 of these um in use. It's sometimes called the zoo. Um sometimes called Jacob's ladder because they can be arranged in a kind of sequence. Some some cheap ones that are not very accurate or some very expensive ones that are quite accurate. And so there's a trade-off there. Um but these functionals, these uh handcrafted functionals, uh each of them applies to some particular subdomain. Some are applied to molecules, some to crystals, some are good for metals. Um and and and so on. Um what's missing is this this universal exchange correlation functional. And there's this nice paper written in 2002 in science called in pursuit of the divine functional which really sort of lamenting the fact that the whole field had had gone down this rabbit hole of these handcrafted approximate functionals and lost sight of the fact that exchange that uh that density functional theory is an exact theory and there is this universal functional. We should get back to the heart of this of trying to find that universal functional.

35:57 So the DFT zoo then is this gives us this trade-off. This is a graph on the vertical axis is um the error um in kilo calories per mole. So we want to get down into that blue region that chemical accuracy or better. That that's what we're aiming for. And here are a bunch of different um handcrafted functionals. You can see on the left we have some that have cubic scaling. So they sit within the overall cubic framework of density functional theory. That's wonderful. They're very affordable, but the accuracy is terrible. Much much worse than chemical accuracy. Um, we we then have uh functionals that are much more expensive like n the^ 5. They're starting to approach chemical accuracy, but they're getting very very expensive.

36:40 And yet we know from con and his Nobel Prize that there's a single universal functional that applies across the whole of chemical space that should give us chemical accuracy or better um with cubic scaling. So that leads to the idea of could we not learn the exchange correlation functional using machine learning. kind of an obvious idea really and quite a few people have looked at this over the years and sometimes the results have been quite good in terms of accuracy but the most accurate of these are very slow and so nobody has been able to produce an exchange correlation a machine learn exchange correlation functional that is both accurate and fast so we've had a project in the AFS science team that's been going since the beginning so more than three years now we've we've been doing it kind of under the radar but we sort of went public a few months ago. Um we've been doing um a couple of things that are different. So let me just explain. Um so we've produced a machine learned exchange correlation functional called Scala. So Scala is from the Italian for for ladder. Um and the idea is we've got electron density.

37:51 So within the cone sham equations at any given time we're updating electron density. We need to know its energy. um there's a bunch of these big terms. We can easily calculate those, but that's not accurate enough. We need this final term, the exchange correlation term. And that's where machine learning comes in. And so, um essentially the electron density is represented by a point cloud. Maybe have a million points. And one of the challenges here is that the dependence of the energy is non-local on that uh on that electron density. and uh even pair-wise interactions on a million times a million would be out of hand. So what we do is we have um a lower dimensional representation centered on the atoms and we use message passing from the point cloud to the atom centers then between the atom centers then back out to the to the electron point cloud and that gives us a scalable model that's also able to learn uh the exchange correlation functional accurately. So that's one innovation.

38:47 The other is just uh you guessed it more data. So we spent a lot of time over the last three years solving Shreddinger's equation using standard quantum chemistry methods to produce highly accurate training data for the exchange correlation functional. In fact an order of magnitude and the totality of all the public data um previously available. Um and so here's just a glimpse at some of the results. So this is a graph. The vertical axis is just the the total energy of the of the molecule and the horizontal axis is another standard benchmark looking at more general uh chemistry properties. Um some of which go beyond what we actually use in in training. And you can see that uh Scala achieves chemical accuracy on the fundamental question of um uh atomization energy and is comparable with um other functionals having cortic scaling for uh for more general chemistry. The the limitation here of Scala as exists today is that it applies only to molecules not yet to materials and it deals only with the so-called main group elements. So we don't yet cover transition metals, but it's very interesting that even even in its current form, it generalizes quite well.

39:58 Um perhaps to do with the fact that we're learning a universal functional. So perhaps that generalization is is less surprising. But we're just tremendously excited about this. We're really hitting hard on a a 50-year-old grand challenge at the at the heart of physics. So I'm going to um finish by just asking a question. Well, how far can we push this? We said direct solution of Shreddinger's equation for the caffeine molecule that's out of reach. Biology looks crazy big and complicated. Can we push this to biology? So let's just see how far we can go. So here's the whole of biology in one slide in case you were wondering how biology works. So we start with sequence. DNA sequence leads to uh an RNA sequence and then that is um uh then uh specifies the amino acid sequence uh which is the protein. That protein then folds up into a three-dimensional structure and then that structure uh along with the dynamics of the proteins determines function. So sequence determines structure determines function.

40:58 Um sequence um DNA sequence and protein sequence those were actually two Nobel prizes both awarded to Fred Sanger in Cambridge at the the other Cambridge. Sorry about that. Um and and of course yeah before last now I guess we had uh Nobel prizes in um in protein structure the threedimensional structure what we really care about is the function and the function of proteins is not about static structure it's about dynamics proteins open and close they dock with other proteins they interact with molecules and so on. So the last big piece of this puzzle is to do with the dynamics of how proteins change shape and how they interact.

41:40 So we can we can again in principle we know all the equations and so we can just compute and and this field is called molecular dynamics. So here's a molecule um each uh each of those balls of course represents one of the atoms the the positions of the nuclei. Um so we can represent each each atom in terms of its position and its velocity at each instant of time. And all we have to do is solve Newton's second law of motion.

42:04 So we can treat them classically and we just update the positions of each atom uh step by step through time just following Newton's second law of motion. Um the challenge here is that the force term which is the gradient of the energy that energy comes from quantum physics. So that's determined by the electrons. And so there are two massive challenges here. One is that at each step of that integration in principle we should go back and solve shredding's equation which we've already seen is extremely hard. There's another challenge here as well. It's called the curse of sequentiality. So this shows you on a kind of log scale the different time scales involved. Right on the on the left hand side there we have the the fundamental vibrations and rotations of the atoms. That's happening on a phentoc time scale. So if we're to integrate Newton's laws for the atoms, we have to follow those vibrations. That means we have to have a time step which is of the order of a phentoc.

43:01 But the interesting stuff happens on much larger time scales. Things like protein folding happening on time scales of milliseconds for example. So there's our integration step. And the problem is we need something between 10^ the 6 or even 10^ the 15 sequential steps to follow that dynamics. Remember at each step we should compute the forces using uh quantum physics. Um and you may think well that's okay. Computers are getting exponentially faster. We just have to wait a few weeks. But the problem is this is intrinsically sequential and computers haven't got faster for many years now.

43:33 If anything, they're getting slightly slower. What they are doing is becoming massively parallel. So the speed up we see in computers through GPUs is through massive parallelism. The clock speeds haven't really changed for quite a while. So this is intrinsically sequential. So it's sort of a double whammy. It's a really really hard problem. Um so can we use emulation to to get a handle on this? And this is something from um a couple of years ago now where what we do is um we we make several carefully controlled approximations. The first is to take we take a protein very large molecule and we look at individual chunks of that protein. So we don't model individual atoms. We group them together into chunks and then we compute the forces between the chunks. We use density functional theory for that. So the bigger the chunks, the fewer the things we have to integrate. that as the chunks get bigger, so the force the relationship of the forces become more complicated and there's a trade-off. So the chunks are as big as we can make them while still producing an accurate machine learned emulator. So we we compute forces for lots of different configurations using DFT, train an emulator and that emulator is now very fast. So now we can run molecular dynamics on these chunks of protein but at each stage the update is just a call to a trained emulator. So extremely fast, no iteration needed. So here we can do um molecular dynamic simulation for proteins up to about 10,000 atoms. So that's an extraordinary advance. Look back to that caffeine molecule and where we were before and here we are doing molecular dynamics at uh the scale of a 10,000 atom protein. So again you can see that sort of hierarchical process of using data from one level to train the emulator at the next level. And we can take that even a step further and use data from molecular dynamic simulations combined with experimental data to avoid having to do the dynamics alto together and actually directly sample from the equilibrium distribution. So this is work that was published in science um a few months ago and we're super happy that the editors chose to put it on the front cover which is which is always nice.

45:38 So yes, we can by using the power of emulators, we can extend this even to the scale of uh of biological molecules. Um I'm going to finish, I have to admit, with just a shameless plug for a book. This is a course on deep learning. So I'd like you all to at least uh be aware of this book. It's called Deep Learning. Um published at the end of uh 2003. Um it's free uh freely available online. If you go to bishopbook.com um you can get a see a get a high resolution um uh digital version of the book. Um but also if like me you quite like old-fashioned physical books um then you might be interested to know that this is actually produced using a traditional binding technique called stitch signatures. So most books today are produced by gluing the pages in on edge. They don't fold flat and if you open them more than two or three times the pages fall out. This is good oldfashioned stitch signatures.

46:29 You print on a big sheet of paper and fold it, fold it, fold it, trim it and then stitched in with cord. So the book lays flat and it should should last for many many years. Um, also delighted I just heard that for the second year running this is Spring of Nature's bestselling book. So Spring of Nature and all their subsidiary publishing houses publish together something like 13,000 new titles every year. And for two years running since this was published, it's the overall number one bestselling book. So really, really delighted by that. Uh but actually the thing that I'm most proud of about this book is it was written with my son Hugh who's also in the machine learning field. So thank you all for your attention. I think we have a little bit of time. I'm happy to take questions.

47:09 Thank you. >> Thank you. So >> thank you so much >> Chris for a fantastic and inspiring talk. So we're going to take questions. We're going to run the mic so that we ensure that we capture that. So please if you have a question raise your hand and we'll go from there.

47:40 >> So previously you said that uh biology is mostly sequence to structure to function and you talked a lot about the work you've been doing with protein structure and molecule structure in general. I was wondering uh what kind of work have you been doing in molecule and protein function especially? >> So the main project that we work on is called the biomolelecular emulator and so the first the first major output of that was the science paper that that you you saw the front cover of and the focus there is really addressing that that third part. So we view sequence you know we have this high throughput sequencing that allows us to you know sequence your entire DNA for $1,000 or whatever it is.

48:18 incredible technology. Uh structure is sort of large static structure is sort of largely solved. What's unsolved is this is the dynamics of proteins. So our focus is on the protein dynamics. And we're really doing something quite ambitious because we're trying to solve what's sometimes called the sampling problem. It's not specific to proteins. It's it's just very generic of how you sample from these complex distributions efficiently. And the goal is to so one way you can always do this is just to run molecular dynamics forever. And you know in the infinite time limit it will sample the different think of an energy well with you know an energy surface with different wells. Um you will sample them all in the correct proportion and that's essentially what you want to know. Um that's that's an infinitely expensive route. We want to be able to avoid the dynamics altogether and just know what are the what are the sort of the energy wells and what are the relative proportions of the time the protein will spend in those in those wells. So what those those wells correspond to is say a protein which um I don't know maybe the spike protein on the on the um um uh co 19 virus has an open confirmation a closed confirmation but if you if you look at down a microscope all the atoms are wiggling and jiggling and it's kind of wiggling and jiggling in this sort of closed thing and then bing springs open and it'll wiggle and jiggle for a long time in this open configuration. When it springs from open to close it's crossed one of those energy barriers. If those barriers are high that's a very rare event. So in molecular dynamics you simulate for a very very very very long time and then you'd see one of these rare events. That's kind of crazy right?

49:47 And as you get as you get high energy barriers it becomes extremely inefficient. So we want to find ways to just you know given the protein just tell me just sample directly from that equilibrium distribution. And what we saw in that in that first paper is the sort of the first light dawn really that yes this looks like it could be possible. What we've done so far is one protein in vacuum. What we really need to do now is proteins interacting with each other, proteins in the presence of water. Then it becomes, you know, we get closer to clinical and biological relevance then. But very exciting that we can do one protein in isolation. So yeah.

50:22 >> Uh thanks for a great talk. Um I was wondering for emulators um we still have to know free lunch theorem and um how can you expect or not even guarantee but expect that these emulators will apply um I guess for problems like solving like a 10 electron shorting equation. How can an emulator that's only seen like a two or three electron systems be expected to solve these more complex like completely intractable problems that are you can't solve using a single even a single time and then I guess the second question is um for uh systems like uh weather which was one of your examples where I imagine data is already um you already have a ton of data why would you want to use emulators as opposed to just training it on the data?

51:13 Right. So I think there are several there was at least three questions in there. So let me make several comments I hope will address the question. The first one about the no free lunch theorem. Um you know you could have a a model that that encodes as I mentioned let's say rotational equivariance. You bake it into the structure of the model. Much simpler thing to do is just replicate the data. So if you know this molecule has a certain energy um it's trivial to just rotate the molecule in the computer. We just it's just simple 3D geometry. And so let's have a hundred copies of that molecule in different orientations all labeled with the same energy. So 100 times as much data that's called data augmentation really really simple to do and eventually ultimately that and that means you can use much more general architectures and you learn the invariance. The downside you don't learn it exactly if you bake it in it's exactly true but really what the bitter lesson is telling is at the end of the day just more data and just learn these things. Um I think the second question you're asking there is about it was a great question actually very insightful question. So let me just explain how I interpret that question then. So we spent a ton of time and energy and effort over three years generating high accuracy quantum chemistry data in order to learn the exchange correlation functional in order that we can then run DFT because DFT is still an iterative process still has cubic scaling. It's not a a single pass. The functional is a is a is a fast emulator is embedded in this and and then we might run DFT to generate training data to learn a thing that we call a force field that would predict the properties of molecules very fast. Why do we bother? Why don't we just take that expensive quantum chemistry data, forget DFT, just train a fast emulator? That emulator would be certainly be very fast. What we miss out is effectively a way in which we've incorporated a lot of inductive bias.

52:47 That machinery of density functional theory has a lot of physics baked into it. In particular, I glossed over all of this and I did apologize to the experts earlier, but that wave function for instance has to be anti-ymmetric. Um that that that's why atoms exist. It's Pal's exclusion principle. The the electrons can't occupy the same state. they have to fill up different different shells. That's really critical physics to understanding the atom. In principle, if we could if we could generate vast amounts of quantum chemistry data with high accuracy very cheaply, yeah, maybe we could learn that stuff in the in the in a sort of data augmentation sense.

53:21 But but in reality, the the power of DFT is it encodes so much physics. You saw, I think, in the diagram there, there's a lot of physics that we can calculate fairly straightforwardly. It's the exchange correlation term that's that's hard. But but but but you but to throw away all of that physics is to give up an enormous amount. So so that's why we want to use DFT in that hierarchy. Um and then I think you know part of it too you're you're asking about generalization partly is it embeds a lot of physics but also and and we're working out good ways to test this but there's this beautiful fact that the exchange correlation function is universal. So in that in that zoo, somebody invents one that's good at crystals, somebody invents one that's good at, you know, molecules or whatever. Um, but here because it's because we're learning a universal functional, we hope that should help with generalization and we do see remarkable generalization to chemistry problems outside the domain in which it was tested and we hypothesized that in part that may be due to the universality of the exchange coination functional. So I hope that in a roundabout way addressed your questions but very insightful question. Thank you.

54:27 >> Amazing. We'll take two more questions. >> So I have a question related to the no free lunch theorem. It seems to me that here you have no free lunch and no free dinner in a way because you I mean typically in the no free lunch is kind of a bias variance tradeoff with the capacity of the model or the models that you're considering but here you're simulating data. So you're essentially adding kind of a second distribution over which you are uh having a no free algorithm right and a second bias variance trade-off. So I'm curious about how that plays out in your like fundamental trilmma if like simulating data is better than actually getting real world data and if at some point you measure somehow like the sim to real bound between the simulated data and the real world data essentially this happens a lot in reinforcement learning you in robotics for example.

55:33 >> Yeah. So I I um I mean two comments I guess one is that the experimental data will always be fundamental at the end of the day that we're not changing the scientific method. It doesn't matter how brilliant our theory is or how many nature paper we get. If it doesn't agree with the experiment we're wrong we have to re rethink. So there's one role for experiment. But also um what I've talked about today is what I think is a is a is a very substantial advance by many orders of magnitude in our ability to do computational science. But that doesn't mean we're going to just throw away experiments and and and so on. So uh so the second role of experimental data is is is um very much to complement um the computational data. In fact in the case of the um uh protein dynamics that I showed you the science paper that's actually trained on a lot of molecular dynamics data but then actually fine-tuned using experimental data for X-ray defraction measurements. So there's clearly a role there. So in terms of the you know the no free dinner theorem. I like the that's a good good good terminology there. Um but you know it it's still a fundamental cornerstone of machine learning that we have to combine um prior knowledge with um with data and this is a very interesting way I think of extracting that prior knowledge in the form of data. So it really I think takes on board the the idea of the bidder lesson and the fact that uh scaling the data is always a very powerful thing to do but by generating the data from the equations then then we're effectively incorporating that prior knowledge. So I'm not sure if that really addressed your your question there but um uh >> amazing final question.

57:05 Hi. Um so about u cellular and genetic therapeutics um understanding of uh protein protein interaction between um therapeutic trans genes and host cellular machinery. It is extremely crucial to have an idea about this before we dive into invivo studies for any kind of therapeutics. So do you think um AIdriven biomolecular dynamics could help the predicting of offtarget interactions or any kind of stability issues before one moves towards the invivo studies of that particular therapeutic?

57:45 >> Yeah, that's such a great question and and of course the answer is I certainly hope so. I think we've been surprised at how far I mean we're really starting from Shredding's equation and the idea that we push this all the way to you know the dynamics of a single protein seems remarkable what you're talking about with you know invivo studies and the whole machinery of the cell this is orders and orders of magnitude more complex still so I don't imagine that in my lifetime or you know the next hundred years I don't know that we're going to have a a first principle simulation of an entire human cell that's derived entirely from Shreddinger's equation for example that that seems crazily out of reach But I do think we're approaching the point where things like offtarget interactions of drugs could be informed through um molecular dynamic simulations accelerated through the through the use of emulators. And that would be very exciting because of course in in drug discovery a lot of drugs fail at the clinical stage through toxicity through through offtarget interactions. The more we can do at the discovery phase to eliminate those um the faster drug discovery can be and the cheaper drug discovery can be. So I am hopeful that these techniques will have real clinical relevance within the next uh two or three years or so. But but but we'll we'll we will not for a long time I think be simulating anything as complex as a human cell using using these techniques. But nevertheless I think they can make a very practical contribution.

59:05 >> Amazing. One more big round of applause for Chris Bishop.