When AI Improves Itself | Richard Socher (Recursive)

0:00 Anything you can simulate, AI will solve. But before you know it, you're in this recursive self-improvement loop. And we believe that that will be a great unlock. Boy, are we far away from the true upper bounds of any of the spaces of intelligence and there's still so much further that AI can go. Hi, I'm Matt Tur. Welcome back to the Matt podcast. My guest today is Richard Sosher, one of the most cited researchers in AI and now at the center of the term everyone in AI is suddenly talking about, recursive self-improvement, AI that makes better AI. Richard just raised $650 million for recursive, a company built to do exactly that. And his new book, The Eureka Machine, is a fascinating blueprint for how AI and RSI are about to revolutionize science. Please enjoy my conversation with the always excellent Richard Social.

0:52 >> Hey Richard, welcome back. >> Great to be back. Thanks for having me. >> All right, so lots to catch up on. We're going to talk about recursive intelligence. We're going to talk about recursive the company but first and foremost and most importantly perhaps we're going to talk about your new book entitled the Eura machine which I read with great interest and would strongly recommend coming out in a couple weeks I believe. The book opens with a premise that I think a lot of people find surprising and shocking which is this claim that scientific progress has slowed down which feels counterintuitive given the number of researchers we have around the world and the sheer amount of money that goes into the space. So why is that? Yeah, it's a somewhat surprising fact and you may argue clearly not like we're making so much little progress on so many different things, but when you think about, you know, how much progress have we made on antibiotics? Like bacterial infections went from like a death sentence and the plague to like a nuisance. Like we now have antibiotics for almost all the different bacteria and we truly solved that. And like we have clearly not solved viruses or cancer the same way we solved bacterial infections. when you think about foundational novel things like E= MC^² and general relativity, we clearly have not made progress on many theories and physics either. when it comes to such foundational things that then literally led to nuclear energy and fusion and fision and other kinds of research that could be conceptually done. And so a lot of the fields have kind of gone through from we understood some foundational pieces to we can now do a lot of engineering but they've also split up into thousands of different sub fields.

2:41 It is almost impossible nowadays to be the sort of general genius that can dabble in all of these different fields because each field takes years and years and years to get really deep into. And so what we found is that as there are more and more sub fields and niches, it's actually hard to have enough people in each of these sub fields. And Stanisav and others have talked about that and predicted that that will be a big part of why we're slowing down.

3:08 >> Yeah, you have a a great expression. You talk about how we evolve from a body of knowledge to a labyrinth of knowledge. 34,000 journals that might as well have no trespassing signs. >> That's right. Yeah. It's so hard to like even understand all the lingo. and I've I'm sort of gone through this myself. First when I started studying linguistics and then computer science, but now that I'm sort of trying to study and have studied now over the last few years sort of on the side biology, it's like man, every time you have a conversation with biologists like 10 sentences in, they're just telling you so many abbreviations and terms that you're not familiar with that most people after a while just space out. And so it's really hard to describe and explain very complex deep fields to someone who hasn't been in >> you also mentioned that there is some level like kind of like social human element to this where in academia you're not necessarily encouraged to take risks and that goes to your your own experience as well. Yeah, 100%. Like people often, you know, the way careers work is you want to be kind of novel, but if you're too novel, if you're too far out there, then your papers will get rejected. And that certainly happened to me a lot in the early days like 2010 of neural networks for natural language processing where the majority of my first couple papers got rejected from NLP conferences when they got accepted in like some small subniches and subgroups. I still remember the first sort of deep learning workshops workshop at NIPS back in the day now that was basically like 30 40 people all the now super famous folks but it's just like a couple of us renegades who thought that this would clearly be the right way of going about it. and we came from different directions like feature engineering seemed like not the right path to doing things. Feature learning was a big part of what got us started. some were neuroscience inspired and it was again that sort of combination of different fields that was you know at that intersection where it's interesting but also often hard to publish well in the beginning.

5:15 >> Okay, great. So to play it back is fundamentally that idea that there is too much knowledge everywhere even people within the field encouraged to go super far and and and to come up with crazy ideas and the challenge is to bring everything back together. So the the Renaissance men so to speak had only a small body of knowledge and therefore were able to come up with cross-disciplinary insights but this is no longer possible. Right. >> That's exactly right. Even like a good example again in biology is that you study not all of biology anymore. You study either biology at the cell level or the tissue sort of medical level or the biochemistry level or the protein level. And if you ask a PhD in biology about like a deep question in one of the other layers, they often some like don't know either.

6:00 >> Great. All right. So the premise of the the whole book is that AI is about to usher or in the process of ushering a whole new paradigm of scientific progress. So just walk us through the the the high level idea. I love the idea is that when you know a like science has gotten really good at understanding smaller and smaller pieces but to bring them back up and bring them together we actually have to use AI and AI is kind of what calculus did for physics AI will do for biology in the sense that it'll help us weave back together lots of very complex pieces that build these complex systems that then have certain properties. So concretely for instance your microbiome or the brain they have so many different pieces and we actually understand many of the individual like neurons okay this neuron is at these synapses and this is how it fires and so on and this is the chemistry which often people still ignore in a lot of neural models and so on but then as they get as we put more and more of them together we stop understanding why the whole brain can now have a thought or you know do certain things and I think AI is the perfect language the perfect way of thinking about some of these complex systems and has the ability to also exhibit sometimes patterns that are almost hard for us to then understand. But of course it's always easier to understand and study a neural network than it is to study the original biological system. And so I think when you put all these together and I'll talk about sort of the details of the Eureka machine and its four columns and so on and you look at the data you look at sort of different fields everything from physics chemistry biology neuroscience medicine economics astrophysics and so on. you look at all of these levels and you see so many small improvements in all of them that can help you extrapolate that AI will usher in this new age. There's still a a good number of people out there that think of generative AI as a next token predictor, as a chatbot, as something for language. Here the claim is very different. It's predicting protein structures. Why is the technology that's good at being the best chatbot in the world also good at scientific discovery?

8:21 >> I think it's it's quite counterintuitive. So, I'm glad you asked because if you ask a biologist, if we'll have a model that can predict something as complex as aging or like general like different cancers and so on, they'll say no, this is like decades out. And in a similar fashion, 10 20 years ago, natural language processing researchers would have told you that it is impossible to build one neural network that could answer any and all kinds of questions. In fact, you can find this online on open review my paper where I described prompt engineering like one neural network that you can just prompt with any kind of questions called DECA NLP. The paper that paper was wildly wildly rejected by the like basically all the reviewers and the the area chair and so on as just like completely useless like too crowded like made no sense. not even humans have one system to answer all these different kinds of questions. And so it was like so like something that's so obvious now that people like you can't even invent prompt engineering because it's so obvious. It was like such a non-obvious thing to the field. And so we we've seen that time and time again throughout different different fields. But now I think when once you showed that we didn't have to actually truly understand and have perfect rules for every single aspect of translation or question answering we will see similar things in and have seen already in for instance the language of proteins. so you have just sequences of amino acids. No human has been sort of evolutionarily trained and has learned to speak the language of proteins. But just like the language of natural language of English and so on, AI doesn't really care if it's English or a sequence of amino acids. And so you can now generate completely new kinds of proteins like you can generate new kinds of sentences that have never been in that combination in the training data. And I think you know when you now apply similar ideas to chemistry and even lower levels such as molecules you'll see that that idea that you have very complex systems that with a lot of data can then make very useful predictions that explore the overall space better and better than than any human could have manually. that helps you then say okay like this is much more this next token prediction is such a beautiful yet simple idea that incorporates knowledge about almost any domain >> because there is a concept of world model and I realize that term is has a has a precise meaning that I may not be using precisely here but a concept of world model that's built into what enables the system to predict the next token. I think in the book you have a a lovely example about driving south from Dresden. Maybe unpack that.

11:19 >> Yeah. So, so next token prediction like why is it so powerful? you know imagine you could try to have a model where you understand the world's geography and where every city is. And here we're in New York so maybe I'll use a New York example. >> driving south to New Jersey >> for example. Yeah. To Princeton or somewhere or north to Boston. So like and and so you you basically just by trying to predict the next token, you will stumble upon sentences where someone somewhere wrote like, "Oh, I was in New York and I was driving north too, right? And now it might be Yale, but maybe more likely it's Boston, right?

11:53 And the more likely it is probably the bigger the city is." And so you now incorporate by trying to predict the next word in that one sentence. I was in New York driving north too. that next token being Boston. Now you predicted something about geography and locations of cities. and so by virtue of doing that billions and billions of times in fact trillions of times nowadays like basically as much data as you can you you actually incorporate knowledge about geography in that and what we're seeing very similar fashions. A lot of people are familiar with protein folding. when when proteins fold, some proteins are actually closer to each other in 3D space than they would be in a sequence of these folded proteins. And it turns out that you can analyze the neural network that was trained to do next token prediction. And you see that indeed the ones that are closer in physical space after being folded, the neural network just by doing next token prediction is also like has those correlations. and and so we basically know that large neural nets trained on very large specific domains will incorporate the knowledge of that domain just by trying to predict the next token quote unquote in a sequence and token for for the nontechnical folks means it can mean an English word can mean a subphrase of an English word just like sequence of characters that together make up a word but a token can also mean a protein it can also mean a piece of few pixels in an image or in a video or in a piece of sound. So everything can be kind of tokenized basically discretized into a set of vocabulary tokens that are then able to be predicted.

13:32 >> By the way, if I may, you know, you're talking to an extraordinary accomplished AI researcher when you sense of east coast geographies at what's south of New York is Princeton and what's north of New York is Yale. Is there a part of the analogy that that breaks down if you have this implied idea that across physics and bio and what have you this this language does that truly translate as an analogy? >> So there are clearly some places where it feels weird like it feels oversimplified and you could clearly say like all models are wrong but some are useful. and that's also true like we know proteins aren't just sequences they have a 3D structure for example but it's surprising how far this analogy is being able to be pushed like even in chemistry you know you have like various loops within molecules but somehow if there's just like a standard way of describing molecules as a sequence as a string and using that is very helpful for a variety of different aspects of chemistry too. So yes it's oversimplified yes all models are wrong but this particular set of models is quite useful for all the different sciences and then of course you can go into arguing but how truly novel can ideas be from some of these models and the truth is that like in almost all of science we stand on the shoulders of giants and just exploring the recombination the clever combination of all the different ideas that are already out there will lead to incredible progress like we have a lot of the foundational pieces figured out in small pieces like the smaller sort of subsets but to weave them back together will lead to incredible progress especially in biology. and then you can also explore how well these like these models can create novel ideas. And there we also have real examples of researchers having had an AI ideate and then a few months later people publishing this a paper with essentially the same idea. Jeff Cloon, one of our co-founders at recursive has tweeted about several such things happening where he had novel evolutionary ideas or evolutionary algorithms create ideas that later have then be published by also people. So clearly the novelty threshold has been met in several cases. Now will it derive like a completely novel way of looking at the universe? maybe be able to disprove or prove any of the many string theories that are out there and things like that.

16:06 There's still some research that has to be done. But I'd argue that, you know, we can with existing technology and giving it more data and more compute. We can already cure a lot of diseases. We can already cure many of the aspects of aging. we can already build better fusion like reactors and systems. we can already create new materials. you can already help with economic questions of how much to tax or subsidize certain populations if you have a certain objective or reward that you're looking for to achieve inside your economy. All of these things are already within our grasp.

16:44 >> I was going to visit that idea later but like since we're on it and it's such a fascinating concept and so central to the whole discussion let's double click on it. So indeed as we talked about recursive self-improvement RSI the one of the big questions has been creativity and the ability to sort of think out of the box and you know you also hear people talk about moves 37 which was the move in in in go nobody expected and that enabled the the demand model to to beat the leel. What is the mechanism by which this can or cannot happen? Do we understand that or or is precisely the fact that we do not understand it? The reason why we we don't think I can be as creative as it could be.

17:32 >> I think we can basically predict where AI will certainly have superhuman capabilities. and those are all scenarios and all domains where we can either have a simulation and or a verification tool. any kind of domain that we can simulate will basically result in a world where the eye can essentially infinitely many times experiment inside that simulation. and assuming the simulation doesn't take you know years to run every time you want something useful from it you then know that you can solve the problems in that domain. And so all the games I was never that surprised that AI will eventually fairly soon be better than us in games that are especially the games that are perfectly visible. They don't have hidden variables like you don't know other people's cards like in chess and and go you see everything and it's all on the board. Now there were too many combinations in the board to just do brute force kind of winning of of those games. But if you have enough training data, the eye can learn the intuitions also behind it by playing many many games. And in this case in particular, it's even easier because the eye can play against itself. and that idea to play against yourself is also a big part of open-endedness and recursive self-improvement that we've applied and I can talk about rainbow teaming and and security and safety research for example with LMS also but yeah just to go back simulations anything you can simulate AI will solve now what else can be simulated that's interesting and can be verified that's useful math math is like going to change massively within the next few at Tao and the most famous and most sort of frontier mathematicians already fully aware of it. The whole field will change just like the field of AI has changed and a lot of sort of skills that used to be useful where you manually feature engineer and then you manually architecture engineer and you manually do these things like are not that useful anymore. That will be true for a lot of mathematics also. And so I think that's a great sort of situation if you cared about solving as many things, proving as many theorems as possible in math. It's not a great thing if you're just love the sort of pursuit of math for intellectual sake and for fun. and we'll actually see that play out in many different ways.

20:03 You know, chess is now more popular. despite being dominated by AI if you wanted to. I think most sports, intellectual and and physical in the future, will probably benefit. I don't know if you saw the robot running really funny in the Chinese Olympics recently. My hunch is humans will try to see if they can emulate that funky run to then run faster too because AI in simulations in the like robotic simulations tried many different ways of running and found this weird new way that somehow humans despite having run all our existence haven't thought of yet right and so there will be just like the best chess players and best go players are better now because they can compete against an almost perfect AI. I think even that will be true for athletes in the future. So I think it will push the field forward and what's the most powerful thing we can simulate and verify and build verifiers for is programming. Software is eating the world famously said I think Mark Andreen and so AI is eating software. So now you can basically and one example a simple example is you can show the picture of a website and you say make it program such that it looks exactly like that right and then you can create infinitely many examples like that and then have a perfectly capable AI building frontends for websites and the truth is you can create all kinds of verifiers like that and so all of programming will change and that will be a major impact on the entire digital economy which is the knowledge economy and so on and then The question is where does it stop? Well, what things are hard to simulate right now? And that's where it becomes interesting to look at the natural sciences where we cannot yet perfectly simulate a complex cell let alone tissues or organs and and full humans and we do need to collect a lot more data in various robotic forms and those are essentially the four columns that I talk about in the Eureka machine 2. Start with human knowledge and LMS you then the second pillar are all the measurements we can take and more and more of those the that we already have and we should incorporate that into the model. The third thing is a simulation and the fourth thing fourth pillar is essentially robotic process automation to collect even more data and verify whether the inventions really made sense. And on top of those four you have an agent swarm and a community of scientists.

22:18 >> So we'll definitely unpack that part in a minute. just going back to u so there's the question of the intuition and creativity which is one of the unexplored frontier and then related to what you just said there's also the big question of generalization so starting with coding and math that seem to be increasingly conquered domains by AI you seem to be saying that then there's like the the life sciences that that could be next what is your sense sense for how far we can go in making AI truly general and what is the process to get there. Is that like brute force RL for this domain and then that domain and that domain or or is there a more sort of sweeping generalization effort that could be produced? So at recursive we are fairly sure that we have to start with AI for AI and then make it really really good at doing research on creating better AI so that it has the equivalent of 50,000 PhDs in terms of knowledge and its own capabilities and only then go after the physical natural sciences like physics, chemistry and biology especially biology I think will be most interesting. I do believe that in the next two or three years while we're focused on recursive self-improvement also we will have more and more data collection. Tahoe Therapeutics is a great example.

23:48 Parallel bio is another one. I think I mentioned both of them in the book. that basically help create much much more training data. And then when you have in case of biology for instance you can do these perturbation studies like you take a cell you try to knock out one gene and you see what happens when I knock out this one gene or I add this one molecule to it and I see what happens. So you do if if you do many many perturbation studies eventually maybe the AI will learn the underlying patterns behind it just like it learned the underlying patterns of like oh I'm in New York and I'm driving north too and then predicting Boston and might be like oh I add this molecule to this kind of cell and then I get the output of you know and then it's just like it can start to eventually generalize if but we're just nowhere near having enough training data for biology and so we need organoids we need eventually all these perturbation studies to come together so that we can then try to build a virtual cell and then in that virtual cell the eye can then go and experiment many times.

24:50 >> There is another counterintuitive idea in the book that I thought was fascinating which is that when it comes to AI based science hallucination might be a feature rather than a bug. Can you explain? >> Yeah. So a lot of folks struggle with hallucinations and models for a long time especially in the earlier versions of these models. The book you know I started thinking and writing the first sort of ideas down like three years ago and had to change a lot of chapters to someone should do to someone has done and let me like talk to them and talk about their startups and whatnot.

25:28 but I do think hallucinations can be also very helpful for AI when you want it to explore novel kinds of proteins like yes it can like every AI can memorize things every computer can easily memorize things right but where it's interesting is like how well can you hallucinate how reasonable or just outside of the distribution in some interesting way are your predictions right and we also know that we can just like with humans right you give them like a certain kind molecule and and their visual cortex goes off into a really different world like you can also increase the temperature is what we call it and sort of a term like a technical term that the the AI at the large language model will then generate tokens that are more and more different to things it has seen before and so I think hallucinations are in some cases a feature and not a bug of course when you ask a factual question online to a search engine or an LM then you want to have it like be correct. And when we know that that's the kind of question you're asking, it's easy to prime the model and say, well, here are search results. That's what you.com does, of course, to like basically take the facts from a real search engine built for agents, plug them into the prompt, and then the eye will kind of summarize that. And so I think the initially people thought, oh, we need symbolic reasoning blah blah to do all this. We just needed more examples of don't hallucinate now like take real facts from a search engine and then mostly summarize those and then those like hallucination problems were to a large degree resolved and then if you want to write a poem for your wife you don't want it to just look sound like the other poems that were out there you want to create a new one you can also do that >> and as a funny moment in the book you mentioned that actually a lot of scientific discoveries were made by scientists in semi-state of hallucination through diseases or otherwise.

27:24 >> That's right. Yeah. Like I mean Heisenberg and and like other physicists and there's all kinds of interesting stories about apps and in some cases also just like actual like mental states that were like eventually quite unhealthy and just psychosis and so on have in some cases push the field forward. >> All right. So you alluded to some of this, but let's take some of the life sciences as examples just to unpack some of the thinking there. So starting with medicine, the deeper shift that you described is going from from reading biology to writing it. and your own team did that was one of the first teams to do it. So do you want to sort of tell us what you guys did and what that means in terms of where science is going?

28:09 Yeah, I think when I started studying the first time I studied psych biology was in in high school and I never to be honest loved it back in high school because it's just like memorize these like processes with all of these different pieces. You write them out, you get an A and then like 6 months later you mostly forgot about that process. And so that wasn't that interesting to me. U but what's changed in the last few years is that biology is becoming a programmable science. It's becoming an engineering science. And that's often the case I think in sort of the the transition of different sciences. Once you've understood most of the basic pieces, you now want to learn how to put them together in novel ways such that they are useful for you and there's like lowhanging fruit when a field transitions into that becoming sort of an engineering science. And I think biology is in that state right now where we we know okay this protein does this but if we change that protein a little bit maybe it can do something else. And you can package you know sometimes you can connect different things that you know one piece for instance attaches to a cell but then you can have different loads like connectors to it. So once it's attached to the cell, you can actually inject something into the cell and now you can recombine these these molecules. And so I think that like engineering aspect of it I think is truly exciting. And the first sort of aha moments for us was I think in 2018 when we trained the largest language models for proteins. It's called Progen.

29:33 Ali Madani is the first author of of that paper. that was back in the day when I was the chief scientist at Salesforce still. and he's since started ProFluent. They've now closed like multi-billion dollar contracts with Eli Liy at Profil and his company because they've created new kinds of proteins that are for instance even better than crisper cast 9 and at gene editing and being even more specific and targeted for changing certain genes inside living people potentially and creating new kinds of therapies from that. And so proteins being such an important piece of all the like building blocks of of life and disease and and and and health making them programmable will unlock very very obviously many many exciting use cases. And I think you're starting to see this sort of in in this recent trial that is making a lot of progress where they basically created a different drug for every different patient in the trial.

30:36 And this is a first for the FDA too. and and more will happen there. It's actually unfortunate how hard it is has become in the US and and certainly in in Europe to run clinical trials. and so a lot of folks are now moving to either China or Australia for their clinical trials. Interesting enough, in China it's cheaper, it's faster, but you also have to worry a little bit whether your IP gets sort of sucked into the ether and is gone. And in Australia, they had a clever move where they actually decentralized clinical trials and every hospital can run its own clinical trials. So all of a sudden, you get competition instead of having one centralized sort of decider on on which clinical trials to run and how to sort them and and all of that. And so yeah anyway there's a lot not not enough people in Australia. So it would be great to get that kind of system happening in the US too but yeah clinical trials will be more and more efficient over time. We'll collect more data and then the will be able to automate more and more of that.

31:31 >> And when you think about the drug discovery and creation life cycle from initial intuition to being available that take what 10 or 15 years. what we talking about here in terms of accelerating discovery. What realistically what portion of the process does it shave off? >> It's a good question and sort of touches upon what some people call the hard takeoff to where some people think once we have RSI and and generally with AI there will be this really hard takeoff and then everything will just happen very quickly. And as bullish and excited as I am about AI, I'm not a believer in this crazy hard takeoff. I think yes things will accelerate but there are certain things that will just require time because of physics and constraints in the real world such as like long-term trials that you want to know whether people have some issue like 3 years after the you know they stop taking the drug and things like that and so there will be some delays but the biggest difference is that the whole bio market and somewhat contrarian take that we have at AX ventures too a lot of folks think the like bio is just a terrible space to invest in because in the past a lot of drug companies kind of spend 8 10 years they finally get you know they have to be public because there's not enough latestage bio investors. so they go public with a a one drug or maybe two drugs in late stage trials like stage three and then the stage three trial fails and then the whole company is dead.

33:03 Now what we're seeing the difference is like we now have companies that instead of having one molecular drug after eight years in late stage trials they actually within six to 18 months have multiple different drugs in like late stage two trials already. and by the time they'll go public it'll be with like eight plus different drugs that are then also much more likely to succeed because we have better predictive models. So clearly we are living in a moment when AI has become quite controversial.

33:38 whether that's the job question or the data center question. The number one thing the industry keeps saying as a way to justify why AI is a great thing is AI is going to cure cancer. >> what is your sense of the reality of that claim and what it's going to take to get there? There's a lot to unpack there. Maybe at a very high level, I think if you care about the outputs of an industry or a company, then you love AI. If you get paid hourly, you probably hate AI. and so AI in the positive instantiation of this future is a huge force towards more entrepreneurial thinking. If you're an entrepreneur, generally you kind of love AI because it's making your things more efficient. you just get more done. You have to like you have an unlimited list of things to do if you're a startup founder or just running a company and to have any I do many of those things for you just means you can do a lot more. but if you're like basically being told you're training your replacement and you're like all your data is being collected hourly then you know at some point those hours will end and then the eye will just do the thing you just taught it how to do. And so it's understandable that people like if they have this like very uninpreneurial mindset of just getting paid by the hour and they don't own any equity in like creating that IP then they they're understandably unhappy.

35:08 and then you can go one level deeper and think about well what is the impact on jobs and my my theory here after thinking about this for quite some time is largely dependent on the elasticity of the demand of the product when its prices go down. and so concretely for instance illustrators illustrators hate AI. the the world needs a certain amount of illustrations because of AI. you cannot charge 200 bucks anymore for one illustrations. So now any little blog post has illustrations.

35:42 If your goal was just to see more illustrations in the world that are specific to a text, you love AI. If you got paid 200 bucks for one illustration and now it's worth 2 cents, maybe you hate it, right? and so the the problem was that the demand for illustrations didn't go a 1000x. when the price went down by a thousandx it didn't grow because you just don't need that many illustrations in the world. Now in coding it was a very different world actually as coding got cheaper and cheaper you had this this famous Jevans paradox that everyone's talking about now I think I was the first at least I didn't see it online for a while it's like an interesting fact from history and you actually like the thing got cheaper and cheaper but we actually used more and more of it and for coding that will definitely be the case and so we're seeing actually more demand for programmers now because they're so much more productive when they use AI and anyone ultimately could have like dozens of apps on their phone that unique to that person that are modified in some way and and very special and there's so many other ideas that people didn't explore because it was like maybe the market wasn't that big but now that you can just create an app really quickly why not and so I think that is another aspect of of jobs and so go back to to cancer yes I do believe actually AI will play a big role in curing multiple cancers we're seeing trials now where AI is being used to make a specific specific cocktail of drugs create specific RNA sequences and so on for the types of cancer and you know each cancer often is also not one like homogeneous thing. it has different types of subcancers in it and so on and you need to specialize treatments for each person and for the various different forms of the different cancers that you can have and so all of that is much much more feasible to be done with AI and we're seeing it >> is that precisely the point that cancer is just extremely complex and ultimately a system problem >> exactly that that AI is uniquely equipped to solve >> exactly exactly so yeah it will take it will take some time and obviously like even if AI let's say had the perfect molecule and like okay for this type of cancer this is the molecule came up like and I came up with it you'd have to still run it through many clinical trials it will still take years to to come out so everything in biology just just takes longer than it does in software what else are you optimistic about in that in that field your predictions for the next decade rare disease cures organs design for individual patients pollution eating synthetic cells else.

38:16 You covered some of this in the book. Like what what are you most excited about in terms of u what may come first? >> Yeah, I'm excited about all of these things. I think we can like design bacteria that eat microplastics and once there's no more plastics, they just die. I think that would be extremely helpful for the oceans and so on. Obviously have to be very very careful that they don't somehow mutate into eating other things and so on. So when you mess with the environment at large scales, it's important that humans have done that many times and sometimes it worked out pretty well. Many other cases maybe not so much like forests are a good example. People deal with forests too much. They don't let small forest fires happen and then they get even bigger because the small ones didn't clear out the under brush and so on.

38:57 >> Everything is a system. >> Everything is a complex system. and we need AI more and more to do some of that engineering better than we've done in the past. And so I'm excited for at at all the different levels. You know when you look at like how to balance a plasma and tokomax for for nuclear fusion that's already a machine learning control problem. I think we'll we'll have a better handle on that. so sort of at the lowest level of physics clearly there are more and more materials more efficient solar cells and solar panels that we can design with AI. There's companies I've invested in that do that. better batteries better materials so we don't need only lithium. We can try to build batteries with more abundant molecules that that are easier to get and mine of less less pollution. we especially again in biology seeing a lot of things. I think it sounds like science fiction and I understand sort of the famous saying of like if you want to know why something doesn't work ask the experts. I think that that was true in natural language processing and neural nets and I think it is currently also true for longevity and cancer and other kinds of research for neural nets applied to to biology and medicine. I do think we will see we'll make more progress than the the most skeptical people think but we also want to have a hard takeoff again because things do require careful experimentation in medicine. and I'm I'm personally excited for all of these things. But I think if you're mostly interested in like making humanity more productive and more efficient and create more outputs and grow then you're going to love AI. But I also in some ways it becomes a philosophical question and I think you've already we already observe many subivilizations or like you know subgroups of people I mean and and cultures that have essentially offramped from progress. Like if you're living on some beautiful island in Greece, you don't really think about AI. You don't have to think about AI and you just enjoy life. You go fishing and you know sometimes there's a storm and things are bad, but most of the time like the weather is good, the fish are abundant and you just kind of live your life. And so I think there will be different like groups of people who will want to offramp from civilization progress, right? there's already, you know, people who prefer to live way deep in the countryside and never go into the big city and so on. And I think we'll have more of that. And in some ways, I personally love progress. I think scientific progress especially, is what helped humans solve most of the hard problems that were in our biosphere. David Deutsch has a a whole section in his book, The Beginning of Infinity, which I I I highly recommend people read too, where he talks about, you know, how there all these different material problems and we came up with solutions thanks to science and better explanations and and better research. and I'm personally all for that, but you know, some people will not want to participate in that world anymore. And I think AI is such an accelerant that it makes that question even more pertinent for people.

42:01 >> That's fascinating. I mean just to keep going down that that path before we go back to our little tour of Frontier Science. how would that manifest? I mean so we would end up with like groups of people that would deliberately opt to just not participate in progress. I guess progress has been sort of jagged throughout humanity in different regions obviously. but as it spreads and as the world keeps going more global those people make a political decision to organize around a principle of nonparticipation in AI.

42:36 >> Yeah. I mean like is that city states that kind of stuff? >> Yeah. I mean like a a sort of example that I'd love to visit actually is Bhutan. Bhutan decided we will not measure our gross domestic product based on money but based on happiness. and happiness mostly for people who want to keep it simple and and have a simple life, not want to build, you know, startups and so on. I'm pretty sure those folks aren't quite as happy in Bhutan, but like overall Bhutan is just very green and like it cares about the environment and cares about like a a specific subset of religions and like and and people are more often content in keeping things the way they are rather than trying to like progress in various different ways. This is why this book and this conversation today from my perspective is so important, right? I think the AI industry has done a terrible PR job in general. so if you and you know others can clearly articulate why AI is good, that may hopefully unlock some of this debate.

43:40 >> Yeah, it's really interesting because clearly people use the technology. It's like if no one used chatvt or cloud code like there wouldn't be a problem. People clearly like it. It's just that the people get a lot of use out of it are not quite as vocal and you are there are negative things. There's also some amount of moral panic about chatbot friends similar fashion to how novels used to be a really bad thing. Like there's all kinds of stories of older people saying oh these novels are ruining the youth. They're now living in these dream worlds and are distracting themselves from the real world. And like you know like the the the lightness is a very famous book in Germany actually led to some suicides is really sad and like now it's like the book every German kid has to read in high school and it's just like a high form of literature in Germany and then comic books were really bad and computer games are really bad and like you know there are various sort of levels of that and currently the chatbots are really bad but there also clearly a lot of people get a ton of value out of these chat bots and now all of a sudden you make access to like medical advice case cheaper, legal advice cheaper, and sometimes also emotional advice cheaper.

44:46 But you don't hear many people like or the the many people that clearly exist who are like hundreds of millions of users of these technologies talk about how much this helped them not commit suicide or something or not be very sad and dysfunctional and so on. So I do think you're right like in some ways not just AI but feel like the future as a whole needs better marketing. All right, going back to our tour because I want to make sure we we cover some of the fascinating parts of the book. so we talked about drug discovery. We talked about computational biology. Another fun example or domain that you mentioned is economics with a fun stat where you said economists failed to predict 148 of the last 150 recessions. And so your team while you were at Salesforce built an AI economist that basically operated on a simulated society and you came up with policy recommendations that were better than the state-of-the-art coordinate of quote. Walk us through that. Yeah, economics is a really interesting field that unfortunately doesn't have obvious benchmarks the way computer science and many other sciences have where you just say if you do better in this benchmark you clearly have the better ideas the better algorithms and we should all learn and study those. when we submitted these papers on two-level reinforcement learning systems to nature and science, they just deskrejected them. In one case, some random like ethicist who had no idea about AI, it was just like desk reject. I'm not even going to read the full paper because AI for economics with reinforcement learning is just a weird thing. And so it was just like gone. and so because of that, economics often becomes just a political field. And if you're in one economics department that has a political certain slant and and direction they want to see the world move into. You just have to write papers that make sense for that political ideology. And so that unfortunately makes it very hard to do more objective research. And so we tried to create the simulations very simple simulation where you have a bunch of agents you know like this is from 2018. The agents were much much simpler back then. They just had a certain utility function. They had certain hours of days, hours in the day that they would be willing to work. they were sampled from, you know, certain priors that you may make assumptions about. You know, not everyone wants to work 14-hour days. but some people, you know, you basically make all these assumptions.

47:18 And then you let these agents collect resources, build houses, they can block other agents from those resources to try to build monopolies and become even wealthier. And then you had a sort of meta agent that looked at all of these other agents and basically chose how to tax and subsidize different groups of of agents. and in that fairly simple simulation, you could essentially give it an overall reward. Like in our case, we said let's maybe start with equality times productivity.

47:46 You want the you know economy to grow, but you also don't want like one agent to have access to everything and everyone else is really poor. And so you obviously you don't want just equality and you want don't just want productivity. so you have a combination of these two multiplicatively. and now if you agree that that's a good reward, you could have politicians say well I'm going to do this and that to help for instance the middle class or like to do this and that. But if we had a much a larger scale up simulation, you could then run their one proposal through billions and billions of years of simulations and of taxation and subsidization to say well will that proposal really result in that outcome that you say you you have the goal that you have. Or maybe probably if you simulate billions and billions of years of of different tax you know tax years maybe there are better ways. And and what we found is that the agents will try to avoid taxes by like dumping a bunch of stuff before or making like a bunch of gains just after the tax year and so on. and the funny thing is that paper basically the the baselines that the field uses one very famous formula is called the SAS formula in economics and basically it's beautiful math and it shows that provably it's the optimal taxation scheme but it's the optimal taxation scheme in a one-step economy where you make one economic decision and then no other decision again. And so we showed that this very complex RL system basically recovers that thing and does come up with the same solution. But now you can actually deal with the fact that economics is a temporal sequence of many different decisions and you can learn and adapt and there are counter adaptations from the agents to certain taxes and subsidy schemes. They're trying to play things and then you can still simulate it. And so my hope is eventually that that paper will have kind of a GPD3 moment where someone actually scales it up, builds a really realistic simulation, and then we could have AI give us feedback. Obviously, we don't want to let the eye like make those decisions without any human oversight, but at least have some economic policy suggestions on how to most objectively try to achieve the goals we want to set. And of course, humans then have to really formalize kind of what is the goal of our society.

50:02 And in many ways this these are very like deep questions that philosophy and political philosophy have asked many times. Socialism, capitalism like maybe social market economies where some regulation in like healthcare but maybe not in other areas and you want competition you can actually define once like what your real goals are. So I think hopefully over the years this kind of system will help us run economics much better and make it a much more objective science. Do you think that's realistic that we could model all of the economy you know with all its nuances? There is a an emerging space around simulation of of of worlds and you know a couple of exciting companies in the in the space but at the same time the economy is a lot of rational decision but a lot of irrational stuff. It's very human.

50:56 There's fears there's greed. can all of this be modeled by AI? >> All models are wrong. Some are useful. I think we can make those models more and more useful. and they'll be less and less wrong. I think we've seen surprising results where you can prompt an LM and say you are now, you know, a 43y old like from this region blah blah blah blah blah. Give them all kinds of sort of prompts on what they're supposed to act like. And then after having trained on tens of trillions of tokens on the internet, you can say similar things to what people might say from that setting. And so I do think these models will get better and better. The fidelity of the simulations will get higher and once they cross a certain threshold then the recommendations from such a simulation with an AI could become more useful.

51:48 I don't think this is very feasible in the United States for a very very long time. is just so much identity politics and special interest groups and how you know super PACs and and so on like get funded that it's very very unlikely to be used. My hunch is like Singapore, China will probably be more likely to try to use those ideas say hey we all agree or we at least make it very clear that this is our objective function and then you know we're going to really try our best to set the various taxes and subsidies and so on in a way that really achieves that objective function. Okay. great. All right. So that we talked about again drug discovery computational biology that was the economics aspect. You talk about astronomy. You talk about neuroscience.

52:33 so would again strongly encourage people to read the book and hear all the stories and all the nuances. let's talk about the York machine itself like you alluded to to four stages and and and maybe as as as we get into that question there is also the question of the the quality of the data that is fed in all those machines because if you train AI on a lot of AI data you don't you inherit like all the the biases and the assumptions and all the stuff that is just wrong that is spread out through all of human history >> yes I I think AI often is only as good as the people, the data, the systems, the infrastructure, the rewards that we give it. and we have to be very careful about how we design and filter out all of those things.

53:25 I think, we have more and more control over it. but it is still surprising how poorly engineered some of the environments are and some of the sandboxes are. that that Frontier Lab use for AI. >> So let let's get into the machine itself. So you got four core pillars. Walk us through the first one. >> So yeah, the four pillars I briefly alluded to them earlier where like the first one is just large language models essentially to try to ingest the world's knowledge into the Eureka machine. And I think the interesting bit here actually is that in some ways there's this weird cycle that happened that I don't talk about in the book as much but it sort of lived through this now which is the the few large closed labs anthropic and open AI took almost everything they could from the open internet trained a model but then the Chinese open-source companies basically siphoned a lot of that knowledge out of those closed source models by distilling it. but then they open sourced the model back into the open domain. So now the knowledge is back in the open internet where it started. And so I think it's very clear that that first pillar of just like having access to all the world's information being able to reason through all these different concepts and the crazy large commentorial space of existing knowledge is the first pillar.

55:01 >> So pillar two is a model of reality itself. So what does that mean? If you think about how limited human perception and the current set of human knowledge is and how we could actually expand that you have to look at scientific measurements right we cannot observe gravitational waves we cannot observe sort of gamma rays but we can build tools and scientific machines that measure these things for us and so that is a clear second pillar that is different into human knowledge that in some cases hasn't been fully sort of described in human language and in some cases might be very complicated to describe in human language. Like we can already say, oh like a neural network predicted this word because of these five million parameters, but it's like okay well you just list them all out but you don't gain an intuition because the system is so complex.

56:00 similar to how no one can really say why did you move this muscle fiber in your pinky when you try to move the steering wheel. no one has access to that in their brain. And even if they did, it would just be like because of this very complex system. and so that is basically the ability of an AI to take in all of these measurements and try to start actually like digesting it and and and taking real knowledge extracting it from scientific measurements.

56:26 >> And pillow one sounds like it already exists. Does pillow 2 exist? How do you teach a machine the rules of the universe? >> So one you'd have to really collaborate with a lot of different sciences to put together this kind of foundational model of physics, chemistry, biology and and larger and larger systems and then have many universities and labs work together to bring all of that into one model. so I think you know we've we've had sort of the projection of the humanity's knowledge onto the internet but there are just lots of things that just don't make sense to put up on the internet and so those things are still hidden. and no one people have many many companies are now working on quote unquote foundational models. Some of them now rebranded them as world models when there similar technology below where they basically try to ingest as much information about one domain and then they build like a first example of a virtual cell that is like particularly good at estimating like particular gene variance or something but not lots of other aspects of a virtual cell. A virtual cell is a good example of like a goalpost where many different teams would have to come together and bring all of that data into one unified model.

57:41 No one in the in its full glory that doesn't exist yet. >> Okay. And then pillar three again we're describing the the the four pillars of the Europe machine, what you call the Euro machine, which is this superpowered AI scientific discovery machine. So pillar three is simulation. So that that goes a little bit to what we were discussing about economics. So would you create different simulations for different domains or one simulation for everything? In a perfect world, we'd create one crazy simulation for everything, but it there obviously sort of different levels of abstraction. And sometimes for most aspects, you actually get away with not having to simulate all the quantum details of a very complex like subatomic particle. you can just say all right these are the molecules and then you you know how in chemistry those molecules will work together based on a veilance shells blah blah and then in biology you sometimes just can abstract from like oh I don't even care about that molecule I'll just say this is overall this protein and that protein has like connects to a cell at that level so in as you you try to build it all together but then you have to have sort of computational efficiencies and abstractions that humanity has been good at building and computer science is particularly good as a field in building like no one has to program in zeros and ones anymore. They can now program in English and a lot of the abstractions can be ignored. I think similarly in these physical simulations we can ignore more and more levels down but sometimes like there is sort of quantum biology and there are maybe some effects that we didn't realize and we're oversimplified and those might come out from a model where you like from one large simulation in which the eye can then try to experiment >> and that level three or pillar three exists in bits and pieces >> in many small bits and pieces right the simplest example is like a simulation of go or chess right that's like okay we we have it it's easy an interesting new one that many people are working towards now is a virtual cell. If we had that, I mean, a virtual cell is so complex, like a real human cell is so complex. We're very far away from that. But I can see how with enough people coming together with enough funding, we can eventually get to a a fairly useful model of a virtual cell.

59:54 >> Okay, great. And then I 4 is the real world. >> That's right. At some point like especially in biology but in all other fields you have to engineer a system. You have to really put it together to see if you missed any thing in your simulation any confounding variables and so on. You have to really run experiments in the real world. And obviously in the smaller case of physics, chemistry and biology you can do that in a lab. At some point you have to you know build real machines and really get out there build satellites and whatnot and take measurements of you know the universe and and all kinds of scales. And so you know there I think we like it makes sense for us to put more and more resources behind that as AI has gotten really really good in the first three pillars. So there's some sequence to it >> and is the future a concept of self-driving robotic labs and if so how far away are we?

60:48 >> You know I love that there are like first efforts in this like periodic labs is a great example of that. I I love that we're starting to think about this. personally from investing perspective, I feel like it's a little bit early, but in like two to three years, I think it'll be right on time. We'll have figured out a lot of the software. will have gotten will be really good at at you know the LMS the scientific sort of data and like connecting that also potentially to LMS like building even more highfidelity simulations and then we can ask the eye to ask like to come up with really good expensive experiments that can take sometimes hours or days or weeks to really run through. will have better organoids or like tiny cell systems where you can you know based human derived stem cells parallel bio disease for instance for human lymph nodes like immune cells and then you can experiment with those cells more quickly and that that is done with robotics already so few real examples of that exist I think there's a chem computer too in chemistry that can put together a small set of of molecules there are first examples of this with parallel bio with like organoids and doing clinical trials with that. By the way that alone that company alone has already gotten FDA approval to skip certain animal trials. So you save many many millions over the next few years and lives of animals that are just bred to then be tested upon and then dissected and evaluated. And this like if you love animals, you can also love AI because AI is now already, not just will eventually, but through this one company, Peril Bio, it's already saving animal lives that are just again bred for being like tested upon. And so I think there's like tons of really amazing work that is very targeted to build these out in more and more generality is kind of what is required to then allow the AI and the agent swarms to sit on all of these four pillars more efficiently.

62:51 >> Yeah. And to finish the tour, so there is like this agent swarm. So what what do the agents do? Do they decide which experiment to run or is a human still deciding what the machine runs and do the agents measure what's coming out or is it a human measuring it? How does that work? >> The agents will ideally work on as much of the scientific process as possible. similar to how scientific communities do it. And in many cases the evolution of science and culture and even biology has aspects of open-endedness which is very inspiring for us at recursive also and are actually basically exploring interestingly different ideas highly in parallel that then can then be recombined. and so these open-ended processes have led in biology to everything from you know our fingers, eyes and brains in in technology.

63:51 there are lots of examples where and and Jeff Cloon, one of our co-founders at Recursive talks about this a lot how you can't get a microwave if you just say make this pot faster in heating up my food, right? and you just like you add all kinds of pressure and so on, but you had to work on radar technology and realize like some like chocolate bar in in your pocket was melting as you work in radar to then eventually get to a microwave to then like like warm up your food faster. And so there are like these different paths and and recombinations of different research ideas that can be coming together and we can model that better and better with with agents worms. It sounds like an incredibly compute hungry and data hungry machine given the complexity of what it is that we're trying to model especially as we think about cross domain pollination.

64:44 Do do we have enough compute? Do we have enough data? I mean Ilia said we've reached big data. Does the machine need to create its own data? What are the constraints? >> Indeed. compute is the biggest constraint. I think in the future more and more humanity and already we see this inside different companies will have to decide like what problem is worth solving how much compute do we give to solving that problem and then there will be new kinds of scaling laws where we give enough compute to really solve different kinds of problems and yes I think there's like the majority of the public internet has been digested by a lot of these labs but there's always new data there's always new things that happen in the news. That's why u you know.com we work with a lot of like Neolabs and other labs also to just give them constantly new search results when they ask about something that just happened last week and wasn't yet part of any training data set.

65:44 >> All right, thanks for that. So a lot of those ideas are embedded in your new startup called Recursive. Tell us about the company. Yeah. So, recursive started with the goal of building recursive self-improving super intelligence to automate knowledge discovery and scientific discovery. And it came actually the the eight co-founders came together and we all in one form or another came to the same realization but actually from very different directions. Tim Rocktachel and Jeff Cloon for instance came very much from this open-endedness research direction of evolutionary algorithms and so on. I came very much from this idea of well we automated feature engineering to have word vectors we have neural nets then we automated architecture engineering by just having one unified architecture what's the next level of automation is like the actual ideation and implementation validation of general ideas in all of AI research and that's like clearly and obviously the next level to unlock a new set of capabilities and when you think about the automation of science and then you apply the automation of AI research to you AI like itself, before you know it, you're in this recursive self-improvement loop. and we believe that that will be a great unlock to then apply that kind of intelligence to all kinds of other scientific discoveries.

67:04 >> And you guys raised a massive round of 650 million and interestingly to the comput discussion that we were just having I read that you committed 410 million basically most of what you've raised to a single compute deal with Amazon. So that goes to show the fundamental importance of compute. >> Yeah, we raised yeah in the end like 670 uhish and yeah that will probably be one of the smallest comput deals that will happen in in our future.

67:33 >> And so what what can we expect from the company? What is it that you guys are going to release first? by when? >> There will be a couple of interesting things coming up. I can guarantee you they will happen this year. we are and I struggle with this sometimes the neolab category I don't love it because I think a lot of them will not succeed we are a real company not an academic lab we are building real products we're talking to real customers and we're very excited of taking this technology and making it useful for for real companies I can't share the details yet of what we're going to release but I think it'll be exciting and we already know from some first conversations that it is exciting >> but is it going to be horizontal or or focus on a specific vertical along the lines of what we discuss.

68:19 >> I would like there will be different there's a sequence to it and some things will be general but then obviously at some point it'll be more and more specific. I think we we did publish a blog post that gives you a little bit of a glimpse of things we're we're thinking about that are essentially milestones towards full recursive self-improvement that show that for instance when a lot of people use AI or do some auto research on small models our system the sort of first instantiation of this Eureka machine in a very narrow domain can already outperform months and sometimes years of human endeavor on particular problems. we also showed that they can build new CUDA kernels which is very useful for faster inference. which is very useful for all like large hyperscalers and people who provide tokens and run models. and we're very excited to keep pushing those and we've heard very positive feedback from from folks who are using these kernels now and have like at NVIDIA folks that created these benchmarks. the sole exact bench as a particular example there. so yeah, those are all just simple examples of artifacts that this Eureka machine can produce when it comes to AI research on the path to full RSI.

69:31 >> And as an aside, I cannot resist asking the question, why are most Neolabs not real companies? >> I mean they're just like ideas of like we want to explore you know this particular idea. and that particular idea is like, you know, it's one, it's one of the many useful artifacts our Euro machine could also produce. but it's not really a product like, you know, if you just want to try to think about how humans interact with, AI in the future, that's not quite like anything very concrete. and so you have to be very careful about, you know, what are, you know, founding teams and so on that have not just done amazing research, but also shipped real products.

70:10 >> All right. To end, I want to talk about intelligence and super intelligence and where all of this is leading. So the book ends by asking a huge question. How far can intelligence go? and you propose your own definition. So maybe talk to this. >> Yeah, that one will take us more than more than the time we have left. I feel like it's almost like a new book. I had to wrap it up at that point.

70:34 the book and so one I'm surprised no one has really defined intelligence in all of its complexity really well neither in terms of the very foundational building blocks which I currently think are prediction action and goals and a combination of those three those are sort of the three principal components just like sort of energy has one unit we don't yet know what this is a unit of intelligence something I'm thinking about a lot right Now we don't yet then have a proper definition sort of physics inspired you know in physics we have kinetic and potential energy but then it also makes sense to study chemical energy and mechanical energy and electrical energy and different forms and and some are like still pure science fields and others are very much engineering fields and so I think a similar thing has happened in AI where we have visual intelligence language intelligence physical intelligence and robotics takes and and I define these 10 different spaces of intelligence and each space basically has many different dimensions and I'll just give you this one example on visual intelligence right we have AI can only sorry humans can only see in a specific part of the electromagnetic frequency spectrum but you can go much beyond humans when you think about what are what are the bounds of visual intelligence how far could an AI or any kind of intelligent life form or entity in the universe push visual intelligence and then you get into very interesting sort of often physics inspired kind of like thoughts and and loops. For example, like you can see everything from gamma rays to gravitational waves. so very different, you know, like the whole spectrum of electromagnetic frequencies.

72:22 You can try to have not just two eyes, but you can have millions and billions of different sensors all throughout. And but then how far could they go? Oh, well, at some point you have communication bounds of like the speed of light and you have like each sensor has sort of a speed of light cone around what it can see and like you quickly get into these these thoughts around bounds. And what you then realize is that boy are we far away from the true upper bounds of any of the spaces of intelligence. And there's still so much further that AI can go in in research. And so when people think oh you know this set of algorithms or the field of AI is sort of like a bubble is going to burst like I mean maybe like energy right the cost the unit cost of intelligence may fluctuate depending on a bunch of factors but we can still go so much further as a field and as a civilization and pushing that field forward. All right, Richard, this has been another fascinating conversation and I could keep you for another couple of hours. but I know you have a actually a couple of companies to run.

73:30 So, thank you for spending time with us. The book again is called The Eura Machine. It comes out on September 22nd. >> That's right. >> And where else can people follow your work >> on Twitter? x Richard >> and recursive.com. >> That's right, recursive.com. >> Wonderful. Thank you so much. We appreciate it. >> Thanks for having me and wonderful questions. Great chatting with you always. >> Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you on the next episode.