AI Pioneer Jürgen Schmidhuber on the State of AI Today

0:00 Sounds like you think a crash is coming. >> There would be one of these stock market crashes. What about like robots? Robot hardware is really inferior compared to human bodies. And there's no humanmade technology that compares to this path. >> It seems a lot more than a hardware problem. >> You can't have AGI without hardware like that. You can't have AGI just behind the screen. Are we kind of close to that? >> It will come, but it will take a >> Jurgen Schmidber has been cited as the father of AI by the New York Times, Forbes, and more. He is behind some of the most important advances in the field that really power the AI revolution today. And it was a real privilege on unsupervised learning to get to sit down with him and talk about everything that's top of mind in the ecosystem today. We talked about what's missing in models today and what he thinks is required to get artificial scientists that can push them forward. We talked about why he thinks today's capex boom is massively overdone and why he's very optimistic on AI technology but deeply pessimistic on the model companies and how he doesn't think recursive self-improvement will actually be a moat for the companies. We also talked about AI safety and why he's notably less worried than a lot of others in the field. It's just an awesome opportunity to get to sit down with a legend in the field and ask him all these questions. I think folks will really enjoy hearing his perspective. Without further ado, here's Jurgen.

1:13 Well, thanks so much for for coming on the podcast. Really excited about this. >> It's my pleasure, Jacob. >> Well, I feel like there there's so many different things I want to talk about today. You've obviously been a pioneer of of a ton of different parts of this current AI moment. You know, I figured where where I'd start at the highest level was, as I understand it, you've had this goal for a long time, which was to build an AI smarter than yourself.

1:32 How close are we to that right now? >> From a cosmic perspective, we are as close as we were in the 1970s when I first wish. So we are very close. But is it going to be a couple of years or a couple of decades? I'm not totally sure about that because true AI is not just the AI behind the stream which is working very well and which is passing the touring test. Now true AI is also you know real robots real machinery outside of the screen in the real world in the physical world and that's not working as well. So the hardware in the real world has lots of limitations that human bodies don't have and we still have a way to go to be able to compete with human bodies and physical AI. Have there been any current, you know, AI results over the last few years that have surprised you?

2:31 >> Not really. Not to the extent that it surprised people who had no contact to, you know, neural networks and artificial neural networks in the previous decades. And suddenly there was a a chat CPT moment and suddenly people started being interested in that thing which they had never seen before. So they they didn't know that there was a long history of large language models and an even longer history of basic insights and algorithms for training these large language models that goes back to the previous millennium. So for a guy who was in the in the center of all of that, it was much easier to predict that than for someone who had totally different interests. Well, you know, I definitely want to hit on recursive self-improvement and metalarning because I feel like you've you know, it's been a huge focus of yours for a while. It obviously seems to be a main focus of of a lot of the major labs these days. and you pioneered a bunch of the research here. How do you kind of articulate the you know where we are today on the on the path of of getting to RSI and and kind of what still needs to be solved? So in 1987 this was about using meta evolution evolutionary programming for evolving better programs that learn to do better kinds of evolution. Meta evolution I called it and it was very darinistic in many ways. And so over time you had better and better learning algorithms, learning to com combine code from previous programs and in better and better ways to solve problems better and better. And then in well yeah and and in 1994 we had reinforcement learning techniques self-referential machines that were able to that basically were using a universal programming language to generate arbitrary self modifications of the code that was running the machine interacting with some environment. And then in 2003 that was a a mathematically optimal way of generating self-improvements by a machine that has some software and it's interacting with an environment and this environment you know sometimes punishes you or provides reward and you want to maximize the sum of all the rewards in your life until the end of your life and you want to minimize the sum of the pain signals and then there's an initial software in that machine and this machine then can in principle write programs that modify the initial software. But before it modifies itself like that, it first has to prove which means there's a proof search in the software. It has to prove that the that the modification that will be caused by the execution of this particular program is useful in the sense that it will lead to more expected reward than than the alternative which would be not to execute this program. and and then it has to generate a formal proof which was called the girdle machine. It is less practical than certain other things that we did you know neural networks that change their own weight matrix by running a learning algorithm on the network itself. That is what we started in 1992 and that's currently more or less the most popular kind of selfm modification and self-reference >> today as you kind of alluded to like one of the limitations is you have you know humans defining the start and end of these of these trials versus you know kind of a mathematical way to determine hey is this going to be helpful beforehand and I guess obviously the the trade-off of that being that it's very comput intensive right to do to do some of these proofs I guess as you think forward to what the the path to RSI might be do Do you think it will involve this kind of ability to to kind of do these proofs beforehand or is it like the modification of weights or what yeah I guess what percent would you know likelihood would you think that that the answer lies in in in one of those?

6:42 >> Yeah. So I would say most of the current self-improving systems are scaled back versions you know toned down versions of the girdle machine of 2003 the mathematically optimal thing and they are more like what we had earlier you know neural networks where where you have the weights and the program of a neural network is basically the weight matrix althyp of neural networks that are general purpose computers, recurrent networks for example, they are general purpose computers because on a recurrent neural network you can implement the processing unit of your Apple laptop or something like that. So if you have certain instructions that allow you to to modify the weights itself then you can you can basically run arbitrary learning algorithms on the network which has to see the errors or negative reward signals that are coming in. So that has to be part of the input which is essential and that's all what we did in the early 90s and back then compute was so expensive 10 million times more expensive than today that we could do only little tiny toy experiments but today you can really show nicely methods like that can learn to generalize and learn new tasks much faster than if you don't have this metalarning capacity and and this is currently the the most popular way of doing recursive self-improvement. However, one has to admit that it is limited because there the the learning algorithm is invented through gradient descent. So everything is differentiable and then the whole network learns through gradient descent to generate way changes that are better than what's caused by grain descent.

8:49 >> Yeah. >> Yeah. So that is that is has the limitations of grain descent. So it's not like the optimal girdle machine but it works really nicely in practice. I I think a big question people have around recursive self-improvement is it is it you know that is in retrospect is it going to feel like hey it was this gradual and boring improvement or is there some like huge discontinuity around the horizon where suddenly you know models take off in in capabilities what's sort of your gut instinct around that when when we get there and and are kind of looking back >> yeah well from a cosmic perspective it will look just like a you know like a stick there was no self-improvement and no real AI and certainly there as AI but then from a cosmic perspective this is also true for all of civilization so civilization started roughly 13,000 years ago and before that there was no artificial intelligence and then only 13,000 years later there was artificial intelligence so and and and basically the first guy who had agriculture 13,000 years ago and domestication of the animals that was almost the same guy who had the first AI and and and the 13,000 years of civilization. They are just 1 millionth of world history, which is about 13.8 billion years. So, it's just a a flash in world history. In hindsight, it will look like that.

10:12 Civilization was almost occurred almost at the same time where AI occurred because over these past 13,000 years, more and more stuff was automated and more and more of agriculture was automated and more and more of human labor was automated. And at some point thinking started to become automated a couple of hundreds of years ago. So the first calculators and then the calculators become became less expensive and then they became faster and now you can calculate much more than 100 years ago for the same price and then suddenly it was there. Now from from this global perspective it's really like there was nothing and suddenly there was a law. from the perspective of a guy who is living through that age it looks like a lot. I think one of the the hopes I guess of of RSI is that you know it may end up being less comput intensive over time right to to actually continue making some of these developments which I guess we'll we'll see but that that that I think is would be would be one of the hopes there right >> yeah absolutely and whenever you are talking about intelligence you basically are talking about laziness so an intelligent be being wants to be lazy it wants to achieve whatever it does with with the least possible effort with the least poss possible energy consumption.

11:34 So all of our self improving systems they they have this extra reward for for being efficient or in other words they get they have an extra cost for every time they wake up a neuron and use energy to wake up that neuron or use energy to do other things. So all the costs the the computational costs and the the the other energy costs have to be taken into account in the and the function and the objective function that our self-improving systems are optimizing.

12:14 >> Yeah. And and to the extent that they are good at that they will do the same thing with less and less resources computational resources other resources. So a natural consequence of intelligent behavior is that whatever is being done is done more and more efficiently. >> As you reflect on maybe the broader work that's going on at the AI labs. I mean obviously there's tons of money and compute going into a bunch of the research questions that the labs are are going after. And I'm wondering like if you were if you were kind of running one of these labs or you know I'm sure you talked to folks there or thinking about advising them like what do you think they're maybe there that you advised them to do differently today from from what they're doing. So today you use large pre-train pre-trained systems, large language models that already have read a lot of papers about coding of course >> and then now a modern way of using a pre-trained model like that is you you let it code something and you only improve the way it codes or recodes its own code. So that is now a pretty obvious way of doing it.

13:24 And quite a few labs are interested in exactly that. And of course you have to have safeguards as it doesn't rewrite its own code in a way that is totally stupid and makes things worse than better rather than better. But there are ways of of dealing with that. >> No. that makes that makes sense. I mean I guess is there is there anything you you'd be you'd be kind of doing differently there? >> It's it's a reasonable approach. it's not the most general approach in the sense that if you rely on anything that is pre-trained on human generated data then you know you are ignoring many of the other possibilities. So look at our current large language models they are super biased towards humans. Why? Because they are trained on all the data on the worldwide web. All the data on the worldwide web is there for the only reason that at least one guy or one person at some point thought this is interesting from a human perspective.

14:24 And then all this material that at least some guy thought is interesting is used to train the the systems and they are for that reason super biased towards human language towards videos that humans find interesting towards behavior that humans find interesting and so on. So they are very aligned in a certain way with humans. Maybe they are more aligned with certain humans than with others. Nevertheless, there's a tremendous human bias. Have an artificial scientist who is living in some unknown environment and tries to build a model of the world by just predicting the consequences of its actions and then use the model of the world for planning. such a such an agent will have to create through its own actions the data that trains the world model. So suddenly you you have something like an artificial scientist who through its own actions generates the data on which the one model is being trained and and this is much more like what humans do what babies do. Babies don't learn by downloading the web or something. No, they learn by predicting the consequences of the actions. So if they move their fingers like that, then the video changes which comes in through the cameras and they learn to predict these these changes and that's how they learn about the physics of the world and about how the world works and about how they their fingers work and everything.

15:55 And they they are being trained on a lot of data that is not on the worldwide web, you know, that is collected through their own experiments and and it's exactly the kind of data that the baby needs to better understand what it can do and that it needs to to maximize its own reward. So now you you see that that all of the data that is collected on the worldwide web that seems to be a lot but it's just a tiny tiny tiny fraction of all the parcel data that you could collect out there through your own experiments. So the future of of of AI is going to lie in such systems that through their own actions through artificial curiosity as I called it in 1990 collect all the data that is train is used to train the world models and these world models they will not depend on human language and they will be very focused in many ways on this particular robot that is collecting the data and yes of course it is going to communicate with other robots living in different environments and it's going to incorporate that knowledge such that it can better generalize and so on. But you know suddenly you have a situation where where the systems are going to be much less human biased. It >> was interesting. I mean a lot of what you you just described obviously there there's companies that are trying to build you know automated labs for you know material science discovery or for biology or you know even in in robotics there's all these folks doing you know teleaoperated data but obviously today it's it's kind of humans deciding what the data that should be gathered is or designing those kind of experiments and and trying to feed it back and I guess the the dream is to have you know a fully AIdriven loop of that of that discovery right >> that's true yeah 20 years ago I I wrote this paper about the formal theory of fun and creativity which which is about exactly that. So what should an what should an artificial scientist or any scientist or any artist or any comedian do when when its main needs you know like eating three times a day are satisfied and and it has extra time to do stuff. What do you do?

18:11 Well, you listen to music or maybe you compose your own music or you generate art or you are a scientist who not only is trying to solve problems given to you from by by other people. No, you also try to invent your own new problems. Ask your own questions. Not just answer questions given to you by somebody else, but invent your own new good questions. That's what scientists do. And and all of science is about two things. is not only taking an existing question and investing a lot of time into solving it and finding an answer. No, inventing the good questions and the the basic principle is very simple. So an artificial scientist or any scientist in my point of view a human scientist as well is driven by one simple thing which is try to find through your actions your own self-invented experiments data that has the property that in the data there is some regularity some pattern that you didn't know but can quickly learn that you didn't know but you can quickly learn it because it's at the limit of what you already almost understand, you know, near the horizon of what you don't know and what you know. And and then you you you generate this data through your own actions through your experiments, the sequences of actions which are experiments. And the data comes in and if there is something interesting in the data, what does that mean? It means that there's a pattern in there which you didn't know of. All patterns mean there's some way of compressing those things in space and time. You're not talking about t time right here to make things not too complicated. But to compress to compress the patterns in a way that you didn't know before, which means that before you understood the regularity and the pattern which is coming in, you needed so many internal bits and hidden units and so on to encode it. And afterwards after you learn to see the pattern after you learn to compress it you need so only so many and the difference between before and after that's the fun that you have you know that's just a real number which says how much internal joy does this scientist now have from recognizing from realizing oh there is a regularity that I didn't know in the data which is coming in it tells me something about gravity and then this becomes the reward of the controller who's generating the actions that lead to the data. Which means now the control is motivated to, you know, generate more and more experiments that lead to insights about the world that it didn't have before. And everything that it understands becomes boring. And then it it wants to create more complicated experiments to, you know, after it has grabbed the lowhanging fruits, more complicated experiments. And maybe the the same the baby which first learned about gravity when it was a little being maybe a year old or something.

21:19 Maybe 20 years later, the same baby is working at the particle collider at the CERN and is part of the team that discovers the the the Hicks Bzon or something and and the only difference is that the experiments are more expensive. And I guess is that really the the blocker to as you think about, you know, what what stands between us and getting to this AI scientist? Obviously, like the ability to stand up experiments, right, and and do them, you know, both at an affordable price point, but also kind of feedback into models is is suddenly that a lot of people are working on, but certainly remains a blocker. And it also seems like there's all sorts of algorithmic problems to be solved to actually kind of create this this this setup that you said.

21:58 How do you think about like yeah, what what needs to be solved to make this AI scientist a reality? And then how how soon do you think we'll get there? >> Yeah. Well, we have simple AI scientists. We have had them for a long time. it's just that maybe they haven't seen their chat GBT moment yet. You know, Chad GBT. When was that? 22, 23. >> Yeah. End of 22. >> It was based on stuff that was really old. We don't have the same kind of chat GPT moment yet for these artificial scientists. But the artificial scientists to a certain extent they already exist and and they are being used in lots of special applications.

22:32 For example, in chemistry you have lots of pairs of inputs and outputs and you train your neural network to predict to predict these these new substances given the previous inputs. And then over time if you show it millions of experiments it learns to become an artificial chemist, an intuitive chemist. So it's not a chemist who understands all these reactions from first principles from valence electrons or whatever. No, it it just becomes an intuitive chemist that better better understands what can be done in chemistry. And then you have certain objectives. For example, you want to create a material that is that is twice as efficient against a certain maybe maybe an insecticide or something that is twice as efficient as the most the the best thing known so far. And then you can say okay let's have a desired output like that which encodes that it should be twice as efficient as the best thing I know and then you can work the whole chemist backwards and can look at the input side can say how much should I change my experiment which is visible at the input side to get this thing that I would like to see to fulfill my wish and then you the chemist will basically give you a suggestion >> and do you think like within a decade we'll we'll figure out like AI chemistry?

24:10 >> Yeah. So at cow we have an interesting project where the goal is to to use certain patented structures metal MOFS they are called and the the goal is to extract carbon dioxide from thin air which is important for improving the climate. And at the moment all of that is very expensive and the goal is to make it so cheap that that you really can make a dent and maybe maybe improve the global warming situation.

24:47 so that is one of the potential applications and and there are lots of applications in all kinds of chemistry fields. What about like robotics? What how do you kind of characterize where we are? What's been happening there? And you know, everyone everyone has to stream of an at home robot. you know, are we kind of close to that? >> I remember again in the 70s when I told my mom about the future of AI and how AI is going to colonize the entire universe and she said, "Just come build me a robot that cleans my kitchen."

25:27 >> The age-old dream. >> Yeah. Exactly. And and back then that didn't work. And still it still doesn't work because robot hardware is really inferior compared to human bodies. And there's no handmade technology, no humanmade technology that compares to this hand. This hand is full of sensors, millions of little sensors and little cables that connected to the control center. And and I wouldn't even know where to put all these cables in a in an artificial hand. And the the crazy thing is you you harm it. You cut it and it starts healing itself. This is super advanced technology. We have nothing like that and man-made tech. And and that's the reason why you know why the robots in the movies are all played by humans because humans are much better robots than the robots that >> but it seems a lot more than a hardware problem right I mean even even with the hardware we have if we if we had good models I'm sure we could do we could do way more right >> but you know you can't have AGI without hardware like that you you can't have AGI just behind the screen yeah you can have a superhuman chess player behind the screen and something that passes the touring test but if it does master the real world. It's not an AGI, you know, it's just a maybe a fancy text editor behind the screen or something that has an idea of how this robot should move or whatever, but it's not the real thing. And if you want to master the real world through a real AGI, a physical AGI, so much more has to be done. So our robots have to become much better than the limited stuff that we have today. How much longer will that take? We can easily predict how how compute per dollar is going to evolve. It's probably going to stay like what we have seen for decades now. Factor of 10 every 5 years roughly like that. Which means also that the guys who are investing a thousand billion dollars into GPUs for data centers today within the next five years they are going to lose 900 billion dollars. You know, there's no business model that can recuperate that last, which already is an indication of the coming crash maybe. But in in robots >> in in robot technology, how long is it going to take to to get something that is compatible to this hand which can do both strong grips and very delicate, you know, finger movements that just manipulate tiny little things in a in a way that is infeasible for what what kind of robots can do. That is at least for me much harder to predict. It will come but it will take maybe it will not take just a few years, it will take another few decades maybe. Well, I I do want to switch over to the business side because you said obviously something very intriguing there and I've heard you say this before that like you know this massive investment in the AI data center build out you know ultimately with the improvement in in compute performance you're investing a ton up front in something that's going to be you know kind of outdated and and worth far less 5 years from now. The like counter-argument or what folks would say to that is that there's just going to be like infinite need for compute and that ultimately like we're going to be, you know, bottlenecked in our ability to to produce enough compute such that like even if you have hardware that's legacy and way less efficient, it's actually going to still people will still want to use it because there's just going to be such demand to run inference for for all of these different interesting at least today digital and in the future physical use cases. And so even if it's less efficient or or not the best compute, it's like every chip on the planet will will need to be used in some way. What's kind of your reaction to that?

29:14 >> Yeah, maybe there will be more and more demand for compute, but somebody has to pay for it, right? And if it turns out that those guys who are currently paying that they are losing a lot of money, you know, at the moment a couple of of companies are investing hundreds of billions per year. so in total maybe a trillion per year or something like that into GPUs for for data centers and and and and these companies which used to be nimble software companies and you know had a little team improving some shitty operating system and then rolling it out for billions of people who had their own phones and their own computers to run the operating system. Suddenly they are providing clouds and and data centers and and suddenly they have to become like utilities like electricity companies and they have to invest in nuclear power plants and gas turbines and whatever and suddenly the cash flow the free cash flow of these companies goes down from 100 billion down to 10 billion or maybe minus 10 billion like for some of these companies. So at the moment while everybody is trying to get market share these these services are are really not efficient in the sense that the guys who are providing the services they are losing a lot of money. It doesn't show up in the price earning ratio because the free cash flow which really should be considered is not showing up there. But all these companies are getting less and less efficient. So at some point they will have to stop. You know now they're taking on debt to finance even more data centers and this will be possible only to a certain extent and then these companies will become less and less valuable. So they because they misinvested the money. That means that that at some point for or later if you can't pay for all these services, if the entire economy is not set up to pay for all these services because compute isn't cheap enough yet and because people want to compensate for it by buying even more computer computers, if if you have a situation like that which which doesn't lead to to a profitable economy, then it's going to disappear due to the laws of supply and demand.

31:45 >> I mean, I guess there's kind of like two two questions embedded in that, right? One is on on the inference itself, you know, are there enough use cases out there that are valuable enough for companies to to want to pay you know, well well above and beyond the the kind of cost of these data center buildouts? And certainly with AI coding, it seems like there's you know, as that is as the first market with a ton of fit, it seems like there's been at least some use cases there that that certainly meet that bar. I you know certainly on the training side I think there's a big question and I know you've talked about this before and I I think our listeners would love your thoughts on whether these you know closed source model providers who obviously are spending a ton on training whether there's a business there right or whether you know over time open source models just catch up and and ultimately you know I think there there's there's two sides of that argument but would love for you to share just how you think about for the closed source model providers like is there is there a business staying you know maybe three six months ahead of of the open source models. Yeah, as you say, the open source models are one of the major reasons why the big companies cannot simply raise the prices because they are so close behind and somebody some commercial large language model is breaks another benchmark record here. But a few months later, there's an open source model that catches up with that. which means there's enormous pressure on pricing which means all these expensive data centers and GPUs where these companies invested in and and took on debt to pay for that they are not being profitable at the moment you know so a stupid way would be just wait a little bit just wait for 5 years then computer is going to be 10 times cheaper you will be able to do the same thing for onetenth of the price or wait 10 is and you will be able to do the same thing for 1% of the price. But then of course the the the big companies say, "Oh, but if I do that, if I wait until the others do it, then I I will lose maybe my market share or whatever or I miss out on the opportunity to be the first to have a true AGI or something like that." And in in that case once they have a true AGI that can solve everything then I can take over the world and it will and and the investment will be nothing compared to the value of that. But but I think all of these these outlooks are over optimistic and and and probably we will see that that the current way of u of investing a lot in computers that aren't fast enough yet is going to backfire. You know, I think related to our earlier conversation, I think there's this belief that, hey, if you can be the first to recursive self-improvement or the first to, you know, have models that make it easier to develop the next models that it becomes this kind of like self-perpetuating cycle that makes it really hard to catch the the folks that are that are that are, you know, in the lead, you know. What do you think of that?

34:45 >> Not really because many of the guys who are interested in self-improvement, they are open source guys and and everybody's cooking with the same water, you know. Yeah, >> everybody's cooking with the same water and almost all of the important algorithms in AI, they were not invented at big companies or whatever. No, they are they were invented at little labs and without much funding and especially the recursive self-improvement stuff and and the basic ideas for that all come from little labs you know academic labs and and any company that wants to somehow grab that and make it its own doesn't have a mode because there are all these other PhD students out there who are super excited about recursive self-improvement and and there's no way to you know keep a little advantage there for long enough to make a lot of money. So I think I think it will be very very difficult for the for the huge companies to be very profitable in this business.

35:55 >> Yeah. because basically like these ideas, you know, permeate throughout the broader ecosystem, you know, and so as a result, you know, when when one lab gets to to some level of RSI, it'll be kind of out in the open as well. And, unless there's some massive compute need required for it or some kind of data advantage to distribution, it'll just be kind of available, you know, more broadly. >> Nobody knows exactly. Maybe maybe maybe you you are lucky and you you stumble across a really good new way of conducting incremental self-improvement using the enormous resources that your particular company has for now and then achieve the AGI that you need to take over the world stock market and and and finish the whole game. I of course there's a remote little chance of that happening but all the indications point the other direction and instead AI is getting cheaper and cheaper and there are more and more people who are you know involved in de developing new AIs and many of them are poor PhD students somewhere haven't even founded their companies yet and so on and then you really will have this AI for all situation where you know everything that seems u impressive today, 30 years from now, will seem trivial because you can do a million times as much for the same price and and it will be, you know, it will be just like with with the smartphones. Yeah.

37:29 >> Yeah. But it sounds it sounds like you think a crash is coming. >> Well, not a crash in the sense of a civilization crash, but there will be one of these stock market crashes because at the moment I think there's a lot of misallocation. Of course, the stock market is going to learn from these misallocations and it's going to pursue different approaches, but and and it's not going to be the end of civilization or something. No, but it's going to be a renormalization of what you currently have because at the moment we have just super expensive companies which don't really have much to offer.

38:06 >> I feel like you've kind of publicly been less worried about AI safety today than maybe some of the you know other other folks in the community. And I'm wondering, you know, these past years, has anything changed your your mind at all or are you kind of still in the in the same camp? >> I remember in the 2010s, we had lots of these safety conferences conferences and alignment conferences and you know, and people writing letters that certain types of recursive self-improvement should be forbidden. Of course, I never signed any of these letters. but but back then it it was already a big deal, you know, for at least for for the machine learning specialists and how can we prevent AIS from from becoming dangerous. Now I think all of these approaches were misguided in many ways and it's kind of obvious that well first of all this whole alignment business means that there is somebody who decides what is the nature of this alignment. How do I align these AIs to the needs of humans? But if you have 10 different humans in one room, they all will have different needs and they will have different opinions about what is good for humans and what is not good.

39:25 And and and so this whole perspective is dominated by the idea that you give one objective function to this AI which is supposed to optimize you know and that's then the alignment that you create through that. On the other hand, for you know since since 1990 or so we have built these artificial scientists which >> are not really aligned much with anything because they all the time they invent their own new objective functions. They invent their own new problems all the time.

39:59 They they they change their own objective functions and really starting in 1990 we had systems like that. So this whole premise of having systems that don't change their object objectives didn't make any sense to me. It was very naive in my point of view. So I didn't sign those letters. I just couldn't. and and today we we have these extreme cases where people are using AI to fight against each other. If you look at the war between Ukraine and Russia, it's about AI based drones on this side, fighting against AI based drones on the other side. And there's no alignment whatsoever. And of course, you cannot expect in our world, you know, a universal government that is going to align all these AIs because all these different secret services and militaries, they all have different objectives which are often totally conflicting with each other. So so it seems to me that that that all these efforts of the 2010s were kind of naive. Now on the other hand I think that and and if you want to get really smart AIs you have to give them the opportunity to set themselves their own goals. you know like little artificial scientists that that now ask a new question about what happens if I do this and how does the world respond if I do that so only if you give them the freedom to ask their own questions and to solve their own self-invented problems only then they will become really smart on the other hand they will become less predictable that is of course the issue that is being addressed here and and and now people are asking but but in a in a world like that where you have all kinds of AIs that set themselves their own goals like they have done in my lab for many decades. isn't that super dangerous and so on? I think it's not going to be more dangerous than what you have now with humans. Humans also set themselves their own goals all the time and and come up with new questions to to ask. And and yes, it is true that the kid my kids are unpredictable. I'm not sure what they are going to do in the future but at least I can contribute as a as a parent to to making them useful members of society you know and whenever they come up with with bad experiments like for example let me take this magnifying glass and and focus the sunlight on this little ant here then I'm going to punish them and say that's bad you shouldn't do that and and that's how I taught my kids to become reasonable members of society. And the same thing we are going to do with our robots and machines. And in the long run once they are much smarter than we are, I think we have a certain we can expect a certain kind of protection through the fact that they are scientists because artificial scientists will be super interested by life and by their own origins and civilization and by everything that led to their current state. and they will they will be fascinated by life and they will be greatly motivated to protect the source of interesting patterns and and and they will be motivated to protect it rather than destroy it. So from from this high level perspective I think there's a very strong reason why you shouldn't be too afraid of you know Arnold Schwitzer Terminator scenarios.

43:43 We always like to end our interviews with a with a bunch of like you know quickfire questions where we we we shove a bunch of things into the end. and so maybe to start actually I'd love just give our listeners context on you've obviously done so many interesting things over your career. How are you kind of splitting your time today and thinking about like what's most interesting to work on going forward. I'm a conservative guy and I think the most interesting way of going forward is still the one that I pursued in the 70s and 80s when I try to build this general purpose AI that learns to become smarter than myself such that I can retire. I'm still working on the same old thing.

44:19 >> You've obviously, you know, have this kind of network of of a ton of amazing folks that have come through and worked under you and then gone on to do great things as well in addition to you. Like what have you learned about what makes a really good researcher? many of the best PhD students that I have had the pleasure to work with, they usually focused on a particular question. So yeah, there's the general big problem that we want to solve, but then there is this particular little thing that currently doesn't work and then you have to look at the details and how do these weights change and this neural network which is trained by this particular learning algorithm and why doesn't u the network do what you want it to do and and you have to debug a little tiny thing and suddenly you see the devil is in the detail and there is a little detail detail which you have overlooked so far and if you fix that one then suddenly everything everything works out and suddenly you have a breakthrough and then the same PhD student can come up with one research paper at a major conference after another just because it's such a rich source of additional improvements that was triggered by this little devil in the detail solution. Do you think the transformer will persist as like the dominant architecture you know over the next 5 10 years?

45:46 >> I guess some sort of transformer will but I think it's going to be a little bit more like the efficient transformer that we had in 1991 the linear transformer. Today it's called the unnormalized linear transformer. I call it the fast weight controller. And and and what is the interesting thing about it? it scales linearly which means that if you have 1,000 times more more text then you need 1,000 times more compute while the modern quadratic transformer of 2017 if you have 1,000 times more text you need 1 million times more compute. So everybody's worried about that. It's one of the reasons why why these data centers are now so super expensive. you have to do so much compute and everybody wants to bring down this complexity into to a more reasonable linear complexity or maybe log linear complexity and there are quite a few transformer variants that are going in this direction or there is stuff that SEO right is doing the XLSTM which is a which is a a thing that combines aspects of the old linear transformers. So you you want to get the complexity down and it's related to what you asked before. intelligence is about doing the same thing with less effort. Yeah, we want to reduce the effort.

47:10 >> Well, I love that. look, this has been a fascinating conversation. I'm sure there's a ton of threads that folks will want to pull on, you know, in in addition to what we've discussed here. So, I want to make sure to leave the last word to you. where can folks go to learn more about your work and really anything you want to point our listeners to? The the the mic is yours. >> Where can you learn what I think is interesting? Look at my blog. It's easy to Google if you Google for Jurgen.

47:36 >> Yeah, we'll we'll link to it and you will find it. And and there are overview pages with links to the original papers on metal learning on artificial curiosity on the formal theory of fun and creativity. Also on the history of our field, the field of machine learning is the is is is about the science of credit assignment and apply that to the field itself and who invented convolutional neural networks and who invented deep learning. All that stuff is nicely listed and explained there. And that also explains what I consider the most important stuff and including what what I just mentioned u you know the important thing is physical AI outside the screen so not behind the screen but in the real world and and once we have a robot and and for you know for hundreds of years people have talked about self-replicating machines but nobody had any idea how to get there but now we see an opening Yeah, for the first time we now can have or maybe we'll we will soon have robots that can learn through through imitation or reinforcement learning to operate the existing machines all the existing machines that already are part of our civilization. Once you have a machine that can operate all the machines that humans currently are operating, then you have a new kind of life, you know.

49:06 Then you have a a way of implementing this ultimate scaling machine because you can have robots that make more of themselves. And I've said that for decades and now it's getting closer to reality. You don't need super smart robots for doing that. just smart enough to learn to operate all the existing machines and a collection of machines like that can make more of itself and something like that is can can also improve itself not only make replicas of itself because all the concepts of machine learning that we already have for AI behind the screen you know in the in the virtual world all these concepts we are going to apply to self-improving robot societies and something like that is not only going to work in the biosphere but also on the moon or on Mercury where there's a lot of material for building infrastructure and bigger AIs and more AIs and more robots and huge spacecraft and all kinds of stuff that we need to colonize the solar system.

50:10 >> Well, I think that's the the perfect note to to end on. incredibly exciting vision for the for the future and and and seriously, thank you so much for for taking the time to chat through everything here. It's such a privilege to to get a chance to talk and I know our listeners will enjoy it, too. It was my pleasure. Thank you, Jacob. >> I'm Jacob Efron and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so, please consider doing that. And thank you so much for your support and listening. We'll see you next episode.