VIDEO: Will AI do your job? with Raza Habib, Member of Technical Staff at Anthropic

0:00 Raza Habib, welcome to Onward. >> It's a pleasure to be here. Thanks so much for having me, Benjamin. >> Okay, so here we are. You work at one of the major AI research labs. So I'm interested to know what you know that other people in the wider public don't understand about AI. But before we get into it, I just want to make sure when we talk about artificial general general intelligence, you know, that people understand what AGI means.

0:25 So like how would you define AGI before we talk about like how revolutionary it might be? >> Yeah, I think that's a it's a great starting question and I don't think that there is a clear or an accepted definition of AGI. Different people say different things. In fact, one of the two founders of uh Deep Mind, Shane Le, has an entire chapter of his PhD dedicated to trying to like define what intelligence is. goes about gathering like 30 different definitions from like operations research and psychology and and and the reality is that I think it's not it's not a well- agreed upon term and because AI has gotten so hypy as well, it's made AGI even more of a moving target. So the definition that Shane Le came up with in his PhD thesis was that intelligence measures the ability of an agent to achieve its goals in a wide variety of domains.

1:17 So in that case, you know, you're measuring intelligence on two axes. One is how general is the thing? Like how many different environments can you achieve your goals in and how good are you at achieving your goals? So I quite like that as a definition of intelligence. It still doesn't tell you where the line for for AGI is. And so I think maybe a thing that's more useful to talk about is something like economically transformative AI or something like that where you could write down a set of criteria or evaluations of things that the system is capable of and then measure whether or not we've crossed those thresholds. then I think it becomes easier to say do we have that thing or not and you can just say okay across this range of economically valuable tasks or something like that can the AI do it but um yeah the reality is there is not a I don't think there is a good definition and AGI has become a bit of a marketing term >> in a way it's like how many jobs can it do right >> the other way you could think about this is like give me a a bucket of tasks that you care about maybe all the things that you can do at a computer and then what's the probability that a AI can do this better than a than a on on that task or something like that and maybe when that probability is below some threshold you say I have AGI there's there's this idea in machine learning of like probably approximately correct algorithms but it's very hard when you're making a classifier to to say that it can sort of do everything accurately on on all possible data sets. So if you want to prove bounds about learning algorithms then the best you can you can say is like it's probably approximately correct and you set some threshold of accuracy and some probability of being right. I think you could probably do something similar like we can probably approximately solve all the tasks that humans can solve or something like that.

2:51 >> Do we have liftoff? >> I think so. >> Okay. Beg you to beg you to say yes >> to be clear because you know the com seem anthropic will get after me otherwise like I'm speaking entirely you know in my capacity as a as an as an individual here and not on not on behalf of the company I work for. But look, I think there's a lot of uncertainty and we won't know like it's making predictions is hard, especially about the future as the old line goes. But I think there are enough there are enough pieces of evidence stacked up that make me feel like yes, we're we're on a trajectory where this thing is happening and it's happening quite plausibly soon.

3:32 For me, the last release that that of models that we had from Anthropic and also others, you know, Opus 4.5 was a big update where I was like, "Oh, no, actually things are still just improving on a very steep trajectory and and I I don't see any sign of that slowing down." >> Yeah. And we saw that I mean, as a consumer code interpreter and immediately put into a bunch of our products. So, and it's awesome. I hope you guys don't move on from that too soon.

4:03 But anyways, I knew you before you joined an AI research lab and you were uh at um human lube and I want to know are you more AGI pilled? Like were you already AGI pill? Like kind of like what's your evolution personally? >> I thought I was AGI pled before I joined Entropic like >> you know my first I did a PhD in machine learning something like a decade ago. I came in to the PhD not very AGI pled and then was like consistently just beaten over the head by this excessive deep learning and I kept thinking oh X thing won't happen for however many years and then it just kept happening so by the time I joined anthropic I thought I was pretty AGI filled already like part of the reason I wanted to join is because I think this technology is likely to be transformative I am much more AGI filled now than I was before >> so it's not even like a red pill blue pill apparently it's this like it's a spectrum Yeah, like you know how how powerful and how quickly and I think I my probabilities that we will get transformative stuff soon have gone up for sure.

5:09 >> Well, I want to talk about what that could mean for society for for companies. You know, one of the things that actually I I have a 14-year-old son and he he keeps asking me what job he should do when he grows up and what because he's worried about AI taking his job. He's asking what's the most resilient to AI and he's never been happy with my answer. So, so okay, how do you respond to that? Like what what what should I tell him? He's going to listen to the podcast probably.

5:44 Yeah, I get this question a lot ever, you know, even before I was at Anthropic, like when I've been working AI for a long time. So, people people sort of uh especially parents, parents that are coming like what should I what should I teach my kids like what should people learn? >> And I honestly don't feel like I have a great answer either. I can tell you some things that I think will be resilient to automation, but I'm not sure that's the answer you really want to hear. So like things that I think will be resilient to automation are well one would be things like professional sports things that like basically have their meaning because they have a large social element. I don't think any of us are going to go watch the Super Bowl with two teams of robots playing each other, right? Like I think that is in my mind like something that's like very fundamentally human and will stick around for a long time. In terms of skills or what to learn, it is so hard to know. But I think that on a time horizon of like 10, 20 years, which is like what young people might be thinking about, I sort of assume that machines will be able to do all the things that we do cognitively as well or better.

6:47 And so actually I heard something interesting. I was listening to um a podcast, the wreath lectures with Rutgar Bergman, Bregman, the Dutch historian, and he he gave these lectures this year on moral ambition and you know, trying to get the the most ambitious people of today to focus on things that will be like society beneficial. I think it's a great lecture series. And he said that, you know, if you went back, he had this statistic, I don't know if it's true, but he said if you go back 20 years, something like that, they did a survey of like incoming undergrads to Harvard.

7:17 And 20 years ago, you asked them like what they wanted to get out of their educational experience and a lot of people said things like an understanding of like a philosophy of life or what it means to live well or something like that. You ask the same question today and people say they want to get a highpaying job and like maybe some of the answer is like revisiting those old questions, right? like the things that will give life meaning and purpose, our community and connection to others and craft. And it doesn't necessarily have to be economically valuable work that we end up spending our time doing, but it depends on how well we as a society distribute the benefits that come from the technology.

7:54 >> Yes. Oh, don't get me. >> Now you're on my mission. I wanted to do this idea of walking through maybe like a timeline. You know, this is obviously super contingent and but it gives I think people like me who are on the outside a sense of like pace and I'm generally of the opinion when I talk to people that they they're really skeptical of the types of things coming out of San Francisco coming out of AI labs like they really are like no that's just doesn't I don't believe you don't make sense all hype you guys are just selling you know selling your wares And so very spec I mean I I don't know how specific you want to be but I think a sense of timeline of like year one so 20 end of 26 end of 27 you know and it's end of 2030 for example like something like that I'd be super interested to hear kind of what what you're you think is like happening with >> it. I hear the skepticism a lot as well and it's interesting like I often ask the people who are most skeptical like whether they're using the tools and I like often hear the answer is no and so I do think there's no substitute for like playing with the models trying stuff out but Nat Friedman had this great line he said you know they alternate between spooky and kooky that's increasingly less true but like there are moments of like true brilliance and then they sort of make dumb mistakes and I think undermines people's confidence. But yeah, on the timelines point, maybe we can like go back a little bit and then do timelines up till this point and forwards because I think it's helpful to get a sense of like the pace of acceleration or how things have like been happening. So I would say that there was like this moment back in in the early 2010s when we got like convolutional neural networks for images working and that was considered like this huge seinal moment. people started to get excited about deep learning. Like the Europe's conference started to have like more people with like with titles of like VC and other things show up at what was previously a small academic conference. That was like a little over 10 years now. And if you go back and you look at like what the things the models could do that got the world so excited, it was like, oh, they could look at a blurry 20x 20 pixel of like a car and recognize that that thing was a car. And that was like mindblowing to everybody at the time. And then right up until say 2020, if you wanted if you were talking about like natural language processing or understanding, you know, if you were buying a legal AI product on the market, like what you were getting was something that could do things like classification and named entity recognition. So this is like five or six years ago. And what it would do is like it could take in a contract and it could like extract with some difficulty and I genuinely mean with some difficulty like who like all the named entities and like the key terms of the contract and that was like really impressive in 2020 and to build that system you know you needed a team of like experts to annotate thousands of example documents and then you would train a specialist model and the model could do just that thing. That's it.

11:08 That's all it could do. And by 2022, right, with GPG3 now, you have models that like, okay, you got one model that can do extraction, can do classification, can do some amount of question answering, but it's still kind of crappy. Like the early use cases of of GPT3 were basically just writing assistance of various kinds. It was like writing marketing copy, you know, copy AI and Jasper were like the early successes there. And then you go forwards like just a year or two later, right? 2025 and you now have systems that are able to take actions on long time horizons and like complete full tasks such that it's the case that we now have coding assistants where you know there are senior engineers at anthropic who say to me they have not written a line of code themselves in in a couple of months. Someone said to me you know I said sort of so how are you how are you checking all of this stuff?

12:00 He's like, you know, like I I still read every line, but like I don't really feel like I need to. Like I do because it's built in braces, but the the fraction of the time that I'm actually finding errors now is pretty close. So that's, you know, we've gone from blurry image being classified to, you know, and needing millions of examples to, oh, I can reasonably just expect this thing to write an entire software application like in the space of like a very small number of years. And the progress in the last few years like the I would say um three or four years has been quite consistent with projections that were made on the basis of scaling models.

12:40 >> So it's not just that we've had like very rapid progress but the people who made quantitative predictions on how much intelligence we would have as a function of model size have been borne out now across something like 10 orders of magnitude of scaling. And so so that's just to say like we're it seems like we've been on an exponential for a while and the exponential could continue for a little while longer. So what would happen if it continued for another two years? I think it's very plausible that by the end of 2026 we can do the work of a senior level software engineer like end to end completely like as good as any top engineer can which is like already on its own quite mind-blowing. I think we'll increasingly have agents that can do long range complex judgmentbased tasks. So we'll see a lot of enterprise automation. We might start to see the first early signs of models that like are doing sort of creative work or sort of pioneering or solving problems on their own, but it'll still be early.

13:37 These are my personal predictions. And I think we'll start to see continual learning starting to work a lot more. So today, if you get one of these AI assistants, you know, it's maybe really smart on day one, but like by day three, you're like getting very frustrated at the fact that like not gotten any better despite the fact that you've given it so much context. models are getting better and better at that and I expect that to to improve in a big way as well and honestly 2027 feels like a time horizon that's too far away for me to like make any reasonable prediction.

14:04 >> Okay, so 2027 is too far away which is next year. So >> I feel like I can see through the haze maybe maybe towards late 2026. >> Let me just like ground some of the things you said in in my experience and then come back to the 2027. So you saw this we this how I got to know you originally was we we were building real AI we're building real estate AI product and um and we did it very different than I think a lot of the other AI applications because we we built a a huge huge data set that's updated every day so that so because it's not just enough to be smart like a person you have to actually also know the information you have to know what's the rent of this property >> you have to have the right context >> I now tell real estate people you can go had like a you know the smartest person who graduated from Harvard but he's no he hasn't spent five years in real estate he's not going to be that good at real estate and so in the last like 30 days we started we we now could produce proformas and underwriting memos and so like like from end to end the real estate professional you know the analysts like you can do a lot of their work I mean this is this is like as you said the rate of progress in the last 12 months it went from like you know it was like a like a not a glorified Google search but like some kind of closer Google search than than like intelligence to me and now it's like it's doing whole tasks whole like a job of real estate analyst because I'm trying to use another example but beyond software because people aren't as familiar with software programming >> is like looking at a lot of data synthesizing that down to the sort of conclusion and and like key data points might be might it might be five pages of data turning that into an output like a perform or investment memo and it can do all of that all of it.

16:00 >> Yeah. >> And that's today and nobody in the real estate industry is using it yet like we are like whatever like basically it's so new that the diffusion has happened yet. >> It's crazy how early it still is. I think a lot of people mistakenly think that the lack of diffusion is a signal that like there's snake oil here or there isn't there there in the technology and I actually just think that part of what's going on is that the rate of change is so fast that you have to revisit the same task like every couple of months to have a sense of the capabilities because you try something two months ago and it's like oh it doesn't work two months later it might work and so having an accurate mental model of the frontier capabilities is like very difficult unless you're like just trying things a lot. So what what what I find now just like in my personal day-to-day life is if I have a hard task to do, I will try to give it to the AI first and and even if I don't expect the AI to be able to do the thing, I just try it with the model first because I I want to have a good intuition of like where the boundary is and I keep being surprised by like oh that thing I didn't expect we could do yet. Now we can.

17:08 >> Do you have like a special internal model? Are you talking or do I guess are you using are you doing that with the same model I I am? >> I can't really I don't have a reason. >> Okay, fine. I didn't I was okay forget it. I definitely don't want to cross any lines either. So, um >> No, that's that that's totally fine. You can imagine that like we have, you know, slightly early access to things internally, but but nothing I'm saying I think wouldn't be true were I to be outside.

17:33 >> Yeah, for sure. Your edge may be different than my edge. I mean, we haven't adopted. It's hard, you know, when we're in the application layer to change the underlying models for certain, you know, tooling like at the rapid pace because you have to change it. You basically have to change a lot. There's a lot of more things that you can't just change the model because all the all of the prompts and the language, all the things that don't work at the same way. Tooling ends up being different. So like um that's a kind of a minor diffusion problem like a very like kind of like logistical one. biggest diffusion challenge is is sort of human behavior. And it's like interesting because like it's almost like the big AI companies have just like they don't even care if the person I mean they do care but they like they the the edge is usually the adoption product market fit but in this case it's sort of not product market fit. It's something else >> like as a company at scale like that's something new.

18:33 >> I agree with that. Let me just go back to something else you said as when you were doing the leadup because I I had had this impression that the scaling laws have sort of or like not laws but they call them laws right the scaling progress had diminished and that what was happening was there's a lot of application work and sort of like post train all other things all these other things were happening to make progress not in your but in your sort of you're painting broad brushes >> I'm definitely painting broad brushes But no, I don't think you're misunderstanding. I think that there are many different dimensions along which we can scale. So you're scaling the size of pre-training, you're scaling how much RL you're doing, and then you're like, you know, there there are sort of continuous small innovations in that whole process as well that allow you to like unlock effective scale because you figure out how to unhobble the model in some way.

19:24 So I think that um you can sort of like map all of this onto compute in some way like how much effective compute is the model able to use in its training process across all of these different dimensions and then as a function of like compute I think we're still seeing >> power law returns to scale right it was always a power law so it is still always diminishing returns you have to scale more to get >> to get the equivalent increase in the loss the loss of the model though the thing that's maybe not so intuitive is like when you're training these models, there's some loss curve, right? That's going down as you train the model that gets better. And the same amount of like change of loss like going from loss five to loss three is not necessarily the same in terms of capabilities as going from, you know, uh 3 to one, right? Even though those might be both reductions of two, >> improvements like late in training might be small improvements in loss, but very big improvements in capability. So it's not it's not true that it's exactly a power law, but there are some diminishing returns to scale, but it's still it's still it's still giving returns. So there's there's still lots of benefit to scaling.

20:31 >> Okay. So now let's go back to this this fuzzy 2027 because I'm like a many layers outside where where you are. But I do spend time trying to like like you know advocate to people. I'm in Washington DC. I I try to advocate to people that the policy makers aren't thinking about this the right way and I and I I you know my little way I did a podcast in December basically saying let's just like stop talking about AI is a bubble and just let's what if they're right like what if this happens like I've been thinking about this a lot >> yes >> I think if you sort of take the perspective of a policy maker I understand why they are skeptical right like they are faced with this like there's a huge amount of uncertainty there's a lot of noise there are people on both sides It's some people like myself or others who are saying, "Hey, we think there's going to be this like very transformative thing in the very near future." And then you've got other people who are saying it's all hype, it's a bubble, like don't worry about it. And they've got these very like quitian problems that they're like voters are actually caring about today and someone's asking them to take very drastic action on this like very uncertain thing for the future is very reasonable for them to have skepticism.

21:43 And so I think there's like two things that I think about there. one is the payoffs here are in in the two scenarios very different. So like if you imagine that like the that AI turns out not to be like that big a deal and it fizzles out or it's not it's not very extreme and you acted early and you sort of started campaigning for you know sort of some kind of redistribution or you campaign for other policy that would like help mitigate the economic impact you probably end up looking pretty embarrassed maybe it's not good for your career like I can understand that there's a risk there but the risk on the other side of it turns out it is transformative that we huge labor impacts that we have AI systems that are smarter than most people in the not too distant future and you don't act like this is potentially like quite catastrophic. And so I do think that that asymmetry at least should motivate like some amount of action.

22:38 And then I think the other thing that's really important is I think that the the frontier companies and the other people in the space need to bring the data. It's not enough to like tell people, hey, we think that this stuff is going to be transformative because I think the skepticism is very warranted. And so I think things like the anthropic economic index or other projects where people are trying to measure like what are we seeing already in the economy today can at least give policy makers some cover and some sort of foundation on which to do bold things. But I think that we need to take much bolder action and and much sooner than most people realize. I think people should start thinking about and and and campaigning for now. I think there will be a crisis that will come at some point as a result of this and I think like what's the I forget the sort of Freriedman quote but sort of you know when the crisis happen the idea that are lying around are the ones that get picked up and so now is the time to like start seeding the the ideas that we want people to pick up.

23:35 >> Yes. Let's do government policy a little later because it's kind of the most high level and do like okay you're I I I have to run a fairly large investment business. We have lots of real estate, right? >> For me, one of the lessons I've learned because I've been doing real estate investing and and and some tech for, you know, since 200 I mean, since 1999, right? I mean, it was a long time. I went through 2001, I went through 2008, went through 2020.

24:04 And so, my takeaway is that the macro is like everything. Like macro is the most important thing. If you get the macro wrong, everything else is wiped away. you can be complete idiot the macro right and you look like a genius and so and I think the AI uh conquest are very very macro you're saying like you know potentially like I don't know what the some super relative would be but very extreme and so let me just walk through some like basic economic questions because that sort of sets the stage of what the policies would need to be so okay so let me just go with like first like so is AI inflationary or deflationary Why would you expect it to be inflationary?

24:49 >> Well, I don't, but everybody thinks inflation's the problem today and they're obsessed with inflation. And I'm like, don't you see that there's this massive deflationary wave coming? But I So I mean, it was kind of like a a softball. But okay, but so okay, so what does that mean that it's deflationary? So like tell me how that gets like uh gets its way into prices. What what is that? And it says it's deflationary. I feel like you're taking me now outside of uh my circle of competence and I I I don't I don't know whether I I have a right to a pine here, but my my just like intuition is that if you can if you can do a lot of the work that you were doing before at far lower cost and you can make it more efficient that that should be a deflationary process. Like it seems to me intuitively hard to see how you manage to do a lot of existing processes more cheaply and drive prices up.

25:42 >> Yes. But it it's complicated, right? We're also like dramatically increasing demand for energy and data centers and chips and like one part of the economy is like having but but but overall it appeals to me like like intuitively that it should be deflationary. But I am not an economist. >> I have an economics degree, but this is one of those things that economists always get wrong. And so it's it's highly it's it's highly uncertain because there's also these sort of like reflexive dynamics that that surprise people.

26:10 But yeah, I think it's deflationary. I think it's when I think about the internet what did the internet do you know internet I mean like take the e-commerce right fundamentally just it just is electronic commerce right so it got rid of retail and then that amount of retail square footage amount of retailers you know department stores and all these you know they eventually got eaten up by e-commerce and I think about AI it's just fundamentally it does cognitive labor so it's it will it will have an effect on the job market and then there's like this other side of the equation which is that it's productivity driving, it creates wealth, it drives efficiency and so you have like these two dynamics and so like yeah you can have inflation and defl 2010 sorry I'm going on a long time here but 2010s you had both because you had massive inflation in San Francisco and New York and you have massive deflation in this in the in the rust belt and rust belt was collapsing because of the China China trade and there was huge you know cloud and mobile you know technology boom on the coast So we could have both in America and in the world. I haven't thought about AI's effect on like the world, right? Because AI is mostly an American product, right? Mostly we probably export AI >> consumed globally.

27:27 >> Consumed globally. >> Yeah, >> it could be our biggest export of America in a few years. >> Yeah, it does. It does seem very plausible globally. It feels today like there are essentially only two players like us and and China and China seems somewhat far behind even now. I mean catching up potentially and we can't be complacent but still not at the frontier >> and just it seems like different too like they had like a it's it's hard for me to say here but it just seems like it's such a different kind of AI ultimately >> but they're also training training frontier large language models. I think the politics of China will make it make it different but again I try to stay within my circle of competence and and this is definitely on the boundary of it but my intuition again like if I if I was to give my man in the pub opinion and people can take from this what they want is from what I understand of the CCP is that they're they don't like things that are varants increasing right there's some sense in which like things that threaten their supremacy are are difficult for them to do as aggressively and I think AI is sort of a challenging one for them because on the one hand it has the potential to give enormous power to whoever produces it and so maybe for that reason you could expect them to like pursue it very aggressively and at the same time I think it has like the potential to be a bit destabilizing and I don't know enough about China you know politics in China to know where that's netting out right now.

28:49 >> Yeah. Yeah. Who does right? So I was just trying to set the fundamental economic challenge that the policy maker has to address and I think there's an economic one which which is that that if if uh a lot of cognitive work which is mostly white collar work right but knowledge work whatever you want to call it gets uh done by done or or co-produced by a AI then there's ends up being some displacement or job suppression right that's problem one then there's Like so much of policy has been downstream of media. Social media had this huge effect on policy and AI is going to change media like radically radically. I mean it's so radical I think people can't appreciate.

29:38 I don't even know what media is once you have AI. Like >> as I said I struggle to see beyond the fog of the end of the year. Like if you plausibly have AI systems, maybe not in a couple years, but like not that far away. It seems not implausible to me that these systems might not just be smarter than like any given human, but they could be smarter than all humans. And again, like this is just it is a very difficult like world to reason with when you have the equivalent of, you know, a country of geniuses in a data center is the line that Dario uses. And I really like that line. It's very hard to make forecasts.

30:14 Like I I do I do think that just the uncertainty is so high on what the impact of that is. >> But I haven't heard great ideas lying around that that somebody's likely to pick up. The UBI feels like a terrible idea to me. Like doesn't seem like it addresses what people really want out of their life. >> Yeah. What I worry about with UBI as well is that um >> it it's very it concentrates power, right? It feels like it will make the majority of people somewhat powerless or dependent on the state. It's very difficult if you're fully dependent for your livelihood on the state to to really have any meaningful freedom or ability to protest or complain. And that feels like it could it could be pretty pretty uncomfortable.

31:00 I guess like historically, you know, you you have balance of power between like elites and everyone else through like labor and violence. I guess violence doesn't go away as a threat, but labor might and and labor is a big is a big negotiating, you know, uh, lever here that maintains the balance of power in society. >> So, what are some of the ideas you think that you got to you got to place strategically around the decision makers when the crisis happens? Like what like do you have like a top three?

31:29 >> I honestly don't right now and it is something I've been like trying to think about and speak to people who are policy experts more. The role I think I can play and where I feel I can personally add value is helping people who are in politics to understand the state of technology, the rate of change, the likely implications and motivate some form of action, get people to believe that something drastic is needed. I don't feel like I'm in a good position to suggest solutions. But it feels to me that like some form of redistribution from the AI companies themselves, from the people who are generating the huge amounts of wealth towards everyone else will be necessary. But the details of what that looks like or how you do it, I don't know. Again, I I work for one of these labs, so I got to be a little bit careful about what I say. But in my capacity purely as a as a citizen like I would love to see some form of redistribution from the AI wealth and profits towards the the rest of society.

32:27 >> Yeah. I mean typically we try to do that through an ownership society, you know, through a through a 401k. >> Taking equity is like one possible way of doing that, right? >> I actually thought that some AI lab would offer to put some of their equity into the Trump accounts for every kid in America. Like I sort of imagine that idea as something somebody would suggest >> for lots of reasons. >> Yeah, I I Yeah, I can see I can see for lots of reasons. Yeah, maybe. I mean, one of the things that I think is amazing that is true at least for us is that there's a donation match. So, a lot of like the employees at least and the founders of the company have pledged to donate the vast majority of their or the founders certainly the vast majority. It's public that like all the founders of have pledged 80% of their wealth. But you know if you if you as an employee you can donate a significant fraction of your stock and the company will match that. But but that's you know scratching the surface. It's nowhere near enough what we need to do.

33:24 >> Yeah. And the problem unless you had this job you don't really appreciate but like making money is a like make building things is a skill right and actually managing money a lot of people who make things end up going bankrupt. They can't manage it after they've made the wealth. But there's an entirely different skill and capability and whatever philosophy of actually being able to give away money successfully. >> Oh yeah, absolutely. >> And so the challenge of what is you create sort of you try to create a whole new problem that we're bad at, right?

33:52 Which is how do you okay fine you you know open AI has this massive foundation but we don't actually know how to give away money and then be effective. >> Yeah. I think it's it's sort of well documented that the difference in outcomes or sort of you know quality adapted life years or something like this of like the median uh charity or philanthropy versus like the stuff that's in the top 1% isn't like it's not like it's 20% better or 30% better. It's often like orders of magnitude better.

34:19 So the difference between doing philanthropy well and doing it badly is is often pretty extreme. >> And there's a scale challenge too. I think you're saying this and I I but I so I may put words in your mouth but I don't think good ideas come out of politics. I don't think they come out of the people who are like in power. So like going to them and saying you need to come home with some good ideas because this is going to be a big crisis. Like that's not I'm I grew up in Washington DC live in Washington DC.

34:48 That is never how I've seen good change happen. I believe you and I wish I could come to you and say, "Hey, these are the three things I think we should do." But honestly, I I don't I don't have good answers myself right now. It's not something that I feel qualified to to to answer. And you know that it's unsatisfactory, but it's the truth. >> I'm not demanding it. I just it's like uh I don't think anybody's qualified just to make you feel a little better about suggesting ideas. So, but um let's let's try to go to another area. So, if AI progress were to slow and plateau, like why would that be the case? I'm sure people have thought about this, but you're you're not expecting that. I'm not expecting that, but there's a bunch of reasons why it could happen. And and I still I still put significant probability mass on that possibility, right? It's not it's not that I think we're like 90% sure to get, you know, AGI or superhuman AI, whatever you want to call it transformation in the next couple years. I just think the probability is high enough that like it's rational to start behaving as if it's going to happen and preparing for that outcome.

35:57 And and the more I see that like my my my probability estimates have only been going up, not down for some time. But I guess like if you if you said, "Hey Resa, it's 2032. Like it turns out that AI fizzled out. It didn't happen. Like what what happened? Like what's your story?" I guess like my explanations would be I mean one is like there is just some genuine science risk like these things that we call scaling laws are empirically observed and so like there's no guarantee that like this thing continues to work as we scale like that's one possibility.

36:25 I think more likely possibilities are like factors from outside of AI end up actually impacting the industry itself. So like you have some form of global unrest. I don't know, China invades Taiwan or something like this happens that like disrupts the semiconductor industry and then we can't get the compute capacity we need. And you know, in those kinds of situations, then yeah, I think AI progress would stall. >> What else do I think could happen that would like make progress stall here? I mean, compute and energy constraints really do feel like a big one. There's just like stuff you have to do in the physical world that takes a long time that you know if that becomes the like rate limiting step then it will take longer.

37:05 >> Yeah, I'm familiar with that part of the world. I'm loving when when people were obsessed with AI being a bubble amount of data centers people were building going to build like whatever two months ago. I was like have you ever built anything? Good luck building good luck building all of these data centers in the timeline they're talking about like this like it's so hard >> for so many reasons there's just a million reasons why building things in the real world in America maybe in like China you can just like you know clear mountains and things like that but here I mean it's like so hard >> I could be a be a rate limiting step >> yeah slowing but okay so do you have a contrarian take like Okay. You're you're you're sort of like right now giving the sort of like the a lot of what AI research lab people say, right? So, do you have something you're like, well, I think that they're right about like a lot of stuff, but here's where I think they're wrong.

38:03 >> It's a question of degree, right? Like you said, how AGI pilled are you and like where are you on that spectrum? if you buy the the sort of most extreme timeline predictions and they don't you don't even most extreme but like I don't know you read some of the like super forecasting results or even like I don't know Dario thinks like maybe 2027 2028 we get something like a country of genius and data center >> let's let's say that you buy that timeline like what happens six months after that right like progress is reasonably fast right now you've got a few thousand good researchers at the top labs like doing their work Um, but you know, you're rate limited effectively by how many of the smart people you have?

38:42 Like what happens when you've got a bunch of like, you know, you got some form of recursive self-improvement? Like I find it really hard to make predictions. So, I don't know. Do I have do I have an a take that's like particularly contrarian to the labs? Honestly, probably not. Like, there's a reason I fought my way to be here. I think that this is going to be transformative. I think it's like very important that we do everything we can to try and make it go right for society.

39:06 I think that there is like both huge potential upside but also genuine like very real risk and trying to capture the upside and mitigate the downside like just feels to me like the most important thing we can do in the world right now. And so in some sense like yes I have the same belief as the other people here because that's why I'm here. >> I used to doing something else. I changed it to get here. In the last couple weeks, the public software as a service businesses, the public SAS companies have been getting gored by the stock market and they're down tens of percent, some of them are down 80%, 60% in like two weeks because they they are afraid that AI is going to eat up a lot of their business.

39:49 Do you have a more nuanced take on that? >> I guess yes and no in that like I think it's directionally correct, right? Like I think the marginal cost of producing software is going down. There's this think tank the the Golden Gate Institute and one of their founders had a blog post that I thought was uh was really excellent. There's this analogy that people have been using for AI for a long time where they talk about what happened to factories when electricity came in and the fact that it took a long time for electricity to actually make a difference. And the analogy has been kind of crude up till now. So like the analogy was like oh factories used to have this single central power shaft.

40:22 Everyone connect off of it. And so when you brought in an electric motor, you initially just replaced your steam powered engine and like nothing really changed inside the factory. So you didn't really see some benefit until people like shrunk down their uh electric motors to be small enough that every piece of equipment had its own motor and then the entire factory could change. And people have sort of been using that as a metaphor to say that like AI diffusion will be slow because you have to restructure things. This is the first time where I actually think the metaphor is like extremely spot-on or extremely accurate because when you think about building software like building software for an individual or for a small group of people that runs locally is not that hard. Like actually you know there's product challenges etc.

41:04 But the smaller the group of people and the more bespoke the software is in some sense the easier it is because you really are building for that person. What makes like software products hard is some part of it is like infrastructure challenges, distributed systems, scaling and some part of it is like products because you're building for a distribution and you have to understand how to build something that's delightful to a group of people who you can't go and speak to all of them. And what AI is making possible is you can shrink down that engine where you had that one big steam engine that was powering the entire factory and give every person an individualized piece of software. Now, not all software products fit that description. There are some things that are inherently multiplayer that they have, you know, that you really do want something to be in the cloud for the software to work. But for things that you really could imagine there being more bespoke versions within a company where they might be paying a huge amount of money to to an external company for something they could reproduce themselves.

42:01 I think that that seems like very clear to me that it's going to happen. I think build like anytime where you think about like a build versus buy by like build has suddenly become like phenomenally more appealing than it used to be. >> I know what that's like our team and for the first few years were were really biased towards build and then we and then we learned that we should mostly buy and so now it's basically oh we should actually build again but we're not a normal organization. We have a lot more software programmers than most companies. So you have to like get to a place where normal people can become product developers essentially.

42:38 >> I think we'll get there. It's very interesting right now. I think like the the roles are all blurring. The boundary between PM, designer, engineer, like all of these are starting to overlap way more. >> I'm much less. Yeah, >> for me it's >> I think for the high agency ideas person and for the you know CEO of the company or whatever, it's wonderful. You amazed how many seuite executives I speak to who are like I'm I'm all over cloud code every weekend or whatever.

43:05 >> This is something I I texted you about. So like how do you build software in an AGI pill way? Like like is that a thing? >> Yeah, I think so. I think so. And I think you've probably experienced this a little bit yourself. The most important part of it, there's a bunch of things, but I think the most important part is one to be building with the expectation of significant improvement in place. So what does that mean? Like you should be able to ask yourself like someone hands me a model from six months in the future that's significantly smarter like do I need to rearchitect my product to get the benefits or will my product just instantly improve? If you think about like what that means is if you are doing a lot of hard-coded workflows with a lot of deterministic logic where you're relegating the AI model to be making small decisions and you're not allowing it to orchestrate the flows or take key decisions because you're trying to mitigate risk then you're going to have to rearchitect a lot to get the benefit of when a new model comes in. Whereas if you trust the fact that the models will get smarter, you put more of that logic into the model, you allow, you know, agents to run the show, say, and you put the the guardrails and the safety in, then I think you're going to you're going to end up with a thing where as the model gets smarter, your system just naturally improves. So I think that's like one way to be agi pill and how you think about product development. I think the other way is in how you think about pricing as well. So I think we're used to pricing software in ways where we charge on the basis of seats or things like this that are somewhat fixed and that makes it hard for you to like constantly be adopting like then you have to really worry about margins. So so if you want like you have to start thinking about like whether or not to adopt the smartest model because usage of that model goes into your cogs. And I think that like shifting towards more usage based pricing and thinking about how do I price my product in such a way where I can pass on some of those costs to my user actually means I'll be able to give my user a better experience they're willing to pay for because I can use I can always be using the most frontier model. Whereas if I'm trying to pack everything into an all you can eat price of some kind, I'm going to constantly be thinking about like optimizations on cost. So that's another way in which I think you can be a pill in in product development. Honestly, the way I think about it a lot is like how do I get out of the model's way? So, how do I like get the right context into the model? How do I make sure the model has the right permissions to access data in a way that like is trusted and secure where it has like sandbox execution environments? Like, how do I put the scaffolding in place so that then I can safely trust the model to get on with things and benefit from the exponentially improve improving capabilities of the model?

45:43 >> Yeah. Right now, my biggest challenge with real AI is that the user struggles with the way to use AI. Like it's just they're they don't find the they're not used to using language as the interface. They're not used to sort of blank screen problems or >> there's all these challenges around actually again maybe this is just the state of AI that the people are the bottleneck. You'd be amazed at like how good the models can be at elicitation as well and actually you can make this a like the AI's problem rather than the person's problem and and sort of you know going to a prompt box and having to figure out what to put in is like very daunting. But if you can put in something that's just like a starting point or there's some suggestion and then the model can elicit from you more information, ask good questions, make proposals, I think you can get a lot further.

46:38 >> And actually the models are pretty good at this already. >> Yeah, I don't want to get too far down this little rabbit hole because it's it's just people people find like when the model's asking them for stuff, they get kind of annoyed like no, just leave me alone. >> You can definitely put too much on the user. Yeah, >> you see great examples of this in products like Cloud Code that have for example plan mode.

46:58 >> So they've like thought of like like you know good ways to in a in a high bandwidth way get feedback from the user sort of when to come to ask for permission versus just do stuff but they're also products that ask you to like give a lot of trust. As we sort of sum this up like the thing I feel like I'm going to try to articulate well I almost want you to articulate it but I'll take a shot. People don't like inequality, not just wealth, but like you know if a thousand people are making all the decisions or so it's power, wealth, popularity, things like that.

47:34 But you're talking about a world where that's becoming like, you know, the nature of technology has been that it it amplifies certain people way more than others. And so we're headed into a world that is not going to be wellliked for this transition period. when you get to the other side, there's enough maybe there's enough abundance. Maybe we figure something out, but we got to get there. And so, you know, the societal consequences you keep bringing up, but like do you have any like uh parting thoughts, recommendations, you know, for individuals, companies doesn't have to be, you know, some magic magic policies prescription.

48:12 >> You know, at the start of the conversation, you said to me like there's still a lot of people who are very skeptical or just like not that many people who know yet. And so I feel like the first thing we need to do is like persuade people that this thing is real. And I think persuading yourself as well because if we're going to get the the politicians and others to take the like size of drastic action that I think will be needed, then they have to believe that there is a crisis that's, you know, unfolding.

48:40 And and I don't think we're quite there yet. I think there's still a lot of skepticism. And I think that that can come bottoms up as well. I think that uh right so I so so I think we need to persuade more people that this technology is real that it's not snake oil that the consequences are real and I think we're going to have to bring the receipts to do that. So to the extent that like we can communicate to others with data and with examples and really show that this is actually happening I think that's like a very powerful thing in and of itself and then sort of at the individual level you know I again I find it I find it very unsatisfactory. I think that that we need to try and find ways to democratize control or decision-m about what's happening. But a prerequisite of all of that in my mind is like first accepting, you know, got to accept the situation we're in and persuade more people of the reality because people are just not very well oriented right now.

49:31 >> When I listen to all the things you were saying, it reminds me of of every major war we ever got into. up until that war that World War II people were fighting us getting into it until Pearl Harbor like like pe like every war World War I like surprised everybody like even though retrospect you look back and so it feels like there's no getting ahead of the crisis realistically and there's just things you would do you wish you'd done ahead of time when the crisis happens not this maybe this is too pessimistic but I just that's just like I think the lesson of history is that like generally people you just don't see governments get ahead of these types of crisis.

50:12 >> Look, all of this is uh makes it all sound very doom and gloom and I think that there is there is like reason to be concerned and there are things that we need to do and I do think we should be trying to and I and I am personally doing things to try and get more people aware and and to drive advocacy. At the same time, I do think it's worth thinking about like, okay, what are the potential upsides if this goes well? And there are so many things that like I'm very personally excited about in terms of being able to give like every person on Earth access to in their pocket, you know, potentially an angel of their better natures. like something that can allow them to self-actualize their own ambitions that can act as that little coach and partner that like nudges you in the right way that helps you learn things that you know you and I are like trying to your 14-year-old son who's like trying to figure out what he wants to do with his life in some sense the question like discovering your values and finding out like how to to get there and the teachers that I remember or the people who have had the most impact on my life were very emotionally aware and they were like they helped me self-actualize in some way but it's a very rare thing like you have to be very lucky to to have someone who can see you in that way, who can understand what you care about and who sort of knows when to push you or not push you and give you that nudge, etc. Like we can maybe give that to everyone. I think we could potentially see huge impacts in healthcare, potentially see huge impacts in science and drug discovery. Those ones like we'll need progress not just in AI but in in data in other in other ways as well. But there are there are huge upsides here as well, right? And also it might be very deflationary. So like just a huge amount of wealth might be created in general.

51:49 We just need to figure out how to how to distribute it. So I try I try to keep both sides of this equation like in in balance. And yes, we need to take pretty drastic action, but if we succeed in this, I think there is a there's a lot of upside on the other side, too. >> Yeah. I mean, it's not like war. It's a golden swan. It's not a black swan, >> right? Yeah. just eventually like you mess it up because you're about to get this massive amount of abundance and the mess up is that you don't figure out how it's shared in a way that everybody wins.

52:21 >> Yeah, that seems right to me. >> So, I mean that's definitely what I believe and I mean I'm super excited for you. I'm so glad you're just you're just crushing it. I can't wait. You even working on some new product. Can't wait to find out what it is. >> I'm excited to be able to to talk about it soon in the future. >> Yeah, you got to let me know. So, well, hopefully this podcast uh you know did a little bit of work in in the efficacy and um yeah, like onward.

52:48 >> Yeah, absolutely. Thanks. Thanks so much for having me, Ben. And yeah, hope hopefully the podcast like raised awareness a little bit and I do think that uh people should start thinking about like what are the mechanisms for distributing the benefit.