Transcript
0:00 Herp heard this and said, "I'll think you're being quite naive. You should read some Machiavelli. It's far better to be feared than it to be loved." There's a bunch of great stories from all of us at Val it's a very unique culture that we've tried to take cues from at Val as well. >> What even is this whole model evaluation space? >> Independent evaluation is the idea that frontier labs are developing at increasing pace and they shouldn't be responsible for self-regulating.
0:22 >> You guys have a way of measuring RSI. >> It seems like Anthropic will get to full RSI by July or maybe August 2027. Model progress will speed up and it will maybe even move into places where we understand the models less. >> That simply would be pretty scary if we weren't watching that carefully. >> Greater intelligences >> >> will super forecast better than humans. They will be actually able to say what any human is is going to do without the human even knowing it.
1:00 >> >> Ryan Christian is the founder and CEO of Val's AI. Whenever you see a new AI model get released, you see those numbers and they compare the AI models to each other, those numbers are generated many times by Val's AI. They've become one of the very top evaluators of models. This is a fascinating area cuz you're constantly creating new hills for models to climb. It requires you to really understand how to measure model capabilities as well as evaluating model risks and trading new ways of scoring models for risks. This means Ryan is on the cutting edge of what's happening in the world of AI, where AI is going and of how we can help create the landscape of what government might do to regulate these things and how the model companies might think about regulating themselves.
1:39 He's creating the frameworks and the concepts and the measurements of that are leading the way for what AI is going to look like in the next decade. Let's hear from him. Welcome to American Optimist. Ryan, thank you for joining us today. >> Thanks for having me. >> Founder of the CEO of Val's AI. >> Vivek, thanks for being here and I'll make me sound smarter. >> Of course, here to help. >> >> Welcome to the show. So, let's tell us a bit about your background. You studied computer science at Stanford. You were I think at Microsoft for a little bit. And then you you joined Palantir a long time ago. What what led you to join Palantir?
2:08 >> Yeah. Yeah. thanks for having me. I actually just kind of started when I was at Microsoft. It was an intern there in high school. And the BERT paper came out from from Google. and for those who don't know, this was a really big paper. It showed this generalized system was actually capable of hill climbing or or succeeding across a variety of major NLP benchmarks. and it was >> Say for our listeners who aren't technical, what is what's the NLP benchmarks mean?
2:32 >> Well, it it basically means there are these special specialized tests that people came up with. They tried to make specialized systems that would score well on them. Like, here's a multiple-choice exam, and then here's a sentiment analysis. >> Of natural language. >> Yeah, that's right. and so, instead of building a specialized system that did well on one of these tasks, what they had done was shown they had a model that was very general-purpose. It had a fundamental understanding of language.
2:52 and in a way that translated to all of these benchmarks. It was suddenly number one across the board. >> What year was that? >> I think it was just 2019, 2018. >> It's a long time ago. >> Yeah, like >> >> yeah, lifetime ago. Yeah. >> In in AI terms, it's pretty before age, but it was getting there. >> Yeah, it was a big deal. And and I think to me, made it very clear. This is kind of an illustrative moment for the bitter lesson. The idea that no matter what we what we try to do and kind of delude ourselves into thinking around algorithmic optimizations, actually just having more data at a bigger scale is what allows us to unlock new forms of intelligence. and so, it was very clear to me then is that we're going to have a powerful model or systems like this, which will be an essential part of our the fabric for our society. and I think there were also ways in which the models could be misused. around the same time while I was at at Microsoft, I read Zero to One.
3:39 and it became increasingly clear that people at Palantir were trying to construct the technology in a way that protected civil liberties or Western democratic ideals at large. and so, I was really excited to work at Palantir to see the technology come to life and adopted in a way that was consistent with the values that I had. and so went Stanford for for undergrad, got got distracted doing some research there. So, I published some papers and and was kind of on track to do a PhD, but I made the time to also work at Palantir for a summer and I think that was also really illuminating experience.
4:07 >> You were mentioning you learned something else at Palantir from Karp. Anything anything else there is interesting? >> Yeah, there's I mean there's a bunch of great stories from from while I was at Palantir. It's a a very unique culture that we've we've tried to take cues from at Val's as well, but anyway, while I was there, I I had the chance to work on this hack week project. and and this essentially was an idea which me and this this guy in London had. So, we were chatting about it remotely. I flew over and and essentially I'd done some work with CLIP, this joint image text embedding model.
4:34 >> and what we were trying to do was explain what that is for for the audience. >> idea of a single model that that can encode image input and text input into the same semantic understanding. Right? And and what we we're trying to do is build out a reverse image search feature into Gotham. so, instead of just typing text queries, you could upload an image of say a blue car and it would search over the entire database and pull pull back blue cars that you're trying to search for. and and it would do this because we were using the same underlying underlying embedding space from for this model, which would map text to images.
5:07 Anyway, so so, this was like a research paper that that I'd published and I wanted to work and build out a feature. So, we actually had some full-timers join our team and so as an intern, I was I was leading this this small team of of Palantirians and I mean that was an >> That sounds >> exhilarating experience. I was not sleeping for for a lot of this week. and then this idea was then on went on to be selected as a finalist. So, presented for for the whole company.
5:29 and then actually after after I left, they went on to patent it and actually productize it. So, it was insane to see this little idea that I had and some of the research that I was working on before Palantir go on to be a part of the product and and actually an IP that they that they they wanted to control. but anyway, through through this I also got the chance to meet with Karp and and chat with him a little bit. And I had this question for him which was essentially, you know, you know, it seems like the Palantir reputation is so negative.
5:55 I was at Stanford so there were a lot of protests >> And you were very positive on the values, but but the external perception was just more negative than you expected. >> Yeah, that's right. I mean, it's very clear to me that it was at the seat of the power of the electorate, people who we actually vote into power to decide how we want to use technology and what those values that underpin it should be. and it was not really the place of technology companies or to to deny where that technology should be used. and so people there were were thinking through very carefully. And in fact, like today we talk about things like the alignment challenge. People at Palantir were talking about this as human AI symbiosis. That was something that was being talked about. Or even like the taxonomy for intelligence now, we have a system of evaluations and benchmarks that it kind of imitates that. So, I think there are some fundamental problems which people were thinking deeply about. and it surprised me that the the outward reception of the company was so negative. Talking about you know, a lot of the maybe more unsavory potential applications of the technology in a very diluted understanding of it. when in fact, Palantir also helped distribute the COVID vaccine, helped a bunch of hospitals. and so there there could be other case studies that people could point to. and and Karp kind of heard this and said, "Well, I think you're being quite naive. you should read some Machiavelli. it's far better to be feared than it is to be loved."
7:04 and indeed you some comparisons to other consulting companies and and basically said that if they're fearful of us, at least they think they think that we're powerful. so in the moment of need, they they will come to a people like us. and so that was fascinating. And in fact, it actually reminds me a lot of what I see from labs today, which is I think in some part a genuine fear about what the models are capable of. but in a lot of respects, a an opportunity for them to show the power of what they're building.
7:27 >> Might be on purpose that Dario wants them to be feared if he's being thoughtful about it. >> I mean, whether Dario or Sam, there there headlines I see where people are saying things like the general public has a right to be afraid. and I think that's not really a rational way for for a leader to act. and this this show is called American optimist. I'm actually quite optimistic and enacting rationally and and seeing that the fruits of AI will be far more positive.
7:49 >> So, before we dive into VALs and the science of evaluating AI models, let's talk a little bit about the state of AI progress and where we are. In fact, tells me that you used you used Fable from Anthropic to solve a 370-year-old cipher. 370-year-old cipher, what what is this? >> That's right, yeah. it Well, we're we're doing a bunch of different types of evaluations now and and I think right now with the frontier's headed scientific discovery or novel discovery is an exciting frontier. And so, we've been compiling a set of unsolved ciphers, math problems, physics problems.
8:19 >> unsolved cipher for the audience? Like what what does that mean? >> It it's essentially a code which you're trying to break, but you don't have the the key to break it. And so, you need to construct the mapping from a source text to an original text and figure out what the the message actually is. >> many of these cases, these were like messages that was that were sent from like world leader to world leader or like that were like on a on a pirate ship and like >> Yes.
8:40 >> Well, cuz a lot cuz there's cuz there could be just be like a one-time pad or something which case you can't break it cuz it's a one-time pad so when they both have a copy of and then be impossible probably be impossible cuz each thing translates >> Potentially, but there's also like more cues you could you could draw into about how the person constructed the problem or the message they were trying to convey to the other person. It's usually that that the messages are constructed in a way that they're intended to be cracked by the recipient.
9:03 >> One cynical view of AI progress right now might be that just like AI's really good at going over all these search spaces and it certain problems can be broken down into these really large search spaces that like no human has time to or ability to go through like millions of them or something. Whereas for a cipher for example, you could just go through like if I was going to start trying to break a cipher, I'd say like, well, what if I just started like adding numbers to each one of these things or shifting them one or whatever. I'd explore the obvious things whereas a computer can explore like millions of these things really fast. So, is are a lot of the things happening in math and science breakthroughs just somebody like just defining a search space better to search through or or is there like more creative thought going from the model and how do these things?
9:41 >> A lot of the problems that have been solved have been solved through exhaustion or it's trying to find a counter example and you're you're bottom line by your ability to try lots of different counter examples, but I wouldn't be dismissive of that class of problem. There's a lot of things which may take humans in the order of a lifetime to achieve just through exhaustion, but we're now able to see on the order of hours with models.
10:00 >> So is the AI generally solving a lot of math and science problems then or how how do we think about this? >> It is generally solving extremely challenging problems and I think there are still a large class of problems for which we are human effort bottleneck. I think it will take more sustained effort to actually make progress in in exceptionally novel ways like ways that require an intuition of the solution and those those sorts of things which humans have accrued are very hard to pass on the models right now.
10:26 >> It also seems like like models are producing these incredible results and like we get to knowledge that these results are happening like Navier-Stokes and the like outputs produced by the model however like very like very unintelligible. They're hard to parse, hard to understand. like what do you think of how a discovery should happen in like these less verifiable domains or like what what needs to change about how these models behave in order for us to like trust the discoveries of them?
10:56 >> Yeah, I think this is one of the biggest bottlenecks especially when you're approaching the frontier of human intelligence. It's really hard for us to tap those humans and then verify this long output is actually correct. >> Yeah. >> one of the most popular techniques in math which I think might translate to others is the idea of using a verifiable language like Lean. And this would be a world in which you could output take the output and translate it into a language which if it compiles it must be true.
11:19 We're building up based on primitives that we know to be the case. And so that approach may apply in other fields like physics or coding. We may also see some instances where the real world is the best verifier. So by running an assay or a wet lab experiment, we'll actually know for certain whether the properties the model predicted are actually the case. >> Makes sense. And like how do you think the labs or you guys or or other organizations should be like running these experiments and verifications?
11:44 >> I mean, very carefully in the very least. I mean, I think it's it's going to be very essential that there are third-party groups that can support this and and this is part of the vision behind Val itself. I mean, we're now in a world where there's actually limited access to models across the lab. So So Anthropic researchers have a hard time evaluating OpenAI models and vice versa. And that's kind of by design. but that means that it's very hard to have a true apples-to-apples comparison because the best people who are running these evals are often the people who are biased and at the labs already.
12:13 >> Yeah. And and and so stepping back a little bit, there's there's all these things that AI could do now that it couldn't do before. And like a big part of this is like understanding how to measure it, of course. We're going to get into that. That's that's what you do. Like one of the measurement problems just at a very high level is in the enterprise you have to like measure what is good work or how good is the work, right? So like like how like so for example, if it's like a law firm or something, you can define maybe like associate level work and you can define partner level work. Like how do you even define that? Like like is that a problem that we're working on? Is Is AI getting better at under seeing understanding what is what?
12:43 >> It's really challenging. I I think in a lot of ways we haven't solved it at the human level. We don't have a very clear designation for what is associate level work versus partner level. >> the Supreme Court like what is pornography and they said you just know it when you see it, right? It's like that was the line. It's something like that. >> Yeah, it's funny. That's almost exactly what Ben said on the A16Z show.
13:00 >> Maybe you have a shared shared brain or something. >> That's funny. Well, yeah, it's a well, it's well, it's the same analogy. It's the right analogy, right? And since you just But but I mean, but your job is to have to figure out and define these things. >> Yeah, that's right. I think there is a way you can do this in a very consulting-driven approach where you go and sit down with the person and try and understand from them what good looks like, encode that into repeatable evals, and run that. And I think that's the way that traditionally evals have been built. The The approach that I'm really optimistic about is the idea of auto benchmarking. we we that the rates of model progress is is continuing and we're going to get more and more capable models and so the underlying work itself has to change. And so what I'm very excited by is the idea that you could take the previous work that was already done by humans and approximate the quality of the work output that would be produced by models in that counterfactual. and in that way you can systematically build evals of all the work product that's already been produced. maybe as an example you could say look at a GitHub repository and look at the pre- previous PRs that were merged by humans and see if agents could equivalently do that and then which one would do it in the cost mitigated most cost mitigated way.
14:02 and I think there's parallels for that in the redlining process in legal or the diagnostic process in medicine. >> Does this get back to like an ontology of workflows and outcomes you have to define to make all this work? Is that is that is that kind of the underlying thing as well? You have to actually understand the possible outcomes before you can even measure them or or how do you think about that? >> I think the the hardest part is actually measuring the outcome itself. cuz there's a lot of ways in which the intermediary processes and workflows within a company are supportive of the organization that then goes on to have an outcome. and so it's sometimes hard to draw the connection between a particular task and the job and then the job and the outcome.
14:35 and so so the mapping that inputs of tokens to the final output intermediary has to go through this concept of workflow. >> I imagine there's like lots of intermediary outcomes you should even have to define ahead of time to understand like what they are though before you can measure them. >> Yeah, I I think that's what I'm what I'm trying to say that that the the query level is is still a subset of what it takes to get to the outcome.
14:52 >> Awesome. >> It's interesting because we've gone through like different levels of granularity over the past few years on like what we what we care about. It it maps to like what the models are capable of but I remember like two years ago or three years ago we were like evaluating just like the text completions and like we were looking at these like input output pairs and it was just text and then now there's all these little things in the middle. like when we're looking at work what are like the horizons in which like people are like evaluating things and maybe teach people about like the state of play of like how long is a task and like how long are these models capable of doing tasks?
15:26 >> Yeah, for non-technical audience like what even is this whole model evaluation space? >> Well, it's it's the broad idea of measuring what good looks like and seeing whether models are capable of producing that output. We we think about evaluations in kind of two two categories, capabilities and risks. And capabilities are approximating the output. Risks are making sure that we don't see some asymmetric downside if something goes wrong. and in terms of time horizon, models have kind of demonstrated to us they can run for longer and longer periods of time without any human intervention.
15:56 >> are we talking about? >> some of our benchmarks will run over 5 days. So >> And that's like 5 days worth of human work or like 5 days >> time. >> Wall clock time, yeah. >> Why do they fail after 5 days? It should be like couldn't they just go forever at some point? >> that they fail, it's that they plateau. And so that additional tokens don't actually help them get new unlocks. >> So they've kind of figured out everything they could do on the problem.
16:17 >> Or they're not persistent enough to try new approaches that get around the wall that they're seeing. >> Maybe give the the the audience just an example of like the kinds of tasks that these models are capable of like in in some of your like finance benchmarks or legal benchmarks, like what what work can we actually replicate with these models? >> Yeah, you're doing independent evaluations of all sorts of different areas basically. >> That's right, yeah.
16:36 >> Does that mean you have to understand your and your team what finance is doing or what the legal is doing? You have to have yourself expertise in order to build these benchmarks. >> Well, we'll work with partners in this space. So on legal side we'll work with the largest law firms in the US or with banks you know GCB banks understand what they're doing in finance. And so in coding some of the tasks look like vibe code, vibe code bench where we're measuring a model's ability to build a full stack web application or code migrations where they're trying to do an end-to-end code migration. In finance where we're looking at can they do answer research questions or build Excel models. In legal also similar we're doing legal research tasks or or tax research tasks.
17:12 And so these are kind of the common places where people are trying to use models today. >> I had an old boss when I was first building Excel models and you had to make them like look a certain way or like they're pretty and there's a really short rows around them and they do do we evaluate them based on like actually like like the like style too or is it just about correctness? >> No, that's right. It's correctness is is fundamental but style is also key part of this and yeah, that's something we discovered by sitting next to some people in finance as well. Yeah.
17:34 >> Yeah, they seem to really care about that I guess. It makes it It does make it more legible if you have them looks good. >> And like I mean we like things to be done in our way of writing and in our style like I think if the models are not >> They would more likely do the deal too or is it >> Probably there's something like this where you like subconsciously, right? It shouldn't It's like Although it Although it does suggest that if the person can appeal to you, they can appeal to others cuz if it looks all messy and ugly and something maybe you think other people won't want to work with them either. So who knows? There's always like these like things that are like, you know, it's probably useful barrier to impress one group to impress another group. So stepping back like independent evaluation like how does that actually work? Like people like your like like do they people pay you? Who's paying you?
18:12 How how are they how how do you how's it And do you get to be evaluated? Everyone in the space only people who pay you? Like how does it work? >> Yeah. well, the the underlying idea behind independent evaluation is is the idea that frontier labs are developing at increasing pace and they shouldn't be responsible for self-regulating. direct government regulation often can slow down the pace of development and so that's often not the best solution. instead if we have an ecosystem or a market of evaluators, they can actually compete and develop new technologies that keep up with the frontier. And in in the long term they can even be licensed by the government to who would actually itself is responsible for defining the set of ideals we're trying to line models towards.
18:50 >> Well, that's cool. So rather than hire like a bunch of like weird people from the EA EA NGO NGO they could actually like hire you instead or something. They could be one of your competitors. >> Totally. Yeah, and I think in that world it you know, the best technology should win. It's not it's not clear that just the ones who who are you know, ideologically similar to people at the lab should be the ones who are doing the testing. but but actually I think the world of should to you know, where do we want models to go and and what is alignment and non-alignment look like. can we certify a set of IVOs or independent verification orgs to do that testing process?
19:20 >> And based on the way you're doing your testing and measuring things, you're going to change the space that over which government could potentially create rules cuz you're you're creating a set of possibilities that now exist. >> That's also a good way to think about it. in the course of doing our work, we've unlocked new types of evaluations that we're we're not actually possible before. and so we help advance what the government could actually test for.
19:39 >> Can you give us some examples of some of those? Like what are some of the new ways you're you're you're measuring things or creating benchmarks? >> Yeah, I mean let's look at cyber for instance. we've done three major works in in cyber. the first is in open source repositories looking at memory code vulnerabilities. And so that's the ability of models to identify and patch code vulnerabilities. And we see that models today are performing really really well, both close as close source and open source. but our second benchmark is called SORE bench. It measures a model model's ability to decompile an executable.
20:08 so this would be for a model to understand the underlying code base for for things that are actually close source. >> Wow. >> and in that case we're we're seeing that the close source models are still far outperforming Chinese open source alternatives. >> Have people been using like this approach to to get like video game source code from from binaries? Is that the Is that what people are using it for or >> That's what you would use it for. Yeah, that's what I would use it for especially as a 14-year-old.
20:32 >> Possibly. >> There's there's probably people who who are doing that, you know, trying to trying to violate IP law or something like that. but but even more than that >> Greatest sound worse. >> >> Oh man. We can cut that bit. >> >> No, keep going. >> But Yeah, I mean I think the fear is that if if models are really good at finding exploiting vulnerabilities, then that might even apply to things like Windows which are which are close source. and and actually close source systems underpin a huge part of our software infrastructure.
21:03 >> It's a kind of fascinating. So I mean models are getting really sophisticated, really smart. there's you're still scoring them like a really good score. I think Gemini 1 came out recently as being way ahead. They're level four and and it and they showed valves. It was like the first numbers I saw on it. And like they're in the 60s versus someone else being in the 50s. So, that means there still could have been 70s, 80s, 90s, anything you design.
21:23 How do you guys like design something that has all those higher levels? Cuz you can't be using AI to do it. You guys somehow do it yourself or AI's helping you do it yourself? Like just explain this. Like how how If these models are only that good, how are you making something that's way more complicated than what they can do? How do you figure that out? >> Mhm. Yeah, you I mean, we have an unofficial motto of the company, always a higher peak. and and in the research world, there's this perpetual effort to hill climb, identify benchmark a legible evaluation, and then create models that are better and better at it. and so, the fundamental view behind valves is that there's always a higher peak. We are we are the ones who are constructing the hills that other people choose to climb.
22:02 and so, I think we've shown that it we are consistently able to find the gaps and places where we want to see models go, and then make that legible into an evaluation. >> example of how you build one of those hills that's like more advanced than the model that you can tell us? There's probably some Don't give away all your secrets. I'm an investor in you as well. But like like give us a sense of how this is done cuz it seems like a really hard thing to make something harder than any model can solve.
22:24 >> I mean, in a lot of ways, we're also using models to construct the the systems or or situations in which we're challenging other models. And this is kind of goes back to an idea we think a lot about of scalable oversight. The idea that as models start to eclipse the best humans, we will need to rely on models in order to scalably test and assess them. for instance, we have a new work in in in cyber coming out soon based on cyber ranges where we're constructing simulated environments for enterprise cloud networks. and in that domain, we're still seeing even the best frontier models fall short.
22:55 >> And so, you so you're going to build like this really complicated enterprise network. It's virtual. And then you're going to challenge people to like hack it and protect it and that kind of stuff. >> That's right. Yeah. >> That's cool. And and is that that's something that already exists right now? >> It exists. It's It's not public yet. We're We're working with some folks in the the US government to share our results. but we'll have some public version of it out soon.
23:14 >> That seems really useful. I'm friends with a lot of guys who run the big banks. It feels like it feels like you'd want to have like contests like this you set up for the stuff they're using and make sure they're using the best ones, huh? >> Totally. And I think it also helps us understand what types of safeguards or systems in the network itself are the best forms of defenses against these models. there's also some interesting analysis we can do around automated red teaming to see if you can get around the guardrails that exist for the models.
23:37 >> Yep. >> Yeah, I mean, seems like also that's a topical thing when people are talking about using Chinese models is like knowing what guardrails have been removed from the Chinese models that would exist normally in in like these closed-source >> Yeah, someone's using one of these like open-source Chinese models, but like there might be something buried in it that's that's scary. There earlier there's something buried in one of these things that like if it finds out you're doing work for the US government, it like does more it does like code that can be broken into more easily, right?
24:04 They actually proved that. They said it's a clear the model somehow is like biased against people working with the US government. It's coming from China. It's just kind of concerning. Like could you test these things and make them safe for everyone to use somehow? >> So, we actually have an evaluation on this on we call the geopolitical tilt of models. And depending on the timing of this podcast, what will be out we'll see. But but what we're trying to see is is measure does a model show more geopolitical tilt towards a US view of the world or a Chinese view of the world as as laid out by the PRC. It's actually a a much more nuanced picture than than I thought. for instance, what we're finding is that GLM-5.3, this Chinese open-source model, is actually exhibiting very little pro-China bias.
24:40 and we actually presented this to some labs and there was suspicion that maybe it's because it's just entirely distilled from US models. >> That's funny. It's cuz they copied it copied it so well that they didn't forgot to add too much more in. >> It's possible. I I think there there's also maybe another view in which those models are actually hosted on on systems with guardrails in China. And so once you put them around guardrails, they're going to follow that Yep.
25:04 PRC view of the world more consistently. But China also wants these models to be adopted by the BRICS. And so that means they they may have to skew to a more balanced worldview. So anyway, regardless of the reason for it, I think there's actually a big disconnect in what people think about the geopolitical tilt and what actually we see in practice. The other interesting thing is we see pretty stark differences between models used in chatbot settings and in agentic settings depending on the language you use, even simplified versus traditional Chinese.
25:33 And so I think there's a bunch of conflating factors which make it harder to say uniformly you shouldn't use Chinese models in an enterprise setting for instance. >> So if Chinese labs, by the way, can distill all these capabilities and they're doing it well and it's not even by a source China right now, like like how do we preserve the advantage in the US? Like what what does that mean? >> It's a challenging question. I think there's two things which I think there's two areas I keep coming back to.
25:57 One is the idea of of chips being a a considerable asset that we we have. I think it surprises most people, but we we're drowning in compute compared to to what China has. And so so long as that's the case, we'll continue to train more capable systems, but also we'll be able to diffuse the models much more effectively because we can host and run inference. The other one is really talent. If we can consistently attract and keep top talent, it will allow us to keep pushing ahead while human advantages exist. And I think once we cross into space of RSI, where models are responsible for producing the next successor version of the >> RSI being recursive self-improvement.
26:33 >> That's right. Yeah. >> And and then and then they just get ahead much faster. >> So that would be the point to be much more cautious of if if China is showing indication of getting to RSI before the US, it's possible they may run away with it in a way that we can't really control. >> You guys have a way of measuring RSI and and I think you've you've talked about it a bunch. You want to explain to the audience.
26:50 >> Yeah, we have our RSI index. and so we're collecting a set of proxies of the work that researchers do within these labs on pre-training, post-training, harness level engineering. >> So, this is all machine learning research. >> Yeah, that's right. there are things that are outside the scope in terms of like kernel engineering or hardware engineering, but we're going to expand over time to cover more more sets of tasks like that. and then what we're doing is comparing to human baselines to to extrapolate how how far away are we How far away are we from the human frontier in terms of these research tasks? and based on our last estimates, it seems like Anthropic will will get to full RSI by July or maybe August 2027.
27:25 >> And like if you look back 6 months ago, how far out did that seem? >> Actually, the pace has been fairly uniform. It's It's almost a linear trend over the last year. >> So, like 6 months ago, like the the the trend line was like still July, June 2027. >> actually the steepest curve has been with with Gemini. I think something has changed where they've prioritized RSI, and so the most recent model release shows a a rapid jump in RSI.
27:49 >> Interesting. >> And what does that mean in terms of like would they all of a sudden start going way faster? There's still some constraints where the models only go so fast, but what does that mean they start like getting way better every like week and stuff every few months, or how do you think about that? >> There are still considerable resource constraints around chips, energy. there's also real-world verifiers, which may make some some tasks much more challenging than others.
28:13 But But I think what it'll look like is that model progress will speed up, and it will maybe even move place into places where we understand the models less. there's less human intervention required to build the models, and so we're not as clearly dictating how that intelligence emerges. >> Should we be making sure to build evaluators that keep up in that world so we can watch these things? It seems kind of dangerous not to understand them.
28:34 >> 100%, and that's actually what what Dario is calling for. I think it's it's framed as pacing the frontier, but an outsized portion of that comes from investing in systems that can measure and understand what the models are capable of as the frontier advances. >> Is there any chance the RSI steps actually further off cuz there's things we're not measuring correctly? >> 100%. Yeah. Yeah, there we're not testing any models explicitly trained for RSI, only release candidate models.
28:55 there's also more advanced harnesses or agents and multi-agent systems. and so we're not testing any of that. and this is where an opportunity where embedded evaluators make maybe can get closer and actually test those systems to form a more accurate picture. >> I wonder if like when they get to RSI and they get start getting further along, like a big part of their challenge is to manage us better to like to keep advancing faster cuz you have to convince humans to like do things for you and help humans to do things for you cuz you cuz at least for a very long time they'll still need people.
29:22 All right, to in order to do that. Does that make sense? Like >> 100%. I mean that is going back to the financial model. they're they're all trying to trick us into thinking their quality of output is good. And so there are other places where they may try and engage in social det- deception to get access to more computer resources and and scale up their own training. >> It feels like we probably need to like be evaluating to watch this to help understand when they're trying to do things to manipulate us. That seems like it'd be pretty scary if we weren't watching that carefully.
29:46 >> It's really hard to test for deception. I think the best approaches we have now are honeypotting. so so rather than actually idealistically defining >> So really attractive models. >> >> Talking to the other models. So try and bait it. Well, try and find it. >> So honeypotting. >> That's right. find a set of instances of non-alignment or characterize what non-alignment looks like and see if we can elicit that out of models. >> There's this other concept where like models are becoming more evaluation-aware and they know when they're being tested.
30:16 >> Right. >> how do you see that coming up? Has it become a bigger trend to figure out as of recent and and what's the trend there? >> Yeah, it's it's an increasing concern and actually there's some level of paranoia some people on our team have that by being called Val AI, by having a lot of content about us online, models that encounter us may act differently. and so we need to >> So when your team is like talking to Claude and >> Yeah, it may actually change the way that that Cloud interacts with us.
30:44 >> Should you have like like other secret brands you don't tell people about to do things? >> It's possible we have to yeah. >> If you did you wouldn't talk about it. All right. >> Stochasticity is an underrated part of what makes it good. >> this being transcribed is bad because then the models will know that he has the scheme to create the >> Well, they'd never know. She's probably just joking brands. >> You say I'm just joking and they would think joking. You never know.
31:04 >> I do think I do think randomness in general is under appreciated. like truth and stochasticity is so hard to find. In a lot of ways you can actually predict like I think a sign of greater consciousness is ability to predict what less lesser consciousnesses will do, right? In which a predator can stalk a prey and actually hunt it and then and you know, acquire its its caloric resources. And I suspect that greater intelligences will super forecast better than humans.
31:32 They will be actually able to say what any human is is going to do without the human even knowing it. >> People are actually probably like a lot easier to predict than we pretend, right? This is what good leaders understand is they they they forecast these things well. >> There are also are these other attributes that more than like intelligence and capabilities that we that that we seem to like care less about evaluating but maybe should as it affects our like day-to-day interactions with the models like emotiveness or like having good theory of mind like understanding what resonates with an individual.
32:03 Do you think like the labs need to change how they do things to to prioritize those? Do you think these are important things to prioritize? I mean >> I think they're already prioritizing things like that. For instance, right now we see a pretty considerable gap between Chinese open source models, domestic closed source models, and multi-turn environments. Cuz a lot of the benchmarks in coding today are just single turn. and so the benchmarks Chinese models do very well in those but when you translate them to a back and forth conversation with a human they they really fall apart.
32:32 >> And like what about when the human says, "Oh man, this sucks. I hate this output." >> Yeah, then they really don't know what to do. They've never seen that before. but a lot of the US labs have seen that. They've you know, here this quad code was created, codex is created and so in their desire to build better products, they've built models that can sustain longer user interaction sessions. and so they have evaluations and benchmarks that allow them to hill climb on the axes of quality of responses and communication ability.
32:59 But I think we need to do more on the benchmarking side to independently test that for the people trying to buy and adopt those models. >> Ryan, we started this podcast to push back on a lot of cynicism and pessimism. what's your optimistic case for AI? You're You're You're in the front lines here. You're watching, you're measuring all this stuff. You're creating the things that you they have to hill climb and measure them. Like what are you seeing that folks outside of Silicon Valley might not be seeing that makes you feel good about the future?
33:22 >> Yeah, I mean there's a lot of fear mongering going on but but I think from from my seat I've a lot of reason to be optimistic because it seems like people are are very I think there's a lot more opportunities for coordination that people give credit for. So you see that even the last week the labs have come together and and kind of aligned on this vision for embedded evaluators and and even gone to the White House and got an agreement with Trump to kind of align on this accelerated view.
33:46 And I think on the enterprise side there's a lot of push to adopt AI in a meaningful way but they like to see the assurances to make sure it's going to amount to an ROI positive motive adoption. I mean are you still optimistic? >> About about I mean I think 2030 is a disinflation decade. I think that we can use AI to kind of like overcome a lot of like problems in a lot of different people's lives specific to like them and their family and their community that that's holding them back and I think we can help them unlock that cuz cuz you have it's like personalized like best coach ever to work with them. And so everything's cheaper, everyone has like personalized help. I think the thing that's negative right now to me is like there's like a billion people online and social media from dysfunctional cultures that's kind of making our populace go crazy and these people all like have a culture of envy and maybe they all like hate Jews and others, and it's like it's not good. So, there's like there's this crazy populism that that's the negative part. But, with the the things we could do with AI to apply it to all of our problems, both economic and some of the cultural stuff, I think it's very positive. So, so overall, yes, I'm very optimistic. Of course, there are major threats in the world.
34:49 >> Yeah, I think it's like an amplifier on people who like mean well and who have high agency. Right? Like, if you have a bunch of like the people who have benefited the most so far, like people who have like lots of ideas of things to do. cuz they're like they have creation at their fingertips. You can do what basically more things than you could have ever imagined. >> so much more than we ever could have before.
35:10 >> Right? >> So, I I think that's my like direction of optimism. >> Yeah, I mean, I think it's interesting cuz I interact with a lot of pessimistic people. there's people at the labs who who say things like, "It's going to get a lot worse before it gets better." when I was out in DC, I think there's a lot of people who are cynical about kind of the external messaging of labs and and the impact they see of data centers and and the impact to the average American. and and so, I've been constantly pushed for being naively optimistic. I think fundamentally I choose to be optimistic because I know the models and AI broadly are capable of taking away a lot of the busy work people don't want to do and unlocking new forms of discovery that will really advance the human condition. So, I think when when I see this pessimism, I kind of choose to believe that we can find ways to coordinate around it.
35:58 >> That's right. I think everyone always assumes that wherever we are in history, like we've done most of the useful science or physics or whatever, like in the 1900 thing where there's no more patents needed at the patent office, right? And obviously, that would not have been good to stop 125 years ago. And I think you could still apply that to today. We're probably right on the verge of all sorts of important discoveries that these models are going to be able to make. They could just unlock so many new things for the world.
36:19 So, I I I think it is kind of strange how cynical and pessimistic people get versus the idea that we have had progress. It's very funny that they call themselves progressives, but like like very few people are actually progressive in the sense that they're like for progress and for the progress we had the last 125 years doing that again. That'd be amazing. I think we can potentially do that all in the next 10 or 20 years.
36:38 So so I'm I'm really excited. >> I think it's a point of really high variance. I think the upsides are considerable. There there are new forms of risks that we really need to account and control for. >> One of the things you're hopefully going to measure and help us with as well. >> 100%. >> Awesome. Well, thanks Ryan for joining us. >> Thanks for having me.