Transcript
0:00 GDPX taking the world by storm. >> Something like that. >> Okay, nice job, guys. Your your benchmark matches your what what did you say? >> Our subjective impression of the models. >> Yes. You're pleasantly surprised that the benchmark matches your subjective impression of the model. >> Yeah. Because because to be clear, right, we actually had no idea what the final leaderboard was going to look like until about two hours before publishing, maybe four hours, right? Because what we did is like we had all the partners difficulty calibrate on four models like they ran Astra, they ran Opus, they ran 3.8 8 flash and then some Argon because we were kind of working on the benchmark sort of before Argon came out and then also GLM 5.3. And so we only had results for those four models up until like 3 four hours before the benchmark actually launched. And then Andrew, you know, switched to polyphasic sleep over the past week so he could build all the infra. You know, he set up a thing so that Claude would wake him up if there's like something that broke and he needed to like, you know, provide input. but then he like set up all the infra started running on the rollouts. You know, these are incredibly long like rollouts, right? sort of doing the task plus grading could easily take like 24 hours. And so we didn't have all the final results until literally, you know, 3 or four hours before we published.
1:35 when we saw his leaderboard, we're like, "Yo, I think you might have, dude." >> Some of these models like some of these models like we haven't used like or even seen like out in the wild. Like carbon like we got access to carbon on like Thursday night or something like we haven't used it for anything and it was like the first time we're trying it out. it did well. It was it was surprising. And same with Muse Wonderful. Like Muse Wonderful is not available in the API.
2:02 We got access to it. It was did great on Cyber. >> Yeah. When did you fix the tasks? When when did those get finalized? >> like Thursday night. So we launched on Sunday. The tasks were finalized like Thursday night because we were doing a lot of sort of back and forth iteration with our partners where like our Q would flag something and be like, "Hey, take a look at this." And then they'd be like, "Oh, like these two are false positives, but actually this one's for real. So we got updated blah blah blah blah blah." And it was really really last minute. Like I'm on I'm I'm very happy that we got it out in time.
2:39 It was very hard though. I'm also glad to be taking a break now. >> but yeah, it was a crazy process. >> I think the work starts now, man. I don't know. >> The work Okay, I'm going to take a I'm going to take a couple of chill days, you know. >> Yeah. >> And then and then we can go through the process of Yeah. I guess kind of reaping the rewards of our effort, >> maintaining the dashboard as new models gets released. Did you did you impose this deadline on yourself? What what was the motivation for that?
3:10 >> It was mostly self-imposed. I mean, we were we're decently high confidence that, you know, a new OpenAI and Enthropic flagship are probably both coming out this week. And you obviously cannot launch your benchmark on the same day as like Fable 55. and >> especially if your benchmark doesn't have it. >> Yeah, I think we were like I think we were decently confident that like Monday would not be the release date for these models. Maybe it's like Tuesday or Thursday or something, but if there's like any risk of the model coming out on Monday, we were like, "Fuck it. You got to we're just going to push Sunday instead." And then there's some fear that like, okay, Sunday night, you know, maybe people aren't as online, so you're not going to sort of have the varietality component, but I think we were very happy with how the launch turned out. So, >> yeah, generally generally happy.
4:02 >> Take a step back, explain GDPX from scratch for people who maybe haven't seen the article or just hearing about it for the first time or >> didn't read it as many people who were commenting about it. >> Yeah. Yeah, I think maybe going taking like sort of a step all the way back, I think it's important to understand that a lot of the really popular benchmarks out there are kind of trash. this is largely because they're made by people who don't have expertise and like making eval environments or sort of train data in general. And it's like very easy for you to make a task that kind of seems good on a surface level, but it's actually unfair in some way such that it's a poor test of model capabilities. yeah, a lot of people make their tasks harder by making it unfair rather than actually making the task like difficult. And and it's like very hard to figure this out unless you run like 10 rollouts, look at the transcripts and see why the model fail and determine like manually like was this actually like a legitimate failure.
5:15 >> Yeah. Can you explain the distinction between difficult and impossible here? Like you did this in the article. It has to do with both the prompt and the greater. >> Yeah, I think maybe impossible is not quite the right word. It's more like difficult versus unfair. I think one classic example of a task would be unfair is if you have like a coding problem that asks them all to fix some bug maybe in some open source repository and then you could have sort of an expert human software engineer look at it and they would say like oh there are sort of two ways to fix this bug. You can either use like module A or module B and I using my human expertise recognize that both are kind of equally fair equally valid. yeah you should be able to pick whichever one you want.
5:57 but when we're making these benchmarks, you also need to include a verifier for each task so you can check the model succeeded or not. And it's totally possible that someone might make a verifier that's sort of scoped to just the module A solution. And then when sort of AI comes and tries the problem, maybe it will sort of rightfully pick module B instead of module A some of the time and then the verifier would mark that as a failure. and sort of you know, the the AI would perform worse on the task. I think this is an example of the task being unfair because it's like the model did technically solve the problem. It's just that your greater is not faithful to the prompt. and it turns out there are actually lots like most popular benchmarks out there like a terminal bench 4 or a sweet bench pro or a deep suite or stuff like that.
6:46 they actually have tons of issues of this type that people have kind of funny enough started really noticing over the past like month or so. There've been a couple kind of big blog posts on this because it actually just it's actually a lot of data companies who were sort of just have agents look at all the transcripts look at all their prompts sort of run the task for their own QA process and they see like oh actually 60% of the Turtle Bench 4 tasks are broken or whatever. The funny thing is like I feel like everybody knew it. Like all the labs definitely know it. Anyone who looks at data like definitely knew that the benchmarks are broken. It's just like no one really like spends the time to like write about it. because again like to like most people in the industry it's like not it's not new information. It's like unsurprising.
7:29 >> Yeah. So so both benchmarks are bad. That's sort of one thing to know. And then we assemble analysis like we knew we wanted to make a benchmark ourselves because it's kind of the only way you can consistently say like interesting things about new model releases. because the team is not going to sort of spend meaningful time and effort trying out I don't know the new MiniAX model when it comes out to see if it's actually good or not. No, no disrespect to Miniax, but like you we're just not going to do that. and so and so you really want an eb and we're not going to do like like no one tested, you know, GPG 6.1 Terra either, right?
8:05 but if you have sort of a really high quality eBay that you can norally do models on, it can go a long way in sort of tracking the performance there. >> but at the same time, like we also recognized that a lot of public benchmarks are bad sort of not because of lack of effort on people's parts. It is actually also just really hard to make like a task that is fair but also sufficiently difficult and also interesting and reflective of like real work. And it would have been sort of very naive of us to think that like oh semi analysis is going to be able to come up with like 200 really high quality tasks ourselves. and so the strategy we took is we partnered with 14 different sort of specialized data companies >> who used like a lot of their expertise to actually supply the task for each domain. but we were responsible for sort of actually QAing all of them. we actually we like dropped more data providers than we actually included in the final benchmark because a lot of them just failed QA.
9:10 and also we were sort of very heavily involved in with like task iteration for the ones who passed QA as well as we kind of mentioned at the the start of the episode. That was sort of like I guess the philosophy/methodology for how we actually made the benchmark in terms of what the benchmark covers. this is obviously sort of another kind of big question we need to figure out because you can I mean you can you can kind of pick any domain right? AI is affecting everything these days. but one of the things that we sort of pretty strongly believe on the tokconomics team is that the first trillion dollars of AI revenue are going to be highly concentrated within a few domains that can actually absorb sort of increasingly smarter frontier intelligence and earn high ROI. I think a lot of people recently have seen sort of these token budgeting trends or people talking about how like oh you actually don't need frontier intelligence for majority of tasks and they view that as bearish for the AI trade. we sort of totally agree with the fact that you know most tasks a lot of tasks will not need frontier intelligence but we think there are going to be a lot of domains like chip design or biology or cyber security where we actually are just going to need sort of the increasingly smarter frontier model and you will continue to get high ROI from Fable 6, Fable 7, Fable 8, whatever. and so we decided to pick domains for our benchmark GPX which sort of match these domains that can absorb frontier intelligence.
10:50 and that was kind of our guiding philosophy there. >> Sweet. Okay. maybe we can talk through the specific domains in a little bit more detail. for those who don't know, there's a website up gdpx.smalysis.com. I'll share it on screen here for a sec. And as we scroll through the domains, recursive self-improvement or RSI, chip design, cyber security, biology, finance, legal, quant trading, hardware design, material science, alignment, and software engineering. I think this is not really a breakdown that a lot of people have on their tip of their tongue when they're talking about like the domains AI is coming. So even just introducing each of these one by one I think would be interesting. But maybe you can start by saying how did you arrive at these categories broadly and then how did you find the specific data vendors that you were going with in the given categories that you were focusing on?
11:46 >> Yeah. Yeah. So arriving at the domains was a lot of just like sort of brainstorming ourselves trying to pick the domains that are verifiable and also the ones that seem like they would have sort of the capacity to absorb a lot of intelligence. And after we sort of had our initial list, we kind of talked to a bunch of people at the labs people sort of working on similar problems. And then we we kind of did this process for maybe a few weeks until we finally arrived at our final seven domains. in terms of how we pick the partners, we're sort of fortunately decently well connected in the data space and we actually just happen to know sort of a lot of the environment companies and what each of their specialties are. so we reached out to sort of a bunch of people who work in the various domains. we got sample tasks from them. We ceueed them.
12:44 We sort of dropped the ones that are bad. In the cases where we had like more than one good person who wanted to be exclusive with us in a particular domain, we would tie break based on things like tastes and how interesting we thought the problems were and representative their domain. and that was yeah that was basically the whole process there. Eventually, people actually started like hearing about GDPX like on their own and started reaching out to us to be included in the eval which was kind of interesting. It kind of spread around like the data space.
13:14 >> Yeah. Yeah. Some of them were more opportunistic like alignment is a good example where >> it's not something you'd probably immediately reach for if you're trying to make a benchmark that measures agent capabilities. but I really do think it is like one of the most important things that you need to evaluate future models on. And so it sort of deserves its place in any comprehensive benchmark. at the same time, Fleet is sort of one of the biggest like most wellrespected data companies today. I have a few friends who work there and so we're when we were sort of brainstorming domains like that we could potentially include fleet 4 alignment ended up just being one that like made a ton of sense. yeah.
13:57 >> Yeah, makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like first of all these are clearly domains where people using Frontier AI today are burning a ton of tokens. >> Absolutely. The domains are really interesting though because like it's it varies so much like some of these are like very established data companies who've been selling data for like a few years now but then some of them like material science are just like one PhD who just like really really good at material science and they just like delivered like these insane tasks that are difficult enough for like even Astra and like I found that like super cool because they just use like papers that like they wrote or like papers that like people that they know like wrote so shout out for those. Like those those are pretty cool emails.
14:45 >> Yeah, my literally my favorite part of the article is just going through and reading about these pretty esoteric domains that just had so much there's just such a it was so clear by reading through the details of the task that the person who had written it felt to me like a domain expert. They certainly knew more than me. >> Mhm. And maybe you can talk a little bit about the reaction from the guys that you've worked with since the release of the article and how it seems like a lot of people were surprised by the results. Like I know Brian was specifically saying like he's shocked that some of the tasks were solved by the >> Yeah, I think Brian Brian had like a brief moment where like you know initially like he didn't actually expect like Astra to be so good at his tasks when we first like asked him for stuff.
15:28 He's like actually yeah some of these workflows that I initially thought Astro was not good on like once he sat down and made tasks he realized that actually no like I was just prompting it wrong or something or I didn't provide enough context and it just ends up solving. so you have to make like another set that was like actually difficult and frontier level because I think a lot of people don't realize how good the models are. and and he even said this like he said like this material science like don't know how good the models are and like they think that the models can do certain things but really they can't.
15:58 >> Well, you should talk about like the experience you had with software engineering at the start of the year and then how that relates to Brian. >> Yeah. >> Yeah. I think I went through a similar thing like it's actually a year ago at this point like >> you know after I left my job to start like an RL company with Max here like at first like you know I I was still of the opinion that like humans plus AI is just like a lot better and you know I can't really take software engineering job like we'll still be coding by hand like but like you know AI will automate the easy things but eventually just became like you know sonnet 37 was really really good sonnet 4 came out sonnet five and then Opus 46 and that Opus 46 was like a moment. Sounds like weird.
16:42 >> Opus 45 first one. >> Opus 45 is really good, but Opus 46 was also really really good. >> Like those two models were just like >> yeah like I think you know give it a few more iterations like there's there'll be no need for me to look at a lot of this code and it was like a very clear trend and I had to like reconcile that myself because I really like coding at the time. and that was like my thing. Like I was just like good at coding and systems. But yeah, I just had to like figure out like, oh, actually, you know, I have to take the higher level things now. Like this is no longer a job that like I'll always be doing like writing code by hand. though I still think like, you know, human plus AI is still better than AI alone just because like there's like little pieces of entropy that you can provide that the models don't have themselves. and so, you know, >> still valuable.
17:28 >> Yeah. But funny enough that the reason why I wanted Andrew to talk about that is >> I think Brian had a very similar moment for material science when he made the task for a benchmark because he was like, "Holy shit." You know, if if any human material scientist could do what like Astra just did, they're probably, you know, a top like they're definitely better than the media material scientists already. and like and like I just watched Asher do this and so I think we're probably going to sort of have these similar moments in lots of other in domains like chip design and material science and sort of the other GDPX domains in the imminent future.
18:12 and that's something that you also hope to be able to see sort of in actual benchmark results as we're able to evaluate you know Fable 55 Astra 61 62 or whatever. I think we got like a really good tweet. Someone asked about whether like you know GDP bal had human baselines for how long tasks take you know to like put things in context and this was really funny because like I responded to this and I was like actually like it doesn't even make sense to have human baselines anymore. like in the material science tasks like Astro solved a subtask of a task better than like a published state-of-the-art reference.
18:47 >> >> like when the models are like essentially past expert level, how do you even put a human baseline to that? And some of these chip design tasks are just so long horizon and like would take teams at companies to even like do and like those like there's no human baseline there. Like how do you even put a number on it? >> It's a lot of money. Whatever the hourly equivalent is in human work hours is a lot of money. And I just threw up on screen the the cost per task per for the material science works where the biggest ones like was at $40 or $50 per task and and many of the models are solving these tasks in some of them in $10 which is I think a pretty unbelievable statement.
19:34 >> maybe you can talk about like reading some of the rollouts in in more detail. Like what was what were some of the pieces of model behavior that surprised you the most? >> Yeah, I mean the models find very creative ways to solve the tasks and like this is again like going back to the whole unfair verifier thing. people don't expect the way that models solve tasks because models are not human. they just solve things in very weird ways like you know maybe use module B instead of module A for example but sometimes just like have completely different taste on like how things are done and sometimes it's correct and sometimes it's not and that's like that's why we have to like go through these transcripts and it's like hard to even go through them sometimes because you know we're not human experts but luckily you can benefit a lot from chatting with AI about certain rollouts to like figure out whether this stuff is correct or not. the models are actually so smart like on their own that with a with with human probing you can figure out whether something is like fair or not fair.
20:40 >> yeah, >> it's a very interesting experience for sure. >> Maybe we can talk about the results. I'm getting I'm realizing that I haven't even shared the like flagship leaderboard screenshot that's at the top of the article yet. you guys can talk about like what you learned about the models themselves. I think particularly people we've been going back and forth and and trying to explain the difference between like the flagship leaderboard across all domains and then compare it to the next one where you've got the leaderboard where you filter out biology, cyber security and alignment where safety classifiers are causing models that have better performance you know in the weights somewhere to not actually demonstrate it on the tasks in the benchmark.
21:28 Yeah. Yeah. I think that's a really good point worth emphasizing because we kind of went back and forth between which of those screenshots was the one that should have been the headline result. the one where you include all the domains versus the one where you exclude cyber, biology and alignment because those were the three domains where sometimes safety classifiers kicked in and therefore sort of sand back the score for some models like if you look at Astra on cyber for example the score is like 0% because it just refused to do the cyber task.
22:05 and ultimately we decided that the one including all domains is better but I think it actually is sort of fair to penalize overly strict safety classifiers to some degree. I mean I know you've experienced this yourself Jordan sort of just doing you know your day-to-day work at semi analysis and then you're hit with a cyber safety classifier from like opus rational and you're like well I guess got to use like GLM 5.3 now or whatever. No, Jordan Jordan was Jordan had an very interesting insight. He actually figured out that Muse Wonder is like a top tier cyber model independently and you were right like you were totally right like Muse Wonder and Muse Wonderful were just top tier for like what you can get out of the box for a cyber bottle.
22:50 >> Yeah. Yeah. >> That which was again a very surprising result for us because I guess Jordan already knew but we know we didn't have the Muse Wonderful number until >> two hours before launch. So that was pretty cool. And we were like, "Yeah, wow. He's wonderful. Just crushes it." >> Yeah, you guys you guys gave away all my alpha there. Look, using the models yourself for something that you think you're pushing the frontier on absolutely gives you insight into which models are good and which models are bad. But you have to actually have that work to do. And I had that work to do in one of the 11 domains we're talking about here.
23:24 >> I could Yeah, I could figure that out. I and I do think probably the biggest benefit of this benchmark is the ability to filter the results as you see fit and interpret them as it is relevant to you. people with Daybreak access or with CVP access from Anthropic which we have sometimes >> well there there's like three or four tiers and then we only had like you know the bottom or maybe like the second to like worst tier and actually that tier I think is more or less useless because you still get flagged doing basically everything and it's just I think they both have like very confusing sort of access programs for the malls today which is hopefully something they can like you know get better at.
24:10 >> Yeah. Yeah. >> The crazy thing is like you know Muse wonderful does like that well in cyber like I wonder like what red access on like Astra or do >> no one can use it. >> Yeah. >> Unless you work like I guess >> we'll get there. >> but okay maybe the the better question about this is just like maybe guys can emphasize again the fact that this benchmark is meant to measure the performance of the models at the frontiers rather than on day-to-day tasks.
24:41 >> Yeah. Yeah. For sure. Because again our our core thesis here is that day-to-day tasks just don't actually matter, right? >> they're more or less solved. >> Yeah. Like like I think I think most of them the majority of them are like more or less solved. I think heading into 27 you're not going to need frontier models for most of these. and we like not only do we think they're sort of poor task to use to measure frontier model capabilities. We also think it's probably going to sort of not be the majority of revenue for OpenAI and Enthropic and the other frontier model companies. And sort of because the thing that we're most interested in tracking with this benchmark is sort of a where frontier model companies sorry where frontier model capabilities are going and then b can you sort of like do any like TAM or actually revenue analysis based off of that you sort of very explicitly focus on just the frontier domains.
25:46 >> Yeah. Yeah. And like when we say like day-to-day tasks like that actually doesn't even exclude like some of the domains people want to use like these models for like it it was surprising to see how bad the models are in like finance and legal for how much people talk about like they will automate finance and legal cuz like when we looked at some of the tasks that we got from like living context and chakra like >> we went through those rolets ourselves cuz we were confused. I actually thought the models were better at these.
26:11 >> Yeah. Andrew is like a tech bro supremist and he was like, "Dude, >> the mall's definitely saturated finance already." That was his prior coming into this. >> Yeah. And we actually had like some of our finance friends look at some of these tasks and like when we were like choosing partners for these, you know, we had different >> rollouts and tasks and like some of them were like, "Yeah, these tasks are >> like pretty bad." Yeah, there there were some tasks from some potential finance partners where we were like whoa this is like not realistic of sort of actual finance work like and the only reason why it's sort of sufficiently difficult for you know Opus and Astra is because it's like lacks realism and it's also unfair in these ways and then after you correct sort of unfairness like oh look opus is like saturated >> and so that would make you think that these tasks are solved but then we got like the really high quality tasks >> and then like we had like our finance friends look at them and they're like, "Yeah, these are this is totally realistic of like a real thing I would have done in like banking and PE."
27:10 and the models just like failed this. >> Yeah. Actually, I want to talk more about the finance task in particular because I think it's probably the most understandable kind of domain to the average listener and I think this sort of be useful in giving people a sense of like why GDPX is such a high quality benchmark. sort of how people have historically made finance tasks in the past. If you look at something like GDP valve from open AAI is they would go to say a company like Merkore which has this vast like expert network of finance bros and be like you finance bro sit down for two hours and try to like come up with a task that's like representative of something that you would actually do in your job and then also give me sort of the golden solution for that task and then you know that's going to be the task that I'm going to give you about and obviously you sort of have a ton of sort of biases when you're telling someone like, "Hey, you know, make a task for my for my benchmark and just please try your best to make it realistic." Man, what like our finance partner did is they took a real private equity transaction where like some private equity firm bought a like family-owned snack company. I think they they paid like a little over 800 million for like 62% of the company.
28:33 they again they they sort of had this actual transaction like all 4,000 files sort of corresponding to it separated into the deal rooms for the private firm that was involved, the investment bank that was involved, the accounting firm that was involved and then also sort of the snack company's sort of own BDR because you know they were selling themselves >> and it was organized or disorganized exactly how they would >> Yeah, it it was like it was like the real the real like files for this real >> I was looking at this and anybody who has worked in a VDR like a a data room for any M&A transaction has seen exactly this and >> they had the real files and then obviously they sort of like deanmized it. but along with the files, they actually had sort of all of the real tasks that, you know, analysts and associates and other people like working on this deal had to do. And they essentially converted this like, you know, actual deal room into an environment they could evaluate the models in. And then they like found the emails that, you know, some accounting partner told their associate to do as part of this task and then they had the model do it. I think this is this is obviously sort of just like a categorically different thing in terms of realism and interestingness and quality. versus like just taking some random, you know, banker off the street and being like, you come up with a task for me.
29:59 >> Okay. One one more thing about this, Max, is the life cycle. I thought this was fascinating, which is like they could effectively replay what the data room looked like when that task signed, right? So like >> you're going to add in these thousand files into the data over time, but not every task is assigned at that same time in the data room. I thought that was like >> yeah, obviously super fascinating that this is >> the depth at which people can can go to to try to make this task realistic.
30:27 >> But some of the failure modes are like so like esoteric when like I looked at them, >> but then like I had to ask like my finance friends and and they told me like, "Yeah, you would totally be expected to just have done this. >> >> It'd be like some email from somebody who was like, "No, actually adjust these numbers to like whatever Q4 or something." It's it's just one email where it mentioned this and if the model didn't find that email, it would fail because, you know, it had to it had to find that email to like actually evaluate all the context relevant to the task and the models didn't look hard enough sometimes and so they just miss it and then like have the wrong numbers go out >> and if the human associate failed, they would have gotten yelled at. So, it's the >> Yeah, apparently that's actually that's exactly how it would have worked. If you were if you were an analyst and you did not read all that one email, if you were not on top of it, you you would have you would have gotten yelled at for sure.
31:14 >> They would have said, "Yeah, this unacceptable behavior in a real banking environment." >> I know some people on the tokconomics team who I think are possibly going to be listening to this podcast having some flashbacks psychard stairs. can you talk about the grading in this case because I think a lot of people would be confused about the partial credit versus binary grading approach that you have to apply to some of these things that don't necessarily have a binary outcome.
31:40 >> Yeah. Yeah. So in the case so I'll say our finance tasks use binary grading. So the model either gets a zero or one. but there's sort of a continuous sort of scoring component there because you might have a task where the final deliverable is like some Excel workbook. maybe it's like a quality of earnings book that like the accountant had to make for this private equity deal. and obviously like that workbook can kind of be decomposed into a bunch of different parts. you know, different cells and the model has to find the right number for and you kind of assign partial grading for each of these atomic criteria. but then we still use kind of binary grading at the end because if you were a real accounting associate and then you gave your boss a workbook where like half the numbers are right and half of them are wrong, they're not going to be like, "Oh man, like >> great job, little Jimmy. Like you got half of it right. Like I'll take from here." They're going to be like, >> "What the man? Like I'm firing your ass right now." so I think in those cases like it makes sense to do binary grading even though there is like a partial scoring component. So yeah maybe Jordan Jordan can show here that like we have 0% when you use binary in the finance basically except for carbon who kind of got like one task right but you switch to partial and you sort of get much more words so you don't have the models actually did. that being said, okay, actually I'll say one, a lot of people on Twitter were kind of on us using binary grading because there are other benchmarks that kind of use that to hide the fact that their tasks are actually quite easy.
33:20 I won't name any particular names here, but there are some popular benchmarks where you look at binary grading and it's like, oh, 25% on opus, like this is probably pretty hard. But then you switch to partial grading, it's like 97% opus, and it's like, okay, like that last 3 3% is probably just like some esoteric requirement that there's no reason you shouldn't have expected them. >> Yeah. Or your task is just like unfair and broken in some way. And yeah, so like I agree if you look at our finance task and it was like 95% on opus or partial grading, they're probably bad tasks. but you saw it was like 50 something%. same with our legal. The legal is all zeros, but then the best model even when you switch to partial grading I think is like it's in the 30s or 40s. at the same time most of our domains actually use continuous grading because it makes sense for those you know you have a chip design task where maybe the model is sort of asked to do some like PPA optimization and then you just have like zero as some naive solution and then you have one as sort of some like max theorical achievable solution and you kind of just give the model partial credit depending on how much it can optimize like your chip design. we sort of picked the grading strategy that like makes the most sense given the particular domain.
34:34 we did not do you know binary grading for everything so we could kind of artificially lower the score. >> Yeah. And I think this is maybe goes back to the comment about frontier intelligence versus day-to-day tasks. I mean, I don't know what frontier and finance necessarily represents the same way in science where it's pretty clear like you're pursuing a new discovery or something like that. But finance like the frontier is being accurate and >> yeah I think I think one of the things to really note for finance is that maybe sort of at first glance do you think it's like different from you know chip design or material science but we actually think it is going to be one of the domains that sort of spends very heavily on tokens in 2027. It is actually already one of the main domains that people spend tokens on in 2026.
35:25 largely because like the margin structures for the businesses sort of supports it. like they can actually just afford to spend a ton of money on tokens. but then also there are and this is actually this is even more true for legal but there are some sort of like PVP components there where if we if we take legal for example if the AI sort of could theoretically do a better job like winning this lawsuit or like redlining some contract like that actually does have implications especially if like maybe your your like opponent in the scenario is not using the super intelligent AI to those things. so I actually think finance and legal like are domains that will be able to absorb a lot of frontier intelligence. whereas maybe things like real estate or like government or something might not be >> makes sense. And and just to like like if you could expand on that a little bit more because I I think I know the backdrop. You're specifically talking about the reason for focusing on these domains. specifically rather than like the choice to do that rather than just taking a cross-section of all knowledge work in the economy and saying we're going to have tasks for every single thing that people do across the economy. Right.
36:47 >> Yeah. >> I think the thing that the listeners are probably listen thinking about right now is just >> the concept of AI driven job loss. if people are using Frontier Intelligence and they're gaining these productivity improvements or they're making more money or they're, you know, winning more things, maybe you can talk about like how does running through a benchmark like this inform your point of view on whether AI is going to yeah cause job loss in the next year, two years, even five year time horizon.
37:21 I actually think they're like orthogonal in some sense because we we don't we supposedly choose not to cover a lot of sort of the white collar knowledge work domains that people are probably most afraid of draws occurring like I don't know customer service or something like that. in terms of whether or not we will have like massive job loss from AI, it's really hard to say. I think a lot of the doomers have probably been very pleasantly surprised by sort of how AI plus human like co-working has evolved over the past year where it's like every any any software engineer that's sort of like has their eyes on the ball will probably tell you that opus when given sort of like a well specified problem is just like a better software engineer than me period. but at the same time like good software engineers are more valuable today than they've ever been. now is it possible we're sort of in this like Goldilock zone where the humans can like a lot of humans can still provide kind of that like taste and supervision and judgment that makes it so that like them plus the model is like way way more effective than just the model. but then like maybe 2 years from now you actually only need like one dude at each company who's like supervising all the software engineering and like basically everyone else got fired. Maybe. I don't know. I think it's possible that's the world we end up in. but it's also possible it's not. I think we just have to you really got to pay close attention to the model capabilities to see.
39:04 I'll say it's definitely raised the bar for the kind of person you're looking for to like work like on stuff like when you when you're looking at like software engineering for example like >> I feel bad for the interns. >> Yes. Essentially like you have to be really good to like provide value higher than just pure like opus at this point as a software engineer. >> Yeah. Because I think a lot of people viewed interns as like oh I'm going to these are the people who I give my wellsp specified well- definfined problems. I don't want to do and they're going to go do them for me. But obviously now you just tell claw to do that. and then sort of all the additional overhead of managing an intern often times become sort of just like unbearable in those cases. I know one of the big quant trading firms had a post around this. It was like Yeah. Really? Yeah. They they said their interns like finished their projects like in the first like few weeks this year compared to previous years.
39:59 >> Hey hey hey at least interns know how to use AI then. That's good. >> Yeah. And I think the typical response from people is that you need to get more ambitious. it makes sense. I mean going back to cyber security or I've been doing some work recently. We saw this recently just with the increase in CVEes. I posted this on October 1st. It's been a month. I don't have an update. There's a lot more than just this at this point.
40:26 >> >> dude, is the whole internet just like broken now? Dude, what's what's going to happen? What are you going to do, Jordan? >> Well, I think it's a Yeah, I can >> I mean that in a lot more detail. Let's save that for another time. But let's use it as a way to make the point that there's I think there's a lot more demand for cyber security experience right now even when using the models. >> yeah, for sure >> and I think that's one of the first domains as a product of software engineering that's getting knocked down. But if you're pushing the frontiers of material science, for example, you're probably not automating away the jobs of the material scientist. You're probably starting to discover a whole bunch of new materials that need to go and through the whole discovery phase. And a lot >> Well, no, I I think it is very possible that you have automated the job of your like PhD candidate assistant if you're like a cracked material scientist.
41:20 >> I I think this is actually exactly what we we've heard from from >> Yeah. If they if they have no ambition to go beyond what they were doing today or last year, >> yeah, >> then yeah, there's nothing useful for them to do. But if they have ambition to accomplish more with the tools that are at the >> if the if the person who would have been forced to just be, you know, kind of some PhD candidate grunt worker two years ago can now kind of leverage their infinite PhD grunt workers on tap to elevate themselves and, you know, just do way more interesting material science. That's great.
41:57 we'll just have to see sort of I I think I'm generally very optimistic kind of human agency and people being able to kind of take their life in their own hands and like, you know, do whatever makes them happy. And so I am kind of bullish people kind of using these tools to be just upleveling themselves essentially. we'll have to see what actually happened. It >> was interesting. I actually completely disagree with that. >> Andrew Andrew's a hater. that I think most people >> well I mean when you look at it like most people in America they just want to go in put their hours in and go home.
42:30 >> They like don't care that much about their careers and like and like maybe maybe actually like this is what forces them to care because like otherwise like you you won't be able to provide value but you know we'll see. >> Yeah. Interesting. Okay. let me go back and get some quick reactions to results that we kind of glossed over. So Andrew, you love Opus 5.5. Were you super happy to see it come out on top on a lot of these leaderboard charts or >> justified what where did you get that insight?
43:01 >> I mean obviously like you know I'm I'm glad that my subjective experience was validated. though you know some domains that I'm not particularly involved in like it was very clear that Astra was just like ridiculous ridiculously strong on some of these like material science but also you know like some of the engineering stuff it was it was very clear that like Astra was like a step ahead there and maybe that's just because we don't have table 55 yet but you know we'll see but it's good to see that like different labs have different strengths because you know there's spike here in different things so you leverage whatever is best for what you're actually doing.
43:44 >> yeah, makes sense. How about Gemini 4 carbon? Quick quick take there. We saw Sundar quote tweeted everything there. You guys are excited about that. Hey, like >> Yeah, that was sick. >> Shout out Sundar. >> I mean, we're excited about it, too. Like, >> hey, more Frontier models is better for all of us. we definitely want, you know, better models from everybody. it only benefits the consumer as there more competition. >> Yeah, I do I do however think that the right sort of apples to apples comparison for where the frontier labs currently sit as if you exclude cyber and bio and alignment and look at that GDP score. Maybe even switch to partial scoring too because that kind of gives you some more signal on legal and finance where no one's smart enough to do binary grading. at least get reward for binary grading. and if you do that sort of ranking, you you see that carbon is still kind of behind Astra and Opus etc. but that being said, they it is certainly sort of an inflection in the trend for where Google's model releases were going because I think at one point it kind of felt that they were going to fall farther and farther behind even like Meta in particular, potentially even XAI and potentially even some of the Chinese open source guys. but they've like kind of really reestablished themselves as the third place player with Carbon.
45:15 >> how much of this is a timing thing then? Like, sorry to interrupt, but Fable 55, you guys are teasing that might come out soon. Obviously, Opus, Sonnet are on the list of >> Yeah. So, I I think I think the gap between Enthropic/ OpenAI versus Google today is likely still very big because like we're going to have new flagship model releases for them quite soon. but like Muse wonderful came out pretty recently, right? and so I think you're sort of evaluating Google's position versus Meta and XAI. I think there's a pretty strong argument to me that actually they're like the top of that kind of tier 2 cohort. but I definitely still believe that there's like a pretty meaningful gap between open eyeopic versus >> and you know there's like the thing like you know open eye andropic are obviously not releasing their best models. so it's it's just a comparison between the public models. Well like Gemini and like Meta like they are releasing like the best that they have so far. Yeah.
46:19 >> Enthropic could just be on like Mythos 4 at this point. Who knows? >> Yeah, true. we're going to get like Fable plus one cuz I I think Enthropic is still kind of rolling with Fable one level models essentially or like Mythos one I guess it's called Fable 5. internally they're probably like at least two generations above that and then we're finally going to get you know one generation above that hopefully this week. so I think the the gap between sort of the actual best model each lab has including internal versus well actually that gap is just sort of very wide across OpenAI, Anthropic, Google, Meta, SpaceX.
46:59 >> Yeah. Well, maybe the the biggest takeaway for me there is just how relentless the pace of model releases is. And I mean the trend is for it to get faster and faster where they improve post training and improve the data mixes and >> RSI bro it's RSI. >> Yeah. can you talk about next steps like where do you go from here? You think it's possible actually one thing we didn't talk about at all is the concept of releasing one task and one roll out for the category for each of the categories or well for each of the data vendors. So multiple from some of the categories.
47:38 >> Can you talk about that approach that you guys took there of of the mix of kind of partially public, partially private and how other benchmarks approach it by comparison? >> So I would say the tradeoff that you have to make for any benchmark is the more public you make it, you know, obviously the more experts are able to QA everything themselves and kind of verify that your stuff is actually good. at the same time if you make it public it just becomes like way too easy to ho climb on the benchmark and oftent times like your benchmark >> no longer becomes a measure of like model capability and then more becomes a measure of like which lab bought more data that looks just like the task in my benchmark. and again I'm not going to name any particular names but I think there are a lot of public benchmarks that have already sort of like fit into that category. It's really just a measure of like, okay, you know, which which lab spent the most effort to buy the most data of this shape. And priority number one for us is that we don't want people to be able to hill climb benchmark. We sort of want our benchmark to be an accurate measure of sort of real model capabilities. but at the same time we we obviously know that establishing credibility is important and so releasing like the rollouts for one task per partner we thought was a good trade-off there where kind of people can get some visibility into each partner and how confident their tasks are good. at the same time sort of us going pretty indepth in the actual methodology for like each partner and explaining how like you know our finance guys actually use this like real deal room or our our cyber guys like found their own zero days and for them to task. I think sort of binding all those details there is the other way that we can sort of instill confidence in the quality of the benchmark. also just like giving details in our QA process where we say like yo you know we ran our own agent to QA then if you take bench 4 literally flags like 50% of the task is broken but then we ran out of every partner we made sure that like if you couldn't pass this bar you just weren't included in the benchmark. I think stuff like that is what allows us to sort of give people confidence in the quality of the bench without making all the task public.
49:53 as for what's next, we will really hope we can get sort of version one on some major new model release cards when they come out because we do think it's sort of like the best benchmark for measuring printer capabilities in the world today. at the same time, we're definitely like just getting started and we plan to do like future versions as models become increasingly capable. they saturate the task in GDPX v1. there are sort of more domains that get unlocked that can absorb lots of like frontier tokens and like be super valuable, earn higher ROI. and so we definitely want to release like future versions with more you know data partners always using sort of the highest quality data partners at any given point in time. Always picking the most important domains at any given point in time. hopefully we can do things like have more partners and have more tasks per domain and also run all the models at every reasoning level and stuff like that now that we have you know more clout and more traction with the benchmark.
50:59 >> Awesome man. Okay, I think you answered the final question about where we go from here. Andrew, any like closing thoughts from you about the future, next steps, initial reaction from this stuff? Pretty fun week, huh? >> Yeah, I mean definitely agree with Max. Like I do want this to be like the benchmark everyone thinks about when they think about like what model has which capabilities and where they stand. and this is only the first iteration like we're going to we're going to keep working on it. We're going to get hopefully more data partners and if anyone listening to this is interested in evals and you know maintaining this please reach out. We are hiring on this team of course just email gdpxian.com.
51:37 Yeah. Awesome. Nice way to end off. Thanks guys for joining. Appreciate it. >> Thank you Jordan. Thank you Jordan.