The Great Evals Debate — Ankur Goyal & Malte Ubl

0:03 Hi, welcome to the Lin Space Lightning Pod. We are here with for a very special episode because this is not really about a company or some launch that someone's doing. This is actually just a debate that I accidentally set off about coding agents mostly not having evals or they do have evals. It's mostly it's not really like standardized or and entirely it's a lot of it's vioded and I think a lot of it is based on the fact of this clip that I had from Boris of cloud code saying that they basically just vibe it and that's a 500 probably higher than that $500 million business that doesn't have evals. So I wanted to invite Anker and Malta from Brain Trust and Verscell to just talk about like the state of EVAL's here and like the EVEL's debate. You guys have your own perspective. Anker, you got a bit in a in a bit of like a blogging back and forth on evals. What are you guys feeling like what's maybe a position statement from each of you?

1:03 >> Yeah, I I mean I can start a little bit like I think the like one maybe important background on me is also that I worked on Google search just before leaving for Versel. It's almost four years ago, but like I do vaguely remember it. And obviously that was also before JGBD moment and before kind of engineering becoming a mainstream thing. And so I left a world in which I was doing evolves every single day and kind of thinking that that would be it, right? Like that I wasn't really seeing that in my future as someone joining an infrastructure company. And so it is both kind of fun to see it coming back and I think it does give me a little bit of extra experience for what like how it helps and how it doesn't help. One thing I tell people for sure is that like you have to put in the work, right?

1:47 And and even vibe checking like as the eval like is important like if you are a product manager in Google search, you're probably doing like 500,000 searches a day. and if you are but then you want to do revals because if you want to do 50,000 searches, you're not going to do that yourself. And I think there's something very similar here where essentially you just to see like, hey, I want to know if I'm doing well and how fast can I find out and like, yeah, the vibe check can tell you really fast, but if you want to know tomorrow, then the only way to do it is to have someone do the 50,000 searches or the 50,000 coding exercises, right? And if you want to know in in let's say in in 3 weeks, you can write an AB test. And if you don't care, then you you know do neither of those things. But but it is like I I think like there the way I think about EVA is essentially like it's the thing that that can tell me tomorrow whether my change is good and I can operate without that knowledge. But it's but it's super super helpful.

2:39 >> Yeah, I have very little to add to that. I I strongly agree. I think like fundamentally if you're building an AI product, you're dealing with this like non-deterministic magic and you don't you can't like type whether it's a model or prompt whatever you can't really type something and know what's going to happen. So you need a feedback loop and I think that fundamentally evaluate was about are both the most challenging to build feedback loop and also the most efficient once built. And then I think AB tests are a little bit less challenging to build and a little bit less efficient. And then like pure vibes are basically like zero effort to build but then also the the least efficient.

3:23 And you have to do a little bit of all of these things. And I think what I've seen both firsthand from building Loop, which is our own agent and brain trust, but much more secondhand from working with Malta and a bunch of really great companies building AI products, is that the best people are extremely deliberate about the investment in each of the feedback loops and they're constantly questioning whether they've put enough energy into each of the feedback loops and making them efficient. And then I think coding is it's like such an interesting topic for a few reasons. The first is it is the use case that has the most product market fit in AI I think other than chat GPT and so it is just an important and interesting use case. The second thing is that to a large extent it's very verifiable and so it's like it's almost insane that it's not eval or at least the fact that it's not evaluating because unlike something like let's say creative writing or summarization etc where there's some nuance there's a lot of objective stuff that you can evaluate. Now, if you talk to the people that are really deep into it, they'll disagree and say that refactoring a codebase is something that's very hard to evaluate or the visual quality of an artifact that you produce on visor very hard to evaluate. And I think therein lies the the subtlety. But then, you know, the the third point which I've watched Boris's podcast which I thought was like ultra ultra insightful and I think the the thing that's very clear from what he said is that based on how people are using Claude Code internally and the enthusiasm in the community, he feels like they have a pretty good feedback loop. And I I think that's awesome. Like it doesn't really matter whether they're running a script that has a for loop that iterates through a bunch of test cases and you know, whatever. That's their feedback loop. What matters is that they have a good feedback loop. And I think that's ultimately what you need to do. And however you get there is is how you get there. And if you're happy with the feedback loop that you have, that's awesome. But I I do think it's important for everyone to think about, do they have a good feedback loop? And if not, how do they make it more efficient?

5:18 >> Yeah. Maybe one one thing to add on what Anker was saying just yesterday, I think we saw both Cognition, congrats, Swix, and and Cursard ship RL fine-tunes of unnamed open source models. And the reason why you can do that is because you can implement those RL loops based on what is effectively an eval that you built, right? That that you run at every iteration. So that's the reason why this coding stuff works so well is because you can write these evals, right? And I I I have this like hot take and this might be like not fair to Boris, but I think he comes from a from a particular position of extreme unusual privilege, which is that he works at an AI lab where like people in the office next door are like writing the the evolves and are are are like training the model like every day in exactly that way. And so it might be like there there I'm sure there's some pretty deep hand inhand work in making the agent work well and be and the model be particularly well trained to what the agent needs and so forth like there's going to be feedback loop like that. I think we have to consider that when we make recommendations for the world out there which is different right like where where you don't have this privilege and where you have to kind of make do with what you got from the either big model apps or the open source models etc.

6:30 >> Yeah I don't know if there was like a particular point to respond on that. I will say like I think all the agent labs if you can call it that you know Versell has your own benchmarks cognition has their own cursor has their own and typically everyone trusts their own benchmarks much more than the public ones but then they're not comparable because they're not open source because probably because their IP or because they use proprietary code >> is open source >> or at least they they have a big open source one. Yeah. Well, next year JS the next year one >> in and out like yeah we open source nextJS we intend to do it for the other tech that we're building and that's probably because we come from a slightly different permission like our goal is just that all the AI models and agents are just incredibly good at the tech we've put out right so we're incentivized >> self-interest in doing >> 100% yeah so so you know and then and then you know cognition like a lot there's a lot of banking customers that like would not appreciate some some of the that those emails work being publicized but I think like yeah anyway so The the point being like actually we spent a lot of effort on public benchmarks and in fact like the team like really burned themselves out over doing this and in the end like people look at a chart and they go like all right like it's it's and then let's check the vibes right so well yeah I mean I think those eval are there are not a public benchmarks that people publish are not in service of creating a feedback loop which is not to say that they're stupid or anything like that it's just that the value proposition of publishing an eval is completely orthogonal to the value proposition of building evals in service of building a good product. I think the purpose of publishing benchmarks is marketing and it's it's good marketing like I come from database land where TPCC and TPCDS all this those are standard benchmarks and then people like click house make clickbench or whatever they make their own benchmarks it's it's very it's a very sort of like mostly objective way of communicating the value proposition and trade-offs of of your technology but I think it's totally independent from the practice of using evals as a as a feedback loop mechanism for for iterating faster Sure.

8:32 Yeah. Okay. Awesome. I want to go back to a little bit something that Anker you touched on. Let's let's just discuss call like the primary dimensions of evals. You talked about effort. You talked about what is it effectiveness or efficiency. I think there's one thing that I don't know how to capture which I'm curious how you guys think about it is open-endedness. This is not like this sounds like a madeup term but it's a real team at Google that is actually exploring open-endedness. And I think Malta to make this relevant to your background, search is open-ended, but it is not as open-ended as codegen. And like I think the reason people lean on vibes more and and spot checks more than traditional evals or offline evals is because they actually cannot predict what was. It's very easy to kill client things that you've already written evals for, right? It's it's it's always the stuff that you don't know that you don't know that is going to catch you off guard like some competitor is going to come in and eat your lunch or whatever, right? So to me that is the I think the debate of like why people are moving from offline to online or online to five checks is because it costs a lot of effort to do offline and it probably lasts maybe 6 months if you're lucky.

9:42 Well, I think I think you're conflating two different things. One thing is sitting in a room and enumerating a set of scenarios or tests that you think represent a workload. And then the other thing is choosing to run those tests outside of production. And both of those things are activities that can happen offline. But I don't think anyone who's legit who's doing offline evals is doing the former anymore. Like in fact when we started Brain Trust and we talked to customers were sort of very excited about Brain Trust helping them create golden data sets and now I actually think that term is kind of a dirty term.

10:17 People don't really want to create golden data sets. It's I think it's it's it's often a wasted effort to to the point that you're making. I think the best teams view offline evals as mechanism of reconciling what they see in production with real users who are using the product with tests and iteration that they can do offline. Like the best teams that we work with on a daily or even more frequently sometimes basis are discovering use cases from logs online and then pulling them into their environments and then playing with them. And I think the difference between doing offline evals and online evals or offline evals and just pure vibe checks is I think the pure vibe check or pure online version of this is you observe some behavior in your logs. You're like, "Oh yeah, let me like play with it."

11:04 Maybe you tweak the prompt a little bit and then you deploy it and then to Malt's point like either statistically unsignificant insignificantly now or statistically significantly 3 weeks later you observe the impact of that change. And then I think the offline eval way of thinking about it is you discover some interesting behavior from prod and then you have one or two clicks that allow you to capture that behavior and repro it offline. Iterate and as you iterate you not only test that but you also test a bunch of other scenarios from other users that may also be recent and you gain a little bit more confidence about the impact of the change and then maybe you ship it still like today or something like frequently when I'm using Vzero I will complain to there's an engineer at Verscell named Max who's like an incredible engineer and when I complain to him like I know that two clicks later he's taken my complaint and is playing with it not in prod but on his laptop and I think That's that is what offline evals are really about. It's not about manufacturing a golden data set anymore.

12:04 >> Anything to add? >> Yeah, I think I think there's definitely the the you know workflow velocity. There's the like the understanding of what the evolve tells you, right? I think that the common quip that if the vibes and the data disagree, the vibes are probably right is true, right? So, you have to like be honest with yourself like do they do they agree and kind of iterate on them over time. the other thing that like I don't know it's just my my weird hat who thinks about it in this terms but like to me often the I like to say the first derivative controls for methodology like like if on a on you have your number right and it's more important that you know that it's good if it goes up than that you understand what one or two on it means that's very difficult to understand sometimes and so you need to like if you if you if you invested in your evolves the al the opposite is true right like so it's so nice to be able to be aggressive in the changes you make knowing that your encoding of what you thought yesterday was a good idea for your product hasn't regressed right even though the world has changed like that that is just like very traditional kind of having testing in software engineering that allows you to move faster because you have higher confidence and I think this is like you just have to understand the trade-offs between that and and potential extra work obviously >> one of the things that I've noticed along these lines I like I hate writing unit tests personally We I write them.

13:25 Actually, Claude writes a lot of them now, but I've always hated them because they're so brittle. Like, I change the design of my system and and then the unit test slows me down rather than enables me to verify whether my system still works. I've always been like a hardcore integration test person. And I think the same or maybe an analogous set of design principles and trade-offs apply to thinking about evals. Like if you construct evals in a way that represent the true north star of the problem that you're solving, then they tend not to break as you change the underlying system. Whereas if you hardcode them to a narrow subset of like an implementation detail of your problem, they can be very very brittle.

14:04 And so I I also I think you have to if eval are not working very well for you. Obviously some of that is like the cost of evals and maybe the tools that you're using or the framework or whatever. But some of it you you kind of have to zoom out and say like okay how do I how do I reframe how I'm thinking about what I'm evaluating so that this can be productive for me or if you choose not to maybe your competitor will right? So may use that as a guiding principle.

14:30 >> Yeah I think eval as a competitive advantage is underrated. I don't know if that's that's spicy to say. Okay. So and I wanted to maybe ask Malta like are there any best practices that you guys have have developed on on eval maybe around brain trust or like principles that you that you think that maybe Anker hasn't really covered because he is from his point of view. You're from sort of the user point of view just I just want to leave it open-ended as to stuff that you've seen.

14:55 >> Yeah. I think from a just pure workflow point of view and Anker kind of already like highlighted some of this but I can maybe go a little bit more concrete. Right? Like the the actual workflow is that with this tool like Brain Trust, right? You essentially are logging and you you're collecting facts about your production system. And what's really interesting is then how do I turn the thing that I know people did in production into into into something that I can validate in my my codebase. And so what we what we do there is that we try to on some relatively regular level kind of upgrade failure modes that we observe in production into something that where we then in the future can measure how how well we do based on on those previous failure modes, right? And so you're in this kind of constant loop that is actually in a way very vibes driven, right? because there's the ground truth of like what actually matters which you can obviously sometimes you envision what it is but like in practice what actually happens is that you are driven by by the failures and the successes of your production users and then you you lift that into something that that's expressible in in on the code side. I think the the other thing that maybe is worth mentioning is that because we're talking about coding in particular, it is in fact possible to to do the essentially like eval like does this program compile? Does this program start up? Does this program start up and render the first page without client side error? Right? You can you can create some of these very trivial like in quote unquote trivial evols, but like what we found is that they're actually they're extremely valuable, right?

16:31 That's actually where so so the like what we then what we actually literally did is we turned those those into into our RL pipeline, right? And so we we essentially trained a model to to fix the errors that the EVA was previously detecting. And that that is extremely effective and it's while you know basically what we do is we have this like composite model architecture where we run the frontier model and then we run the fine tune model after to fix it errors that's doesn't perform better than in a gentic loop but it's orders of magnitude faster right because all like you know like the agent might all the time find out oh yeah I made a mistake and then it totally knows how to fix it right that's not the problem but it it's fixing it using a model that has somewhere between 30 and now maybe the fastest want 900 tokens per second which is still snail pace for trivial errors that these models make all the time like const a equals 1 next line a equals 2 which doesn't compile but it's very very easy to fix >> okay so so you're saying you don't do an agentic loop you only do frontier draft and and fine-tuned fix is is that >> that's not what I'm that's not what I'm saying I'm saying >> we do avoid having to do the agentic loop in more cases so that the total time goes down because you use the agentic loop for fixing difficult things and not for fixing trivial things.

17:52 >> So I'll briefly mention that in the sweet gap work part of the RL objective was was time to complete and so you can just actually constrain the number of turns it takes if you if you have the agentic driver model not be incentivized to take forever because we want to prioritize human interaction. Yeah, I think there's another point I'll inject a little bit of spice into this discussion that Multi made, which is using EVELs as a way to get product insight. And I and I think this may be where Eval has caught the attention of Lenny. I don't know for sure, but we've we've we've seen like a massive surge of product manager, product managers, and designers getting interested in participating in the eval process among our customers.

18:36 And and I think the reason is that in many ways coming to the table of product building with representative examples and criteria that articulate what good versus bad is for a use case is just a more precise and usable form of product management than writing a spec. And so what we've seen from some of the best teams is like actually leaning into getting product managers very very involved in the eval process. And if a product manager who has a very good instinct about what the vibes should be wants to communicate that to an engineer who is maybe less familiar with the vibes for a particular use case and in coding this doesn't make any sense because the engineer has the best vibes but if you think about use cases like finance for example or healthcare where AI is providing like a massive amount of impact the product manager has a way better usually has a way better understanding of what the user actually wants.

19:33 One thing they could do in like 2015 is write a spec that describes like, hey, you know, the user wants this and therefore blah blah blah blah blah. Now, what they can do is construct evals that help illustrate that point or even construct scoring functions or LLM as a judge as they're called that are they're kind of written like specs. They have the same structure and and they're kind of looking for certain criteria, but they just end up getting utilized differently. And and I I actually think there's a lot I think I think there's a lot there. And it and it's it's it's also somewhat coincidentally interesting that that particular phenomenon is mostly irrelevant to the coding domain, but is like highly relevant to all the other domains in which someone might use AI.

20:18 >> I think that's right. And it's actually I think the magic keyword that everyone's saying is rubrics these days for LMS judge. Really? Isn't it? I don't know. >> That's what we use. >> Okay. Right. So like yeah we we we built this thing called the junior dev eval. We're we're upgrading the senior dev eval working with like mccor and like other a lot of these like high-end annotator companies to to to do all these and coming back to the point on product management. Yes, I think it's right. I think it's both managers love like you know would love to have more of a say in engineering right and they write this like 50page PRD. no one reads it or they they have to go engage in this back and forth and say like well you didn't exactly get what I meant. But here they can actually come up with a number and if the number went out they they're happy everyone's happy right and they get to treat the products as a black box. So it's a >> I don't even think it's just the number though. I I think it it it if you if you just think about vibes like forget about the term eval and you're working in the finance domain and like I don't know much about finance but if I like let's say I build something that I don't know processes and invoice or something and Swix maybe you're like a you know finance wizard and you're like hey Aner the vibes are off. I'm like I what what do you mean dude? I have no I have no idea what you mean.

21:28 >> how are you going to communicate that to me please? >> Exactly. Yeah. So, so you could write a spec or you could just give me some examples. And I think the beauty of AI is that give me some examples is actually a very powerful way of communicating product intuition. And so again, I think like if you just forget about the term evals or or and and you don't like I I think vibes are actually a form of evals. They're just like a very very expensive scoring function that is extraordinarily accurate in a lot of use cases. Maybe not for compiling code, but certainly for like aesthetic quality, for example.

22:04 So if you if you just generalize and you think of evals as including all these things, I I think it's a great way of communicating product intuition. Yeah. I I I agree with that. I I I would say like the the so what that's one of the responses to the great evals debate which was well vibes are also part of evals. And I'm like, well then you're then just everything is eval right like you're always doing evals that you know because you are an evals person you obviously see the whole world in in in your point of view. But I think we're trying to to like push towards people towards >> we can we can stop using the term evals and just call it feedback loop because I think that's I I think that's not there's no feedback loop vendor or something like that's not the the perspective I'm I'm trying to communicate here.

22:44 >> No no no I'm not not kissing you. Yeah. I I think feedback loop is right. I think that's why there's also all these like RL startups that are coming out. We're working with applied compute. and like there's like 20 different ones. I just I just had a call the other day with another one and it like feels like it should be viodable all these environments. I don't know why they're so hard, but I wonder if you guys have have interacted with the RL environment startup groups and like is that a part of the evals movement that is is happening or is that separate?

23:11 Yeah, I I think we obviously are familiar and friendly with a bunch of these companies and keep an eye on it. And I think that I like to think of evals as like really like three components like you know data or examples and then some thing that you're trying to evaluate like an agent or whatever and scorers which help you compute whether you know your thing works or not. And I think RL environments they're they're like definitely a powerful form of the third thing and not necessarily a powerful form of the first thing data.

23:44 And I think that what we have really tried to solve for at Brain Trust is helping people solve the data problem. Like we we we we are really really about helping you capture data from prod and figure out what's worth looking at and then pull that offline. And and I think you can use that to help inform how you use an RL environment. down down I I would say practically speaking outside of companies like yourself, Swix, that are at the very frontier and doing like training your own models and and doing like really crazy stuff. We haven't seen RL trickle into the average company that's trying to figure out how to just use a few prompts and maybe build like a basic agentic loop and get it working. I'd be quite excited for that to happen because I think that the RL paradigm is more programmable than manual. And I think that's quite cool.

24:34 But I think what I've heard and experienced so far is that you have to have a pretty specific set of expertise, which maybe what you're alluding to with the vibe coding thing, you need to have like a pretty specific expertise to design the RL environment in a way that's not vulnerable to reward hacking. And I I think that either you'll end up with some fixed number of very well-engineered RL environments or you need to somehow employ this expertise to apply it to the N plus first problem.

25:01 And I don't I don't see any evidence yet that the average company that's trying to build an AI application will be able to recruit that expertise. It's kind of an open question to me how how far it will propagate outside of the companies like yourself who are at the very frontier. >> I'll just briefly interject in case Malta has something to follow on. Actually I I see a lot of it in computer use in in terms of the oral environment and I think you're coming from a very human ccentric point of view of data like humans are the source humans and domain experts are the source of truth and they're effectively distilling their knowledge into the environment we using as the data set. I think that the what I see the potential of our environments is is that these are a lot of encoded opinions in a realistic environment which would not be conveyable at human scale. So that you're trying to scale up individual humans by just having them construct realistic environments that embed a lot of like normal assumptions that an RL agent can go in there and just work on.

25:57 Yeah. So I think the difference is kind of the trying to decouple from the humans because humans are expensive and slow into an environment which we can parallelize, control and play back and forward. yeah, that'll be my my twist. One comment I I wanted to kind of dig in a little bit and it's a little bit not directly related to what we were discussing but I think there's a there's an interesting link is that speakers I think you were saying earlier in the conversation that there is basically obviously people have proprietary evos people publish some notion of leaderboards publicly which are marketing and they're like you know people find that interesting primarily but they don't have a you know particular purpose and I think there is a like the nextG eval that we released that you already mentioned and which we you know we're planning to do more in that space I think points to an interesting future where the question is not so much like you know is there going to be an RL process somewhere or is there going to be a better model but like can there be kind of an inversion of control of like who's writing the emails versus who is building the agent and who's building the model like right now it's always the same entity and I I could see in the future that this actually changes, right? Like if if if I'm a business and I want an agent to be incredibly good at like automating my business because my business model isn't that someone clicks around my UI. My business model is that something else happens on the back end. Then I'm incredibly incentivized to say, hey, you know, model lab, hey, agent lab, how can we work together so that you have an easy time getting really good at the tasks are relevant for my business? And so I think that that is a there there's sort of this new way of of of thinking about whose responsibility is big making the evolves. It might not be the same entity that then takes those into production in terms of making a better model or a better agent.

27:50 >> Yeah. I think that's why like companies like artificial analysis Ella Marina these are these are like the independents like the new garners of like independent reading agencies for various companies. Yeah. Makes sense. >> Yeah. I think that's a that's a bit it's a little little bit of a different take than from what I what I meant because they are independent, right? >> Yes. >> But I don't I don't I exactly don't mean that. I'm like I'm not independent like because I'm I'm Versell and I publish in Evo >> that I want OpenAI and and Anthropic to use to make sure when they ship the next model that they're better at the stuff that I care about. And that's I think somewhat different from now obviously like you could have a middle person like element arena in there to to facilitate that process but then my my main main point was like the the the creator of the eval is an entity that has that wants to achieve a goal. It's just that the goal isn't that their own models get better. It's the goal that someone else's model get better. Yeah. This this idea has been around forever like OpenAI the EVEL's repo and the sort of expressed not they weren't like shy about it. The expressed intent of the repo was for people to contribute evals that they could use to make their models better at and you know the trade-off is hey we're open AI we'd appreciate having more evals excuse me to to test with and then in exchange if you contribute evals that you care about you'd get you know perform better on those evals with future versions of our model. The interesting thing that that we saw and we we've talked to all the labs about this because we work with basically all the commercial customers that they would want to optimize for. The interesting thing is that most customers or actually I'd say a stronger statement like all customers are quite afraid and reticent to to just hand over the data that they use to do evals on to to labs. I wouldn't say I fully understand it that position myself and I think that in some utopian sort of construct it it could be very valuable for everyone to exchange data this way but it just hasn't happened and maybe RL environments are kind of a cool solution to that. I definitely see I'm like maybe to illustrate Malta's point, let's say that I'm starting like a healthcare startup and I'm I'm really you know my job is to figure out like based on your like a picture of your tongue what diseases you might have or something like that and you know GBT 5 is like okay at this but I wanted to be like really great at this problem.

30:10 How do I do that? Well, I could like train my own model. That's going to be really hard. I could get a bunch of evals together, like take a bunch of my friends who have diseases, you know, pictures of their tongues, and then upload them to OpenAI's evals repo, or I could create an RL environment that I publish on some marketplace or something that people can use to actually test. Maybe I have a way of generating photos of tongues that have illustrate various diseases. Maybe I could somehow publish that and then that's a very efficient way for people to get better at that.

30:39 And this is kind of a whites space thing right now. like if I'm starting this tongue disease business like over the next several you know hours weeks months that this mechanism should exist like it's in a capitalist standpoint it like makes sense for this mechanism to exist that's that's kind of what you're saying right Malta >> yeah exactly I think it's always important to track the incentives right like I don't do that if I'm the medical agent startup because then it's then it's for me privately but if I am something that I want the world to use through AI, then I'm incentivized to make it the eye extremely good at this.

31:16 so and that always happens when there's a bit of a detachment in the business model from the from that task itself. Yeah, I guess Nex.js is kind of like a a good subtle example of that, right? Like you don't you may not want your competitors to be able to generate as aesthetically pleasing designs as VZO, but you certainly want every model to be able to generate Nex.js with extreme precision. >> Exactly. Like that's our explicit goal.

31:40 and we also want to be able to ship new versions of the framework and then have quick update in the base models to be able to like to to essentially learn the new skills. >> Have you tried building an RL environment for Nex.js? >> Well, we just shipped evol last week, but I think it's a it's a very interesting idea. Well, I think like RL environment for NexJS is just npm install next.js. Is it? No. Is is it or can you can you actually design something that is better? I don't I don't know. I mean, I think it's I'm not enough of an expert to to to know the answer to that question, but I'd imagine that you can probably apply some creativity and and create something that's better. You know what? The next templates page is the set of RM cuz NexJS itself is too horizontal to be meaningful. But actually focusing on the top like five 10 use cases of Nex.js. Yeah, I would say so. Okay. I don't want I'm mindful of your time. you you guys are both very busy important people and like but I also I feel like we come to a good spot in terms of like you know if you want to control your AI engineers you want to control your labs you should build eval so that it's like a tool of communication and also control might be a little bit of an overstatement I just I'm being spicy I'm being spicy no you know like he who writes the evals you know rules the world whatever right like so something something of that nature but no I think a very good place and I think very educational for people who are like haven't really thought deeply about this obviously you know people can reach out to both of you who are very online and very generous with your time. and yeah I mean I I think this this really is is a very deep debate is is for people who are serious about building a good AI systems and I mean I fully agree there's more to have with with this debate about like how do you like push the state of evals it's something I'm interested in exploring at cognition I don't have the time yet or the material I'm just saying like you know if we want to define what's next for coding agents we have to build the evals and then we have to kill climb the evals and that's like basically what my job is so I'm excited for for that anything to plug for for both of you anything coming or any calls to action?

33:41 >> well, we're shipping crazy stuff every week, so I I don't I'm not sure I have anything specific to plug, but come visit brainfrust.dev and and join join the party. >> My plug is to listen to the other podcast with you that I recorded this week, which does plug everything we're doing. >> While we're all here, I will plug the fact that you can now with one click send AI traces from a Verscell application to Brainfrust.

34:06 We're very proud to be part of the Versell marketplace. It's it's it's super exciting. >> Yeah. How did that sleep on mind? These are literally the the best two guys to talk talk about that on. Okay. Go test it out. We'll put it some links to that in the show notes if we can find it. thank you guys.