How to Test New AI Models Before They Break Production

0:00 They had announced the deprecation for 2.5 flash before they had even released 2.5 flash. Keeping up with constant model deprecation became a real headache for us. >> One day like one test case was working and then a week later like three of them were broken and there was three other ones that were now working and you're just kind of like flailing a little bit. >> You know these things are going to get deprecated. Build your system in a way that expects it and makes it easy to to upgrade.

0:26 >> No, this is great. I mean this is this is the proper engineering right? Okay, so today on A that works, we got producer Kevin on talking about how he builds evals for models because sometimes the model you're using gets deprecated and you need to know if a new model or a different model is going to solve your needs. And ideally, you don't want this to become a fire drill in your company. And the way you do that is you set up your product in your deployment.

0:50 You have production, you have your codebase, you collect data, you look at user reports, and the most important thing you do is you want to build up a test case. So when the new model comes out, you can have your 2,00 test cases. You can look at the outputs from the old and the new one. You can diff them. You can see where they're the same. You can see where they're different. You maybe even review them manually and reabel them and keep growing your test case.

1:10 almost all of this ends up being structured output. So you have raw data or documents or images and then you know that some structured output should come out of it. we talked about some like kind of prior art here and API migration. And then Kevin showed off a bunch of cool code of generating basically a matrix of scores on cost and latency and accuracy for a bunch of different models versus a baseline. And then we did some riffing with Claude at the end to visualize the data which was also super fun. so a lot of good alpha in this one. I hope you guys enjoy it. Kevin, thank you so much for coming and let's get into it. What's up y'all? This is AI that works. We got producer Kevin on for a very spec special episode.

1:49 >> I'm on >> I'm Dex. I'm the CEO and co-founder of Human Layer. we build the world's best way to use AI agents to solve hard problems and complex code bases called Human Layer. We got Kevin over here who is one of the best AI engineers that I know. he's done a couple episodes with us before. His eval project in what was it? A Streamllet app? >> Streamlet app. Y >> super sick. Like just like one of the cleanest, tightest, like no fluff just like here's how to get your make your AI pipelines better. and all the way down to the like labeling of the data set. So, very esteemed. Kevin runs all the AI automations that make this show work. including putting out the emails, including setting up all the all the release notes, including writing scripts for the shorts, like he's just he's done he's done all of it. and so, we're super stoked to have him here. And every once in a while he pops in and does an episode and drops some drops some great knowledge on us from his work in the fields as kind of an ICI engineer solving like production problems at like how how big is Evolution IQ?

2:54 >> Gosh, we've got I think something like 150 engineers or 150 employees somewhere between 150 to 200 employees and most of us are engineers maybe half to 70%. >> Real real software teams not me and Viob's tiny you know 5 10 5 10 15 person startups. That's right. That's right. Big big company. Yeah. And we were acquired gosh a year and a half ago at this point for 730 million. So that was a fun >> Yeah, that was a fun >> Did you get paid on that?

3:24 >> a little bit. Little bit. >> Little bit. Nice. >> Yeah. >> I wasn't I wasn't mad. It was a good year, I got to say. >> Yep. >> Yeah. cool. I don't know. I don't know if that was the full intro or if you got anything else. maybe give us some context about why we're talking about what we're talking about and then we can get into it. >> Yeah. So what we're talking about today is essentially model deprecation and model obsolescence. So something that we dealt with a lot at evolution IQ is we use our we use Gemini models very very heavily in our all of our workflows.

3:55 We're a Google shop. Everything's to GCP and and Vert.Ex AI and because we deal with we work so what EIQ does is we build claims guidance systems for disability and workers comp insurance carriers. So we see a lot of PII and PHI come through our systems and so we need to have a really secure environment where LLMs can process that and our answer to that has been keep everything in Google Cloud via Vert.ex AI. So we've done pretty much all of our production workflows are in either Gemini Pro or Gemini Flash. Most of most of which are in Gemini Flash and we initially started when we first started building these workflows, it was late 2023 and we were using Tech Spyson.

4:36 and then Gemini 1.5 Flash came out and so we migrated everything to Gemini 1.5 Flash and then I think it was September of last year when or thereabouts maybe it was July sometime last year Gemini 1.5 flash was deprecated and you had to switch to Gemini 2 and 2.5 and then I remember distinctly there was a moment where Google had announced they that they were they had announced the deprecation ation for 2.5 flash before they had even released 2.5 flash. So they had a plan release date and a planned deprecation date that was they were both in the future and keeping up with all like all of our workflows dependent on these models and keeping up with constant model deprecation became a real headache for us because we had all these workflows based in these models. So we had to come up with a way to build a system that can easily swap models and give you the eval that you need on the dimensions that you need and just a harness where you can kind of throw new models at it and see like how does this actually perform?

5:48 Can I just swap it in or do I have more work that I need to do? So that's kind of the that is the the the inspiration for this episode. And so what we're going to be talking about today is kind of a another way to do that. I know we did it our way. This is kind of a different way to do that that's I think really flexible and kind of answers a lot more questions that that we didn't that we couldn't really we just had some organizational constraints kind of just the way our things work that we weren't able to to implement kind of this this solution. And I think this really kind of pays for itself in three ways. when you have basically the question that you want to answer is you want to say okay if a new model comes out can I use it right and so you need a harness that you can just throw a new model at and essentially get a yes no on and if it's a no what do I need to do to get it to a yes and that kind of pays for itself in I think three ways one is these like every model that we are using in all of our workflows now will be deprecated at some point in the future we're living in a world where there are There's so many new models coming out and you know you can't just in the olden days you could have an XG boost and that XG boost as long as it was still performing well you could just keep it going. You can't do that anymore. The deprecations are coming and if our workflows depend on whatever models they depend on at some point we will have to change and we will be forced to change unless you're self-hosting.

7:17 >> I was going to say the the the headline for the last week has been open source solves this doesn't it? It does and that's actually something that we're switching to now specifically because it has been such a headache. we're switching to self-hosting open source models, but but open source does solve this, right? But the there's still the other two pieces. Another is, hey, a new model came out. Will it improve my workflow? Right? And so the same harness can answer that question. And then the third question is say you're running Opus or Sonnet, will Haiku work? Right? It's cheaper.

7:50 It's faster. Like is it good enough? How much cheaper? How much faster? So you need a harness that can answer all these questions. And it's it's the same question just kind of phrased three different ways, right? The question of how will a new model perform in my workflow has three different applications. >> Yeah, that's great. Tell us about your your recipe. How do we how do we cook with this? >> So the the first thing that you need to do is you need to define like what you care about, right? What the obvious one is accuracy, right? Like does it actually perform well? So, but you need to define the the dimensions along which you're going to grade these models.

8:34 The I think the obvious three are accuracy, cost, and latency. I'd say those are the the big ones. So, you define those three and then you build up a series of test cases. And again, this is all just like really high level. You'll build you build up a series of test cases to to measure those three and then you run your incumbent on you have a harness that runs your incumbent on those test cases and you can just swap a new model in and it compares against your your current or the incumbent performance and it can tell you performance or accuracy did this cost did this latency did this.

9:20 >> okay. So, the slop code bench thing was actually kind of a nice amuse boost to what we're talking about of, hey, Opus 5 came out. I want to see compared to the other models that we were using before. >> Well, that's the that's the thing that like, you know, it's it's funny because, you know, we we've been talking about this episode for a while >> and when we first talked about it, I I looked at the models that had come out.

9:41 I'm like, oh, like it's a shame we can't do this now. And this was a couple months ago. was like cuz all these models have come out you know recently and now here we are and that is even more true now than it was then right in the past month I mean soul has come out Grock 4.5 Gemini 36 cyber I'm I'm forgetting a bunch opus 5 sonnet 5 right all these new models have come out in the past just in the past month and I mean Haiku 5 I imagine is going to come out at any point now because that's so on 4.5, but new models are coming out all the time and we need some sort of easy way to say hey like like one like is my agent or my workflow better with this model and then two like if I'm still running Sonnet 4 or Sonnet 4.5 like that is going to get deprecated sooner rather than later and so every time a new model comes out that's a sign that hey my I need to I need to keep up.

10:41 Mhm. So, yeah, so it's funny that you started with Opus 5, right? cuz that is exactly the kind of problem that we're we're looking at. And I think another thing, one thing that's important about the harness is swapping out the model is pretty easy, right? It's that's easy to do in general. What's hard is defining what success looks like, right? So maybe you say I want this to be strictly at least as good as my incumbent model. Maybe you allow a 5% like okay well if it goes from 95% to 90% that's still okay. So that definition of figuring out what does the acceptable bar look like that is the hardest part of this I think. And so that is the easy you should make that the easiest thing to change in your harness. So you can just easily change out these different thresholds, run it and you get your answer.

11:45 >> Well, and this was also I think we talked about this in vibe. I was talking about like image gen, right? Is like you don't just want to know like to start you want to see one number, right? This is why all these things like even if you look at the I'll grab a screenshot from slop codebench and drop it on here. let's see, right? Like usually you get a table like this, right? which is like >> okay you have a bunch of different categories like they have strict pass isolated pass core pass partial pass >> and then you can see okay like GPT 5.5 got a 14.8% you can see opus opus 4.7 got an 8.2% 2% and like this like for me this first column is the most interesting one of like can the model ace the problem and actually get everything correct and what percentage of of the problems can it do that >> because all these other data like cost and time and also they have these like quality metrics here >> and so I guess like >> the the the general idea is is is this one around like okay cool like what is our what is our like north star right because if if you if you drop in a model and you were getting 20% or let's say you were getting 90%. And you drop in a new model, you get 50%. That's a very easy like, nope, weird. It is whatever the amount of work it might take to make this thing get back to 90%. Is probably not not not worth it.

13:09 >> Right. Right. And you know that's something else that's interesting is when a new model comes out I mean I'm I also just glance at the benchmarks to see is it is it generally better than the other model but I mean you and Vivov just did an episode about the state-of-the-art benchmarks having let's say limited purchase power right >> well and it's also the the thing about benchmarks is like every model now can get a 99% on sweepbench and every model now can get like an 80 to 90% on on frontier frontier code or not frontier code on on sweet bench multilingual and the frontier code averages I think for opus 5.5 was like 55% and like soul is like 60% or something like this like I don't I don't know the exact numbers but the idea is like for some benchmarks we want them to be truly unsaturated right which means like hey look like there is a lot of room for improvement And in Slap Codebench, it's like, oh, most models get in the 10 to 20% range. If you're a model builder or you are someone who cares about like, okay, what is the frontier? What is the hardest problem we can solve? Then you want an unsaturated benchmark. But when you're building your evals for yourself, you probably actually want a very saturated benchmark because your whole goal is to get to 99% accuracy or 99.9% accuracy, >> right? And even looking at the state-of-the-art benchmarks or the release benchmarks that only has limited utility because you don't care you don't actually care about the performance of those benchmarks. They care about the performance in your workflow.

14:46 >> Right. Exactly. >> General benchmarks are very rarely to like likely to give like good enough signal on on the exact work that you're doing which is >> out of distribution and is necessarily almost like eval become very proprietary to your company like they're an important part of your IP, right? your intellectual property, >> right? And another interesting question here is how how do we get our test cases, right? and so I think I mean there the obvious answer is just label a bunch of data, right? You get data, you label data, you sample data from production to make sure as your system, you know, as time goes on, the data the shape of the data doesn't change.

15:31 >> yeah, let's let's talk about that. like what is what is your actual actual loop for doing that in in this method? So you have like some app running in production and you have some like codebase and then you have some users somewhere using your thing. Right. >> Right. Right. >> I don't know why these are going to be these colors but I love it. And you kind of have this loop of like, okay, you you you manage and then I guess like we can add one more. We can have your like engineering team.

16:04 >> Yeah. >> What color did they get? I guess they can be blue. And so you have users using production and giving you feedback. And then you probably also have some kind of thing where like you're just pulling data from production on some regular basis. And all of these things happen to be like changes in your codebase which contains you know your prompts your pipelines and your evals. >> Exactly. >> I'll call them test cases since eval is a little bit but yeah. Okay.

16:35 >> But something that I think and I'm curious to get your take on this Dex. You can kind of cheat a little bit with this because label data is expensive. you do need a golden set and you need to evolve that golden set over time. But when you're doing something like changing out a model because of a deprecation or an improvement or whatever, the question that you're trying to answer is not actually how accurate this model is. It is how does it compare to the current model in production.

17:05 >> Yep. So for instance, so you can just do essentially a diff, right? So, if you have a model where say you're running, I don't know, Sonnet 4 or Gemini 2.5. Let's say Sonnet 4 because it's easier to say Sonnet 4 and you run Sonnet 5 against it. If the results agree one to one, you don't if because Sonnet 4 is in your production system right now and if it's good enough and Sonnet 5 agrees, you can run these diffs on thousand cases really really really easily or 2,000 or whatever because you don't actually need to handle label any of the data, >> right? All right. So you can get a very very quick Yeah, you can get a very quick signal about >> is this really how similar is this to my current system? And if it's the answer is very very similar again the once you've defined what good enough is right. So you say it it's okay if it differs in 2% 5% whatever the case is you have to decide that and then once you decide that like just checking those diffs is really fast and really cheap and the ons >> so looking at the difference in output from your incumbent model with your candidate model if they're the same right if all the results are the same then there are no diffs, right?

18:33 >> What what does our output look like in this case? Is it a is it a structured object? Is it a plain text summary? Like like help help me understand like just like a little bit more the the shape of the inputs and the outputs to this problem like what is what is a what does an average test case look like? >> Yeah, I mean that's a great point. what I what we do is all not all a lot of what we do is structured extraction. So because we're extracting medical facts, we're extracting surgeries, we're extracting doctor's names, stuff like that. So it's structured extraction. and so that's a pretty straightforward case of you know right versus wrong or these two are equal. Free text summary is a lot more challenging, >> right?

19:22 Because in JSON data you can have you can have an object and you can say you know like patient name something >> you know you diagnosis is some like enum or whatever it is I I don't know and so you can take this like long like sort of like unstructured maybe it's an image or a scan of a doctor's notes or whatever. I'm I'm being very very loose here, but you want to see can I turn this if I if I take this input document, does it generate this structured JSON? And you basically just have like a thousand of these you said, right?

19:59 >> Yeah. Or however many you want, >> right? Yeah. >> And handlabeling those, getting a golden data set from those is expensive, right? That's hard, >> right? having a human read it and then go fill out a form and then you save that somewhere and then it becomes a test case. >> Exactly. That's that's timeconuming and expensive. >> Yep. >> Running two LLMs in parallel on a thousand documents and just comparing the JSON output is very fast and cheap. And that gives you your initial answer of how similar is the output from these. Right? So that gives you that answer. It doesn't tell you about like latency and speed and cost. I mean, you can you can do that analysis, but that is kind of a a a shortcut for test cases. And in ones where they disagree, those are the ones I I think are probably the most interesting cases that I would kick to a human or an engineer or an ex a SME, have that person label it, and then add that to your golden data set for like future.

21:05 Oh, so you look at where Sonnet 4 and Sonnet 5 disagree and you hand those to a human or some subset of them to like confirm whether it's like, okay, is Sonifi actually worse at this or did Son 5 actually do a better job, but it just didn't match what we wanted to extract. >> Exactly. Cuz my instinct is that those are the interesting cases, right? Those are the hard problems that we need to make sure kind of all of our future models are able to solve correctly as well. Okay, cool. So, we have our outputs old, we have our outputs new, and then we have by looking at both of these, we have like basically like three, we have like two sets. We have the ones where like same answer and then we have ones where we have like a different answer.

21:59 and so we can then, so the pass rate gives us a very like gut check like are these about the same? But then these actually give you you take these like this set of different answers and you kind of like can go through and like relabel it or like confirm label it or something, >> right? And then that gets added to your kind of ongoing golden data set. And that's a good way to build a golden data set over time.

22:23 >> And then this goes into your we'll do this. We'll do the 2,00 test cases as like one thing that gets used across both of them, right? >> Mhm. >> The theory makes sense. I'm curious like what have you found that works really well to kind of operationalize this, right? Because like I have a bunch of prompts in our system and you know they're all use specific models to look at coding agent sessions and then decide if they are decide if they are sorry I was going to write diff decide decide if they are whatever it is the next step is being suggested or whatever guidance we need to give the user and like I could go make another version of them and like run the tests and get a pass number or a list of things but like nothing there is actually like snapshotting the outputs I think I think that's what makes like eval from from a lot of like like we like to say like eval are basically just unit tests right and it's the same concept of like Hey, look, like you want to be able to push a button and know in some amount of time like, hey, all of the things that worked last week still work this week. without having to go test a thousand different prompts, like type them into the app, literally.

23:38 >> Yeah. No, I would I would I've never thought about it framed that way, but I would I would definitely agree. >> Yeah, we used to do we did a we did a migration project when I was at Sprout. I wasn't like on this project but they presented it at like the engineering meeting where we had this like old API in Django which is a Python framework and we had like moved some like 10 endpoints to Java right and so I was like okay the highest volume endpoints we want to use a like a higher performance programming language that easier to optimize and easier to maintain and has like strict type checking. This was in 2015. So like there was no there was no types in Python. It like it was like basically a research paper that no one had done anything with yet. and so we had this and and we did this and basically what we did was we like took a sample of real production traffic. So like in in your log you have like git slashendpoint and the response.

24:35 actually you just had git/endpoint postendpoint etc etc. You had like you know thousands of requests. You could just pull these from the log. So like when the app is being used, pull a ton of examples of this. And then what we would do is this was our data set. >> Mhm. >> And then what we would do is we would we wrote a script that would basically it was like step one was like replay the requests against the Python.

25:07 to replay the same requests against the Java. and you would get out of this, you would get like basically you would get a like progressive diff where it was like if the like HTTP status was different, we would kind of like abort any more like diffing. Basically like, okay, if one of them got a 200 and one of them got a 500, then like cool, we have to like go fix that. And then it was like the response body.

25:39 >> and then we would diff that. And then there was literally just like a giant table of cases of all of this data of like status. Oh my god. Okay, let's do this. Status body. And then we also had like a latency number, right? >> Mhm. >> so we skipped the manual labeling part of this. We just like took real production traffic and like pushed it through the system after there was like manual labeling but it was like someone wrote five test cases for like they wrote one happy path test case for each endpoint. and so like this would be like a pass, this would be a fail and then we would have like latency old, latency new.

26:29 and so like obviously if this was slower it would fail. Mhm. >> Or I guess I guess we can have like latency. We can have like also like a latency pass, right? >> And if there was a body failure, you could like expand this row and you would actually see like the the actual diff. I don't know if I'm going to be able to draw a diff, but you would be able to see basically like, hey, cool. The old one had how are we going to do this?

27:05 You know, this a bunch of stuff was the same. >> Mhm. >> And then you would have like okay, the old one has, you know, the old one had this stuff in it and and then the new one had this is really dumb to try to do this in Excal, but you get where I'm going with this. Yeah, I'm impressed how fast you're coming up with this on the fly. >> I mean, like I said, I didn't come up with this. I just watched some someone smarter than me.

27:35 >> No, I mean come up with a way to do this in Excala draw on the fly. >> you know, it's one of my one of my lame superpowers. and so you could go any engineer could go and look at this and they could say like, "Okay, that's a false. I don't care if the time stamps are different." Like we actually don't care. We actually ended up building systems to like strip out all the things that like >> didn't need to be diffed. basically was like okay if these two things are different basically just like slurp all the fields we don't care about out of the JSON and just diff the meaningful stuff >> and then you know you would have a giant table of these things right so you have pass fail you would have you know hundreds or thousands of cases here of like different things that an engineer could go in and you know go inspect and they literally just did this they just ran this and they pulled in new traffic every week and when it was all green. Then they did the cut over and then they moved all the traffic to the new service. But they basically got to like ghost test this thing >> in I don't I don't know how they did the post request, right? Because you can't take customer traffic and like redo because it it was like sending tweets and you don't actually want to like resend a tweet. So there's some there's some art here of like how do you how do you give the system under test kind of like >> make sure it's not actually interacting with the outside world because otherwise you're like doubling the amount of traffic.

28:57 >> yeah. >> But, yeah, this was this was how we did it. And I imagine doing this with AI is like very similar where you basically get a big table full of diffs. >> Yeah. Yeah. That's that's how I would continually kind of build my golden data set. >> yeah. So, I've got so I built kind of a really slow version of this. I have a demo, Dex. >> Amazing. I don't know why you're letting me share the screen the whole time.

29:24 because you're just you're you're just that good at Excaladraw. >> I hope that's not what I end up being known for. >> So, what we have is this is structured extraction. I've got a bunch of test cases just pretending to extract receipts. This is all this is all made up data. And the point here is not which model is better because again, this is made up data. So, when one model is is it says it's wrong, it might actually be correct. So it's this isn't a lesson on which model would I suggest using.

29:55 This is just how do you build a system that can answer these questions. So we have a bunch of test cases here, right? I think there are 30 test cases and then in our clients we have where we can see so a lot of what's interesting is when I was kind of playing around with this our a lot of our repo uses GPT40 which is not deprecated technically the endpoint is still alive but it is disappeared from OpenAI's website the pricing information is really hard to find it it like they pretend it doesn't exist even though it still does for now.

30:33 I'm sure it'll be deprecated eventually. >> So, if we're pretending that we have a system that's running on GPT40 and we say, "Hey, like I know it's going to be deprecated or there are a bunch of new models that have come out. Let's see how they perform." >> Can you code a little bit by the way? >> Yeah. Yeah, definitely. >> Thanks. >> I forget I forget how big my screen is. >> Yeah. >> Okay. Is that good?

30:55 >> Yep. >> All right. So we see here's our incumbent and then we have a bunch of candidates, right? So we have 40 mini which you would just replace if if you want you know faster or cheaper. We have more GBT55 and then we have a bunch of you know sonnet 36 flash 35 flash and then 35 flash light right and you can just make whatever clients you want. and so then we have our actual prompt which is just essentially extract the expenses from this receipt and we have the expense policy that gets dumped into the prompt, >> right?

31:38 >> Very straightforward >> instructions and then you got your your output schema, >> right? And then so going back to kind of what we decided what we talked about before of like what do we care about and what do we consider failure or success to look like, right? I said I care about accuracy, the cost and latency. Those three dimensions are what I want to measure all these new models against. And I created a budget class that has the max. So these are our like our gates, our thresholds, right? So max accuracy drop allowed is zero.

32:18 >> Yep. >> Max cost multiple 20% more and latency P95 latency 50% more. And again like you can this is the hard part of deciding what good enough looks like I think right I think that's the hardest part of this because you can change this to whatever you want right you can change this to 0.9 and the point is that is I would argue kind of the most important part of this and also the hardest and so you make it really really easy to change things and to swap other numbers in.

32:51 Okay, >> so this is the budget and then if we go to our the gate. >> Okay, so this is your Python harness that is actually like executing these evals. >> Yes, exactly. And what you can do is you can create a gate that has these checks. So here we've got a data class called check. And you can see okay, if I go up to the definition of the data class, right, we have the name, okay, which is essentially pass or fail, and then details. And then we have the overall result of the the gate, so to speak.

33:32 >> And all you have to do is you have the accuracy check and then you define okay as less than equal to that max accuracy drop that we just looked at. >> Okay. >> And you have the same for latency. And then cost. I had to create a little helper function for cost because if I just did it like the other two an unpriced model would have passed and we don't we don't want that. So this has a little kind of right if cost per call is none or baseline cost per call. So if either of the models cost per call is none then you know we fail it essentially.

34:13 >> So >> this is just three checks. And what's what's nice about this is >> surfacing to the person who's reviewing the results like by the way we can't give you any info on cost here. We cannot tell you that. Yeah. Okay. >> And and this works really really well because you can see how easy it is to add another check. Right. If you wanted to check I I don't know like the median latency and in addition to P95, you would just add another check. you would add it to the budget and then you would have then added it to your gate.

34:49 >> Okay. >> and so the CLI actually runs it and we can I can show you what that looks like. So let's see. So let's see. I've got a couple commands. Let's see. It's UV around Python. So this is when I ran everything >> inspecting the data. >> I'm sorry. >> This is inspecting the data you already collected.

35:20 >> Yes. So this I mean it's cached. So I wouldn't wouldn't have to sit here and watch it run, right? >> Yeah. >> But if it's not cached, it would run all the different models that we had as our candidates. it would run them against the gates we just looked at, right? And so we can see here this is our candidate, right? Obviously latency and cost 1x that's a good check, right? >> and we can look at all everything else here, right? So we can look at okay, so sonnet got a perfect accuracy, but the latency is two it's it's 2x latency and 1.6 times the cost. That's too expensive, >> right? And we can and and there's another command that'll >> and slow. It's twice the it's twice the time, half the speed.

36:14 >> Not worth a 3.3% accuracy increase, >> right? Or maybe it is, right? I would I would say it's not. And it's probably not, but that's the kind of thing like you know your system. If you're running a BTOC company with millions of users and a 1% difference means a million dollars, then maybe it does make sense. >> Exactly. Maybe it does make sense. And that's that's the hard part. And that's why this is so interesting, right?

36:41 Because you can easily swap these numbers out and you can get you'll this entire table will change. and for this instance, 3.5 flashlights, it only won that one. It it's it's faster, cheaper, and better. So it wins on all three dimensions. >> But incredible. >> Again, these are just madeup numbers, so it might actually be worse. Who knows? >> but the latency is real. >> I did run it, but I didn't handle label all the test cases.

37:10 >> I see. Okay. Okay. >> So, yeah, the test cases are generated. >> Yeah. >> but the latency and cost numbers are real. >> Yeah. Cool. How many test cases is this? >> 30. >> 30 test cases. Nice. Yeah. Yeah. But what's neat is if you do this, right, if you run uv run python harness dot yeah dot show gate and then you do you can change the latency.

37:42 Actually, let's just run show gate. And that shows everything that passed or failed. Right? So I've got two two gates that I defined in the budget. One is survive the swap, which is yes, we can kind of just swap it in. Another is does it actually drop our costs, right? Like if you know that's the that answers the question, can we replace things with haiku or flashlight? >> So, where was I? Doesn't matter.

38:13 >> okay, so you could and that's useful if you want to run this against a 100 models basically with a small test set and just basically say like cool, I don't want to I don't want to read a table of 100 results. just filter it to, you know, max latency is x or y. >> Mhm. >> Or max cost is, you know, 50% or less of the baseline. >> Right. Right. and then what you can do is you can just change one of the variables. So you can do latency 3.0 and say actually I'm okay with the latency being three. And you see life, >> right? Three times as slow.

38:53 here Gemini 3.5 flash and 3.6 six flash now pass, right? So you can change that >> really rich data for every single execution to be able to do this. Huh? >> Because you need P95, right? >> Yeah, but that's not that hard. It's not that hard to get those. >> Can we look at what the data looks like? Like is this just like a big JSON blob? >> we yeah.

39:23 Okay. Okay, cool. So it's actually just aggregate for the run. You're not you're not storing the result of every single case, right? >> No, no, no, no. >> Okay. And you're not catching the actual like logic failures basically. >> Mhm. >> Yeah. >> Nice. >> So, but again, right, this is what makes this really nice is you can play with these parameters and you can instantly see which ones pass, which ones fail. Have you tried plotting this in in in 3D space or 4D space?

39:58 >> No, I haven't. I have not. >> Do you have do you have Opus 5 on here? Do you have a Do you have a Cloud Code shell? >> Oh, do I? >> This is always my favorite. I They've gotten quite surprisingly good at data visualization. >> That is surprising. All right. >> Can you build? >> Yeah, go for it. What should I what should I ask? >> Let's let's say can you build me a visualization of matri you do like atmatrix.json JSON like in 3D on a on a grid I don't know in 3D as an HTML as a static HTML file.

40:46 Yeah, let's try it. So while it's doing this, another thing that you can do that I tried and that didn't work for my specific case is BAML has a prompt optimizer built in. So if you so one of the things I played with was, you know, like we said, Sonnet 5 was perfectly accurate but a lot slower and more expensive. So you can run the optimizer on just the token input, the input and output tokens. And so what you you can build into your harness like because in our case 3.5 flashlight wins on all axes. So you're like great just swap it out.

41:25 >> Drop it in >> right. >> Oh but if you have like what is it they call it like the paro frontier of like one of them is the cheapest but another one is the fastest and the other one is the most accurate then you can kind of try to optimize across those. >> Exactly. And if you're stuck in a situation where your model is deprecating >> Yep. and you have to switch a model and so you have to decide between you know these different situations where no one strictly wins. You can run that Jeepa optimizer and it can take the different pieces of the frontier and create what is you know ideally an optimized prompt that that is your kind of your best of all three. So that's a really neat thing that you can do and the whole point of this is model deprecations are announced far in advance and that can either be a two week scramble where you're trying to change everything it can be a P0 or it can be 30 minutes or an hour on a Tuesday morning right so >> you it's you know these things are going to get deprecated and So you build your systems in a way that expects it and makes it easy to to upgrade is is the lesson of this whole thing.

42:44 >> Yeah. No, this is great. I mean this is this is the proper engineering, right? I don't know. I've worked with some teams in early human layer days. We did a we did some more like forward deployed of like working with larger companies of like hey how do we figure out our AI problem and it was like >> the the engineers were just like kept trying new prompts and new models and new tricks. I just like can we get this to work well does it work over here and it was like okay one day like one like of the product managers like test cases was working and then a week later like three of them were broken and there was three other ones that were now working and you're just kind of like flailing a little bit.

43:23 >> all right. Tell it to open it in your browser. >> and what I did was I was like, you need to label some data. You need someone to like make a spreadsheet of like 30 of these cases. Not that it's going to take you a day or two. It's going to suck. >> But then anytime anyone makes a change, you'll be able to get one number that says this is crazy. >> Wow. Oh, >> yeah. I think you can drag it around too. And there may be like hover.

43:48 >> That is bananas. >> Yeah. So there's flash right all the way up in the very top corner. >> Oh yeah. Yeah. You can see that. That is so cool. >> I mean this is this is not that useful of of a visualization because they're all kind of clustered together. But yeah, this is the this is the >> these are some of the things you can do. >> But they're they're not though. All right. I mean these all are. But you can see 5.5 is way over here.

44:13 >> Yep. >> Flashlight is up here and 4 point or 40 mini is way down here because we don't have the costs. So it kind of just doesn't measure on that axis. But still like this is really cool. >> And if I were presenting this to a non-engineer or a PM or you know a a engagement manager or solution, someone who's not in the code every day, this is how I would present it. Absolutely. >> Yeah. I would probably say there is a there's a projection of this of like okay just take the ones that got 100% accuracy and show them on a 2D grid versus yeah >> whatever else you're looking at. But like yeah in general like the the models have gotten good and so it's like again you can't you can't build this visualization if you don't have the data and you can't generate the data unless you know what accuracy looks like. I mean latency and costs are easy right?

45:04 Those get basically like any any any programming language, any model can just like vibe out latency and cost stats, but like the accuracy stats is the really hard part. And like without that, none of this is really that useful, >> right? And I think >> my temptation as an engineer is to just is to not want to label data because it's boring and it takes a while. >> Yeah. But it is very very important to have a golden label data set. Yep.

45:33 Someone says, " try do a 2D viz on accuracy versus cost with P95 boundary lines." >> What else we got in the chat? What else do you guys want to see? We're going to play we're going to play spend spend Kevin's tokens. >> My favorite game. and P95.

46:05 >> This is fun. >> Anyways, yeah, this is super helpful. I mean, like it's kind of the same thing we say every two or three months. We come back on the show and be like, "Look at your data, label your data, test your stuff." And like again like we said even on the very early evals episode exactly what Vonyy said, is there an intuitive way to know which models would work for your use case instead of going through evals every time? Like yes, before you go build a giant label data set, you should do a little bit of vibes. you just like paste the same paste one one case that you care about of like hey let me just drop this photo of a driver's license and do the same prompt with five different models and just read the outputs >> and see what's different or like diff the outputs across the models and like that intuition is always useful and it's always worth like building up and then again we did with the with the with the JP Morgan like emails or the the Enron emails thing of like we did one to test it and then we did 10 and then we refined it a little bit and then we did a hundred and we refined mind a little bit more. And so it's like, yeah, you should always start from vibes and you should always look at the actual data and then you slowly grow your leverage as you get more confidence and you go from 50% to 80% to 90% to 95% to 99%.

47:16 >> Well, one other thing is when you say go on vibes, right, the as you continue doing this, the cases where the models have disagreed in the past become really, really good cases to check vibes on it at the beginning when you have a new model come out. Yep. Ram has a question. Any advice on how to do this with summaries instead of structured data? Like if the outputs of your model are unstructured? >> That's a great question. Summaries are hard. I mean, you can do there there are a couple of of ways you can do it, I think, depending on the goal of your summary.

47:55 and one and the most obvious thing to do is an LLM as a judge. So, you have an LLM as a judge. just pick which is a quote unquote better summary and you make sure that you have a really fleshed out definition of what makes a summary better for your use case and then you just have it essentially pick which you know which model is better. and then the second one is depending on, and this is kind of I think rare and maybe more kind of niche, but if you have a summary that has a lot of structured information in the summary, you can then ex have another layer that just extracts the summar the the structured information and that is what you check. So you make sure none of the important information is hallucinated or lost.

48:43 And again, that's more specific, but I have seen that use case before where you have you know, you're writing, say you're writing emails where you have information pulled in from a lot of different contracts or things like that. >> Yeah. that's essentially a summary, but what you really care about is that the structured information in that email is correct. >> Right. >> But I'm I'm curious to hear, Dex, if you have any other thoughts as well.

49:09 >> Yeah, I mean it's something basically along those lines, right? I mean, and we did this in the like doing evals on a on a on the email data set too is like you will want to turn it into structured data at some point. I think some people have for I think most people understand that asking a model to score something on like a 1 to 10 scale like doesn't really work that well. yeah, but that but that if you can tell the model like okay does it have this characteristic? Does it have this characteristic? Does it include this data point? Okay, how many times does this thing is this thing mentioned?

49:47 And you can basically turn the even if the answer is is is free text and the thing your product exposes to users is free text, you can still do structured output on that unstructured data so that you can process it in a program. and we do this for ranking algorithms. All the way we run the AI tinkerers website, right, is like everybody who signs up for an AI tinkerers event gets a zero to a,000 score across a bunch of different dimensions. And it's like how often like are you pushing to GitHub and like do you have like software engineering in your LinkedIn and all this stuff to try to tell like are you an active builder who's going to have interesting things to share and interesting things to teach the other people. But it's all comes down to like take this giant blob of data of like unstructured and structured and this whole mix and turn it into like a like scores on a giant rubric, right?

50:35 yes or no, boolean or like enum classifiers, maybe counting, things like that. Some of it you don't even need a model for, right? You can take, hey, cool, like go get this person's commit history, turn that all into a number completely deterministic, and then just include that in the score. but you want to make the actual output of the the accuracy score be as deterministic as possible. Whether the model's output is structured or whether you do structured output on top of what the model output.

51:05 Yes. I think this is how like the very early days of like generic eval platforms for chat bots would do is like they would always give you like a a sentiment score and it's just like if the user seems angry you get you get you get one enum and if they seem happy you get another enum and like that goes into the score of how is the chatbot doing. >> Yeah. Yeah. >> so that's the name of the game.

51:29 That's the same show the same show every single week is do structured outputs. There is no magic. Everything in AI engineering is structured outputs. everything else is just more abstractions that you might not need. >> That's right. Just window dressing around that. That's how you make that works. >> Amazing. Thank you all so much for coming. next week's episode, we might talk about slop code bench. We might talk about something else. We got a lot of fun topics in the hopper. Kevin will be yelling at me and Vibb on Slack to get us to pick something and then we will publish it and we will tell you what it is. But I guarantee you it's going to be dope. So, thank you all for coming and we'll see you next week. Bye everyone.