transcribe

How to Use Decision Models for Evals (Jev & More)

Hamel Husain · 14m · transcribed 17d ago
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Decision Models

What are decision models and how can they be evaluated?

Decision models like Jev from Type-Safe provide fast and cost-effective classification. The video discusses testing six decision models to determine their usefulness for evaluations and how to choose between them.

  • Decision models can classify data quickly and cheaply.
  • Evaluating models involves testing their performance on specific tasks.
  • Choosing the right model depends on the evaluation criteria.
# 2:58

Precision vs. Recall in Business Decisions

How should businesses balance precision and recall?

Businesses must consider the importance of precision and recall based on their specific context. For example, cancer screening prioritizes recall, while criminal law prioritizes precision.

  • Different applications require different balances of precision and recall.
  • Understanding the trade-offs is crucial for effective decision-making.
  • Visualizations can help clarify model performance in terms of precision and recall.
# 5:57

Tuning Decision Thresholds

How can decision thresholds be tuned for better model performance?

Tuning decision thresholds involves selecting a metric that reflects the importance of precision and recall. The F1 score can be used, but businesses should consider their specific costs and benefits.

  • Choosing the right metric is essential for tuning decision thresholds.
  • Businesses should assess the financial implications of false positives and negatives.
  • Using tools like spreadsheets can aid in decision-making processes.
# 8:56

Impact of Threshold Tuning on Model Performance

What is the effect of threshold tuning on model performance?

Threshold tuning can significantly impact model performance, with some models benefiting more than others. It's important to evaluate and visualize the performance and cost of different models.

  • Threshold tuning can lead to improved model performance.
  • Visualizing performance against cost helps in model selection.
  • Different models may require different approaches to threshold tuning.
# 11:55

Calibrating Model Scores

How can model scores be calibrated for better decision-making?

Calibrating model scores can align predicted probabilities with actual outcomes, using techniques like Platt scaling. However, not all models will benefit from the same calibration method.

  • Calibration techniques can improve the reliability of model predictions.
  • Different models may require different calibration approaches.
  • Understanding the context of score usage is key to effective decision-making.

Transcript

0:00 There's been a lot of interest recently in decision models like Jev from Type-Safe. And the reason these models are so exciting is because they offer a fast and cheap classification, which is really useful. Now, one natural question that comes up is, are these models useful for evals? And how do you choose between them? So, what we did is we tested six decision models head-to-head. And in this video, we're going to show you how you should go about choosing between models, but also tune these models to get the best performance on your evals tasks. Let's get right into it.

0:29 >> Just to set the stage of this experiment, we used 100 traces, which were just logs of user sessions of a real production AI assistant for property managers of apartments. And what this assistant does is it helps those property managers field inbound requests from prospective tenants with questions about the property, booking appointments, but also from current residents that have questions about their apartment, like maintenance requests. And so, the task of the decision models is to find issues in each of these sessions. Now, a little bit more than half of these user sessions in this experiment have an issue. And those issues are things like failure to hand off to a human or escalate to a human, or failure to make an appointment correctly, or failure to communicate in the right way to the prospective tenant. And so, we tested six models. We tested Jev 1.13, Perplexity Decider 1.1. We tested Liquid D1 and D1 Ultra. We tested Cloudflare's Clef Flash. And OpenAI's Decisions, which is based on GPT-6 Luna. We gave each model the same prompt, which instructed it to find issues in these sessions and flag which one of these sessions had a problem. And we compared each of these models on two dimensions: performance, which we'll talk about in a little bit, and cost. Before we get into cost versus performance, we need to talk about decision thresholds. Now, just as a reminder, we instructed each of these decision models to output a flag, pass or fail, indicating whether or not it thinks there is an issue with the trace.

2:02 In addition to that, each model also outputs a score on the range of zero to one. Now, by default, each of these models have a default threshold of 0.5. It's really important to think about this threshold and not just use the default cuz it's very likely that the default is not optimal for you. And there's two things that you need to think about when choosing a threshold. The first one is precision. So, precision means out of all the things that the model has flagged as errors, which ones are actually errors? In other words, how many false alarms is this model giving you in terms of what it thinks is bad? And then there's recall.

2:36 Recall is the ability of the model to actually find all the errors. Now, there's an inherent tension or trade-off between precision and recall. So, the more discriminating we are, meaning the less false alarms that we want, the less failures we're going to catch cuz we're going to set the threshold higher in order to flag something as a failure. On the other hand, if we're more interested in catching lots of things, then we're going to have more false alarms. So, your job is to think about what balance of that is important to your business.

3:06 So, let me give you two examples of this. One is cancer screening. In cancer screening, you care a lot about recall. It is okay to have false alarms cuz you can give second and third tests to know if someone really has cancer. Whereas in criminal law, you care more about precision. It's okay not to catch some criminals. You want to have a higher threshold to make sure you don't accidentally convict innocent people. That's the way that we think about it.

3:31 And so, similarly for your business, you want to figure out how much you care about precision and recall. And to make this a little bit more concrete, this is a visualization that helps you understand what this looks like. And these are all six models, and it shows you the distribution of these scores, which are on the range of zero to one for both pass and fails. So, orange is failure, blue is pass. And what you want is each model to assign lower scores for pass, higher scores for fail, but more importantly, we want the models to separate each. We want pass to be on one side and fails to be on another side. In a perfect world, we would want to draw a line somewhere in the middle and perfectly separate the two. Usually, you won't be able to perfectly separate the two because usually there's no such thing as a perfect classifier, unless you're working on a really easy problem.

4:17 So, for example, in this graph, we can see that OpenAI Decisions does the best job of separating pass and fail. You see all the blue kind of cases are on one side and all the orange ones are on another side, and we can discriminate between them. So, in this particular visualization, the dotted line is the default decision threshold. Now, you may want to set that threshold according to what's important to you. But before we get there, what you usually want to do is set a minimum threshold for precision and recall. So, what I mean by that is, "Look, if I'm going to use this eval for a real-world alerting system, I want to make sure that's not giving me too many false alarms, but I also want to make sure that is catching enough failures for it to be minimally useful." So, before you try to tune the threshold, what you want to do is set a minimum bar. And this visualization helps you helps show that. This visualization is called a precision recall curve, and you can see on the Y axis, we show precision, on the X axis is recall, and we plot all of the different precision recall scores at different thresholds for each model. And this green quadrant is where both precision and recall are above 80%. So, in this example, I'm saying my minimum threshold for each is 80%. And when we set that minimum threshold, only three models survive: OpenAI Decisions, Liquid D1 Ultra, and Perplexity. Now, this is not to say those models are always the best. In fact, this is going to change with every task and use case, your results are going to be different. So, don't take anything I say as one model is always better than the other. I'm just showing you how you can do the analysis for your task. So, you can see even though we've set this minimum threshold, we still have to choose the threshold. Like in this OpenAI example, there's a lot of different thresholds to choose even when you set that minimum of 80% recall, we can choose any point along this line depending on how important precision and recall is. The other models have some flexibility, too, of different precision and recall, but not as much as OpenAI in this particular example. So, by the way, you can graph all of these things in the same way I did by just pointing your agent at this blog post, which I'll leave a link to in the description, and ask it to reproduce this graph for you.

6:23 So, don't get overwhelmed by this and think that it's too technical. AI can help you do this visualization. So, for this experiment that I'm showing you today, actually came out question that we got about using Jev for evals. And so, we get a lot of questions like this, and I put together this free AI evals FAQ based upon all of the questions that we get during our AI evals course office hours. We're always adding to it, and I'll leave the link in the description below. So, the next thing you want to do is tune the decision threshold. Now, before tuning a decision threshold, you want to select a metric. And you want to select a metric that represents how important precision and recall are to you. So, in the examples I'm going to show, just for illustration, I'm going to assume precision and recall are equally important. So, I'm going to use something called F1 score. But, it's highly likely that precision and recall are not equally important to you. What I recommend is getting a spreadsheet out and assigning a dollar value to the value of catching failures, and also the dollar cost of what a false alarm is.

7:24 And doing a little bit of a thought experiment of, "Okay, how important is precision? How important is recall? How should I weight them?" There's something called an F beta score that allows you to set a number, like a weight, and you can set beta below one to give precision more weight, and beta above one to give recall more weight. It's like a slider. Don't worry about this too much. You can look up the definition, and you can play with it according to your needs. And you can get AI to help you with this. The idea is pick a metric, either F1 or F beta. If you notice, F1 is really a subset of F beta, where just one is equally balanced between precision and recall because it's a one, so the slider is in the middle. And I have a prompt here that you can give to your agent, and I would recommend modifying this so the prompt says, "Use my saved model scores and binary labels tune a decision threshold to maximize F1." Now again, that's the metric that I'm optimizing while requiring at least minimum of 80% precision and recall. I also say, "Use stratified five-fold cross-validation choosing thresholds on the training examples and measuring performance on the remaining examples." Then I say, "If comparing models, use the same splits only select thresholds that meet both minimums on the training examples." So this sentence is just there to make sure we're not over-fitting our threshold on the data we see. We want the threshold to be good on data that we don't see as well, and this helps with that. So how big of a difference does threshold tuning make? It can make a huge difference. This visualization shows what happened after threshold tuning.

8:55 Now, the models on the right-hand side that are gray, we ended up kicking those out, so we're not using them anyways, but threshold tuning on them would have made a huge difference. On the remaining models, we see that some models didn't need threshold tuning at all. For example, Jev, we could see it didn't really make a difference. The reason that threshold tuning dropped the score a little bit is because we used that five-fold cross-validation and we didn't overfit on the data. If we overfit on the data, it would just be point it would be just overlapping. But you can see for some models, it did make a difference. Like for perplexity, it did raise the score a bit. And so, don't discount this. Even small differences can make a big business impact. Either way, you should try to tune your threshold or just at least look at it because you never know how much performance you're leaving on the table.

9:38 And by the way, this would change completely if you tried to use a different metric or if you had a different data set or a different task, your results are going to be very different than this. But it's important to at least think about your threshold. After you've decided what threshold is right for you and you've tuned that threshold, etc., now you can compare performance and cost. And I highly recommend visualizing it, especially if you have a lot of models to compare. On the Y axis, you have your performance score, which in our case is F1, and on the X axis, you can put cost. And you can can the different models are plotted here, there's six models. These three we took out of consideration because they don't meet our minimum precision recall requirements, and these are the three left over. And this dotted line is like a Pareto frontier. Pareto frontier just means that you likely just want to choose a model somewhere on this dotted line. For example, on this particular task, Liquid D1 Ultra doesn't make a lot of sense because it's offering the same performance as Perplexity, but at a much higher cost. And then, of course, OpenAI Decisions is giving you a lot more performance, but it does cost more. And again, you can point your agent at this blog post and tell it to do the same thing if you're wondering how to do this yourself. There's one more thing that you might want to know about when using decision models, and it's how to interpret that probability score. A lot of people want to use that probability score as some kind of confidence score.

10:57 And the question becomes, if you get a confidence score of 0.8, is there really a 0.8 likelihood of failure? And the answer is usually not. You have to be really careful. You have to check to make sure if that's true. If it's not true, there are ways that you can fix it. What I recommend is plotting the data. And what I'm showing here on the screen is a probability calibration chart. And the way this chart is made is we create bins of similar probability scores. In this case, we have five bins, and we compare the predicted probabilities to the observed failure rate, cuz remember this data is labeled.

11:30 And we want to see that they are similar. For example, a bin that has an average predicted probability of let's say 0.8, we want to see 80% failures in that bin. If it's way off, then we know that the probabilities are not really calibrated to reality. And so, that's what this graph shows. The Y axis is observed failure rate in a bin, and the X axis is the predicted failure probability in that bin, and we want these points to be on the diagonal line.

11:55 We want them to agree with each other. And for example, in the second bin here, we have a predicted probability of about 8% but in that bin, we have about 60% failures. That probability doesn't really match reality, and you shouldn't interpret that score as like some kind of confidence score. you would be way off. There are ways to transform or calibrate these scores to more match reality, and it works like fitting a model. Think of it almost like linear regression. There's a lot of different techniques. For example, there's something called Platt scaling and isotonic regression. We don't want to necessarily get in those rabbit holes.

12:29 What I'm showing on this graph is the result of Platt scaling. And again, you can just tell your agent, you can point at this blog post and say, "Hey, can you plot the data like this and can you try Platt scaling or isotonic regression, whatever you want to do." And it will do that cuz the blog post has a lot of the information in it that you need. And you can see like, "Okay, for Cloudflare, we're able to calibrate it and make its probabilities more aligned or calibrated." But for other models, like this Platt scaling didn't work. Like this is how OpenAI looked. You can see like there's a little bit of a sort of an issue with the the way the probabilities are spread out. And like Platt scaling didn't really help. So when I might need to use a different approach here. But again, this is only if you're interested in using the score for decision making. You may not be interested in it. If you're just trying to use it for some kind of binary decision making, yes or no, which is what you're trying to do for evals, then you really don't need to worry about this. You just need to be setting the right threshold. It's just that so many people are interested in using these scores for other things that I think this is worth thinking about. And again, the post has the prompt that you can use for this. I'm not going to read it, but just know that you can point your agent to this blog post, and again, I'll leave the link in the description, and have it do it for you. So just to recap, here are the things that you should do when choosing a decision model. First, pick a minimum precision and recall threshold that you can use to weed out models.

13:48 After you set a minimum precision recall threshold, then choose a metric that reflects how important precision and recall is to you. And then tune the threshold according to that. Be intentional about what metric you choose. Don't just choose F1 because we did in this blog post. Also, if you want to use scores as confidence estimates, check to make sure whether or not those predicted probabilities match observed outcomes like we did in this probability calibration chart. Then finally, I recommend just plotting your cost versus performance so you can see the Pareto frontier of different models you may want to choose from and reason about what trade-off is important to you. Just remember the process can be iterative, meaning you may loop through it several times, you might change certain things, change your threshold, you may end up refining your judges prompt, revisiting things from time to time. These are things that will help you choose the right decision model and optimize it for your use case. And if you're looking for other ways to improve your eval workflow, check out this other video where I go through eight Claude code skills that you can use right away to improve your eval

Summary

The video discusses the evaluation of six decision models for identifying issues in user sessions of an AI assistant for property managers. It emphasizes the importance of selecting the right model based on performance, cost, and the tuning of decision thresholds to balance precision and recall for specific business needs.

- Decision models like Jev and OpenAI's Decisions are tested for their effectiveness in classifying issues in user sessions.
- A default threshold of 0.5 is often not optimal; users should consider adjusting it based on their precision and recall needs.
- Precision measures the accuracy of flagged errors, while recall measures the model's ability to identify all errors, creating a trade-off that businesses must navigate.
- A minimum threshold for precision and recall should be established to filter out less effective models.
- Visualization tools, such as precision-recall curves and cost-performance graphs, help in selecting the best model.
- Threshold tuning can significantly improve model performance, and different metrics (like F1 or F beta) can be used based on business priorities.
- Probability scores from models should be calibrated to ensure they accurately reflect the likelihood of failure.
- The process of model selection and tuning is iterative, allowing for adjustments based on ongoing evaluations and changing business needs.

Questions Answered

What are decision models and how can they be evaluated?

Decision models like Jev from Type-Safe provide fast and cost-effective classification. The video discusses testing six decision models to determine their usefulness for evaluations and how to choose between them.

How should businesses balance precision and recall?

Businesses must consider the importance of precision and recall based on their specific context. For example, cancer screening prioritizes recall, while criminal law prioritizes precision.

How can decision thresholds be tuned for better model performance?

Tuning decision thresholds involves selecting a metric that reflects the importance of precision and recall. The F1 score can be used, but businesses should consider their specific costs and benefits.

What is the effect of threshold tuning on model performance?

Threshold tuning can significantly impact model performance, with some models benefiting more than others. It's important to evaluate and visualize the performance and cost of different models.

How can model scores be calibrated for better decision-making?

Calibrating model scores can align predicted probabilities with actual outcomes, using techniques like Platt scaling. However, not all models will benefit from the same calibration method.

© transcribe · For agents Built with care and craft by Gokul Rajaram