transcribe

I Ranked Every AI Eval Concept From Worst to Best

Hamel Husain · 13m · transcribed Oct 2026
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to AI Evaluations

What common mistakes are made in AI evaluations?

The speaker introduces the concept of AI evaluations and highlights common mistakes, emphasizing the importance of effective evaluation methods.

  • Generic metrics are often useless and should be avoided.
  • Error analysis is crucial for identifying issues in AI applications.
  • Brainstorming potential failures before building can lead to wasted time.
# 2:40

Evaluating Evals: What Does 100% Passing Mean?

Is having all evals pass 100% a good sign?

Having a suite of evals that always pass is not beneficial; it indicates that the evals may not be challenging enough to provide useful insights.

  • Evals should not always pass; they need to be challenging.
  • Reading traces and analyzing data is essential for improvement.
  • Avoid off-the-shelf LLM judges as they often lack relevance.
# 5:21

The Limitations of Public Benchmarks

Should public benchmarks be used as substitutes for product evaluations?

Public benchmarks are not suitable substitutes for product-specific evaluations, as they may not correlate with specific product failures.

  • Domain experts should annotate data, not just developers.
  • Designing for verification can enhance the evaluation process.
  • Avoid using non-binary rating systems for evaluations.
# 8:02

The Role of a Benevolent Dictator in Data Annotation

What is the benefit of having a single person responsible for data annotation?

Having a 'benevolent dictator' for data annotation streamlines the process and reduces costs associated with committee-based decisions.

  • Focus on the evaluation process rather than just picking a vendor.
  • Avoid generic metrics and create product-specific metrics.
  • Relying on a single judge for all evaluations can be detrimental.
# 10:43

Best Practices for Evaluations

What are some best practices for conducting AI evaluations?

Validating judges against human labels and involving domain experts in prompt writing are key practices for effective evaluations.

  • Avoid off-the-shelf guardrails and generic metrics.
  • Look at data first before deciding on evaluation rubrics.
  • Domain experts should be closely involved in the evaluation process.

Transcript

0:00 Hey, how's it going? If you've been watching this channel, you know that I focus on AI evals. I've been working on evals for over two years and I've worked with over 50 companies improving their AI products with evals and I see the same mistake over and over again. So, what I'm going to do is this fun exercise where I have all of these eval concepts and we're going to rank them from S tier to F. So, let's get right into it. First thing, generic metrics.

0:26 What are generic metrics? These are things like helpfulness score, conciseness score. They're very generic. They usually come with eval tools and they involve prompts written by someone else. I'm going to put this in the F tier because this is a giant waste of time. If you have generic metrics like this on your dashboard, they're probably not telling you anything and they're usually not useful at all. Okay, the second thing, error analysis. What is error analysis? This errors analysis is a process that helps you look at your data. It's like a structured approach to looking at your data quickly so that you can find what's broken in your application. I'm going to put this on the S tier. If you're not doing this, you're doing eval. And you need to be looking at data. We'll talk more about that later. Okay, here's the next one.

1:15 Writing evals before building anything. So, what I mean by this is before you start writing emails, you sit down in a room or maybe a dark corner and you just brainstorm all the things that could possibly go wrong with your application and then you write a whole bunch of evals for that and you start there. I would say I'm going to put that in the C tier. You don't want to do this because there's infinite surface area of things that can go wrong in your AI product.

1:44 you're just going to waste a lot of time if you just come at it with a top down approach like that. It's not the worst thing in the world because at least you're doing something, but it's a bad way to prioritize your time. You should be looking at data first and seeing what's actually broken and let that help you prioritize what emails to write because emails aren't cheap. Custom annotation tool. So, this custom annotation tool basically it like you should be writing a custom annotation tool to look at help you look at your data. So, I'm going to put it in the S tier. And what I mean by this is like use Claude, Codeex, whatever, and create a custom annotation app that removes all the friction from looking at your traces and make it really good. So, like let's say you have an AI system that helps you write emails, render it like an email.

2:31 It's way better than spreadsheets and there's just a lot of alpha there in creating your own custom annotation tool. Okay, the next one. Let's say you have a suite of evals and your evals pass 100%. Like your whole suite of evals, they're all 100% passing against all your test cases. Is that a good thing? I would say that's D tier. If your evals are always passing all the time on all your test cases, they're not telling you anything at all. They're just passing all the time. You should think of evals a little bit different than your unit tests in software. like you don't want them to always be passing. You should always have some hardto-reach emails that are challenging that you're working on so that you can use those evals to help improve your application. Okay. Reading traces yourself. This is again S tier. This goes back to look at the data. There's nothing more valuable than looking at the data. Okay. Offtheshelf LLM judges.

3:30 What does that mean? If you go to a popular eval vendor, a lot of times they'll have like an off-the-shelf LLM judge with a prompt someone has written for you, I would say this is F tier. This is the same thing as generic metrics. Don't do this. Don't use those things. I challenge you to go read those prompts. Those prompts rarely make any sense and they're super generic. They're usually not good for your product. Okay, prompt optimization without evals. This is probably also F tier. Like what are you even optimizing against if you don't have evals? I see this all the time where people just ask claude to improve their product. They give like very vague description of what good looks like.

4:12 It's usually a bad idea. Your agent will end up producing a lot of slop. Don't do this. Think about eval a little bit before you jump into prompt optimization. Look at your data before you jump into prompt optimization and think about what you're trying to optimize against. Okay. binary pass fail metrics. I'm going to put this in a tier. So what I mean by that is if you're building any kind of eval, especially LLM as a judge or classifier, you want the output to be binary rather than like art scale. Binary is just going to cut through a lot of noise and help you clarify your decision- making in terms of what is good and what is bad. If you have a like art scale like one to five or even a score from 0 to 100, often times you're going to hide a lot of complexity and kind of kick the can down the road in terms of decision-m going to hide those hide that in those middle values and th those aren't really that interpretable. So I would discourage people from doing that generally unless you have a good reason to. And I'm going to go through these really fast. By the way, I have a evals guide that talks about each of these things in a lot of detail. You can find it if you go to bit.ly eval-guide. Okay, so the next thing is public benchmarks as a substitute for product eval. Okay, so we know that there's public benchmarks out there. Terminal bench, SWE bench, you name it. Should you just use those at benchmarks somehow as your evals? I would say absolutely not. Those benchmarks may help you choose a model, but they usually don't correlate that well to your product specific failures.

5:47 You need to create evals for your product. The next one, only devs annotate data. I'm going to put this in D tier. So unless you're building a coding agent, your developer should not be the ones annotating data. You should have the domain expert be the ones that are annotating data. Usually that's someone like a product manager or if it's some if it's an expert assistant like a medical AI assistant then you would need like a doctor or somebody with enough expertise to be reviewing the data and not a developer. Designing for verification.

6:22 This one just refers to hey like design your application such that it's easy to verify meaning show your experts intermediate outputs. Don't just show them one shot the answer. I'll put it in B. It's kind of advanced. It's not necessary, but it's a really good idea. One to five ratings. We already talked about this. Don't use like art scales. I'm going to go ahead and put this in F because I hate it. I think it wastes a lot of time and and it just makes you spin your wheels as an engineering team or a product team in general to have non-binary outcomes for your evals.

6:57 Okay. One judge for everything. I see a lot of teams just like push everything into one judge. They have like one giant prompt for the judge of like every thinkable mistake. And honestly, the judge in these situations are scoped way too large. I actually haven't put that in the F tier. I think that's really bad. This your likelihood of success is very low if you just have one judge for everything. Codebased checks. S tier, you should always try to have cheap evals wherever possible. So LM is judged as the most expensive kind of eval you can have. A classifier is even cheaper.

7:34 Something like Jev. Even cheaper than that is code. So try to use the cheapest eval machinery you can. Obviously you want to make sure that your eval agrees with human labels. You want to validate that. But within that, use the right tool for the job and try to see if you can be clever and use cheaper methods because it it pays off. Okay, eval based only on brainstorm failures. This again is F tier. We kind of talked about this like don't sit there and brainstorm failures and use those to drive evals.

8:07 Do error analysis. at least you're doing something, you know. you should look at your data first. So let's make that C tier. Hey, it's just it's really the same as writing emails before building. Generic helpfulness and coherence scores. We already talked about this. That goes in the F tier. Okay. Benevolent dictator. So, if you are annotating your data by committee, that is super painful and super costly. Where possible, try to appoint a benevolent dictator. I'm going to put it in a tier. Benevolent dictator is someone that you trust to annotate your data. Someone that can make the final call. And hopefully that's the person.

8:44 You only need one person for that specific feature, product, or geography to be able to annotate that data. It doesn't have to be one person for everything, but the fewer people that you can have annotating data, the better off you are. Okay. picking an eval vendor first. I'm going to put that in F tier. So, if the first thing you do when trying to solve for evals, if is you go pick a vendor, you're in trouble. Don't do that. It's not about the tools. Focus on the process. Look at the data. Write some minimal evals. Understand what's happening. Don't go straight for the tools. If you go straight for the tools, you're likely going to be led astray.

9:24 Don't do that. Focus on the process. Okay. Rugg and bird score for answer quality. These metrics don't even make sense a lot of times for answer quality. You don't want some kind of string similarity score like Rue for AI outputs. And you don't just generic metrics in general. Just don't use that. create product specific metrics for your failures. Running always passing evals less often. This is a really good thing to keep in mind. I'm going to put it in a tier. So, emails are not free. It costs money a lot of times to run emails, especially if they're not codebased evals, especially if they're LLM. As a judge, you want to weigh the costs and benefits of the eval. And so if it's always passing, that's one sign that the eval is becoming less useful to you and maybe consider retiring it.

10:12 Okay, the next one. Immediately writing an eval for every failure. I'm going to put this in the D tier. I'm put this in the F tier actually. So if you immediately write an eval for every failure you see, it's probably a bad idea because again, eval aren't free. You have to weigh the costs and benefits. So if the email if the failure is really bad and you don't know how to fix it, that's an amazing situation in what's to write email because that eval is probably going to provide you with value. It's not only useful to catch that error, but you're going to have to iterate a lot over a period of time to fix that error. And that eval is going to help you because it's going to give you a goal to hill climb against. Okay, next one. Validating judges against human labels. I'm going to put that in S tier. So most people write an LLM judge, they just write a prompt and they say, "This is my judge." But how do you trust that judge? The only way to trust a judge is to check it with human labels.

11:11 And so validating with human labels is S tier. Okay. Off-the-shelf guard rails. Off-the-shelf guardrails, F tier. Very bad idea for the same reasons that generic metrics are a bad idea. You don't want to just blindly use someone else's prompts or use someone else's classifier. Usually, it's a bad idea. They're not tuned to your product. Don't do this. Most of the time, I see offtheshelf guardrails are just someone else's prompt. Deciding on a rubric before reading data. I'm going to call it Ctier. Again, this is the idea that you are just sitting there brainstorming, ruminating about what can go wrong about your AI product. Don't do this. Look at data. Look at data first.

11:52 It should be the first thing you do is look at data. Next, domain experts writing prompts. I'm going to go ahead and put that in the S tier. So, a lot of people outsource everything that has to do with AI to developers. Don't do this. The whole point of AI is to have the domain expert closer to the product. So, you don't have this separation of concerns. You should have your domain expert, let's say it's a legal assistant, you should have your lawyer writing the prompt. And then finally, the last one is letting AI do all your evals. I'm going to put this in the F tier because this is how you can get led astray. A lot of people just use cloud code or codeex to go find all the problems. That can only catch some issues, but I guarantee you it won't catch all of your issues and it won't catch the most important ones, the tricky ones that take context, taste, and judgment to understand. I went through that really fast. If you want to learn more about evals, I made this detailed guide that walks through everything you need to know about eval depending on what stage you're at with evals. So maybe you don't know anything about evals or maybe you have evals but you don't really know what to test or maybe you don't trust your eval scores or maybe your product feels too difficult to eval.

13:11 Whatever the case may be, I have a detailed guide for you that'll walk you through what you need to know. You can find it at bit.lys-guide. Click the link and we'll send it to you.

Summary

The video discusses the evaluation (eval) processes for AI products, ranking various eval concepts from S tier to F tier based on their effectiveness. The speaker emphasizes the importance of data-driven approaches and warns against relying on generic metrics and off-the-shelf solutions.

- **S Tier**: Error analysis, custom annotation tools, reading traces yourself, validating judges against human labels, domain experts writing prompts.
- **A Tier**: Binary pass/fail metrics, code-based checks, running always passing evals less often, benevolent dictator for annotation.
- **B Tier**: Designing for verification.
- **C Tier**: Writing evals before building, deciding on a rubric before reading data, eval based only on brainstorm failures.
- **D Tier**: Only developers annotating data, having all evals pass 100%, immediately writing an eval for every failure.
- **F Tier**: Generic metrics, off-the-shelf LLM judges, prompt optimization without evals, one judge for everything, letting AI do all evals.

Questions Answered

What common mistakes are made in AI evaluations?

The speaker introduces the concept of AI evaluations and highlights common mistakes, emphasizing the importance of effective evaluation methods.

Is having all evals pass 100% a good sign?

Having a suite of evals that always pass is not beneficial; it indicates that the evals may not be challenging enough to provide useful insights.

Should public benchmarks be used as substitutes for product evaluations?

Public benchmarks are not suitable substitutes for product-specific evaluations, as they may not correlate with specific product failures.

What is the benefit of having a single person responsible for data annotation?

Having a 'benevolent dictator' for data annotation streamlines the process and reduces costs associated with committee-based decisions.

What are some best practices for conducting AI evaluations?

Validating judges against human labels and involving domain experts in prompt writing are key practices for effective evaluations.

© transcribe · For agents Built with care and craft by Gokul Rajaram