transcribe

7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)

Hamel Husain · 12m · transcribed Oct 2026
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Common Mistakes in Evals

What are the common mistakes made when creating Evals?

The speaker discusses the seven most common mistakes in creating Evals, emphasizing the importance of data-driven approaches rather than brainstorming potential issues without data.

  • Creating Evals without data leads to wasted time and unreliable metrics.
  • Starting with specific data points helps focus on relevant issues.
  • Generic metrics are less useful than specific, observable errors.
# 2:33

Importance of Specific Metrics

Why are specific metrics important in Evals?

Specific metrics correlate with observed data and help identify non-trivial issues, making Evals more effective and actionable.

  • Focus on specific metrics rather than generic ones.
  • Ensure metrics relate to observable issues.
  • Avoid creating Evals for easily fixable problems.
# 5:07

Designing Effective Evals

How should Evals be designed to ensure effectiveness?

Evals should be designed with a clear understanding of the data and the context in which they will be used, ensuring that they can be evaluated without excessive effort.

  • Design products with Evals in mind to facilitate evaluation.
  • Avoid creating systems that are difficult to evaluate.
  • Use examples from trusted data to guide the evaluation process.
# 7:41

Avoiding Likert Scales in Evals

What is the issue with using Likert scales in Evals?

Using Likert scales complicates the evaluation process and obscures clarity; Evals should focus on binary outcomes to simplify decision-making.

  • Binary outcomes (pass/fail) are preferable to complex scoring systems.
  • Avoid ambiguity in Evals to streamline decision-making.
  • Align Evals with clear business criteria for success.
# 10:14

Role of Domain Experts in Evals

Who should review the data in Evals?

Domain experts, rather than developers, should review data to ensure accurate evaluations, as they have the necessary context and understanding.

  • Domain experts provide valuable insights for data evaluation.
  • Create customized annotation tools for expert reviews.
  • Avoid relying solely on automated approaches for Evals.

Transcript

0:00 I've taught Evals to more than 5,000 engineers and PMs and worked with over 50 companies on their Evals. And I've seen a lot of the same mistakes which lead to teams wasting their time, posting unreliable Evals, or chasing down the wrong problems. So, in this video, I'm going to walk through the seven most common mistakes people make with Evals so you can build Evals that you actually trust. Number one is creating Evals before you look at any data. So, if you create Evals before looking at any data, you're going to waste a lot of time. If you start with just brainstorming anything that can go wrong, what happens is the space of what can go wrong is infinite. And so, you're going to end up wasting tons of time and you're going to have a dashboard full of metrics that are not going to point you in the direction of what you need to work on. Let me show you an example of what I mean. Here is a simplified conversation from an AI leasing assistant we've worked with. The purpose of this AI leasing assistant is to help property managers handle conversations with prospective tenants. So, this is one real conversation that we found when looking at the data. After a bunch of back and forth, the prospect eventually said, "Hey, this is out of my budget.

1:08 Thank you for your business." And then the leasing assistant went ahead and said, "You're welcome. If your situation changes or if any other questions in the future, feel free to reach out. Have a great day." To some people, this might sound great. If you point AI at a conversation like this, it's going to say everything went well cuz it looks like the customer is happy. But remember, this application is a sales assistant. to nurture the prospective tenant. And so, in this particular situation, it could have done a better job at looking for other properties owned by the same company that would have fit that person's needs. In this specific situation, the AI didn't do anything in the background to try to handle this objection and it just said, "Okay, thank you." And so, this is the example why you need to look at data so that you can remind yourself about what good looks like and ground the Evals that you choose to write in actual problems that are coming up. The second mistake people make is using generic metrics off the shelf. So, if you go to any evals vendor or any evals tool out there, they'll serve you up a menu of generic metrics that you can just put on a dashboard. And it feels like progress, but there is no better way to waste your time. What you'll end up with is a dashboard that looks like this: helpfulness, truthfulness, conciseness score, all on a scale of one to five.

2:22 And it looks good, but nobody knows what it means. What does conciseness 4.9 mean or helpfulness 4.2 mean? It's not actionable, and it's so generic that it's a waste of everyone's time. We don't know what a 3.7 versus a 4.3 means. You don't want to do this. You want to start with looking at your data. An example of a better metric would have been did we handle sales objection when a qualifying opportunity was presented? That is specific. That is something that's going wrong with the application, and that's something you want to measure rather than these generic topics. Just to expand on this topic further, I have this flash card that helps drive home this point. So, you don't want generic metrics like Rou, Blue, faithfulness, helpfulness. And you want these specific metrics like a calendar scheduling failure, interrupted conversation flow, a widget not rendering, maybe the email recipient is incorrect, or maybe you have failed to escalate to a human. See how these errors are specific and not generic like that helpfulness score. And so, what you want to think about is you want to make sure your eval metrics correlate to something you have observed. There's some data that you observed, or at the very least, it's something you have been able to reproduce. If it's some error that you're really worried about happening, you should try really hard to reproduce it before creating the eval. You also want your eval metrics to relate to a non-trivial issue. Don't waste your time creating evals for things that can be fixed super easily, as evals are not free. You also want your evals to be scoped to a specific failure. You don't want a catch-all eval. If you have a catch-all eval with a giant prompt of anything that can go wrong, that's very similar to a generic metric, and ultimately, it's not that helpful. You also want to try to make sure your evals have a binary outcome, pass or fail, instead of some kind of score, because a score hides a lot of complexity in it, and it becomes exponentially more expensive to align your team on on what a one, two, three, four, five means. And finally, you want your evals to be verifiable. You want to use human labels so that you can measure your evals against a ground truth and build confidence around how good that eval is.

4:26 And that brings us to our third mistake, not validating your LLM judge or eval against human labels. The last situation you want to be in is where you are challenged to explain why your LLM as a judge can be trusted or why your eval can be trusted. You don't want to just write a prompt and assume that the eval is good. That would be the same thing as just praying that the eval is good. Instead, what you want to do is have human labels, and you want to test to make sure those human labels align with your LLM as a judge or classifier. And so, I have this flashcard here on the screen, and basically, what you want to do is segment your data. You want around 20% of your data to be train, 40% to be dev, and 40% to be test. And essentially, what you want to do is, when you're writing your LLM as a judge prompt, you want to source examples from that train set. And then that dev set, you want to tune the LLM as a judge to agree with the dev set as much as possible. So basically, want to hill climb against that dev set. And then that test set, you want to hold out or hide from yourself to the very end, just to make sure you didn't overfit to the dev set. You want to make sure that any LLM judge or classifier you create can generalize outside of the data that you are, the LLM or the classifier has seen.

5:40 Now, you shouldn't just measure accuracy, you should measure precision and recall, and that's what TPR and TNR is. I won't go into the details of those calculations, but the most important thing to take away here is that if you have an LLM judge, you need to align it with human labels. You need to measure it against human labels. Beyond these mistakes, there's a lot of questions that come up with evals. So, I've put together this free AI evals FAQ with the most common questions that we get during our evals course office hours. We're always adding to it, and I'll leave a link in the description below. The fourth mistake is not designing your product for success with evals in the first place. Let me show you an example.

6:18 So, what I have on my screen is a data agent. The purpose of the data agent is to answer business questions, like what was net revenue for product A last quarter. Now, a lot of times people design such a thing as a chatbot that just gives you the answer. And the problem is, how are you supposed to evaluate this? The only way to evaluate something like this would be to redo all of the work. And that would become extremely expensive to evaluate. There's no way that you could write evals or look at this data without it consuming a lot of time. And what you want to do is design your product differently such that it sets you up for success with evals, but also is a better user experience. Anytime you have an AI product that caters to an expert user, like this data agent, what you want to do is design for that persona in mind.

7:06 So, instead of something that looks like this, what I'm showing on the screen, you want to design something that maybe looks like this. You ask a question, but it shows you where the answer is coming from, and it shows you the intermediate calculations. It shows you correlated numbers, so on and so forth. And it might also show you a lot of detail around how it was computed and give you the chance to walk through step-by-step how that number was computed, and even change the way it's computed, and give you a bunch of trust signals, such as how was this metric computed previously?

7:38 Are there any related reports where this number is consistent with what was previously reported, etc. If you design your product like this, you have a much better chance of being successful with evals. And it's not just constrained to data agents. There are other examples in this blog post. I talk about a curriculum builder for schools where instead of just an app that just spits out a lesson plan, what you want is an app that walks the teacher through the changes from an existing trusted lesson plan and allows them to accept or reject those changes. And similarly, I have another example in the blog post which I won't go through right now about a medical use case. I'll put the link in the description, so you can go through this more carefully. The next most common mistake, at number five, is using a Likert scale or one to five scale with an LLM judge. And I've shared a flashcard on the screen to help illustrate the point. It's really common when you're building an LLM judge to think you want a score, like a one to five score or zero to 100 score, because in life, a lot of times there are shades of gray. There's no black and white.

8:38 Something is kind of good, but not great. And you don't want to carry that intuition over into the way you construct evals. You want to create evals that mirrors the way that you are going to make product and business decisions. What you want to think about is, is this good enough to ship? Yes or no? And be very clear about what is good and what is not. And it's very likely in your business that you have a line of what is good enough and what is not good enough to ship. If you are constructing evals that output a one to five score, you're going to be trapped in massive complexity. You're going to have to have a rubric that specifies what is a one, two, three, four, five mean. And you're going to have to spend a lot of time aligning humans on exactly what that means. And it's going to slow down the process exponentially. You don't want to do this. Another thing you want to do is scope your evals so that they are focused on a single error. When you scope your evals like this, it becomes much easier to create a binary score.

9:34 So, in this diagram, we're showing you how to do things correctly. Again, you don't want this Likert score. What you want to do is have binary pass-fail outcomes and scope your evals to a very specific error, like is polite or is there a scheduling error or was there a human handoff error and have those be pass or fail. Those are more actionable and going to be way easier and less expensive to manage. Having a one to five Likert scale will just trap you in complexity and result in meetings that go on for hours. You don't want to do this. Number six is having the wrong people review the data. So, it's very tempting to outsource evals to your developers. After all, your developers are the ones who are making the AI application. However, your developers are not the domain expert in most situations, unless you're building a coding application or coding agent. The people that should be reviewing the data are the people who understand the data.

10:25 So, if it's a legal assistant, it should be a lawyer. If it's a medical assistant, it should be a doctor or person with enough medical expertise to understand whether or not the outcome is good or not. To do this well, you want to create your own annotation apps. You can use Cloud Code, Codex, or your favorite coding agents to create a customized annotation app that makes it easy for that expert to review the data. This is an example of a custom annotation app for that apartment leasing AI assistant that we discussed earlier. This is a customized to the data and allows an expert, which is a product manager working on this application, to review the data. It is a really good idea to do this, so your domain experts are the ones who are annotating and reviewing data, not the developers. For number seven, the mistake is relying too much on automated approaches or automated evals. So, I have a meme here on the screen and people literally do this all the time.

11:18 It's just ask a coding agent to find all the errors. Now, a coding agent can find some errors, but it's not going to find all the errors. There's a lot of context that an agent doesn't have about what is good and what is bad. The best way to explore what is good and what is bad is to look at the data yourself. So, we recently did a study comparing all of the leading e-val vendors on their capabilities with automated e-vals. And all of the leading vendors have some sort of feature that allows an agent to look through all of your data to find issues. We compared Observe.ai, LangSmith, and BrainTrust. And we found that they can all find issues that are obvious, but they all fail in the same ways. We used the same apartment leasing AI assistant data that we've discussed before. So, all of these agents failed in the exact same way and failed to catch the same errors. For example, they all missed issues with sales objections.

12:09 They missed formatting issues like having markdown in text messages. They all failed to capture problems with interruptions in the voice agent. And they also all failed to capture problems with human handoff. I'll put a link to the study in the description so you can dig deeper. But, what you need to know is you need to be looking at your data so that you can refine what good looks like. It's impossible for you to know what good looks like without looking at some data. If you were just tasked with brainstorming all of this up front, it's likely you would miss something and your e-vals wouldn't be really good. If you want to avoid these mistakes in practice, check out this video where I go over eight call quality skills that can help you improve your e-vals.

Summary

The video outlines seven common mistakes in creating effective evaluations (Evals) for AI applications, emphasizing the importance of data-driven decisions and specific metrics. By avoiding these pitfalls, teams can develop more reliable Evals that accurately reflect performance and areas for improvement.

- **Mistake 1**: Creating Evals without analyzing data first leads to irrelevant metrics and wasted time.
- **Mistake 2**: Using generic metrics results in unactionable dashboards; specific metrics tied to observed issues are essential.
- **Mistake 3**: Failing to validate LLM judges against human labels can undermine trust in Evals; alignment with human labels is crucial.
- **Mistake 4**: Not designing products for evaluability complicates the assessment process; products should facilitate easy evaluation.
- **Mistake 5**: Relying on Likert scales introduces complexity; binary pass/fail outcomes are more actionable and manageable.
- **Mistake 6**: Involving the wrong reviewers (like developers instead of domain experts) can lead to inaccurate evaluations; domain experts should conduct reviews.
- **Mistake 7**: Over-reliance on automated evaluations misses nuanced errors; human review is necessary for comprehensive assessment.

Questions Answered

What are the common mistakes made when creating Evals?

The speaker discusses the seven most common mistakes in creating Evals, emphasizing the importance of data-driven approaches rather than brainstorming potential issues without data.

Why are specific metrics important in Evals?

Specific metrics correlate with observed data and help identify non-trivial issues, making Evals more effective and actionable.

How should Evals be designed to ensure effectiveness?

Evals should be designed with a clear understanding of the data and the context in which they will be used, ensuring that they can be evaluated without excessive effort.

What is the issue with using Likert scales in Evals?

Using Likert scales complicates the evaluation process and obscures clarity; Evals should focus on binary outcomes to simplify decision-making.

Who should review the data in Evals?

Domain experts, rather than developers, should review data to ensure accurate evaluations, as they have the necessary context and understanding.

© transcribe · For agents Built with care and craft by Gokul Rajaram