Section Insights
Introduction to AI Evaluation Skills
What are the key skills for evaluating AI products?
The speaker introduces eight evaluation skills developed through collaboration with companies and students. These skills help identify failures in AI products before jumping into evaluation.
- Shipping AI products is easier, but ensuring their effectiveness remains challenging.
- Common mistakes include rushing into evaluation without understanding existing failures.
- The skills are available on GitHub and include an entry point for routing to appropriate skills.
Designing Annotation Interfaces
How should one design an annotation interface for data evaluation?
The section discusses the phases of designing an annotation interface, emphasizing the importance of visual design principles and creating a user-friendly experience for data annotation.
- Design principles include color, spacing, and visual encoding rules.
- An effective annotation interface enhances the evaluation process.
- The interface should facilitate easy interaction with the data being evaluated.
Data Sampling Techniques
What are effective strategies for sampling data during evaluation?
The speaker explains the importance of intelligently sampling data to efficiently explore and identify failure modes, emphasizing the need for diversity in samples.
- Data should be sampled both randomly and from identified clusters.
- Prioritizing diversity helps in discovering failure modes rather than just estimating their frequency.
- An interactive review loop allows for adaptive sampling based on ongoing annotations.
Using the Eval Audit Skill
What is the purpose of the Eval Audit skill?
The Eval Audit skill helps identify common mistakes in evaluation setups, ensuring that the evaluation process is robust and effective.
- The skill checks for common pitfalls in eval setups, such as error analysis and appropriate evaluation metrics.
- It emphasizes the importance of using binary evaluations over complex grading systems.
- Regular validation of LLM judges against human labels is crucial for maintaining evaluation quality.
Overview of Additional Evaluation Skills
What other skills complement the main evaluation skills?
The speaker briefly outlines additional skills that assist in generating synthetic data, validating LLM judges, and building evaluation metrics for search and retrieval.
- Generating synthetic data is essential when real user data is unavailable.
- Skills for writing and auditing prompts ensure best practices are followed.
- A comprehensive evaluation pipeline includes various skills to enhance the overall evaluation process.
Transcript
0:00 It's gotten much easier to ship AI products, but figuring out whether the AI actually works the way you want is still hard. And one of the biggest mistakes I see are people jumping into eval before they looked at any failures that are actually happening in their product. So my colleague Shrea Shankar and I build eight eval skills to help with that process. We've iterated on these skills by working with over 50 companies and teaching over 5,000 students in our AI evals course. I'll walk through all eight, what they do, and when to use each one. Okay, so you can find the eval skills on GitHub. I have it pulled up right here. I'm going to share the URL up here above to the GitHub repo. And the GitHub has a nice read me in it. You can see some information. It's important to start with this table here. You'll see a bunch of skills. The entry point is the eval start skill. And the eval start skill just lets your agent, whether it's cloud code or codeex, route to the appropriate skill. So there's this table that your agent can read. My favorite skill out of all of these is the error discovery skill. It's the most valuable skill because it guides you through looking at your data in the right way so that you can build good eval. You will get directed to this skill if you already have traces and you want to find failure modes and you haven't discovered what your failure modes are yet. There's also this eval audit skill, which is really handy if you already have evals and you want to find out what's wrong. And there's a whole bunch of other stuff like getting help with writing a judge prompt, how to validate your LM as a judge and make sure your LM as a judge is good, as well as some other things like how to generate synthetic data or how to write a custom annotation interface or how to evaluate search for your rag pipelines. This is what comes with this eval plugin. I really want to encourage everyone to start with the error discovery skill. It's the most powerful skill in the set and let me walk you through what it does. You can take a sneak peek at the error discovery skill. Now the error discovery skill does a lot of things but it goes through different phases. So first it lets your coding agent understand your data and instructs it to read and parse the data and to identify what the structure is of that data. Then it tells your agent to identify what are the dimensions of variation, which is just a fancy way of saying identify what's the most important things to pay attention to in the data. What dimensions in the data account for the biggest differences and it also instructs your agent to think about what bad means. Phase two instructs your coding agent to design the annotation interface that you're going to use to look at the data. And there's some design principles here.
2:37 There's these core gestalt principles. This is design terminology. It's not something we made up, but this just lays out some design principles around color, spacing, backgrounds, and containers, opacity, and contrast, as well as some visual encoding rules, things like typography, border, container, things like that. And then phase three is where we instruct your agent to build the review interface or use an appropriate architecture. And then finally, build the HTML app. And we don't have to read every part of this, but essentially what it does is it allows you to quickly get an interface for annotating your data.
3:13 And let me show you what that looks like. This is what an annotation interface might look like. What I'm showing here on the screen is an annotation interface for a real estate assistant. And the real estate assistant is a chatbot that has multiple turns. And what we're doing is looking at the logs. And we can see here I can see the system prompt. You are a leasing team member at the Metobrook Gardens apartment. Your name is Taylor. Your goals all to answer questions, encourage customers to schedule tours. And that's kind of gives you an idea of what this agent is about. And then we see multiple turns. We see an assistant followed by user turns. And you can see like this seems like it's a phone call because this call is recorded is the first thing the assistant says. Then you can see there has a there's a back and forth conversation. There's even a tool call where a transfer is taking place and then a success. And we can page through some of these and see the different conversations. We see a user is asking a question about a specific apartment and when it's available. And we see there's several tool calls being made. And there's this tool call that's getting apartment availability and a tool response followed by some other conversations. So you can see this allows us to look at the logs. And this is exactly what this skill allows you to do is create something that has good ergonomics and allows you to flip through the data really fast. Not all annotation interface will look like this. The agent will create an annotation interface that is customized to your data. So, it's likely that your annotation interface is going to look much different than this. The next phase of the skill allows your agent to intelligently sample data. Looking at data is very tedious. You can imagine if you just were to go through logs like this one at a time randomly, it would be very monotonous and you'll get tired quickly. It's really important that you're looking at data as efficiently as possible. So what that means is it's important to look at some data randomly.
5:05 So to make sure you're exploring the space, but also making sure that you are looking at diverse samples so that you are exploring the breadth of your product. And sometimes also drilling into failures that you already uncovered so that you can learn more. And so that's exactly what this part of the skill does is it instructs your agents to cluster your data and sample from the clusters but then it also random sample so that you can do that exploration. As you see here in this phase we prioritize diversity. The goal is discovering failure modes not estimating how common they are. And then finally there's this interactive review loop. This is my favorite part of this skill. This instructs your agent to enter a interactive review loop so that it samples differently as you annotate traces. So as you discover new errors, the agent will update its sampling mechanism and show you appropriate traces. So we can click on this review loop to see what's going on here. See that there's breath and depth. The goal of breath is to cover as much of the data set as possible. And depth is when once you found an error or a failure mode, examine it thoroughly and find more traces like that. And the way it works is it monitors your annotations in the background and processes them as they arrive. This skill talks a little bit more about what depth mode means, how to actually go about that, how to do breath, and some rules around re-reviewing things. and it's instructing to keep serving you samples until you converge or you're not finding any new kinds of failure modes. So that's the error discovery skill. We have a video walking through this skill in a lot of detail where we do it live.
6:48 We'll put the link here so you can get to it. And if you want to dig into how it works, you should totally watch that. By the way, if you're learning AI evals, I put together a set of flashcards that walk you through the most important concepts in our AI evals course. I'll leave the link to them below. The next thing I want to show you is the Eval's audit skill. This is my second favorite skill out of the plug-in. And basically what it does is allows you to quickly identify common mistakes that people make in their eval setup. We've been using this skill successfully for years to uncover real problems in people's eval pipelines. And it's really fast and easy to use. Basically, what it'll do is it'll check against a list of common mistakes to make sure you are not making those mistakes. So here are some of those mistakes. Let's speedrun them really fast. This skill is going to connect your eval infrastructure, do some diagnostic checks, but what it's going to do, it's going to make sure that you've done error analysis, which is the skill we just covered before with the error discovery. It's going to make sure that your failure categories are ones that you observe, so that you have annotations on your data, that any evals you write are based on data and not brainstormed problems that might happen.
7:59 and it'll check to make sure that your evals are binary pass fail and not things like art scales or letter grades. You often don't want letter grades or like art scales because those tend to be very unwieldy and often add a lot of noise. It also makes sure your LLM judges are not over scoped. You're using codebased checks when you can and you're not overrelying on LM judge. It also check for the case where you're abusing similarity metrics. A lot of people try to abuse similarity metrics for evals and similarity metrics are rarely ever useful for this purpose. The eval audit will also make sure that you're validating your LM as a judge. In other words, make sure that your LLM judge is good. How do you test if your LLM judge is good? You have to check it against human labels. And that's what this part of the audit will do. Even if you are checking against human labels, it will make sure that you are splitting up your data properly and that you have good data hygiene so you're not overfitting and you're not peeking at the human labels and allowing the judge to cheat in your evaluation of the judge. This eval audit will also make sure that you're using good review processes when you're annotating data. Next, it will check to make sure you have enough label data for every failure mode. What we have is we target at least 100 traces for every failure mode you have. This is just a suggestion but it will look at that and if you don't have enough, it will try to suggest ways that you can acquire those. Finally, it'll look at your overall eval pipeline hygiene. It will make sure that you're running error analysis on a consistent basis. It will also make sure you're maintaining your evaluators properly. And then finally, it will generate a report for you to review. The eval start skill and the eval skill. Those are the two skills I would spend most of my time on. If you get through just those two skills and you apply those two skills, you'll be far ahead of most people on evals. Let's quickly touch on the other skills. So, we already covered the error discovery skill and the audit skill as well as the start skill. The generate synthetic data skill is useful when you can't acquire any data. Let's say you don't have any users. You are pre-product and you want to stress test your product. You might want to generate synthetic data. There is a right and wrong way to generate synthetic data. This skill will help with that and maximize the diversity of that synthetic data so that you are pushing the boundaries of your application in the right way. Then there's a skill that will help you write an LM as a judge prompt and audit those prompts to make sure they're follow best practices based upon what we found working with many companies on this topic. Next, there's this validate evaluator skill that makes sure that any LLM as a judge you have is vetted and is checked against human labels in the right way so that you can trust it.
10:54 There's also this evaluate rag skill which will help you build evaluation metrics around search and retrieval. And finally, there's this build review interface. The build review interface is actually leveraged by this error discovery skill because in error discovery, you will have to review examples. So, this is kind of a skill used by other skills. There's a little bit more about how the skill works further down in the readme. If you want to know the general process of how this all fits together, you can see this diagram or read this paragraph. If you want to see how a real team is using these skills in production, I recently talked with one of our former students, Lucas, about how his team uses them with cloud code in production. You can watch that conversation
Summary
- Eight evaluation skills have been developed to assist in AI product evaluations.
- The **Error Discovery Skill** is highlighted as the most valuable, guiding users through data analysis to identify failure modes.
- The **Eval Audit Skill** helps identify common mistakes in evaluation setups and ensures proper error analysis and data hygiene.
- Skills include generating synthetic data for pre-product testing and writing prompts for LLM judges.
- The evaluation process emphasizes the importance of diverse data sampling and interactive review loops to discover errors effectively.
- The speaker encourages starting with the Error Discovery and Eval Audit skills for significant improvements in evaluation practices.
- A GitHub repository is available for accessing these skills, along with detailed documentation and examples.
- The speaker shares insights from real-world applications of these skills in production environments.
Questions Answered
What are the key skills for evaluating AI products?
The speaker introduces eight evaluation skills developed through collaboration with companies and students. These skills help identify failures in AI products before jumping into evaluation.
How should one design an annotation interface for data evaluation?
The section discusses the phases of designing an annotation interface, emphasizing the importance of visual design principles and creating a user-friendly experience for data annotation.
What are effective strategies for sampling data during evaluation?
The speaker explains the importance of intelligently sampling data to efficiently explore and identify failure modes, emphasizing the need for diversity in samples.
What is the purpose of the Eval Audit skill?
The Eval Audit skill helps identify common mistakes in evaluation setups, ensuring that the evaluation process is robust and effective.
What other skills complement the main evaluation skills?
The speaker briefly outlines additional skills that assist in generating synthetic data, validating LLM judges, and building evaluation metrics for search and retrieval.