How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix

0:12 All right, well, hi everybody. I'm Cameron Wolfe. I'm a staff research scientist at Netflix. I also write the deep learning focused newsletter. Hopefully you read it. And today I'm going to be talking about evaluations and more specifically trying to evaluate highly subjective tasks. And we're going to go over a lot of information. it's supposed to be kind of like an overview. If there's anything that you want to look into in more depth, you could scan this QR code. It'll take you to the slides. I've also written a lot of stuff that outlines all of these different concepts in detail. So plenty opportunity for further reading if there's anything that you're interested in throughout the talk.

0:50 So, we all know evaluations are difficult. We've seen a lot of presentations on this during AIEE. And if we think about why this is the case, there's just there's evidence of this everywhere within the AI research community. People are talking about benchmarking constantly. There's new evals being released like every day in different revisions the evals. So, we all know that evaluations are difficult. But if we step back and think about why this is the case, there are a couple of different reasons. And one of the primary reasons and the one that we're going to focus on a lot today is noise.

1:22 We're working with several non-deterministic components when we're trying to evaluate models. This creates a lot of fluctuations in behavior and it just makes it really difficult to draw reliable conclusions from eval results. And to understand why this is the case, we can just look at the structure of an evaluation trial. So, we have our evaluation suite here. It's composed of a bunch of different tasks that we want to try and solve. We pass these of those tasks to a non-deterministic agent that gives us a rollout and eventually a solution. And then we take this solution and pass it to a grader which will give us just some some kind of pass fail result. So, at the end of the day we're just trying to get our agents to solve these tasks and seeing if they're passing them or not.

2:03 And because of the cost of evaluation, you know, some of these recent benchmarks will cost like, you know, 50, 100,000 dollars to run them. we can't evaluate over that many tasks. So, the set of tasks within our evaluation benchmark typically is going to be relatively small. And this combined with the fact that we're already working with a non-deterministic agent, it already just makes it difficult to have any sort of like reliance on the eval result. And on top of this, if we look at most of the stuff that people are working on in terms of evals, we're primarily focusing on verifiable tasks. So, this grader that we see at the bottom of the screen here, it's implemented as some sort of like deterministic check, whether it be string matching, symbolic verification, test cases, and so on.

2:42 so, we're mostly focusing on these verifiable tasks where pass-fail is something that we can evaluate with a function. But if we look at the different things that an agent will try and do in the real world, it kind of lies on a spectrum of verifiability. Not all tasks are verifiable. So, on one end of the spectrum, we have things like math where we can verify solutions deterministically with things like string matching or symbolic verification. But on the other end of the spectrum, we have super subjective tasks like creative writing. And here, the definition of pass-fail is extremely subjective and open-ended, and it's hard even for humans to evaluate reliably.

3:17 So, usually we're going to have to draw upon some type of like LM judge or rubric to help us with this. And what we end up with is that we're taking a component that was previously deterministic, this grader, and we're replacing it with another non-deterministic LM-based component. So, we just end up with a lot of noise and subjectivity within an evaluation process that was already difficult to begin with. so, what we end up with with this is just something that looks like this. We're drowning in all this noise. we're completely blind to if our agent is like doing well or not. So, the goal of the talk today is to try and figure out how to not become this person.

3:54 So, what we're going to do is we're going to outline a repeatable three-part framework for evaluating super subjective tasks. This will consist of offline evaluation, human evaluation, and production monitoring. And then we're going to outline the best practices for each of these components, which applies both to subjective tasks and just to evaluation in general. And then along the way we're going to cover a bunch of different tricks that we can use to reduce noise, have more statistical like confidence in these evaluation results, and just dry the most dry dry the most reliable conclusions that we can from our evaluation results.

4:25 So, let's get into it. So, when we're working on a very subjective or open-ended task, the first step is to set up a human evaluation process. And this is going to be the North Star for defining, measuring, and improving the quality of our task over time. But, in order to set up human evaluation, we need to first create a clear definition of what quality means. And usually the way that we do this is by partnering with some small number of subject matter experts to create quality guidelines.

4:52 And within these quality guidelines for our task, we define a small set of criteria that we care about in terms of what should be in the final solution. We define how to rate these criteria, usually using some type of like binary or Likert score. And then we just provide a ton of details defining what each of these criteria are, specific examples of how to rate them, and things like this. So, we want to provide humans as much information possible so that we can minimize the subjectivity of this human evaluation process.

5:19 But, even if we write the best possible human evaluation guidelines, we can't just assume that the human evaluation process is going to be reliable. So, we need to calibrate it extensively. And what this means concretely is that we take multiple human evaluation samples for the same data, look at the agreement between them, and then update our guidelines based upon the sources of disagreement that we see. So, this is kind of an iterative process where you're literally just talking with humans and trying to see, you know, is there some issue in my guideline that makes it unclear how to evaluate this?

5:47 And over time, we want to make our quality guidelines as clear as possible so we can get high agreement between all of these different humans. So, if we do this properly, then we end up with a calibrated human evaluation process and very clear quality guidelines. And we can already use this to start performing evaluations. But, if we want to draw some sort of reliable conclusion from these human evaluations, we also need to think about how we sample data.

6:11 So, we need to think about things like the amount of data that we're going to put into the evaluation process, the type of data that we will evaluate on and so forth. And for the amount of data, we can draw upon a statistical concept called power analysis, which will help us understand the amount of data that we need to pass into the evaluation process. And power is a really interesting area of statistics, but for the purposes of this presentation, the high-level idea is that power analysis allows us to estimate the volume of data that we need to put into the evaluation process to make a meaningful conclusion about the difference in performance between two different models.

6:47 And we make this estimation by looking at things like the effect size that we want to detect, the level of noise that we observe in our eval results. You can estimate this with like resampling model completions and so forth, and the desired significance level. So, how confident we want to be that a difference in two models' performance is actually legitimate and not due to noise. So, we can use power analysis with these factors to estimate the size of our eval that we need for a reliable conclusion.

7:11 And then in terms of the type of data that we want to sample, there are a bunch of different factors, but at a high level, if you read a bunch of different benchmark reports and see what people do, generally they're going to focus upon diversity and the realism of the evaluation. So, we want all of the data that we evaluate on to accurately reflect the types of tasks that the agent is solving in production. We want it to be difficult and discriminative, meaning that it differentiates between the capabilities of different models.

7:37 Usually, this is assessed via some form of like profiling, just sampling rollouts and seeing what types of problems a model can actually solve versus not. And then we want it to be as accurate as possible, which is usually accomplished via some combination of like model and human-based filtering over time. And these concepts for sampling data properly, they're not only applicable to human evaluation. You can kind of use these anywhere. So, we'll also see again in the future when we talk about offline evals, we'll probably use like very similar data sampling techniques.

8:07 So, at this point, we have a reliable human evaluation process, but the problem is if we want to human evaluate every new agent or model that we're looking into, it's quickly going to become a bottleneck, and we can't iterate fast enough. So, we need to invest into creating offline or automatic evaluations. And the key idea here is that instead of verifying the solution quality with a human, we need to find some sort of automatic proxy for this, which in this case is usually going to be some LM judge or rubric.

8:35 And we won't talk deeply about the differences between LM judges and rubrics in this talk, but at a high level, basically LM judges tend to be given high-level quality guidelines, and it's very similar to how humans evaluate tasks. We give them our quality guidelines, and they're going to evaluate whether the solution adheres to that. And it's the same set of quality guidelines that we look at across all different tasks that we're trying to evaluate. Rubrics, on the other hand, are a little bit more specific. So, they define a task-specific checklist of individual items that are each separately verified with with an LM judge. So, one example of this from the Dr. Tulu paper up here, I like this example cuz it's both super subjective, and it's also talking about trying to culturally adapt a book into a next Netflix film. But if we look at all the rubric checks that the model is looking at here, we're not looking at some high-level like quality guideline.

9:25 We're outlining a checklist of specific things that the LM judge needs to verify within the solution that it's looking at. So, it's task-specific, and we're looking and verifying each of them independently to make the evaluation way more objective. And what we end up with here is automatic automatic metrics that we can use to verify solution for quality in a similar way to how a human would evaluate a solution. But the problem with this is that these are all automatic metrics. They're just a proxy, so they're going to be imperfect. So, to solve this, we need to do some sort of meta-evaluation to ensure that our judge is accurate. And what this means is that we'll take data from the human evaluation process, golden data. These are going to be evaluation examples from humans that we're confident in their quality. We know these are good evaluation examples.

10:09 And then we want to measure the agreement between what our judge or rubric is telling us and this golden data set. So, we basically just want to look at basic things like correlation metrics, binary accuracy, and so forth. But meta-evaluation is like a pretty wide space and there's a bunch of different ways that you could approach this. One example that I like to pull on from Netflix, a project that I worked on, we actually took an LLM judge and showed that the outputs of this LLM judge were causally correlated with key streaming metrics like abandonment. So, if your LLM judge tells you something is bad, it's likely that somebody will abandon the show after trying to watch it. So, there's a bunch of different possibilities for how to evaluate judges and I'd encourage you to look into this blog if you're interested in them.

10:50 And then finally, before moving on from from meta-evaluation, one common question is how to figure out what agreement level is good. And for this, it depends a lot on the task. Like more subjective tasks, the level of agreement that you can achieve is going to be a lot lower. But what we can do is just use human evaluation as a baseline. So, we can look at the amount of agreement between different humans who are trying to evaluate this task, and we can shoot for a similar level of agreement of our judge with the humans. So, similar levels of agreement both between judges and humans and mutually between humans.

11:22 And then from there, we have automatic verifiers that we know work well. So, we can just start to create benchmark benchmarks around all of these verifiers. To do this, it's pretty similar to human evaluation. We just sample data properly using the same techniques we just talked about, and then we use LLM judges or rubrics to verify these outputs and then just try and continually revise and improve the data that we're evaluating on over time. The only point that we still need to talk about is that we now have multiple non-deterministic components that we're working with. In human evaluate evaluations, we only had a non-deterministic LLM. Now we have a non-deterministic model and a non-deterministic evaluator. So, we have a lot of noise that we need to figure out how to handle.

12:02 And in order to do this, we can just apply the kitchen sink of different like statistical methods to try and put confidence intervals on all of this. So, similarly to human evaluation, we can use power analysis to tell us what the size of our benchmark should be. We can report error bars along with all of our evaluation results and then use techniques like resampling, which basically just means taking multiple rollouts from your LLM or from the judges well to reduce variance.

12:28 And what we end up with is we don't just report evaluation results. We report our evaluation results with clear confidence signals and then we take interventions to try and make all of these confidence signals as tight as possible. And if we report those together, we can basically more meaningfully detect capability improvements in our model. We know if one score on an eval is better, it's because the model is actually better, not because of noise or something like this.

12:54 So now we have a calibrated human evaluation process. We've created offline evaluations and calibrated judges that we can use to run this automatically. So we can just run this a bunch of different times and test out tons of different ideas to scale the experimentation process. And then from there, we can take whatever is working well and pass it back to human evaluation to have humans evaluate whether this is actually going well. And at this point, we might be asking ourselves whether we're ready to deploy this model or not.

13:22 And the answer could be yes, but if we if we say yes at this point, we'd actually be skipping a super important part of the evaluation process, which is production monitoring. And the idea here is that we can define and track production metrics for our agent. So we're using how the agent is used in production as another mechanism for evaluating whether it's working properly or not. And to do this, you can look at a variety of different metrics. Some of them are explicit like thumbs up, thumbs down feedback, preference feedback and so forth. You'll see this done in like chatbots all the time. But it could also be implicit metrics like the length of a session, abandonment rate and so on. So you can define these different metrics that are some somehow correlated to whether the agent is performing well or not.

14:04 And these are basically just quality signals from production that we could use to see if our agent is doing good. And what we could do is we can take these production metrics and measure them in an AB test to get gain a relative sense of how our agent is doing. So, seeing compared to what we already have in place, is the current agent doing better than that? And again, we can use a lot of statistical techniques to tell us how long we need to run this AB test and so forth. This is related to power analysis, which we talked about before.

14:31 And the nice thing about AB testing is that what we could use what we could do is basically treat this as a soft model deployment process. So, instead of just sending the model out to everybody, we can vet the quality of this model in production with an AB test and make sure everything's going well before pursuing a full rollout. And even after we roll out the model, there's a lot of opportunities to just continually production monitor. We can look for any concerning trends in performance or even find cases where the agent is failing, new use cases and so forth, and use production to ground the other evaluation steps in the real world. So, production monitoring is incredibly useful for extending evals, finding new failure cases, all of these things. We want to find all of these cases and then pass them back to the other parts of the evaluation process.

15:14 So, in this way, we just have to continually work towards improving all of the evaluation components over time. we've seen all of the different components that we're going to cover in this talk, but we can always make all of them better. So, we could define new or better quality criteria, we can improve calibration of humans or judges, or we could look at production to find new error examples that we can work into offline evals and so forth.

15:37 Or we could just take this entire process and repeat it in a new new domain. So, just build an entirely different eval. So, to sum everything up, evaluation is difficult, but for subjective tasks, it's especially difficult because we're dealing with more sources of noise. And we've seen a roadmap and all of these different components and tricks that we can use to reduce the amount of noise as much as possible to draw more confident conclusions. And in this roadmap, human evaluation serves as our North Star for quality. It provides a clear quality definition that humans agree on. And then from this, we could use this definition to calibrate LLM judges or rubrics as a proxy to human evaluation, and this allows us to ex-scale the experimentation process.

16:20 Then anything that goes well, we can pass it to human evaluation and eventually deploy it. And when we deploy, we can use AB testing, production monitoring is another evaluation mechanism to make sure everything's going well in the real world. And this basically is something where we have to fight noise at every step. So because we're working with subjective evaluations, we know that they're going to be noisy. We need to be very skeptical of evaluation results and try and use statistics to give us like the best picture of how our engine is performing. So we can use techniques like power analysis, confidence intervals, resampling and more to reduce the variance of the evaluation process as much as possible and provide clearer confidence signals.

16:57 And if we put all of this together, we basically just end up with a multi-step evaluation process. We run tons of offline evals to give us an idea of what's going to go well. We pass the best ones to human evaluation to give us a higher fidelity view of how something is doing. And then we AB test it to kind of test the waters in terms of well whether it's doing well in production and eventually deploy it. And once we deploy, we can continue to monitor for failure cases that we then work back into all of the other steps of the process.

17:25 All right. So I know that's a lot of information, but for any areas that were confusing or you want to read more deeply into, I created this index here. So this links out to further reading on pretty much everything that we talked about both from myself and other people. So I encourage you to look into all of these details if it's interesting to you. Thank you so much and I hope it was helpful.