Section Insights
Introduction to Regression Testing in RAG Pipelines
What is regression testing and why is it important in RAG pipelines?
Regression testing helps evaluate whether changes in the language model (LLM) improve or degrade performance in a retrieval-augmented generation (RAG) pipeline. It allows for the isolation of instances in the evaluation set to identify improvements or regressions relative to a baseline model.
- Regression testing is crucial for assessing the impact of changing LLMs in RAG pipelines.
- It enables the identification of specific instances that improve or worsen performance.
- A well-defined evaluation set is essential for effective regression testing.
Setting Up the Evaluation Environment
How do you set up an evaluation environment for RAG testing?
The evaluation environment is set up by creating a new evaluation set and indexing relevant documentation. Different LLM providers can be specified to run experiments, allowing for comparisons between models like OpenAI and open-source options.
- Creating a new evaluation set is the first step in regression testing.
- Indexing relevant documentation is crucial for accurate evaluations.
- Flexibility in choosing LLM providers enhances the testing process.
Running Evaluations on Different Models
What steps are involved in running evaluations on different LLMs?
Evaluations are run on selected models, and results are compared against a baseline. The evaluation set created earlier is used to assess the performance of different models, allowing for a clear comparison of their outputs.
- Running evaluations on multiple models provides insights into their relative performance.
- A baseline model serves as a reference point for comparison.
- The evaluation process can highlight specific strengths and weaknesses of each model.
Interpreting Evaluation Results
How can evaluation results be interpreted to assess model performance?
Results are visually represented with color coding to indicate performance relative to the baseline. Red cells indicate worse performance, while green cells indicate improvements. This allows for quick identification of regressions and progressions.
- Color coding in evaluation results simplifies the interpretation of model performance.
- Identifying regressions and improvements is crucial for refining models.
- The ability to audit evaluations enhances understanding of scoring decisions.
Utilizing Additional Metrics and Filters
What additional metrics and filters can be used to enhance regression testing?
Additional metrics such as token usage and latencies can be displayed to provide deeper insights into model performance. Filters allow users to isolate cases of improvement or regression, facilitating a more detailed analysis of results.
- Additional metrics provide a comprehensive view of model performance.
- Filters help in isolating specific cases for further investigation.
- Regression testing is a powerful tool for optimizing LLM applications.
Transcript
0:06 this is last L chain this is the 15th video in our Langs Smith evaluation series we're going to focusing on regression testing so the past few videos talked a lot about rag evaluation just to refresh for example we talked about how to evaluate the rag chain answer versus a reference or the answer versus the documents for hallucinations or even the documents relative to the question for kind of retrieval relevance now here's a common situation that comes
0:31 up when I think about building a rag pipeline what if I want to change the llm so you know good example is some pretty cool and interesting new open source LMS have recently come out like llama 3 53 how can I know whether or not I can replace for example gbd4 as my Baseline with one of these for my particular use case and my particular rag pipeline this is where there a notion of aggression testing comes in so we
0:56 previously had talked about building an eal set running it on you know for example are answers and evaluating answer relevance or or answer you know accuracy in this case relative to ground truth answer regression testing allows me to run this on a few different examples like let's say different llms that perform the generation step and to identify cases that either improve or regress relative to my Baseline so the key point of regression testing is the ability to
1:25 isolate instances in your eval set they even getting better or worse so that's kind of the main intuition so to kick this off I'm just going to build a new eval set so that's all defined here it's going to be called rag QA LCL so it's going to use the same line change expression language documentation I used for the prior rag videos just a few other you get a few slightly different questions so I've built that and I've built my index so
1:49 again we're just indexing the line CH expression language documentation again I'm choosing a large chunk size just like we saw before no major changes so I've laid out my rag pipeline and this looks very similar we saw before the only difference is I've added a slight modification where I can specify different providers so in this case I can choose AMA which allows me to run a number of Open Source llms locally along with open Ai and again
2:14 just like rag prompts for the open source models versus open AI but you know again not no major differences here really I'm just extending my rag model or my rag chain so I can run with different llms that's all I'm doing here and I'm defining three of them so I have my ra chain with open AI rag chain with llama 3 and rag chain with 53 again llama 3 and 53 we use open AI as a
2:36 provider and I'm just specifying the model I've already downloaded these locally just with simply AMA pull llama 3 orama pull 53 and it'll just run locally on my laptop so that's a whole separate thing but that's a pretty cool nice thing about AMA so I've defined an evaluator for answers so we saw this previously this labeled score string evaluator basically lets me Supply some criteria so in this case you know I'm asking is the assistant answer grounded in the ground
3:05 truth answer and I give it you know some criteria for scoring one through 10 10 being the best one being the worst so that's all all that's going on here and I kick off my eval right so I run eval on open AI first I run eval on llama 3 I run eval on th3 so those evals have run now let's go over and look at my data set so here is the data set we created
3:28 again RAB QA LCL let's just confirm that here yeah so this is the this is evil set we're running on and it's the evil set we created up here so that's right here so we created our eval set now we can look at our experiments so this is what we saw before right so this is our experiment page no major changes here right we can see our three experiments run here now what I'm going to do is I'm
3:50 going to select these three experiments so I'm going to go one two three and I'm going to open this comparison mod this is where things are going to get kind of interesting we're going to see some things that are a bit new here and let me just scroll over a little bit so first we're going to see is the three experiments we selected are all here okay and we're seeing this idea of a baseline so what does Baseline mean the
4:13 Baseline is basically our reference so in my particular case so let's just say gbd4 was my prior Baseline right so gbd4 was the model that I had been using and I want to see how th3 and llama 3 compare to GPD 4 so what I can do really easily is so these are our three examples just like we saw so again I can click on this and this will basically open up the example so I can see here's
4:38 the question here's the answer so that's just like you know we we've kind of gone through that before so here's the input here's reference output and now here's my three experiments and you can see just like before we've set a baseline and I can see the scores from my evaluator here now what's interesting is I can scroll over and I can look at my two other experiments so llama 3 here 53 here so what's pretty cool is immediately your attention is
5:09 drawn to these red or green cells and this simply means if it's a red cell it means the performance is worse than my Baseline if it's a green I me it's better so in this case right my Baseline was 0.1 in this case 53 actually for the second example does better it got a score of 0.5 so that's why it's green otherwise you can see there's regressions across the board for questions one and three for both of the
5:34 models now I can do a few different things here first like we saw before I can always click on this button and that that actually opens up the evaluator run itself so I can audit the evaluator so that's again we we talked about that previously it's really useful you can kind of see why particular score was given so that's really cool and I can go back and you can see my Baseline is still preserved so that's actually very convenient now
6:01 this thing at the top tells me the number of total regressions or progressions or improvements relative to my Baseline so I can click this and it'll highlight only the cases that improve click this it'll highlight only the cases that regress again likewise here so again it's really useful and nice way to drill into the cases that are improving or getting worse relative to a baseline in this case there's only three examples but in realistic cases you may have an eal set of maybe dozens
6:28 or hundreds of examples so this can very easily be used to isolate your progressions or your improvements and your regressions so you know it's it's it's really helpful I do want to also show here if you go to the display you can actually open up you know additional metrics this can show you like this will show you token usage this will show you latencies so you can kind of close that down to make lesser of those you can see the full text of
6:52 the generations if you want to see that but again it's it's by default of removed so it's like way cleaner and that's probably the main things I'd like to show you there you can of course choose different experiments or additional experiments if they're not all Chen here it's easy to add experiments so anyway this is a really useful and convenient thing I I do this all the time so I'm always running experiments with different llms I always
7:17 want to isolate cases that are improving or regressing and I want to dig into why and basically using these filters I can very easily isolate the cases that are for example getting worse drill into y again in each of these cases you can see you actually can look at the trace again so you this will drill into the trace that resulted in the generation I can go down I can audit the entire chain so this is like my cadow Lama invocation
7:39 here this is like the prompt which contains yeah the context my rag answer so again a lot of things you can play with here really useful and regression testing is obviously super powerful for identifying cases that your chain for example is getting worse or better with different pertubations like different models different chunks or other things so definitely play with us it's extremely useful and and helpful in building llm applications thanks
Summary
- Regression testing isolates instances in the evaluation set to determine improvements or regressions relative to a baseline model.
- A new evaluation set called RAG QA LCL is created, utilizing the same expression language documentation as previous videos.
- The presenter demonstrates setting up a RAG pipeline that can run different LLMs, including OpenAI and open-source models.
- An evaluator is defined to score answers based on their grounding in the ground truth, using a 1-10 scale.
- The comparison of results from different models highlights performance changes, with red cells indicating regressions and green cells indicating improvements.
- The interface allows users to filter and audit results, making it easy to identify specific cases of improvement or regression.
- Additional metrics like token usage and latency can be displayed for deeper analysis.
- Regression testing is emphasized as a powerful tool for optimizing LLM applications by systematically evaluating model performance changes.
Questions Answered
What is regression testing and why is it important in RAG pipelines?
Regression testing helps evaluate whether changes in the language model (LLM) improve or degrade performance in a retrieval-augmented generation (RAG) pipeline. It allows for the isolation of instances in the evaluation set to identify improvements or regressions relative to a baseline model.
How do you set up an evaluation environment for RAG testing?
The evaluation environment is set up by creating a new evaluation set and indexing relevant documentation. Different LLM providers can be specified to run experiments, allowing for comparisons between models like OpenAI and open-source options.
What steps are involved in running evaluations on different LLMs?
Evaluations are run on selected models, and results are compared against a baseline. The evaluation set created earlier is used to assess the performance of different models, allowing for a clear comparison of their outputs.
How can evaluation results be interpreted to assess model performance?
Results are visually represented with color coding to indicate performance relative to the baseline. Red cells indicate worse performance, while green cells indicate improvements. This allows for quick identification of regressions and progressions.
What additional metrics and filters can be used to enhance regression testing?
Additional metrics such as token usage and latencies can be displayed to provide deeper insights into model performance. Filters allow users to isolate cases of improvement or regression, facilitating a more detailed analysis of results.