Section Insights
Introduction to RAG Evaluation
What are the types of RAG evaluation?
The section introduces the concept of RAG (Retrieval-Augmented Generation) evaluation, outlining four main types: comparing generated answers to reference answers, checking for hallucinations by comparing generated answers to retrieved documents, assessing the relevance of documents to the question, and performing internal sanity checks.
- RAG evaluation is crucial for assessing the quality of generated answers.
- Four main evaluation types include comparison to reference answers and checking for hallucinations.
- Evaluators can help ensure that the generated content is relevant and accurate.
Building a RAG Chain
How do you set up a RAG chain for evaluation?
The section describes the process of building a RAG chain, including indexing documents, defining a system prompt, and creating a dataset of ground truth question-answer pairs for evaluation.
- Setting up a RAG chain involves indexing documents and defining prompts.
- Creating a dataset of ground truth pairs is essential for effective evaluation.
- The RAG chain should return both answers and relevant documents.
Using LM as Judge Evaluators
What are LM as Judge Evaluators and how are they used?
This section explains the use of LM as Judge Evaluators, specifically the QA and CoQA types, which are designed for question answering and integrate with LangChain's string evaluators for effective evaluation.
- LM as Judge Evaluators provide a structured way to assess generated answers.
- CoQA is a specific evaluator type that works well for question answering tasks.
- Integrating evaluators with datasets enhances the evaluation process.
Executing the Evaluation
How is the evaluation executed using LangChain?
The section details the execution of the evaluation process using LangChain, connecting the chain outputs to the evaluator interface, and logging the results of the evaluation.
- Executing the evaluation involves connecting chain outputs to the evaluator interface.
- Results are logged for further inspection and analysis.
- The process is streamlined with a single command to run the evaluation.
Inspecting Evaluation Results
What insights can be gained from inspecting evaluation results?
This section discusses how to inspect the results of the evaluation, including viewing the prompts, the answers provided by the chain, and the reasoning behind the evaluator's grading.
- Inspecting evaluation results provides insights into the reasoning of the evaluator.
- Understanding the grading process helps refine the evaluation setup.
- The evaluation process can be easily repeated and adjusted based on findings.
Transcript
0:00 hi this is Lance from Lang chain this is the 12th video on lsmith evaluation series focused on rag so rag retrieval alment of generations one of those popular LM applications and there's a few types of rag evaluation that have come up that are quite useful so probably the most intuitive is if I look at my rag chain here I start with a question I retrieve documents that are relevant to the question I pass them to llm I can answer
0:25 based upon those retrieve documents I can compare my generated answer to a ground truth or reference answer so that's evaluation type one that can be really useful evaluation type two is I can compare my generated answer to my retrieved documents make sure there wasn't anything like hallucinations or other kinds of erroneous kind of information that was injected into my answer that's not present in my documents so that's kind of step two evaluation three is comparing my
0:50 question toar documents so for example relevance is a common thing you can do here make sure your retrieval is actually work properly and the documents are relevant to my question I can also compare my question to my ultimate answer as like an internal sanity check if I didn't have a reference so those are the some some useful types of EV vals there's also more you can think about but at least those are four that i' like to use so all those are string
1:13 comparisons for example comparing my LM generate answer to a reference answer and because of that we can use LINE chain string evaluators for that and the nice thing about this is all line chain string evaluators provide a common interface of a prediction reference and an input so these are the three things I need to PL into the evaluator for it to do its work now let's go ahead and build a rag chain so I've done a few pip
1:36 installs here I'm going to index some of Lang chain stocks focused on Lang chain expression language and I'm using a text splitter I'm defining a chunk size and I now have an index I'm using chroma as my Vector store I'm not going to find a rag chain so this is just basically pretty standard stuff I'm defining a system prompt here your helpful code assistant with expertise in L expression language use the docs to produce a concise code solution to the question so
2:02 that's really all that's happening now I can ask a question I get an answer so this this chain all kind of works and you'll actually not that it returns both the answer as well as documents so that's actually an important thing to note so now I can build a data set from these documents and here I just have we talked about this previously I have a set of inputs I have a set of outputs so these are ground truth questions ground
2:25 truth answers I'll call that rag test LC I'll create that and now if I go over to lsmith I go to my data sets I can search it's there cool here's my examples no experiments yet so we're all set now let's talk about evaluators again and what I want to show first is this reference evaluation because it's kind of the most intuitive it's what I've done usually the most so we built an ebal set I have ground truth question
2:49 answer pairs I want to compare my generate answers to my reference answer so that's kind of step one and again if you look at our framing previously in terms of data set we've built a data setad of question answer pairs so it's user curated we are going to use LM as a judge evaluator on strings comparing my reference answer to my LM generate answer so that's kind of the flow in the setup again we're going to look at this here I'm going to
3:17 use LM as judge EV Valor which I'll show below so within LM as judge evaluators there's a few different options so QA and coot QA are both really convenient on ones that are often used for for question answering and you'll note that both of those are also Lang chain string evaluators which means that they both expose a common interface again like I said before so here's really the information flow this is the key thing to understand what I'm
3:45 doing here is I'm connecting my data set and my chain outputs to the Lang string evaluator interface for my coqa evaluator so again Ling string evaluator the more General type that exposes these common kind this common interface and coqa is the particular LM is Judge of Val that I'm going to use and it happens to also be a lang string of valer so it exposes those common inputs so that's really what's happening here you can see basically my chain
4:17 answer will get piped into prediction and then my reference answer gets piped into reference my question gets piped into input that's really all that's happening so again if you back all the way up we're doing string comparisons because we're doing string comparison L string evaluators are appropriate within L string evaluators a lot of different options I'm going to choose LM as judge evaluators and I'm going to choose cqa which is just a Chain of Thought QA as
4:43 evaluator type that I want that's all that's happening here and you can actually go and investigate the prompt here as well it's all checked into the prompt tub so here's just basically a little function I'm going to use to wrap my chain and here let's just kick off the eval so I'm kicking that off now and I'll explain what's happening here so again I'm choosing my Lang chain string evaluator here and I'm going to choose cqa as my particular evaluator I
5:07 want to use it's a really nice convenient LM as judge evaluator and here is really where all the heavy lifting happenings happens I'm hooking up my run which we saw here that's just my chain my run answer which we can see right here so basically my aners returned from my run chain I'll hook that up to the prediction and then I'm going to take my example which you can see here this is my data set example the
5:35 answer is going to be pumped to reference the question is going to be piped to input and these are the keys necessary for my Lang string for my Lang chain string evaluator that's all it's happening so that ran again this is the only command I need to run so just run evaluate this is the chain I use so you can see it right here the data set name we just defined earlier here's the evaluator name and that's really it
6:01 that's all it's going on so if I go over now to my data set you can see an experiment was logged that's pretty nice and here we go so we can open that up and you can see here we can actually look at this is the input this is the reference output this is my chain output and you'll see something kind of nice here in each case you have this contextual accuracy logged one or zero and if I want to dig under the hood here
6:28 all I need to do is B click on this little thing here and this will go ahead and open right here so this is going to open the actual evaluator run itself so I can peek in here and I can actually look this is the greater prompt and this is just like the setup and down here is where the actual answer that my chain gave as well as the ground truth are provided and then you can
6:58 actually see here this is what the the LMS judge actually did and here's its reasoning and then here's its final grade and that's all reflected then in here so in this case the grade was indeed one so that's really all I need to do it's pretty nice I've used these all the time for rag evals and this is really the first type of eval that is a kind of is is pretty easy to set up and that I like to do and it's
7:25 comparing my answer to a reference answer and all you need is is basically to build a small eval set of ground truth answers you expect and pretty simply Define right here for example CO2 QA is really kind of nice LM is judge of valare that works really well for question answering and all I need to do is just connect my run or my chain output to prediction and my example so this is my ground truth
7:56 answer to reference and then my input as well so that's really it and we just run an evaluation we can inspect that in lsmith and we're going to be talking about some other types of evaluation but I want to kind of start with answer evaluation as really the the first and kind of the most common evaluation I've typically done thanks
Summary
- RAG models combine document retrieval with language model generation for enhanced responses.
- Four key evaluation types include comparing generated answers to reference answers, checking for hallucinations against retrieved documents, assessing document relevance, and ensuring question-answer coherence.
- LangChain provides string evaluators that facilitate these comparisons through a common interface.
- The setup involves creating a dataset of ground truth question-answer pairs to evaluate model performance.
- The evaluation process utilizes an LM as judge evaluator, specifically the Chain of Thought QA (CoQA) method, for assessing answer quality.
- Results from evaluations can be logged and inspected for accuracy and reasoning behind the model's outputs.
- The ease of setting up evaluations with LangChain makes it a preferred choice for RAG assessments.
Questions Answered
What are the types of RAG evaluation?
The section introduces the concept of RAG (Retrieval-Augmented Generation) evaluation, outlining four main types: comparing generated answers to reference answers, checking for hallucinations by comparing generated answers to retrieved documents, assessing the relevance of documents to the question, and performing internal sanity checks.
How do you set up a RAG chain for evaluation?
The section describes the process of building a RAG chain, including indexing documents, defining a system prompt, and creating a dataset of ground truth question-answer pairs for evaluation.
What are LM as Judge Evaluators and how are they used?
This section explains the use of LM as Judge Evaluators, specifically the QA and CoQA types, which are designed for question answering and integrate with LangChain's string evaluators for effective evaluation.
How is the evaluation executed using LangChain?
The section details the execution of the evaluation process using LangChain, connecting the chain outputs to the evaluator interface, and logging the results of the evaluation.
What insights can be gained from inspecting evaluation results?
This section discusses how to inspect the results of the evaluation, including viewing the prompts, the answers provided by the chain, and the reasoning behind the evaluator's grading.