Section Insights
Introduction to Back Testing with LangChain
How can we evaluate different app variants using production data?
The process involves collecting production logs from an existing app and using them to test new app variants without needing to create a curated dataset. This allows for effective back testing of different approaches, such as context stuffing versus using a vector store.
- Back testing allows for testing new app variants using real user data.
- Production logs can be repurposed as datasets for evaluation.
- This method eliminates the need for curated datasets.
Creating a New Dataset for Evaluation
What steps are involved in creating a new dataset from user logs?
After running user questions through the app, the logs can be filtered and transformed into a new dataset. This dataset can then be used to evaluate a different app variant, allowing for a direct comparison with the production version.
- User logs can be transformed into datasets for evaluation.
- Filtering options allow for tailored dataset creation.
- New app variants can be tested against the same user inputs.
Running Experiments with New App Variants
How do we run experiments to compare app variants?
Once the new dataset is created, an experiment can be initiated using the new app variant. This experiment runs on the same user inputs, generating outputs that can be compared to the production baseline.
- Experiments can be run on new app variants using existing datasets.
- Outputs from new variants can be directly compared to production outputs.
- This process helps in assessing the effectiveness of changes made.
Setting Up Pairwise Evaluation
What is the purpose of a pairwise evaluator in app testing?
A pairwise evaluator allows for a comparative analysis between the outputs of the production baseline and the new app variant. By defining specific evaluation criteria, the evaluator can assess which version performs better based on user interactions.
- Pairwise evaluation provides a structured way to compare app outputs.
- Criteria can be defined to focus the evaluation on specific aspects.
- This method enhances the understanding of performance differences.
Analyzing Evaluation Results
What insights can we gain from the evaluation results?
The evaluation results indicate a preference for the new app variant over the baseline, and detailed explanations can be accessed to understand the reasoning behind this preference. This workflow demonstrates the effectiveness of back testing and comparative evaluations.
- Evaluation results can reveal preferences for app variants.
- Detailed explanations help in understanding performance differences.
- The workflow supports continuous improvement of production apps.
Transcript
0:01 hi this is Lance from Lang chain we're continuing our Langs Smith evaluation series focused on back testing today so to to motivate this let's say we had an app in production say for example it's one of our rag apps that we kind of talked about in some prior videos and the particular rag app in our case is gbd4 Turbo using a vector store so that's in production we're collecting production traffic from users based using this app version now what happens
0:28 if we want to try different variant like let's say I want to try context stuffing rather than using my Vector store one really convenient and nice thing to be able to do there is take a bunch of production logs we've already collected and just run my new app variant on them and see how the output Compares right that's a really common thing you might want to do so if you look at our framework here we're talking about the
0:49 data set actually can come from existing production logs that's really convenient we don't have to build some like curated data set from it we can just take user data that actually has been you know contributed to our existing app and turn that into a data set so that's step one and we can test then a different app variant on those same inputs so that's really useful and we can use like a pairwise evaluator compare our new
1:13 variant of the app to our old or production version that's running currently so that's like a common workflow now let's walk through this so I'm going to create a new project I'll call it back testing I'll create a a new one here back testing let's say V2 and here's my app so let's say this is running a production here here's a few user questions related to L chain expression language so if you call this particular app is actually ingesting
1:38 information about Lang and expression language and so these are all running cool so we can just kick a few of these off great so I've run five different user
2:12 questions through my app now assume this is you know obviously we're doing the notebook but assume this is a production app it's it's just out there these are users interacting with your app right so that's what we're simulating here now I've created this project back testing V2 we can go over to limith I can look at my projects page and it's here and so great so here's all my traces they're all logged here so that's fantastic now what I can do is this
2:42 code right here will take these traces so I can specify my like my my run filters in terms of like start and end time and I can choose the project that I want to grab from so in this case it's this project we just logged to and basically what we can do here is I can create a new data set so I'm taking these user logs effectively right here and what I'm do is I'm turning them into
3:05 a new data set so now we'll see if you go over to our data sets page we have this new data set called backing V2 and it has like the exact time and so forth and what you'll see here is pretty nice those user inputs are now just simply inputs in my data set so I've kind of sucked them into a data set that's all I've done now what I can do here is I can run an evaluation on the states that
3:28 we just created using a different Vari into my app so I'll call this this predict Rag and ser gbd4 turbo is a different app variant this particular variant doesn't use a vector store it does context stuffing which we can talk about a little bit later but let's say kick off that evaluation so I run that right now so if I can I can go back to my data set here we can see a new experiment has been kicked off and
3:50 what's pretty cool about this when I create this data set my initial experiment is the prod Baseline so that's basically my production app that we've been using that we logged from and that is actually what we collected our our inputs and our outputs initially from so this is kind of like our Baseline right this is our inputs these are outputs from that Baseline version we looked at initially now I've just run a new experiment using our variant right
4:14 so this a gp4 turbo so it's what I just kicked off and this is running now you can see it's still running you can see kind of the name here my experiment prefix gp4 turbo and that is what you can see here so gb4 turbo that's great so that's all running and we can check and see the state of that experiment looks like it's still going cool so there's one more to go and it looks like it's done so that's
4:45 great so now we are finished so here's where things get kind of interesting so I've run so maybe just back up what do we do here so if I go all the way back first I had a project that I was logging to so this is simulating for example like a a production app and these are user user interactions that are being logged to some particular project what I've done is I've taken these these these input and output Pairs and I've
5:13 pulled them into a new data set and you can see that right here I suck those in and by default when I do that that Baseline is grabbed as my prod Baseline so this is what my production app inputs and outputs now what's cool is I was able to just run a new experiment with my new app version on those same user inputs but now I have my new output so these are the outputs for my new chain
5:35 so that's great now this all comes together in kind of a cool way because I can use a pairwise evaluator to compare them so I just created a new pairwise evaluation prompt for this particular task and I I think I already have it open here great so this particular task if you recall is code question answering based on the Lang expression language documentation so I set up my pawise prompt and I just say you know previous
5:58 act as a judge to evaluate the quality of of code responses from two assistants related to Lang Str language I give my criteria here begin your evaluation comparing the responses based upon do they contain a clear overview of the problem do they contain code import they contain code solution right so you don't allow length to affect the results and so forth so anyway that's my evaluation prompt now I've I can Define this pairwise Valor we just already
6:27 talked about this in the last video so you've already seen a good example of that I'm creating a pawise evaluator right here and what's pretty cool is I can run that pawise evaluator on my two experiment so this is my this is the name of my most recent experiment it's gp4 turbo boom and my initial experiment I can go back and get that name so it's this prod Baseline so I can just plug those in okay so I'm running a comparative
6:57 evaluation between my prod Baseline and my new app variant that I ran back testing on I kick that off so now this is running and of course it's going to be logged to the same overall data set here so if I go back to my data set recall we had two experiments my prod Baseline my back testing which is running those same prod inputs with my new chain gp4 Turbo with context stuffing and now I've kicked off an
7:25 experiment that's comparing the two so it's comparing my prod Baseline versus my new variant gbd4 Turbo with context stuffing versus the the Baseline if you recall is actually using retrieval from a vector store the varant is actually using gbd4 Turbo with context stuffing the entire lell docs into the into the llm which has some benefits over retrieval because you don't have issues related to retrieval of the correct document chunks you're just passing in everything and so there's
7:55 some benefits there that we could talk about at length different time so this is running and it looks like it might be finished this is done and this is pretty cool so I can go I can go here I can look at here's all the inputs here was my prod Baseline here is my variant this is what I ran back testing on and what's pretty neat about the pair wise value we can really see the preference
8:18 and so in this particular case it prefers the output from our variant over the Baseline and again just as we saw in the previous video we can actually click into here and investigate why so you can go to the evaluator and it actually gives you an explanation as to why so anyway you can really dig into this but this is a pretty nice workflow so if we Zoom all the way back out what did we really do here well we had an
8:42 example of an app running in production this is kind of simulation of that we collected five user inputs we turned those into a data set that's what we did here we then ran what we call back testing on that data set with a new app variant so like I want to test the new app version in this case my Baseline used retrieval from a vector store my new app variant used context stuffing so I ran that
9:10 evaluation and then I so then that that resulted in a generation so that's right here for every input I produced a new generation or output from my variant that's great and then I ran this comparative eval saying hey which one's better here's my prompt and I kicked that off I just added the names of the two experiments and as we saw before you get this pretty nice comparative assessment with detailed evaluation so pretty useful thing to do if you have any app running
9:38 in production you can pretty easily grab those production logs turn them into a data set do back testing on them with like different variants different chains you want to test and you can even do things like pairwise evaluation to say hey which one's better or worse based on some criteria that I can Define so really nice workflow highly convenient for kind of testing at testing different variants of of chains that you want to do you want to put in production
10:03 thanks
Summary
- Back testing allows for testing new app variants using existing production logs.
- User interactions can be transformed into datasets without the need for curated data.
- A pairwise evaluator can compare outputs from the new variant against the production baseline.
- The workflow includes creating a project, logging user interactions, and generating new datasets.
- Context stuffing is highlighted as an alternative to using a vector store for data retrieval.
- The evaluation process provides detailed insights into which app variant performs better based on defined criteria.
- This method is efficient for testing different chains and improving production applications.
Questions Answered
How can we evaluate different app variants using production data?
The process involves collecting production logs from an existing app and using them to test new app variants without needing to create a curated dataset. This allows for effective back testing of different approaches, such as context stuffing versus using a vector store.
What steps are involved in creating a new dataset from user logs?
After running user questions through the app, the logs can be filtered and transformed into a new dataset. This dataset can then be used to evaluate a different app variant, allowing for a direct comparison with the production version.
How do we run experiments to compare app variants?
Once the new dataset is created, an experiment can be initiated using the new app variant. This experiment runs on the same user inputs, generating outputs that can be compared to the production baseline.
What is the purpose of a pairwise evaluator in app testing?
A pairwise evaluator allows for a comparative analysis between the outputs of the production baseline and the new app variant. By defining specific evaluation criteria, the evaluator can assess which version performs better based on user interactions.
What insights can we gain from the evaluation results?
The evaluation results indicate a preference for the new app variant over the baseline, and detailed explanations can be accessed to understand the reasoning behind this preference. This workflow demonstrates the effectiveness of back testing and comparative evaluations.