Create a Dataset from Production Traces in LangSmith

0:01 Hi, this is Katherine from Langchain. Today I'll be showing you how to create an evaluation data set from production traces. So when we build agentic applications, it's really important to be able to test their quality. This can be done through experiments. It's typically a good practice to run experiments against a set of representative examples every time there's a change in your application. So this could be a new prompt, a new model, or a new architecture that you might be considering. Because of this, it's really good to have data sets that represent the most important use cases of the application. There are a couple different ways that you can curate these data sets. You can create examples manually.

0:50 You can use AI to generate and augment your data set. And finally, you can also build these data set from real world production traces. So building examples from production traces is a particularly good way to collect realistic examples based on the real world scenarios and this will be the focus of our video today. We'll be showcasing features in lensmith that help you create a highquality data set which can then be used for offline evaluations.

1:22 There are also additional ways to layer on human audits and to automate the process as well. Now I want to quickly showcase the application that will be collecting from production traces today. This is chat lang. This is an application that we have been built by our team that is designed to answer user questions about our product ecosystem. So for example, I can click on one of the example questions about recursive URL. So first we can see that it's generated three different steps of the research plans. And for each of these research plans, it's fetching relevant vector store documents from a vector database. And as a last step, it consolidates all the information from those documents into a consolidated answer.

2:13 So now that we know how this works, let's hop over to Lansmith to see the tracing. Now clicking into the tracing project for chat link chain, I can see that my question about the recursive URL loader has been traced to the back end in linkmith. I can click on this trace to inspect the inputs and outputs from our application. If I notice, let's say in this case that the response is is a little less than ideal, maybe it's incorrect, incomplete, or not as helpful as I expected, then in this scenario, I probably want to add this example to a data set as a use case I want to iterate and improve on. I can click on this button on the top right. And now I can click on add data set. And this will allow me to save this example. So I can create a new data set here or in this case I've already created one for the sake of this demo.

3:09 Let's choose here. It then automatically populates the inputs and outputs from the trace. I can inspect it, make edits to the output in this view that will better reflect the ideal response from our application. For example, I can delete a portion of the message. And this will allow us to define what a good output should look like so that when we run offline evaluations later, we can measure whether our application is actually improving over time. So once I'm happy with both the inputs and outputs, I can hit submit here and this will be added to the data set.

3:52 Let's click on it and we can see that this run has been added as an example here. Additional edits can also be done in this example view if we want to continue iterating on this data set. Great. Now let's hop back over to the tracing view. We just saw a way to inspect and add a singular trace to a data set. This can actually also be applied for multiple traces as well. So for example, I can multi select traces. I can also click on this button that selects all the runs on this page. You can see that there are two options on the bottom. I can add it to a data set which is a similar view to what we just saw. I can also add it to an annotation queue.

4:43 Let's do that right here. Great. Let's hop over to see what annotation Q looks like. So annotation Q provides a dedicated view for inspecting traces and either leaving feedback or adding them to a data set. This is also very helpful for teams that may have dedicated annotators that review data and traces. They can come in, they can provide feedback on the right hand side. This also allows me to quickly inspect through my traces in batch. So, similar to how we can edit the inputs and outputs before adding the data example in a singular trace view, I can also do this quickly for all of my traces. After I'm finished with modifying the example, I can select the data set, which in this case is the traces data set. I can hit add to data set after corrections, hit done, and now it goes to the next trace for me. Let's do this for another example.

5:49 Done. And now let's view the data set that we just created. So you can see that all three examples have been added to my data set. So now we have shown ways to add production traces directly into a data set or through an annotation cube. Now I want to show how you can automate this entire process. So this is done through rules. Rules are essentially a set of actions that you can take on your production traces. So here I'm filtering for all the root runs and can designate a sampling rate. So if I do 0.1 here, this would just means that this rule will apply to 10% of all the incoming traces to this project. I can choose to apply this to past runs as well.

6:42 So below is the list of actions I can now take from here. I can add it to a data set or I can choose to go through the annotation Q route first. Here I can select the trace Q that we were just using. And now rather than needing to come over to the project view and manually select traces, I can now directly go to annotation Q and the trace will be automatically ported over for us. The rules feature also allow us to apply additional filters to identify on the type of runs we want to focus on. This is very useful for prioritizing the type of examples that should be reviewed or included in a data set. And here are some of the common actions that we see. One common use case is sending runs with negative feedback to an annotation queue first for further review. So let's show how we can set it up here.

7:40 So I can add additional filter here and specify that we want to capture runs where the feedback score is negative. So for this particular application we have defined a feedback called helpfulness and a score of zero would mean that it has negative feedback score. And now by filtering for it this will automatically send any negative tracing into the annotation queue for further review. Another useful filter is identifying runs with errors or unusually long response times. So to do that, we can add a filter that selects all runs that has a status of error. Do that right here. Or we can also choose things like latency that exceeds a certain amount. Finally, we can also use rules to automatically add highquality runs to a data set. So this would be similar to the first example where we will create a filter for the feedback score helpfulness few shot as one and then we can directly add it to a data set in order to curate our examples. Great. From there by building up a data set of these examples we can run offline evaluations to track how well our applications handle comprehensive set of use cases over time.