Getting Started with LangSmith (7/8): Automations & Online Evaluation

0:01 Hi, today we're going to talk about automations and online evaluations in Langmith. Automations are powerful rules you can configure to run over every trace sent to your production application. Online evaluations are a type of automation that help you measure metrics on your application's output. Unlike offline evaluations which we covered in a previous video, online evaluations are run on live production user interactions, not on a curated data set. So let's see how to set up automations.

0:33 We'll start by returning to the tracing project for our application, the explain like 5 chatbot. To create a new automation, you can use the rules button in the top right. I've already configured a couple of automations, but let's click add rule to see how we can make a new one. You can see that we're first given the option to create a name for our automation. We can then use a filter to define which runs in our project our rule will execute on.

1:03 By default, our rule is going to execute on every root run in our project. We also have the ability to configure a sampling rate. Some automations may be more expensive to execute like LM as a judge evaluations. So you can tune the sampling rate to only run your rule on a subset of the traces that meet your filter criteria. You also have the ability to apply your rule to past runs. Finally, you have a list of actions you can actually execute with your automation. You can add the selected traces to an annotation queue or to a data set. You can also apply an evaluator to the selected traces and add feedback to them. This is that concept of online evaluations.

1:49 Just like in offline evaluations, you can create an LLM as a judge or a custom code evaluator to run on your traces. The only difference is that you won't have reference outputs to use in your evaluation because you'll be running over production traces rather than a curated data set. Next, you can also trigger web hooks from automations. This can be useful for workflows like creating a Jira ticket when trace errors occur. For alerting flows, you have the option of configuring an alert directly using pager duty.

2:23 Finally, you can use automations to extend the retention period for your traces as the default trace lifespan is 14 days unless you've added feedback to it or added it to a data set. Now, let's take a look at the automations that I've set up for our project. I've set up one online evaluator using an updated version of our Simplicity evaluator from previous videos. We can click in to see that the prompt is the same.

2:55 We're asking the LLM to evaluate whether our chatbot's answer is simple enough for children to understand. If we scroll down, we can see that we still provide the LLM the input and the output of our trace, but we don't have a reference output to provide anymore. If we scroll further down, you can see that we've also updated the scoring to be from 1 to 10. Our second automation is called the simplicity sample review and it takes a subset of traces with high simplicity scores and adds them to an annotation queue. This allows me to review whether these traces are actually simple explanations.

3:40 And this is pretty cool because these two evaluators are effectively chained. The first evaluator will give traces a simplicity score and the ones that score highly will be sampled by the second evaluator to add them to an annotation queue. We didn't need to do anything special to create this training effect. Rules that add feedback to a trace will send the trace back through our other automations to ensure that any rules triggered off of feedback are properly fired.

4:09 Let's look at the feedback our rules created. We can check on the logs of the rule to see if it ran on our traces. And you can see that it has run on several of the traces I recently added. We can also go back to our tracing project to see our feedback. So, if we click back into the main screen, any feedback added by our online evaluators will show up in the feedback column, and we can scroll all the way to see them.

4:43 These pieces of feedback are also visible if you click on an individual trace. If I open this particular trace, I can click the feedback pane to see our feedback scores, including the simplicity score given by our online evaluator. Finally, we can check our annotation cue to check that some runs with feedback were successfully sampled by our rule. And we can see that some were.

5:16 That's automations in Langmith. As usual, thanks for watching and I'll see you in the next video.