Getting Started with LangSmith (6/8): Annotation Queues

0:00 Hi, today we're going to talk about annotation cues. Annotation cues are a great way for both developers and subject matter experts to directly leave feedback on your application's performance. This allows you to leverage human feedback to improve your application over time. So, let's start by setting one up. You can do so by going to the annotation cues tab and clicking new annotation Q in the top right. This will open up a page where you can specify a name, description, and default data set for your queue. The default data set is a helpful quality of life feature to let you easily save annotations after you're done with them.

0:40 So, let's go ahead and fill in these fields. We're going to be using this cue to review the outputs of our explain like M5 chatbot. Let's use our golden data set as the default. In this next section, we can give our reviewers some instructions. And we can also specify what feedback we would like our reviewers to give. In our case, we're interested in how correct our chatbot's answers are. So, let's use correctness as our metric.

1:18 We can define these categories and move on to the final section. In this final section, we can specify how many reviewers each annotation needs as well as enable this concept called reservations. Reservations simply allow you to lock your review for a set amount of time to let a particular reviewer look at it. Let's go ahead and finish creating this queue. You can see it shows up here. The natural next question is, how can we add traces to this queue? Let's go to the project for our chatbot.

1:57 To add a trace to an annotation queue, you can click into it and select this add to button in the top right. By choosing annotation Q, we can add this to the queue we just created. We can also add traces in bulk to our annotation queue. To do this, select all the traces you're interested in and then choose add to annotation queue at the bottom of your screen. Let's choose the queue we just created. We should now be able to see these traces in our annotation queue. So, let's go and check it out.

2:34 You can see that our traces have populated the queue. We can see the input of the trace, the output of the trace, as well as a space for us to give feedback as the reviewer. In this case, I don't actually think this is a good answer for what trust call is. When I was asking this question, I actually wanted results about Lang Chain's open- source library. So, I could go to the feedback section and give this an incorrect score.

3:02 I would then have the option to move on to the next trace by clicking done or to save this annotated trace to a data set. Alternatively, as the reviewer, I could also modify this output answer so that it is correct before I save it to our golden data set. Let's put in a correct answer. We'll change our feedback to match. And now we can save this to our golden data set. Now that we're done annotating this, we can move on to the next trace.

3:38 This trace is about photosynthesis, and I think it's a pretty good answer. So, we can go ahead and mark it as correct. If I wanted to, I also have this space here where I can leave free form comments on this annotation if I feel the need to. In this case, I don't feel the need to. So, I can mark done and finish all the traces in our annotation queue. With that, we finished reviewing all of our traces in our queue and we've finished explaining how annotation cues work in Linksmith. Thanks for watching and I'll see you in the next video.