Turn Flagged Traces Into a Dataset in 3 Minutes with the LangSmith CLI

0:00 Everyone, I wish to demonstrate a useful workflow for quickly analyzing traces and creating data sets using a coding agent. Here at LangChain, we have a customer support agent which gets thousands of interactions a day. It's quite impossible to comb through these manually, so we recruit the help of evaluators to do the initial screening for us. To help with that screening, LangChain has created an evaluator called perceived error. It detects if the conversation contains evidence that the agent made a mistake, if it misunderstood the user's request, or if it just took their interaction in the wrong direction.

0:30 So, as you can see here, we have many perceived error tags, some of which are true flags. Let's see how we can use these flags as an indicator there was an issue with this thread, and see what the issue actually was. In order to do this, we're going to use the LangSmith CLI and our favorite coding agent. It makes it really easy to interact with LangSmith with an agent that we know how to use quite well.

0:54 Before we get started with our coding agent, let's grab the latest LangSmith skills. These are located at docs.langchain.com, and it's as simple as just copying and pasting it into your coding agent. Great. Looks like the skills are all updated. Now, let's tell Codex to check our chatbot tracing project and grab the 50 latest threads which have been flagged for perceived error. Great. Looks like our agent was able to grab the last 50 thread traces which have been flagged for perceived error.

1:24 And before we go further, let's take a look at agent issue types. There's a bunch that we've defined here in the LangChain docs. Agent looping, context explosion, failed recovery, feature gap, flawed plan, and on. We can actually just copy this and give it to our coding agent to then classify the perceived error failures and see what actually went wrong with our agent, and how many times that issue came up. Great. So, I've now set up the prompt to then classify the threads which have been flagged by reading both the thread and the feedback reasoning, and then categorize them into the categories that we saw in the issues list in our docs.

2:02 Great. Looks like our coding agent was able to classify all 50 threads. Seeing most of the results saying that it's just a flawed plan. So, if we take a look at what flawed plan means, it is when the agent's approach shows a fundamental misunderstanding of the task. The best thing that we should do now is create a data set which contains all of these issues in LangSmith. What would be ideal is if we had data set splits for each issue type. That way we can efficiently retrieve and sort all the different issues.

2:33 So, here's the prompt. It's very simple, just telling Codex to create a native thread data set with these issue threads and create a data set split for each issue type. Great. Looks like Codex was able to create the data set. And I've also asked it to provide a link. So, let's go check it out. Cool. Looks like we have various splits here. And we can take a look at this. And this is the proper thread. You have human-AI pairs, perfect.

2:59 Looks like attachments are rendered properly as well. And let's take a look at how we can actually see these splits. So, we have let's max it out at 50. A couple of different splits here. We can edit the splits as well. Select split. Let's see, we want to see where all the failed error recovery threads were, and here we go. Great. Given this corpus of issues, you can now use them as an evaluation metric, post- training examples, or anything else which your domain calls for. Thank you.