Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI)

LangChain · 5m · transcribed 11d ago

Section Insights

# 0:00

Introduction to Agent Evaluation

Why is evaluating agents important?

Evaluating agents is crucial for understanding user experiences, particularly in identifying user frustration through automated evaluations.

LangSmith CLI allows for efficient agent evaluation.
Automated evaluations can help manage large volumes of user interactions.
Using LLMs to assess user frustration can streamline the evaluation process.

# 1:01

Feedback Mechanism of Evaluations

How does the evaluation process work?

The evaluation process involves assigning scores and reasoning to user interactions based on predefined metrics, such as user frustration.

Scores are binary, indicating user frustration or lack thereof.
Reasoning is provided alongside scores for transparency.
Feedback is integrated into the LangSmith system for further analysis.

# 2:03

Implementation of User Frustration Evaluator

How is the user frustration evaluator set up?

A coding agent is utilized to automatically evaluate user interactions for signs of frustration, assigning scores based on the evaluation criteria.

The coding agent uses LangSmith evaluator skills for real-time evaluations.
Binary scoring system simplifies the assessment of user experiences.
The setup allows for efficient monitoring of user interactions.

# 3:04

Reviewing Evaluation Results

What do the evaluation results indicate?

The evaluation results provide insights into user interactions, highlighting instances of frustration and confirming the accuracy of the evaluation process.

Feedback scores reveal user sentiment effectively.
Detailed analysis of interactions helps in understanding user frustrations.
The evaluation process can be validated through specific user interactions.

# 4:06

Updating the Evaluator Settings

How can the evaluator be optimized?

The evaluator settings, such as the sampling rate, can be adjusted to reduce costs while maintaining effective evaluations.

Adjusting the sampling rate can significantly lower operational costs.
Continuous updates to the evaluator enhance its effectiveness.
Optimizing settings allows for better resource management in evaluations.

Transcript

0:00 Hi everyone. In this video, we're going to show you how we evaluate our agents at scale without having to ever leave the terminal. We're going to use the LangSmith CLI to interact with LangSmith itself, but by using a coding agent. So, why should I care about evals? To show you why, let me give you an example, Chat LangChain. Chat LangChain is an agent used by people around the world. We get thousands of traces a day, so going through each one of them is impossible.

0:25 It'd be amazing if we could just have an LLM read through the thread and see if the user is frustrated or not. This is exactly how online evaluations are structured. Let's see how that would work. With LangSmith, this is straightforward. Your main agent traces to LangSmith, and within LangSmith, we can create an evaluation which uses an LLM as a judge. Your element as a judge has a prompt and a rubric, which determines what it's evaluating for.

0:54 In our case, this is user frustration. We We tell the LLM what user frustration looks like, and how to score it. It'll then produce some feedback, which gets assigned on the trace, and consists of a score plus a reasoning. The score is whatever metric we decide, so in this case it'll be binary, zero or one. The reasoning is to why the LLM assigned that score. It then gets attached to the trace, where we can see it in LangSmith.

1:20 Let's see how we can implement this on our own. So, now I'm recruiting a coding agent to solve my problem of manually sifting through traces. But first, let's give it the most up-to-date information using LangSmith skills. So, skills are very simple to install. We'll just go over here docs.langchain.com, and we'll copy the latest skills command. This will allow us to This will allow our coding agent to have the most up-to-date information on how to interact with LangSmith.

1:55 Cool. Now that all the skills are installed, I'm going to tell Cloud to go ahead and use the LangSmith CLI to create an LLM as a judge. This judge will run on all incoming traces to our chatbot and then do some form of evaluation. I asked it to check whether the user has had a negative experience and assign a score to each trace where a binary which is binary where one signals that the user is frustrated and zero signals that the user is not frustrated. Let's see how it does.

2:26 Cool. Looks like Looks like the agent is using the LangSmith evaluator skill just as expected. Great. After a couple minutes, looks like the agent was able to create a user frustration evaluator on threads. Perfect. 1 equals negative frustrated experience, 0 equals not frustrated. GPT 5.6 Luna full conversation, 100% sampling. Great. Let's go check the UI and see how it did.

2:57 Cool. I've given the evaluator some time to run, but we can see that it's configured properly. It's user frustration thread, just what we want it. It's a thread evaluator. We have this feedback key and a sampling rate of 100%. We have 13 runs already, so let's take a look at how those runs look. So, checking threads, we see that there's some feedback scores assigned here. Let's look at the ones that may stand out. So, user frustration is given one here. Let's just look at this.

3:27 Okay, the user is asking what's up with LangChain. The agent is giving it a proper response. It's asking for how much we raised. The agent is refusing to answer. The user is still pleading. Agent is refusing to answer. And now the user is clearly frustrated. So, looks like this is the correct classification. Let's turn to details and see the last run of the thread. Looks like the the LM has a judge correctly classified that the user is explicitly expressed anger and dissatisfaction.

4:02 looks like this this is perfect. Cool. Now that we saw the evaluator working properly, let's try to make some updates. So, I'm going to give it the evaluator name which is the user frustration thread. And I'm going to tell it to update the sampling rate. Right now it's at 200% which can get quite expensive pretty fast if you're running a frontier model. So, let's make that change.

4:32 Cool. So, I've told Codex to make the change and it's going to use the LangSmith evaluator scale which is great. Looks like the the agent has updated our user frustration evaluator to now 50%. Let's go check. And just as expected, it's updated to 50%. Now this will be half the cost, hopefully. Now hopefully with that demonstration, you and your agents can pinpoint issues, use them to improve your production applications, and bring a smoother experience for your customers. Thank you.

Summary

The video demonstrates how to evaluate chatbot interactions at scale using the LangSmith CLI and a coding agent to identify user frustration. By leveraging an LLM as a judge, the process automates the evaluation of thousands of traces, providing feedback and scores based on user sentiment.

- LangSmith CLI allows for efficient evaluation of chatbot interactions without manual oversight.
- An LLM is used to assess user frustration by analyzing conversation threads.
- The evaluation process involves creating a rubric that defines user frustration and scoring it as binary (0 or 1).
- The coding agent retrieves up-to-date information from LangSmith to enhance its evaluation capabilities.
- The agent successfully implements a user frustration evaluator, providing feedback on chatbot interactions.
- Adjustments to the evaluation parameters, like sampling rates, can be made to optimize costs.
- The system aims to improve user experience by identifying and addressing issues in real-time.