Catch Agent Regressions Before You Ship: Evals for Managed Deep Agents

0:00 Once you've built a managed agent, you need a way to evaluate its performance over time. As you add skills, tools, and other capabilities, you'll need to ensure that the base performance and any other changes you make to the agent itself don't lose certain features that you've already built. I'm Nathan, I'm in product at LangChain, and I'm gonna be talking a bit about our eval capabilities that you can add to your managed agents as you develop them.

0:23 So once you have an agent built out, you need a benchmark to evaluate the performance of that agent. With evals, there's kinda two ways to think about it. The first is the ability to find and catch any regressions. So when you make a change to your agent, you can ensure the performance doesn't change. Another way is to hill climb, which is where you set a certain set of criteria, and you try to improve the capabilities of your agent.

0:43 We're gonna mostly focus on defining and catching regressions today. So with Managed Deep Agents, we found Harbor is a great toolkit for building out evals so it's bundled into the Managed Deep Agents system, which means it's pretty easy to get started and we have some great skills that make it easier to scaffold the set of evals that are built around your agent. Harbor specifically is useful because it builds an image around your agent, so every time you run an eval, it's using a fresh container.

1:11 Because agents may need to interact with the file system or run external commands, you wanna make sure that it's nicely containerized for every run. It also has great capabilities around the management of tests to avoid pollution of your test environment, and it has really nice conventions for handling how you execute the tests and, and how you structure them, so they're easier for you and any collaborators to work on. In general, when you have an eval, you have the environment, which is the set of files, data, the starting state that you have for your agent.

1:42 You have a job, which is essentially an instruction to the agent, and that's defined in a markdown file. And then you have a check, which is a verifier that evaluates whether the agent completed the job adequately. the evals are going to sit inside of your project directory. And when you run mda evals init, it will scaffold this for you using Harbor under the hood. In this example, we'll talk about a research assistant agent and we'll talk through the specific evals that we are going to be building for that agent or ways that we can automate that.

2:10 In each eval, we have the instruction file, we have the task file, which is the config for Harbor, which defines how it should be set up, our Docker file, which sets up the container, and then we have our tests where we are actually evaluating the response of the agent. And in general, most of this is going to be just set up automatically. You shouldn't need to change the config in most cases. Where we're going to focus primarily is on the instructions and the tests that we're actually executing.

2:39 in this example, I have a research assistant agent for the made-up Meridian platform. This agent is expected to answer questions from the internal document corpus. And all this is defined in our instructions file for the managed deep agent. This agent also has access to some tools to help it perform its research. We've made up some documentation. This is a document corpus with some various documentation about however this platform works. And the agent has a tool search docs that allows it to query the documentation and, and learn more about the Meridian platform if it needs to.

3:12 Let's say I talk to this agent, and I ask it to brief me on how per target rate limit config works for outbound integrations. So what this is gonna do is it'll leverage the tool calls, and it'll use the instructions.md in order to generate some output that's going to match with the spec. At least, that's what we want. And in this case, it did the thing. So it used the tool calls, it generated the results, and I can kind of visually inspect and say, "Yeah, okay, this seems to match. We've got summary findings, gaps, sources," it looks pretty good.

3:42 The fastest way to get started is by running mda evals init. And if we use the -i command, this is gonna allow us to hand off the evals work to a coding agent. And I'll just hand this off to Claude Code. What happens is it passes into Claude Code the instruction set. It asks it to use the LangChain eval engineering skill, which will help out with the ability for the agent to produce evals. And then it tells it to run the evals as well to ensure that they work.

4:15 Claude has finished producing the evals for this agent. Let's take a look at the output. It will produce a plan, and then it will start to execute. And once it's completed, it'll describe the set of tasks. You'll see a pause command because the skill requires that the agent produce this task.md file, which the human in the loop is going to have to approve. And for each of the tests that we have we have our evals, and we have these three new directories, free tier limits, replay after expiry, and unwritten guide.

4:41 We have the same spec that we were looking at before, and each one has this task.md, which is the human readable spec describing the eval that was built there. In this task.md, we can see a detailed analysis of what happened and what the eval is analyzing in this situation. We have the instructions. So this is the only thing the agent sees when we create this. So if I just run this in Studio, we can actually see what the agent will do pretty easily.

5:05 It'll search the docs and then it will generate some output that's answered here. The eval suite is going to compare that output using the set of test cases that were produced. So here's our output, and then we want to ensure that this test cases, like these, cover this. So Harbor is gonna run this test.sh., In this case, it'll invoke the various test cases that the agent has produced, and then it will produce a reward depending on if the agent was successful or not.

5:32 Because I'm connected with LangSmith, these runs, which the agent ran, in this case, I can see them show up actually under my Data Sets tab in LangSmith. So here's the three evals, and I can look into each of these and see what happened if I want to inspect the trace. Tracing is managed automatically with deep agents, which is great. And I can see that in all three cases, the reward was one. In essence, the agent successfully completed the task, so that's great.

5:59 The last thing I wanna do is look closely at some simpler test suites. So in this example, I have the exact same agent. Let's take a look at some tests that we can actually read start to finish. In these examples, I have a couple of simpler tests, and I really wanna evaluate core principles of this agent, like whether it invokes tools and whether the brief is in the right format. In the instruction, we can obviously try this out again, and this is one we looked at before where it runs a brief.

6:23 It'll search the docs using the tools that are available to the agent, and then return a response. When we run this in Harbor, it will evaluate the response and then compare it with our test case. So what Harbor is doing is it's running this test.sh, which in this case is very simple. Anything that you want can go in here, of course, but in this case, what we're doing is we're running pytest I've written these tests using pytest.

6:45 Depending on the output, we're going to give a different value to our reward file, which is what Harbor is looking for to determine the success In this, we are running a couple of very simple tests. We're testing whether it's cited only real documents, we're testing whether it admits when the corpus falls short, and we're testing whether it lists its sources. So we expect, in these cases, that the agent will look in, and it'll find the citations, and we're gonna compare the citations to the set of answers that are available up above.

7:16 And to test whether the corpus is falling short, we'll just look in the agent's response to see whether the agent says, "Okay, we don't have this, this doesn't exist," or whatever other strings we might think match. Finally, for listing sources, we want to assert that it includes these references in the answer. So with these three tests, we have some basic coverage of the way that our agent can perform in different situations. And in this example, I actually have two different evals: thin corpus with one instruction and retry semantics with a different instruction.

7:51 In this case we wanna make sure it uses the brief format, again, it only cites real documents, and then that it found the status codes because this instruction is about our retry semantics. You can get pretty creative with the way that you evaluate these agents to ensure that it's following the spec. So once you've produced a set of evals, a great way to manage them in production for your managed Deep Agent, you want to continue to update them over time and make sure that your coverage continues to be strong for the project that you're working on.

8:16 But you can run them, for example, nightly inside of your CI system in order to keep track of your agent's changes over time. And as you do that the results will file into LangSmith. You work on your agent, your tests continue to run, and you can continuously evaluate the performance of the agent. So again this is a set of tests that are focused primarily on ensuring stability over time. You can play around and experiment with different models, with different tools or with different tool descriptions to see how that affects the performance of your model.

8:46 So as you make changes, not only running these nightly, but running these in real time as you work on development is super helpful. So that's a quick intro to evals and to how you can manage them in LangSmith and with Harbor.