Observability and Evals for AI Agents: A Simple Breakdown

0:00 Two of the most crucial things when building production agents is setting up proper observability and setting up proper evaluation. And these are actually tied and coupled, and this is different than in software engineering and the role observability and evaluation play when building agents is different than in software engineering as well. So I wanna talk a little bit about how we view observability and how it powers a lot of agent evaluation. Maybe starting briefly highlighting some of the things that we think are different about building with ai.

0:30 So in the old world of software, it's very deterministic. You have some code, you run it once you run it again, it has the same results In the new world when we're building LLM apps. The first thing that people do is add an LLM feature into their app in some way. And basically all it does is it takes some inputs, passes it through the LLM, gets some outputs and that's the first kind of like, you know, chat experience that people add to apps or smart search or something like that.

0:54 And it's a little bit different now. It's got this LLM and these LMS are inherently, you know, non-deterministic. And so it's got that in, but it's still kind of constrained. You're, you've really just like got this input to the LLM. You've got the output, you have the user pass the input. And then you show the output to the user in some way. There's not that much. It's doing, this starts to change with agents, where in agents you have the LM actually deciding actions that happen.

1:17 It's calling tools. It's, it's, it's making API requests. It's looking up data, it's taking a bunch of actions and so this changes. The way that the applications functions, they go from fully deterministic to this now non-deterministic place where, where behavior is emergent as you run it. This leads to one of the things that we think is fundamentally different between software and agents. With agents. You don't actually know what your agent will do until your users use it.

1:44 With software, you can, you can write a bunch of tests ahead of time. You can test for correctness ahead of time. There's, there's generally, you know, this, this more constrained. Set of actions that the user can do because they're clicking around in the UI or there's maybe some forms that they can fill in with agents. It's completely different. First, the agent itself is not deterministic, but then also you've got this input box. This input box is natural language.

2:06 That natural language is fed into the agent. It's completely unconstrained in its input. And so both of those things combined to mean that you don't actually know what your agent will do until your users use it. And so this changes what you're trying to debug. With observability. So software observability is built for deterministic systems where you have the code. The code is the source of truth. You know what the system's going to do when you read the code and you debug things by identifying failed functions in a stack trace, and you monitor with a bunch of traditional, kind of like a PM tools with agents because they're non-deterministic.

2:42 The traces. The actual traces of what happened. Those are the source of truth. If you look at the code, it just has the prompt and the tools. You don't actually know what will happen when you run the agent until you run it, until your users use it. And so the source of truth is the traces that are present. Debugging these requires diving into the traces. It requires understanding the decision. And the context at each step. The, the places where these agents mess up is when they make LLM calls.

3:10 And so in particular, it's really important to understand the context at each LLM call. And then from there you can debug what's going wrong. And, and in order to have really good observability to the whole agent, you need to track everything. You need to track the threads of the conversation because something said earlier might actually be the reason it's messing up later, either because the agent mentions it or maybe it doesn't mention it, and that's the source of error.

3:31 There's a ton of different sources of error. So there's a few different primitives of agent observability that we think are really important for actually understanding what's going on in your agentic system. The first is a run, and so this is a single execution step within a trace, and so the easiest way to think about this is a single alarm call, and so important things here are the input, which includes kind of like the system prompt and any tools that are defined.

3:56 Then the output, and this can be an AI message, it can be an AI message with tool calls. It can have some reasoning blocks that start to get a little bit more complex. But high level, the important parts of a run are the input and the context that that has and the output that's produced. Then you've got the trace. The trace is basically a collection of these runs in a certain order. And so you can see here that you can see that these runs include kind of like a research plan, which has some code, but then also has this LLM call and then this conduct research step, which again, HA is a subagent itself.

4:27 So it has some more steps like generate query. You can see more LLM calls. This is the trace is a full execution of all runs, including the order that they happen in. And that's really important because again, like previous steps may influence later steps and you need to be able to see them. Kind of like in context. A lot of agent apps have human in the loop, whether it's because they are chat based or because the human's approval in things.

4:52 But the high level point is in order to really understand and complete this picture of agent observability, you can't just look at a single run and you can't just look at a single trace. You actually need to see all these traces for a given kind of like thread. And that's our terminology for this grouping of multi turn interactions. And so, again, the easiest way to think about this is in chat. You can see here. Visualizing a chat.

5:14 You've got the human message, the AI message. The AI message may be a whole trace under the hood, so it could be doing a bunch of calls here. And so you need to be able to see them kind of like stacked up so that if you see a response down here and the third conversation, it may be coming from stuff that happens up here. And so the context of the whole thread is important for really understanding what the agent's doing.

5:39 We've talked about observability and how that's different, but evaluations also different. So the main thing that you're testing with agent evaluation is the reasoning, not necessarily the code pass. So you know, in software you're testing different code paths. When you're debugging something, when you're testing something, you might break down function and into multiple different methods and test the individual pieces and add unit tests for that. And the equivalent of that would be testing the reasoning at certain parts.

6:05 But you can't break down the code because again, logic of the agent doesn't live in code. It lives in the context. So you basically need to break down the context and test the context at different parts. And so you can do this in a bunch of different ways. And again, the reasoning. That's present can be observed as runs, traces, or threads in, in a similar way, you can test the reasoning at those different steps as well.

6:27 So you can do single step tests, which are basically testing a run. Did the agent make the right decision for this particular LLM call? You're not doing any kind of like looping, you're just passing in the context exactly as is looking at the output and judging the output. You can also kind of like test the trace itself. So now you let it run, you let it loop you let it call tools. Did the agent perform well in, in this kind of like end-to-end trace?

6:49 Did it get an expected output? Did it not loop in a row? And then if you take the, you know, analogy to the third level, which is threads, did the agent perform well over time? Did it maintain context across the conversation? Is it remembering things? And so that's one thing that's kind of like different. The second thing that's different is that production is really where you discover what to test for offline. So you know, correctness and what it means for an app to be correct.

7:16 You. You generally know that before you ship to production, you might learn things in production, but that's more about the product itself rather than correctness, but. In agents, you learn where correctness may be wrong or where gaps may be happening from production because you don't know what goes in, you don't know what users will say, and then you don't know how the agent will handle those until again, you actually ship it to production. And so that's.

7:43 A second big difference, which is again, and that really empowers why these things are so tightly coupled. So talking maybe a little bit about evaluating the different pieces, because evaluation is a place where we see a lot of people spending time and investing a lot of effort, and it's not easy. The first place that you can evaluate is single step evaluations. And so this is, again, you can think of unit tests for decisions. You basically run the AI for a single step.

8:06 You don't execute the tool. You just kind of like look at the, the result of the AI message and see what happens. This is pretty fast to execute. It's just a single LM call and it's got. Pretty clear, kind of like pass fail criteria. Usually for like a single alarm call, you know that, oh, it should call tool A, or, oh, it should respond and it should say X, Y, Z in the response. So it's generally pretty quick to kind of like set up and pretty quick to judge.

8:30 For traces. This validates the whole kind of like agent run from start to finish. So you can see here that it's actually executing the tools. This is better because it tests more of the whole trajectory. You can also test kind of like other state changes. So if the tools are modifying files or modifying code or modifying something, you don't just have to look at. This kind of like message list. To test it, you can actually test the files themselves, and then multi turn is the most realistic.

8:58 This is how people interact with your app. The difficulty is, this is actually kind of hard to set up because if you look at the sequence, in order to test this, you have to, you have to message the agent in a test with this, see what happens, and then message it. With this. The issue is that depending on what happens here. This message might be completely irrelevant, so it's a little bit more complicated to set up the inputs and outputs, but it is the most realistic test of, of production behavior.

9:23 And so going through kind of like some of the pros and the cons, and, and by the way, like none of these is right, quote unquote, like all of these have their place. We see people using these together all the time. But the pros and the cons, the single step ones are the easiest to define metrics for. But if your agent logic, and especially kind of like the internals of should you use a skill or should you use a subagent?

9:43 Like if that goes, if that changes, then these tests may go out of date. The trace is the full end, end one. These are easiest to define inputs for because the inputs are just what you would message the the agent wants. But sometimes it's hard to define metrics for because you're judging the kind of like end result of an agent, which could be pretty complex and hard to judge. Threads are the most realistic, but they're hard to define both inputs and metrics for.

10:09 We've also mostly been talking about so far offline evaluation, which is before you ship to production. But I wanna just highlight that evaluation means a bunch of different things. So offline, maybe starting with offline should be really clear. This is where you. Build up a data set of at least inputs and usually some ground truth outputs as well. I mean, you run this before you ship to prod, whether this is on every commit or just once before you ship to prod, kind of up to you.

10:34 We see people doing different things depending on how complex it is to run. You do this before you ship to prod, and this is really good for catching regressions and then benchmarking. So regressions are usually when people have a set of things that they want to always pass, they'll run these as kind of like unit test. But then they also often have kind of like a benchmark set that they're constantly trying to kind of like hill climb and improve on.

10:56 And so these are both forms of like offline evals, online evals happen when you run it in production. It runs right after the agent runs, so the trace gets sent into Lang Smith. We support these in Lang Smith, and you can define evaluators that run right after the agent runs. The reason to do this is that you can flag issues that are actually occurring in production, the types of valuations you do. Could be similar to offline or they could be different.

11:19 So in offline you can have ground truth. In online, you obviously don't have ground truth, and so you're evaluating just based on the input. So sometimes the evaluator metrics are different in here. And then ad hoc is what we call basically doing this on an ad hoc manner. So online will like always run over all traces, but sometimes you just wanna, you want to like dive into things and evaluate things because you have a hunch or you got some feedback from a user that you need to look into X, Y, Z.

11:43 And so you wanna do that in kind of like an ad hoc manner. And you didn't have set up online evals for that particular thing. And so this can be done at any point. This is very similar to kind of like exploratory data analysis. We have a feature in Lang Smith called Insights Agent that does this. You can ask it things like, Hey, when did the user get frustrated? And it will go, it will filter to things for where the user got frustrated and then cluster them as well.

12:05 So it's more exploratory than kind of like catching regressions or anything like that. And one of the big differences in agents is that observability and evals are pretty tightly coupled. So in software they're. Kind of separate things, but in agents, because the source of truth is in these traces, they're much more tightly coupled. They're coupled in a few ways, traces, power, manual debugging. So one of the first things that the best teams do is set up tracing for their agents locally so that when something goes wrong, they can just manually debug it.

12:37 And so this is looking at agent traces on an. Basis. You can look at every step to see what goro went wrong. You can open it up in the playground to debug, and this is a really powerful and effective debugging workflow. They also power the offline evaluation, so production traces become your eval data set. Like you don't know where the agent will fail until you run in production, and then when it. You do discover places where it's mis performing, you need a really easy way to bring those traces into a data set.

13:04 And so an example workflow that we see be really common is user reports, some incorrect behavior. The developer goes in or, or the builder goes in, doesn't have to be a developer, but they go in, they find the production trace, they get kind of like the state at that failure point. So what are the messages, what is the files that it has access to? And then they create a test case from that state. If you can't use actual production data, you can anonymize it or scrub it or something like that.

13:26 So you can have a representative test case, and then after that you, you go, you fix that test case with offline evals and then you're good to go. Traces, obviously power online evals, so online evals. Run over the traces that you're capturing in production. These can, again, these can't check for correctness because they don't know the ground truth, so they can't use these ground truth labels to check for that. But you can check for trajectory or efficiency or quality or other failures that don't require knowing the ground truth.

13:57 And then again, traces power, these ad hoc insights. So these ad hoc insights run over traces and threads, and so you can use these to surface uses patterns, or identify failure modes or compare successful versus failed executions. And so the foundation of Lang Smith is really in this observability layer because we think having really good observability and a really good mental model for observability these runs, traces, threads, having those concepts front and center really powers, debugging, offline evals, online evals, insights, and other things that we haven't even talked about.

14:33 Thanks for watching this video, and I'd encourage you to check out Lang Smith as a way to get started with Agent observability and evaluation. And if you have any questions, leave them in the comments or please reach out to us.