# Observability and Evals for AI Agents: A Simple Breakdown

**Creator:** LangChain
**Platform:** youtube
**Duration:** 14m
**Source:** https://www.youtube.com/watch?v=FDVdLrloFOw

## Summary

Building production agents requires a strong focus on observability and evaluation, which differ significantly from traditional software engineering due to the non-deterministic nature of AI agents. Understanding how to track and assess agent behavior through traces and runs is crucial for effective debugging and evaluation.

- Observability and evaluation are tightly coupled in AI agents, unlike in deterministic software systems.
- AI agents operate non-deterministically, making it impossible to predict their behavior until users interact with them.
- Observability relies on tracing the execution of agents, where the traces become the source of truth for debugging.
- Key components of observability include runs (individual execution steps), traces (sequences of runs), and threads (multi-turn interactions).
- Evaluation focuses on reasoning rather than code paths, requiring context breakdown to assess agent performance.
- Different evaluation methods include single-step tests, trace evaluations, and multi-turn assessments, each with its pros and cons.
- Offline evaluations help catch regressions before production, while online evaluations flag issues in real-time based on user interactions.
- Ad hoc evaluations allow for exploratory analysis of agent behavior based on specific user feedback or observed issues.

## Transcript

[[0:00]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=0s)
Two of the most crucial things
when building production agents is setting up proper observability
and setting up proper evaluation. And these are actually tied and
coupled, and this is different than in software engineering and the role
observability and evaluation play when building agents is different
than in software engineering as well. So I wanna talk a little bit about
how we view observability and how it powers a lot of agent evaluation. Maybe starting briefly highlighting
some of the things that we think are different about building with ai.

[[0:30]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=30s)
So in the old world of software,
it's very deterministic. You have some code, you run it
once you run it again, it has the same results In the new world
when we're building LLM apps. The first thing that people do is add an
LLM feature into their app in some way. And basically all it does is it takes
some inputs, passes it through the LLM, gets some outputs and that's the
first kind of like, you know, chat experience that people add to apps or
smart search or something like that.

[[0:54]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=54s)
And it's a little bit different now. It's got this LLM and these LMS are
inherently, you know, non-deterministic. And so it's got that in, but
it's still kind of constrained. You're, you've really just
like got this input to the LLM. You've got the output, you
have the user pass the input. And then you show the output
to the user in some way. There's not that much. It's doing, this starts to change with
agents, where in agents you have the LM actually deciding actions that happen.

[[1:17]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=77s)
It's calling tools. It's, it's, it's making API requests. It's looking up data, it's taking a
bunch of actions and so this changes. The way that the applications functions,
they go from fully deterministic to this now non-deterministic place where, where
behavior is emergent as you run it. This leads to one of the things
that we think is fundamentally different between software and agents. With agents. You don't actually know what your
agent will do until your users use it.

[[1:44]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=104s)
With software, you can, you can
write a bunch of tests ahead of time. You can test for
correctness ahead of time. There's, there's generally, you
know, this, this more constrained. Set of actions that the user can do
because they're clicking around in the UI or there's maybe some forms
that they can fill in with agents. It's completely different. First, the agent itself is not
deterministic, but then also you've got this input box. This input box is natural language.

[[2:06]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=126s)
That natural language
is fed into the agent. It's completely
unconstrained in its input. And so both of those things
combined to mean that you don't actually know what your agent
will do until your users use it. And so this changes what
you're trying to debug. With observability. So software observability is
built for deterministic systems where you have the code. The code is the source of truth. You know what the system's going to
do when you read the code and you debug things by identifying failed
functions in a stack trace, and you monitor with a bunch of traditional,
kind of like a PM tools with agents because they're non-deterministic.

[[2:42]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=162s)
The traces. The actual traces of what happened. Those are the source of truth. If you look at the code, it just
has the prompt and the tools. You don't actually know what will
happen when you run the agent until you run it, until your users use it. And so the source of truth is
the traces that are present. Debugging these requires
diving into the traces. It requires understanding the decision. And the context at each step. The, the places where these agents
mess up is when they make LLM calls.

[[3:10]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=190s)
And so in particular, it's
really important to understand the context at each LLM call. And then from there you can
debug what's going wrong. And, and in order to have really
good observability to the whole agent, you need to track everything. You need to track the threads of the
conversation because something said earlier might actually be the reason it's
messing up later, either because the agent mentions it or maybe it doesn't mention
it, and that's the source of error.

[[3:31]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=211s)
There's a ton of different
sources of error. So there's a few different
primitives of agent observability that we think are really important
for actually understanding what's going on in your agentic system. The first is a run, and so this is a
single execution step within a trace, and so the easiest way to think about
this is a single alarm call, and so important things here are the input,
which includes kind of like the system prompt and any tools that are defined.

[[3:56]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=236s)
Then the output, and this can
be an AI message, it can be an AI message with tool calls. It can have some reasoning blocks that
start to get a little bit more complex. But high level, the important parts of
a run are the input and the context that that has and the output that's produced. Then you've got the trace. The trace is basically a collection
of these runs in a certain order. And so you can see here that you can
see that these runs include kind of like a research plan, which has some
code, but then also has this LLM call and then this conduct research step,
which again, HA is a subagent itself.

[[4:27]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=267s)
So it has some more steps
like generate query. You can see more LLM calls. This is the trace is a full
execution of all runs, including the order that they happen in. And that's really important because again,
like previous steps may influence later steps and you need to be able to see them. Kind of like in context. A lot of agent apps have human
in the loop, whether it's because they are chat based or because
the human's approval in things.

[[4:52]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=292s)
But the high level point is in order
to really understand and complete this picture of agent observability, you
can't just look at a single run and you can't just look at a single trace. You actually need to see all these
traces for a given kind of like thread. And that's our terminology for this
grouping of multi turn interactions. And so, again, the easiest way
to think about this is in chat. You can see here. Visualizing a chat.

[[5:14]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=314s)
You've got the human
message, the AI message. The AI message may be a whole
trace under the hood, so it could be doing a bunch of calls here. And so you need to be able to see them
kind of like stacked up so that if you see a response down here and the
third conversation, it may be coming from stuff that happens up here. And so the context of the whole
thread is important for really understanding what the agent's doing.

[[5:39]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=339s)
We've talked about observability
and how that's different, but evaluations also different. So the main thing that you're testing
with agent evaluation is the reasoning, not necessarily the code pass. So you know, in software you're
testing different code paths. When you're debugging something, when
you're testing something, you might break down function and into multiple
different methods and test the individual pieces and add unit tests for that. And the equivalent of that would be
testing the reasoning at certain parts.

[[6:05]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=365s)
But you can't break down the
code because again, logic of the agent doesn't live in code. It lives in the context. So you basically need to break
down the context and test the context at different parts. And so you can do this in
a bunch of different ways. And again, the reasoning. That's present can be observed as
runs, traces, or threads in, in a similar way, you can test the reasoning
at those different steps as well.

[[6:27]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=387s)
So you can do single step tests,
which are basically testing a run. Did the agent make the right decision
for this particular LLM call? You're not doing any kind of like
looping, you're just passing in the context exactly as is looking at
the output and judging the output. You can also kind of like
test the trace itself. So now you let it run, you let
it loop you let it call tools. Did the agent perform well in, in
this kind of like end-to-end trace?

[[6:49]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=409s)
Did it get an expected output? Did it not loop in a row? And then if you take the, you
know, analogy to the third level, which is threads, did the
agent perform well over time? Did it maintain context
across the conversation? Is it remembering things? And so that's one thing
that's kind of like different. The second thing that's different is
that production is really where you discover what to test for offline. So you know, correctness and what
it means for an app to be correct.

[[7:16]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=436s)
You. You generally know that before
you ship to production, you might learn things in production, but
that's more about the product itself rather than correctness, but. In agents, you learn where correctness may
be wrong or where gaps may be happening from production because you don't know
what goes in, you don't know what users will say, and then you don't know how
the agent will handle those until again, you actually ship it to production. And so that's.

[[7:43]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=463s)
A second big difference, which is
again, and that really empowers why these things are so tightly coupled. So talking maybe a little bit about
evaluating the different pieces, because evaluation is a place where we see a lot
of people spending time and investing a lot of effort, and it's not easy. The first place that you can
evaluate is single step evaluations. And so this is, again, you can
think of unit tests for decisions. You basically run the
AI for a single step.

[[8:06]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=486s)
You don't execute the tool. You just kind of like look
at the, the result of the AI message and see what happens. This is pretty fast to execute. It's just a single LM call and it's got. Pretty clear, kind of
like pass fail criteria. Usually for like a single alarm call,
you know that, oh, it should call tool A, or, oh, it should respond and it
should say X, Y, Z in the response. So it's generally pretty quick to kind
of like set up and pretty quick to judge.

[[8:30]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=510s)
For traces. This validates the whole kind of
like agent run from start to finish. So you can see here that it's
actually executing the tools. This is better because it tests
more of the whole trajectory. You can also test kind of
like other state changes. So if the tools are modifying files or
modifying code or modifying something, you don't just have to look at. This kind of like message list. To test it, you can actually test
the files themselves, and then multi turn is the most realistic.

[[8:58]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=538s)
This is how people interact with your app. The difficulty is, this is actually
kind of hard to set up because if you look at the sequence, in order to test
this, you have to, you have to message the agent in a test with this, see
what happens, and then message it. With this. The issue is that depending
on what happens here. This message might be completely
irrelevant, so it's a little bit more complicated to set up the inputs and
outputs, but it is the most realistic test of, of production behavior.

[[9:23]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=563s)
And so going through kind of like
some of the pros and the cons, and, and by the way, like none
of these is right, quote unquote, like all of these have their place. We see people using these
together all the time. But the pros and the cons,
the single step ones are the easiest to define metrics for. But if your agent logic, and especially
kind of like the internals of should you use a skill or should you use a subagent?

[[9:43]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=583s)
Like if that goes, if that changes,
then these tests may go out of date. The trace is the full end, end one. These are easiest to define inputs
for because the inputs are just what you would message the the agent wants. But sometimes it's hard to define metrics
for because you're judging the kind of like end result of an agent, which could
be pretty complex and hard to judge. Threads are the most realistic,
but they're hard to define both inputs and metrics for.

[[10:09]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=609s)
We've also mostly been talking about
so far offline evaluation, which is before you ship to production. But I wanna just highlight that evaluation
means a bunch of different things. So offline, maybe starting with
offline should be really clear. This is where you. Build up a data set of at least inputs and
usually some ground truth outputs as well. I mean, you run this before you
ship to prod, whether this is on every commit or just once before
you ship to prod, kind of up to you.

[[10:34]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=634s)
We see people doing different things
depending on how complex it is to run. You do this before you ship to prod,
and this is really good for catching regressions and then benchmarking. So regressions are usually when
people have a set of things that they want to always pass, they'll
run these as kind of like unit test. But then they also often have
kind of like a benchmark set that they're constantly trying to kind
of like hill climb and improve on.

[[10:56]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=656s)
And so these are both forms of like
offline evals, online evals happen when you run it in production. It runs right after the agent runs, so
the trace gets sent into Lang Smith. We support these in Lang Smith,
and you can define evaluators that run right after the agent runs. The reason to do this is that
you can flag issues that are actually occurring in production,
the types of valuations you do. Could be similar to offline
or they could be different.

[[11:19]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=679s)
So in offline you can have ground truth. In online, you obviously don't
have ground truth, and so you're evaluating just based on the input. So sometimes the evaluator
metrics are different in here. And then ad hoc is what we call
basically doing this on an ad hoc manner. So online will like always run over all
traces, but sometimes you just wanna, you want to like dive into things and
evaluate things because you have a hunch or you got some feedback from a user
that you need to look into X, Y, Z.

[[11:43]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=703s)
And so you wanna do that in
kind of like an ad hoc manner. And you didn't have set up online
evals for that particular thing. And so this can be done at any point. This is very similar to kind of
like exploratory data analysis. We have a feature in Lang Smith
called Insights Agent that does this. You can ask it things like, Hey,
when did the user get frustrated? And it will go, it will filter to
things for where the user got frustrated and then cluster them as well.

[[12:05]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=725s)
So it's more exploratory
than kind of like catching regressions or anything like that. And one of the big differences in
agents is that observability and evals are pretty tightly coupled. So in software they're. Kind of separate things, but in agents,
because the source of truth is in these traces, they're much more tightly coupled. They're coupled in a few ways,
traces, power, manual debugging. So one of the first things that the best
teams do is set up tracing for their agents locally so that when something goes
wrong, they can just manually debug it.

[[12:37]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=757s)
And so this is looking
at agent traces on an. Basis. You can look at every step
to see what goro went wrong. You can open it up in the playground
to debug, and this is a really powerful and effective debugging workflow. They also power the offline
evaluation, so production traces become your eval data set. Like you don't know where the
agent will fail until you run in production, and then when it. You do discover places where it's mis
performing, you need a really easy way to bring those traces into a data set.

[[13:04]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=784s)
And so an example workflow that
we see be really common is user reports, some incorrect behavior. The developer goes in or, or the
builder goes in, doesn't have to be a developer, but they go in, they find
the production trace, they get kind of like the state at that failure point. So what are the messages, what is
the files that it has access to? And then they create a
test case from that state. If you can't use actual production
data, you can anonymize it or scrub it or something like that.

[[13:26]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=806s)
So you can have a representative test
case, and then after that you, you go, you fix that test case with offline
evals and then you're good to go. Traces, obviously power
online evals, so online evals. Run over the traces that
you're capturing in production. These can, again, these can't check for
correctness because they don't know the ground truth, so they can't use these
ground truth labels to check for that. But you can check for trajectory
or efficiency or quality or other failures that don't
require knowing the ground truth.

[[13:57]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=837s)
And then again, traces
power, these ad hoc insights. So these ad hoc insights run over
traces and threads, and so you can use these to surface uses patterns,
or identify failure modes or compare successful versus failed executions. And so the foundation of Lang Smith is
really in this observability layer because we think having really good observability
and a really good mental model for observability these runs, traces, threads,
having those concepts front and center really powers, debugging, offline evals,
online evals, insights, and other things that we haven't even talked about.

[[14:33]](https://www.youtube.com/watch?v=FDVdLrloFOw&t=873s)
Thanks for watching this video, and
I'd encourage you to check out Lang Smith as a way to get started with
Agent observability and evaluation. And if you have any questions, leave them
in the comments or please reach out to us.