Day 8: Harness Engineering - The Observability Layer

0:00 The next and final component we're going to be looking at is the observability. The observability part of the harness. Now any instrumentation that is built around the harness built around the model that is exposing what the model is doing is the observability layer. Every instrumentation that you construct around the model that exposes the activities of the model is what can be found at the observability layer or is what is considered to be a component of the observability layer. Now this includes things like logs of each model call. Every time you call your model, we write logs tracking information about that model call. We also have traces of each tool execution. So whenever the model calls a tool, we want to have a trace of the entire execution stack for that tool call so that we can have that information anytime we need to debug it.

0:59 We also have latency and token metrics. Latency and token metrics, the time to first token, how long is taking for us to get our responses from the model and how much tokens are we consuming in a model call. both input tokens and output tokens and even get to compute the total tokens for a particular session. So we have this information also that can be gathered for us to have insight into how fast our request are and how much we are spending on each request. And we also have evals. We also have evals. These help us to score agent behavior. So we pick specific tasks that are critical to the models operation. We write evals around them and they score these operations so that we can be able to have accurate information about how well our agent is performing the quality of responses that we're getting from our model and how we can always improve our earnestness to make sure that we continue to get good responses and also improve for better responses. So yeah these are the types of information that we can gather on the observability layer through the instrument station around the model and this is also a very important component of our Alice. Now why do we have to have this in place?

2:09 Why do we have to have something like this observability layer in place? First is that agents are nondeterministic. Agents can be nondeterministic. They are nondeterministic. They run multiple steps and they can even run for several minutes or even hours. They can run for hours for long running tasks. So when something goes wrong and trust me something would always go wrong. You need to be able to answer the question what did the model do? In what order with what imputees and also why? Why did I why did the agent perform the action in the first place? What did the model do? In what order did it do it? How did he get to that action? And what took place within executing that action with what imputees? what went in and most likely what went out and why why was that step taken and you can't answer any of these questions without observability instrumentation. You can't you can't the model itself if you ask the model directly it will just give you something like a summary. It can't count its own tokens or measure its own latency and stuff like that. It can't give you that.

3:15 So you can't get the answers to this without instrumenting an observability layer. And you can't improve the anest if you can't measure whether your changes made things better or worse. We've seen anes that got a boost on the anest benchmark a boost of like 30% increase just by improving the just by they didn't the model was not touched. the model was not touched. The earnest was simply improved and that just shot them up in the benchmarks leaderboard.

3:46 So you want to have an observability layer to help you gather this type of information or this type of information rather to answer this type of questions so that you can know when your model or your agent is performing optimally and also areas where you can improve. Now what does a good design look like? What does a good design look like? The first thing you want to consider when building your observability layer is that every model call must be logged. Yeah, you have to log every single model call and you have to log it with the full context. You have to log it with the response, the latency and also the token count. You have to log every single model call. This is your ground truth for anything you want to measure in terms of how well your RS is performing.

4:30 And as you're logging every model call, also ensure that every tool call is logged. Yeah, two calls are also first class operations when it comes to an agent execution. So you have to make sure that you're logging every tool call and you have to be logging it with arguments with the results and also the timing information, how long it took, when it was called, when the execution ended. You have to have all these metrics recorded because you're going to thank yourself when you need to debug critical issues. In fact, the issues don't even have to be critical. Even simple issues need this information to be properly investigated. Another information that you want to grab is a trace is a trace. Now, when you call your LLM, input output, that's a single call. That's like one unit of execution.

5:18 Something you call a span in your telemetry terms. Now, if you have a chat session, within that chat session, you're going to have different back and forth with the AI model. So, you send a message, the AI model responds. You send another message, the AI model responds. You send another message and so on and so forth. Now, every single message you send is a single call to the LLM. You send a message, the LLM responds. That is a single call. So, your impute, the LM's output, that's a single LLM call.

5:48 That's a span. You have another one. and then you have another one. A trace is what is used to group all these together into a single session. This entire conversation is one single trace. For every message that is passed in this session, you have a trace ID that is going to connect all of them together. And what this does is that it helps you to build a session in your observability layer. Because you're just logging model calls, you're logging individual tool calls. A trace is what ties everything together for a single session. And with that you can replay a single session end to end to see if you can reproduce issues. You can also replay the session end to end when you fix the issue and you want to test if you now have a passing operation. So you also want to make sure that you're saving traces which are basically identifiers that tie together model calls into a single session into a single conversation session. So that's also something that we want to have in a proper observability layer design for our anests. And finally we'll look at evals.

6:50 Yeah, you should have evals. You see they're also here. You need to have evals because they are what you use to measure whether your agent is giving you the required results that you want. You design a fixed set of tasks that are going to be your standard for measuring the output. You have outputs that you expect and then you run your evals and measure the outputs of the agent against your standard outputs and based on the score you can determine whether your agent is performing well or you will need to improve your RS or sometimes improve the amount of information that you're collecting because you don't want to have false positives. If you don't design your observability layer very very well. If you're collecting too little information that is not going to be able to pinpoint your exact problem or you're collecting too much information that you're now getting so much noise, you won't be getting accurate answers. So you need to have clean evals that have been designed to make sure that you're getting proper answers and then you run tasks against those evals. They are scored against your ground truth and based on those scores, let's say you have a 0.8 that's like 80%. That looks pretty good. You can always improve that to 85 or 0.9.

8:09 But if you are previously getting 8 and you run your browse probably after a model change or you change something in the RNS and you notice that you start getting something like 0.65 or you start getting something like 0.71, then that means your ANS has degraded. So evals are what expose this information to you so that you can know when you're doing the right things with your honest