Day 2: Harness Engineering - The 6 Components

0:00 So, in this video, we're going to be defining the honest. Going to be defining the honest. By the way, I just noticed this definition has an error here. So, let's fix that. Okay, that's good. So, like I said, we're going to be defining the honest. Now, the honest is not just one thing. It's not just one thing. It's actually a stack of several concerns. Six of them are core. So, those are the six we're going to be looking at in this video. and they they translate into every harness you see out there claw codeex any type of harness you see out there open claw they all contain this six components so like I said the harness is not just one thing it's a stack of six concerns and each of which can be designed well or badly and each of these concerns corresponds to a specific class of gap that we have looked at and if you see your agents failing in production or being broken Yeah, this this component designs are going to be one of the reasons why you are having that failure. Subsequent videos after this is going to go into each component in detail, but here we're just going to be looking at each of them at the surface level. So, in the previous lesson, we looked at the gaps.

1:14 We even saw a code demo to show us those gaps. In this lesson, we're going to be naming the parts that fill those gaps. the six components like I said the six core components and once you have these six components properly understood you will be able to participate in any conversation about agents trust me you're going to be able to participate seamlessly in any conversation about agents whether it's about designing them debugging them comparing different tools reading agent papers you would be able to easily flow with the vocabulary yeah you would now stand literate about agents once you can have these six components nailed down. But before we go into the components, let us first look at this formal definition of what an agent harness is. So let's look at it.

2:03 It says the harness is the engineered system surrounding a language model that turns its single output channel which is text into action. persistence and measurable behavior. Let's take that again. The harness is the engineered system surrounding a language model that turns its single output channel which is text into action persistence and measurable behavior.

2:44 Like I said, the ANS is basically the system that is translating the textual output of your models into agentic operations. That's what the AS is doing. And we have agentic actions. We need to persist state and also we need to be able to measure the agent's behavior. We need to be able to understand what is doing. But let us take that one by one. Let's start with action. So action is the first main verb in this definition.

3:13 And what the anest basically does to ensure that models can take actions is that it gives the model a surface area that is beyond text. Yeah. It gives the model a playground. It gives the model a workspace. So it gives it a file system. It gives it a code sandbox and even things like web access. It just gives it different environments for it to play in. file system, code, execution environment, access to the internet, databases, many more. So, it gives it a surface area that is beyond text. And what we've described is basically an environment. Yeah, it gives it an environment to operate in. But not just an environment, it also helps it to execute the things that the model describes. Yeah, it executes the things that the models describe. Can you guess what that is? Can you guess how it does that? Yep. Tools. Tools. Yeah. It gives it tools to operate and do what the model describes. The model describes an action. The anest provides an environment that it can execute the action and it will be able to execute the action through a tool.

4:28 Yeah. Tools execute inside an environment by executing an action described by the model. That is the action part of this definition. Next, let us look at persistence. So we already understand that our model cannot retain state and this is where persistence come in. Persistence come in and this persistence can be within a session or across sessions. Two flavors of persistence for the agent to be able to operate. So it has persistence within a conversational session. What we're going to later on understand as short-term memory. So that is the memory within your conversation your current conversational session with the model and also what is known as long-term memory and long-term memory can be shared between different sessions. It is more like what the agent is learning about the user and some ground knowledge that the agent is always working with.

5:24 So we get to achieve persistence for short-term needs and for long-term needs. And finally we have measurable behavior. We need the honest to be able to expose what the agent is doing so that as engineers we can evaluate. We can debug and we can also get enough information to improve the agent to improve the honestness. To improve the agent's honest must be able to extract logs and traces and any form of visibility into the agent's actions that can help us improve our honest. So from this definition we've seen action which is done by tools in an environment, persistence which is implemented in the short term and the long term and measurable behavior. These are three core behaviors that will build into a harness to get a genic behavior. Now we're still talking about attributes.

6:22 Let us now look at the six components that make this happen. Welcome to the components board. These are the six core components that you're going to find in almost every standard harness. Every standard harness should have this six components. I've not seen a harness out there that doesn't have any of the six. And the first one is the loop. The first one is the loop. Now, why are we calling it the loop? In our demo, you'll realize that we can only call the model once and each call is stateless. So if we are to have a conversation with the model and we need to call it in multiple terms that means we have to wrap a loop around it and this loop is to take the conversation history and feed it into the model for every interaction we have with it to take the conversation history each time and feed it back into the model. That's why we call this the loop.

7:19 But basically it is the control structure that calls the model in multiple turns. It's also in charge of dispatching tools and decides when the conversation stops. It's basically the orchestrator around the model. But like I said, we have a video dedicated to each of those components. So no need to go deep into them now. Just understand that the loop is what guides our conversation with the model and makes it multi-tonon. Component number two is the tools. The tools. Yeah, the items that actually perform the operations. These are basically functions the model can invoke in order to have an effect on the outside world. can be to read a file, to write a file, to delete a file, to create a directory, to query a database, to fetch a web page and many more. The tools, this is where you start getting real agentic behavior by wrapping a harness around your model. Next, we have the context. The context, yeah, this is everything you feed into the model for every call. Remember I said we call our model in a loop in order to have a multi-turn conversation with it. Now every time we call it we have to feed it a prompt and that prompt has to contain everything the model needs to be able to give the appropriate answer.

8:46 Everything the model needs to give us the appropriate answer. That everything is the context. The context that is what is fed into our model on each call. And it contains things like the system prompt, the conversation history, retrieved knowledge from static sources and even tool results. The results of calling tools can also be fed into the context. So that's the third component in a standard RNS. Then we have as number four the environment. Yeah, the environment this is the runtime the tools operate inside. Yeah, the if you say read a file, the tool needs to be able to point to a specific location to pick the file. If you say write a file, the tool needs to be able to point to a specific location to write the file. So the runtime the tools operate inside is the environment. And this can be a file system, this can be a shell, it can be the network or even be a sandbox code environment with boundaries. So the environment is the fourth component in our standard RNS. The fifth component is the memory. Yeah, the memory. We need to persist state. So as we loop around, as we continue to loop around each iterations, we need to persist state for a session. In the short term, we persist state for that session. In the long term, we extract certain useful information from each interaction and plug it into a more persistent storage so that we can feed it in whenever necessary in any conversational turn. So we have the memory that I've described earlier has been divided into short-term and long-term. And finally as our sixth component, we have the observability.

10:38 the observability. The ANS also needs to contain instrumentation that exposes what the agent is doing. Yeah, we need to be able to see everything from logs, from traces. We need to be able to expose the details of every single action that our agent is taking. We also need a vows. We need a vowels to be able to benchmark our agent against known tasks and see its output to see if we're getting quality results. We also need information to be able to debug our agent. So here are your six components for a standard harness. The loop which is a control structure that guides your conversation with the model. The tools which are your action takers. The context which is a construction of information to be fed into the model to invoke the right response. The environment which is where tools are going to operate. The memory which is how you persist state and observability which is how you expose the agent operations in order to debug and improve it for better impact.