Day 9: Harness Engineering - How All Components Connect

0:00 So, one important thing I've learned in systems thinking is that every system is composed of parts, connections, and patterns. Parts, connections, and patterns. If you want to learn a system thoroughly, you need to understand the different parts that make up the system. By the way, let me just move something on the screen so that you know that your screen is not frozen. Yeah. I know that helps a lot of people. Okay. >> >> So, if you want to learn a system thoroughly, you need to learn its different parts, how they connect, and the different patterns, the different proven patterns of building that system. Or let's say, the different proven patterns of composing that system, that's between the parts and the connections. How people have composed that system over the years and the patterns they've discovered to be battle-tested. That's one of the best ways to learn a system.

0:57 So, in this video we're going to be looking at the components we've learned, how they connect, even though we've kind of mentioned it a couple of times. We're going to look at how they connect for canonical harnesses, for standard harnesses. And throughout the course, we're going to be looking at the patterns that have been battle-tested for composing this com- components and their connectors. So, in this video we're going to be looking at the simple standard agent harness architecture in order to understand how the components connect, how they talk to each other.

1:29 Now, this is our harness. This entire box is our harness. This entire frame is our harness. And the most important thing here is the loop. The most important thing in this harness, the coordinator of this harness, is the loop. Is the loop, as I've discussed in its own introduction video. And what the loop does is it basically orchestrates the activities within your harness. If your harness is going be single agent or multi agent, that depends on the loop. If your honest is going to gather the right context, it depends on the loop. If your honest is going to save the right things into memory, it depends on the decisions that the loop is taking. Because an agent is basically a looped call wrapped around an LLM.

2:16 So, the loop is the most important. So, when we go into the loop, what happens? We get our prompt fed in into the model. Get our prompt fed in into the model. This model is going to process our request. Check if we need a tool call or not. Most likely we need a tool call. If we don't need a tool call, we get a response. Yeah. But, most likely we're going to need a tool call. So, a tool is called, let's say web search, file search, whatever it is.

2:43 The tool is dispatched by the loop. The tool then produces results. Produces results. And this tool execution and the results it produced happen in the environment. The tool execution is in the environment, the results are in the environment. So, we observe these results. Have we gotten to the final answer? Most likely no, because the task we want to perform will definitely take several steps. So, these several steps are going to be running in this loop. So, we check in this environment for our result. We observe our result.

3:14 Then, from our result, we determine what we want to save into memory. What do we want to save into memory? Of course, we want to update the conversation history. That is very, very important. So, we want to save that into the short-term memory. We also want to save tool call results and some other stuff that is important to the second cycle or to the subsequent cycles of this loop. So, we save that into the short-term memory.

3:38 Then, for long-term memory, we extract important information from what just happened, if any. If any. So, this is not a compulsory step. But, if you discover any important thing, you extract it. And this might not even be the trigger point for your long-term memory. It might be when the tool is called or when an event happens anywhere within this loop. Anyway, for memory, we have short-term being updated at every turn and long-term being updated by your triggers, whatever necessary. Once we make sure that memory is adequately updated, then we need to fire the next turn. We need to fire the next request to our model.

4:16 And this is where context engineering takes place. This is where context engineering takes place because we now need to arrange the context. We We need to arrange the next context that will be fed into the model. Our conversation history, tool calls that are relevant to this next turn, rag retrieve documents based on what this next turn requires, static data, context rules, so so many things, system instructions, we assemble everything for the next turn and we'll fire our next call. And we'll keep doing this over and over and over and over for every single turn, for every step in the task, until the task is complete.

4:56 Yeah, until the task is complete. And we will resume this loop. We have our long-term memory to feed stuff into our context to help us remember what we're working on. And we continue to run this, continue to run this, continue to run this until we finish the current session that we are. So, this loop continues to run as long as you're interacting with your honest. Everything your honest is doing, whether you are the one interacting with it, whether it's the honest recalling itself, whether you're in a rough loop or whatever, this whole process runs over and over and over until an exit condition is met.

5:33 And outside that, you have all the observability instrumentation that is collecting information from this entire process. Just just mining information from this entire process and saving it into a dashboard, probably using something like LangSmith or LangFuse. Those dashboards are very, very detailed when it comes to all these model operations. They're going to show you what the rag operations are. They're going to show you the full context that your model is being called with, every single tool call, drill down to tool calls, so that you can pinpoint the inputs, the output, the execution time, and everything. Show you how much tokens you're consuming for your model, even show you thinking tokens, the amount of thinking tokens that you are consuming.

6:12 So, yeah. This is the simplified agent or honest architecture. This is your simplified honest architecture. And as we go further in this course, building out our honest, you'll be seeing a lot of decisions that we'll get to make to build what I'll call the honest of our dreams. And even in production honest, you still encounter a lot of intricate decisions. A lot of engineering decisions go into composing all these parts, and even some other sub parts, orchestrating this loop to ensure that the honest or the agent is in the exact shape that we want it.