Day 3: Harness Engineering - The Agent Loop

0:00 In this video, we're going to be looking at the very first harness component, and that is the loop. The loop. Yeah. This is the orchestrator. It is the control flow that wraps the model. Remember we said the model needs to be called multiton. Most usage of the model needs to be called multiton. Now, it's not that you can't just call the model once and get a response. You can always just make a single call to the model and get a response.

0:27 But, when it comes to agentic interactions, most of the calls to the model will be multiton, meaning cycles of this operation. And this is where you need the loop, which is the component we're discussing, to wrap around the model. To be the director, to be the conductor of the interaction with the model. It will take user input, pass it to the model, get the response, and feed back the conversation history into the model. So, at the end of the day, you get to have something like this. This is the agent's loop in its basic form.

1:03 You call the model with a prompt. You get a response. That response gets packaged into a conversation history and sent back to the model. The conversation history is basically a combination of what has been said so far plus the new prompt. This is the most basic form of your loop. You prompt your model, get a response, that response goes into the conversation history, the conversation history is combined with another prompt, and you feed it back into the model.

1:32 Get a response, gets packaged into the conversation history, that conversation history is tacked onto the new prompt, and it keeps going on and on and on like that. Now, this cannot run forever, so definitely you will need a termination point. You will need to know when this is to stop. The loop is also what does that. So, at every response point, we always do a termination check. And this check basically asks, "Are we good enough? Do we have everything we need for a response for the task?" If not, we go again and we prompt. We go again and we prompt. But if we do, we then just send our response to the user. We send the final response to the user. So, this is the basic form of the agent loop. Now, there can also be a tool call, which is very, very common in conversations with the model that require agentic operations. So, we most likely we have a tool call. How does a tool call come into this? If we get a response, and that response has a tool call, then we'll need to execute the tool.

2:40 We need to execute the tool. We get the result of the tool, and those results are fed into the conversation history, and used to re-prompt the model. If we don't get a tool call, we'll simply go on with our regular business, which is just stocking on the new response to the conversation history, and stocking that to our new prompt. So, if there's a tool, we execute the tool in our environment, get the tool results, and send it into the conversation history as a tool call result.

3:11 If we don't get a tool call, business goes on as usual. Now, this is how the loop operates in its most basic form. This can really get very, very fancy. This is where your react pattern comes in. This is where your plan execute pattern comes in. This is where your multi-agent orchestration, sub-agents, and all those kind of stuff come in. It can get that complicated. But in its simple form, you're basically looping calls around your model.

3:38 You are constructing a conversation history, which is a chronological order of all the user messages, response messages, and if there's a tool, tool messages. And you need a termination condition to be able to know when your result is sufficient so that you can pass a response to the user. This loop cannot go on forever. And also, when you encounter a tool, you execute the tool, and you pass the result of the tool into your conversation history.

4:05 If you don't encounter a tool call, you just simply go on with business as usual, maintaining your conversation history, and making sure that every new prompt includes the conversation history. So, this is basically how the responsibility of the loop looks like in your honest. What does a good loop design require? And we have kind of discussed some of them as we were looking at the loop. But first, you need a termination condition. A termination condition. The model needs to say something for you to know that, yeah, this is where we stop iterating.

4:39 And this can be as simple as the agent just saying done. You can tell the model to say done, let's say in capital letters. So, it can scream done, and immediately, your loop sees that particular sequence of characters, it's just going to terminate. It's going to terminate and return the final answer to the user. There's also a stop configuration that you can configure. There's a There's something called stop sequence or finish sequence. When you're When you're using the OpenAI SDK, there's a setting called stop stop sequence, I think, or finish reason or so I think it's finish reason. I think it's finish reason. Some of them probably call it stop sequence that you can configure to make the model stop.

5:15 Once it encounters that, it just stops. You can give it a character. You can say when it sees a pipe, when it sees a dot, and you can make sure that in your prompt, you're specifying that that, okay, when you reach these certain conditions, when you feel it is good, just output the word done or output the character pipe. Also, you can have something like a step budget. You can have a step budget. You can say after every five iterations, just give me the final answer. Yeah.

5:42 Once you this loop count is up to five, just give me the final answer. So, to do that, you know that you have to maintain the loop count so that as you loop for every conversation, you are incrementing the count. 1 2 3 4, and once it's five, you just say, "Yeah, once anything you get at the point of five, just return it. Just return it to me." So, you can have a step budget. Some of these models also have a configuration for this. Some call it max iterations, and some frameworks also, like LangChain, have configurations for this. So, you can have a step budget. Also, you can also terminate when a error threshold is hit. Probably, you keep getting errors, probably from two results, keep getting errors, and you can tell it, "Yeah, if you get three errors, just terminate the process and return what the final answer is. Just just report the error."

6:34 So, that's one attribute of a good design. Another attribute of a good design is to have sensible error handling. Have sensible error handling. And what that means is that don't crash the loop when tools fail. Yeah, don't crash the loop because a tool failed. If a tool fails, undo the error and pass the error message into the conversation history. Undo the error appropriately during the execution, and pass the error message into the conversation history so that the model is the one that gets to report the error.

7:04 The model reports the error. So, basically, the model is reporting the error as an observation. So, that's another thing. Third, every good loop must have a shape. It must have a shape. It must have an architecture. It must have a pattern. What you've described here can be argued to be the react pattern. That's the reason and action pattern, the react pattern. So, every loop must have a shape. There's also the plan then act pattern. So, you can implement that. You can implement a multi-agent pattern. You can implement a multi-agent pattern. You can go that complex if your agent requires it. You can also do deep recursion. You can do deep recursion.

7:44 For example, you must have heard of something called the Ralph loop. Yeah, the Ralph loop. You can do deep recursion or do like a orchestrator worker pattern. Basically, your loop needs to have a shape. The loop around your model needs to have a specific shape. And there are standard architectures that you can use like the react pattern, the plan and act pattern, a multi-agent pattern, or a deep recursion pattern like the Ralph loop. The shape of your agent is what determines the kind of tasks that your agent can do. Just as we know that some agents cannot perform deep work because they don't have something like the Ralph loop implemented. Some agents cannot also be multi-agent because their loop does not have that type of architecture.

8:27 So, the shape of your loop is what determines the capability of your agent, what tasks your agent can perform. Now, to wrap things up in this lesson, let's do a simple case study and look at a harness in the real world, Claude code. And how we can detect the loop behavior that Claude code exhibits. How it fixes bugs. How Claude code fixes bugs. Now, when you ask Claude code to fix a bug, it doesn't just make one model call. It doesn't just make one call to the model and the model will just magically fix the entire bug and return a response.

8:59 No. It makes many calls. Makes many calls. It makes calls to read the file, edit the file, it would also run tests, read the error messages, and go back to editing the file again with what it has learned about the errors. The loop is what makes this multi-step process possible. This multi-step behavior that Claude code exhibits when it is fixing a bug, the loop is what makes it possible. The loop is also what makes it decide when to stop. After running the tests, when it reads the errors and sees that all is clear, it is confident that the task is done, and that is when it now returns a response to you that it is done. This is not a one-shot operation.

9:44 This is evidence that it loop is in operation running these processes, iterating over them until it is clear of errors. It can then report back with confidence that the task is done. So, that is the honest loop. A whole chapter has been dedicated to this component for us to actually build our own loop around an LLM. So, consider this your intro. In that chapter, we're going to be going into more details, and we're actually going to be building our loop from scratch.