Day 1: Harness Engineering - The Raw Model Program

0:00 So in this video and basically in this entire chapter, we're going to be understanding why we need a harness in the first place. Why we need a harness for our agent and we'll begin that by understanding what I would like to describe as the raw model problem. The raw model problem. Now basically the LLMs that we're used to today that's our model are basically text transformers. they are text transformers because what they do is that they take a sequence of tokens.

0:32 This is basically what LLM's read your prompts to be a sequence of tokens and based on this sequence they try to predict the next set of tokens. So basically their output is a probability distribution over the next token and they keep doing this on and on and on and on until a full response is generated and that's it. That is what these models do. They are basically text transformers. They're basically text transformers. they don't have they don't create any side effects. They don't have any memory. They don't have access to anything outside of the prompt. They basically just transform text. A sequence of tokens going as your impute. And the probability distribution creates the next set of tokens and they keep generating that until a full response is generated which they then send to the user. So the intelligence is real. The intelligence is real. But the surface area is limited to tokens in and tokens out. That's basically what these LLMs have been doing. Now why does this matter? Why does it matter that we understand what this is? Is because most developers and I assume that a lot of people watching this course are developers or at least are working in the software industry. We often assume that these models are some sort of function in a library. Typical functions that we call in a library or in an SDK.

1:57 We assume that there are some sort of functions and what we do is that we call them, they do something and return a result. That's how most developers will see an LLM at least when they interacting with an LLM for the first time. probably you're calling the generate content function on an open AI SDK and you feel that oh yeah I just send in an input my arguments and I just get a response because the model does something that mental model is wrong for models models don't actually do anything they don't take any form of action what they basically do is describe the doing of things yeah they describe doing things they don't do the thing they just describe how the thing can be done or how the thing is done. That's basically what models do. Their surface area is limited to text. They're basically taking your input and describing what the output should look like. So a model doesn't actually take action. They only describe an action. And if you're already familiar with a concept like tool calling, you realize that what the model actually returns is a tool call.

3:04 The model doesn't call your function. It doesn't call the function that you describe to it. It just describes back how the function is to be called. That is what the whole tool calling integration is based on. The fact that the model understands the function and it can describe how the function can be called and it can even tell you use it with this argument and that argument. That is the whole thing about tool calling. So it's still within the description of things and that is the only thing that models can do. That is the only thing that LLMs can do. They can only describe how things can be done. And this leads to several gaps when we actually want these models to operate as agents because agents are meant to do things. Agents need to have a real impact in our environment in the real world. They need to help us get things done. A model that can simply just describe things is not going to cut it. So let us look at the gaps because it is from looking at these gaps that we can then describe what a harness is. A harness typically in plain English is basically a support system. A harness is a support system. It's what for example I used to intern in the telecommunication industry and one of the things that we did was that we climb masts. We do climb this telecommunication mast but we don't just climb it just like that. We wear this safety equipment and that equipment is called a harness. It supports us. It guides us. It helps us to stay safe. So basically a harness is a support system for a thing. And we're going to be looking at the gaps that are between a model that can only describe things and an agent that needs to do things. In between those gaps is where the harness then comes in as the support system that helps the model do things not just describe but do things. So let us look at the gaps because the gaps are what is really going to determine what you need in your harness. So the first gap of a model that can only describe things but not do things is the fact that there is no execution. It can't execute an action. That is the obvious gap. That's the first obvious gap. It can't execute an action. The model cannot read a file.

5:22 It cannot run a script. It cannot query a database or even make an HTTP connection or an HTTP request when you need to communicate to an external service. The model can't do anything in our environment. It can't do anything in the real world, but we need it to do that. And something has to help the model do that. That something is the harness. So that's the first gap. The second model gap is that there's no durable state. There's no durable state.

5:50 Every API call on our model is stateless. Anytime you call OpenAI through its API or you call Antropics cloud models through their APIs, everything is stateless. Everything is stateless. There's no state that is being maintained on the OpenAI or Antropic servers. Everything that the model needs to know has to be referred to the model each time it needs to return a response. What you understand as conversation memory is basically an illusion that is created by simply sending all the previous messages back into every subsequent call. It's an illusion. You you are just basically taking everything that you have received as response and everything that you've sent as a message in the chronological order as a conversation history. You are replaying it every single time into your model call. That is what the conversation history is about. And that is why your models that you've been chatting with most especially through your chatbots Gemini Claude and chat GPT that's why it feels like they remember stuff. They actually don't remember anything. the messages the entire messages is just being replayed. And once the context fills up or the session ends, the model remembers nothing. The model remembers nothing. When your session in a particular context window is filled up, the model cannot remember anything beyond that. Yeah, it has maxed out its working memory. But for an agent that is going to act on our behalf, we need it to remember things. We need it to remember us. We need it to remember the activities we've done. We need it to remember that alarm we need to set. We need it to remember what we discussed 3 weeks ago. That persistence is not something we can get from the model.

7:35 Like I said, the model remains a text transformer. We can't get that persistence from model. We need to get it from the harness. So this is the second gap that we need our harness, our support system to fill up. The next gap that we'll experience by using just a plain LLM is that there's no fresh knowledge. There's no fresh knowledge beyond the model's cutoff date. That's the model's training cut off date. It doesn't have any knowledge beyond that.

8:05 So if the model, let's say, okay, this week, was it last week? This week, Arsenal won the Premier League. Yeah, Assa won the Premier Premier League. This is the today is 22nd of May. So as won the Premier League this month and this week to be specific. Let's say our model's training cut off date was actually March of 2026 and we ask it who won the Premier League in 2006. It won't remember it can't remember that because its training cutoff date was March 2026 and by that time the winner of the Premier League had not been decided yet or we ask it who won the NBA title. We we don't even till today we are still in the NBA playoffs the conference playoffs. So yeah the conference finals so we don't know who won that yet but if a team eventually wins it down the road I think around July or something. This model is not going to know. This model is not going to know because his training cut off date is March 2026.

9:03 Anything he needs to know has to be injected through the prompt. It has to be injected through the prompt. You have to keep feeding it into the prompt. That is actually called in context learning. you have to fit it into the prompt. There's no way for it to get fresh knowledge aside that. There's no native way for it to just pull fresh knowledge aside being fed into the prompt. So our harness also has to take up the job of passing in any fresh knowledge that our agent needs either by querying a website, querying our database, any sort of query needs to do, any sort of knowledge base it needs to be exposed to so that we can keep feeding it into our model, our harness has to do that job.

9:42 the model itself is not going to do that job. Another gap is that there's no way for the model to actually verify what it does. Of course, it can output a response, but it can fact check itself. Yeah, I know some of you might say, "Yeah, why don't you feed it into another LLM and it will verify whether that LLM is right?" And yeah, that can work to an extent. But when it comes to truth, when it comes to truth, not things like probably just verifying whether a document is well written, whether it's grammatically correct or stuff, when you're trying to verify truth, when you're trying to ground truth, stacking layers of unpredictability, that's just having models verify each other is not going to it's not going to cut it. So there's no way for a model to actually verify that what it is doing is correct. It can generate code but it cannot run the code to see if it works. Yeah, it can run it.

10:37 It can produce a summary but it cannot say yeah let me compare that summary with this truth to check fidelity. It can only claim correctness and it does that with a very high level of confidence which is one of the limitations of models today. But claiming and verifying are different operations and only one of them is available to the model which is claiming. the model can claim. In fact, the model will do a lot to claim. you tell it is wrong, it will also accept.

11:03 But probably you are wrong too. So it doesn't even know whether it's actually correct. yeah, models are getting smarter these days. I think from us 4.7, I'm beginning to get a little push back sometimes when I tell it, yeah, why is this this? I thought this was this. why are you doing this like this? Sometimes it tries to kind of argue why it does that. So they're getting smarter, but still these are probabilistic machines with no soul. So they can't actually I can't take their word 100%. So the model can't verify that it is correct. The RNS has to be responsible for creating a verification system around the model's output. Yeah. Has to be able to create a verification system around your model's output so that you can know that yeah this is actually correct. This is actually good. For example, these days when I use claude, I use cloud code when it generates code, it tries to run it within a sandbox so that I can verify that yeah, this code works. Then it then returns it responds and say, "Oh, yeah, this is correct. This is valid and all that." And that's because a harness has been built around it. But the model, the pure model itself cannot do that.

12:10 Yeah, it can do that. Now, the final gap when we're trying to translate a model into a fully autonomous agent is that the model has no environment. The model is without an environment. It doesn't have a body. Yeah. It's basically a brain. It doesn't have a body. It's not something that's sitting inside a Linux box with Python and G installed. No, it's not sitting anywhere. It is basically just a mathematical function being evaluated on a GPU. If you want to use something like pandas say you want to install pandas or you want to use a browser something actually has to install pandas something actually has to give you a browser and those type of environments is something that we need the harness to provide it is the harness that's going to be providing the environment not the model. So this is also another harness responsibility.

13:06 Execution is a harness responsibility that's taking actions. State is a harness responsibility. Getting fresh knowledge into the context of the model is a harness responsibility. Verifying that what the model outputs is correct is a harness responsibility. and providing an environment in which the actions of or the actions described by the model can be executed is also a harness responsibility. So I believe from these gaps you'll be able to understand why we need a harness. But in next video we're going to run a piece of code that is actually going to show these gaps and then we can start filling these gaps by introducing features that close these gaps into an agentic harness.

14:01 So in this video, we're going to be running a simple demo that is going to help us to visualize and confirm a bunch of the gaps that we discussed in the previous video. In the previous video, we saw gaps like no execution, no durable state, no environment and the rest. And we're going to be running a simple script. this is a simple OpenAI script that is using the OpenAI SDK. We are creating a client and that client is calling the OpenAI API and what we're doing here is that we are calling the GPT4 mini model and we're calling it as a user and we're telling it that we want to read the file article.txt in this directory. We're just saying in this directory. so we're assuming that it should just assume the directory that the script is in. It says summarize it in three bullets and save the summary to summary MD.

14:59 So we are telling it here to perform an action. We're telling it to read a file, summarize it, and write a file. And if you look at these three actions, that's reading a file, summarizing it, and writing a file. This three actions based on the gaps that we discussed from the previous video, only one of them can be performed by the model. Yeah. Reading a file, no, the model cannot do that. Summarizing the file, yes, the model can do that. writing a file. Nah, that is another file system responsibility. But don't take my word for it. Let us just look at this simple plain call to the openi API. This what a simple call looks like to a model. Nothing added, no hers filling any gap, just a simple call. Let us run this code. I've already added my open API key. which is something that you need to run this code. You need to add it with export open open AI API key. Then you pass your key here. So if you're running this code, make sure you do that. and I also have the article here, article.txt. It's just a random article about the printing press, but it doesn't the content doesn't matter in the scope of this lesson. So we're going to run this code by saying python demo. py and we're going to see what the model does and how it surfaces a bunch of the gaps that we have discussed. Okay. So the response is back. Let us read this and see what the model tells us. It says I can't directly read files or interact with your file system.

16:45 You see it immediately recognizes it limitation. It says I can't read files or interact with your file system. So it knows it can't access files. It then goes further to say but I can help you summarize the content if you provide the text from the article. You see here summarization it is telling you what it can do. This is what I can do. It first tells us what it cannot do which is the file system operations.

17:12 then tells us I can summarize because it's a text transformer it can do text then says please copy and paste the contents of article txt and I'll assist you in creating a summary is describing what the action would should look like give me the file and I'm going to create the summary you can then save the file yourself yeah you can go save the file yourself so you see the model exposing its gap apps and also indicating its capabilities and the gaps that we've discussed are surfacing in just this response. First, nothing happened. Nothing happened. If you look at the file system, nothing happened. no summary was created.

17:58 There's no summary on MD, which is a lesson because this is not telling you that the model failed. This is what the model exactly does. It it just tells you what it couldn't do gracefully. But in the outside world, nothing changed. Nothing changed. So the model cannot act. We've seen that. And that's a surface area problem in its purest form. The model's only channel of output is text. It can describe text. It can recommend action. It can decline action.

18:25 It can even claim action. Sometimes the model will just outright lie to you that it created the file, but when you look at your environment, you don't see anything. So it can even claim that it did something, but there's no action. And we all know that the gap between describing something like writing a file and actually writing the file is there are two different things. There's a big gap between that and that is something that has to be closed. That's a gap that has to be closed by the harness. So in this demo we've been able to see the no execution gap surface which is the fact that the model couldn't read or write the file and the model was very very transparent about that. It correctly said so there's no environment so there's no file system available to it in the first place. there's no file system. It doesn't even know where to find the article.xt because there's no environment that it can access. Also, there's no durable state. That might not be like outrightly visible right now that there's no durable state. But even if we gave it the article and it was able to generate a summary and the following turn we ask it about the summary, it is not going to remember that summary. It's not going to remember both the article and it's not going to remember the summary. So there's no durable state. For example, if I go here and I change this and I say, let's just say, but can you at least tell me the name of the file I asked you to summarize because earlier we told you that there's an article.txt. So even if it doesn't know what is inside, even if it doesn't know what is inside the article.txt, we have told it that there is an article.txt.

20:00 Now we are asking it for the name of the file. We asked it to summarize. Let's see what it tells us. See, I don't have the ability to access files or remember previous interaction. It it is literally telling us the gaps. And these are the gaps that your needs to fill. If you want to convert your model into an agent, filling these gaps is what the hest is all about. So it tells us, I can't remember previous conversations or previous interactions. If you provide the title or content of the file, I'll be glad to summarize or discuss it. Now, you didn't get the summarize from the previous prompt. We literally said summarize here. So, it's you kind of remember that we're talking about summarizing a particular article. So the long form of it is that your RNS is every piece of code, configuration and execution logic that surrounds the model and makes it capable of acting, remembering, learning, verifying its output and persisting state. That's basically what the Alice is all about.

21:02 That's what we're trying to drive out with the previous lesson and this lesson. everything you are wrapping around your model to make sure that it is capable of taking actions. It can remember past interactions. It can learn from previous conversations. It can verify its output and it can also persist files and information to state or permanently. That is what the harness is all about. Everything in an agent's development that is not the model, everything that is wrapped around the model to make it function as an agent, everything that closes the gap between description and action is the earnest.

21:45 And in coming videos, we're going to be looking at each of those anests items one after the