Day 5: Harness Engineering - The Context Layer

0:00 In this video, we're going to be discussing the context. And I want you to listen very carefully here because this is a very interesting topic. And if you've ever been confused about the concept of context engineering, this is where you actually understand what is going on there. This video is not going to give you all the details about the engineering part, but definitely you understand what it's all about and what is involved. When we get to the chapter that focuses on the context, we then get to do some context engineering. And this particular topic was quite interesting for me because it makes me understand how powerful the context is and how it translates to a lot of powerful agent analyses and AI products that we're seeing today. So as we already understand the model our LLM takes a prompt. It takes a prompt on every API call the model is being fed a prompt.

0:51 But what goes into this prompt? What is contained in this prompt? because everything that is in this prompt is what is going to determine the type of response that we're going to get from the AI model is what is going to determine this response. So let's say we first want to make sure that we are passing instructions to the model that defines who it is, what it should do, that's who the agent is, what the agent is all about. Let's say cloud code.

1:19 Claude code is basically a software development tool. what he's supposed to do, what is it role? We might want to add some backstory and everything. We want to put those type of stuff inside what is known as a system prompt. Yeah. because we want this to always have effect on every single call we make to the model. So once you have a system prompt in there, once have a system instruction, we will put a system instruction there where we tell the agent you are a code editor or you are a code generation agent. You are a software development agent. You can write Python at the level of I don't know the best Python programmers on earth, but let's say you can write at those guys level. I will give it instructions like style guides and stuff like that and we'll put everything like that in the system prompt. Yeah, maybe not put style guides probably but just key instructions on how to write code. I want this agent to write code, who it is and everything. Let's say we put it in the system prompt. So, we want to put that because we want it to always have that in its context window and take that into account every time it is generating a response. Let us say we have something like our style guides. We have our architecture documents. We have some programming language documentation that we feel that it is going to need for it to perform its operations. Probably it wants to write a particular piece of code and it needs to use the observer design pattern. So we want to make sure it has the knowledge for let's say Pythons observer design patterns and everything. So what do we do? We retrieve those docs. We retrieve those documents from let's say the Python documentation or probably we have some PDFs we retrieve them and we put it inside this prompt. So we put in retrieve documents through a process called rag retrieval augmented generation. So we augment the generation of our response by retrieving some important documents that this is going to need that this prompt is going to need that need to go into this prompt.

3:17 So we get that too. So we have that in place. Let's say the current response actually called a tool. Let's say called a tool. We also need those two results here. So what do we do? We simply include two results. We simply include two results. Making sense, right? Then let's say we want some files. We want some want to attach some files. probably we have some particular company documents that guide how we write code. I've talked about style guide before, but let's say we have some policies. We have some things that have to do with compliance and stuff like that and want to make sure that we include them because of the type of code we want to write. Probably we're writing for a fintech application or a healthcare application. No, sensitive applications like that that has a lot that would have a lot of compliance requirements and stuff like that. So, we just attach some documents to this prompt. And with that, we automatically have another section for attached documents. attack document and we can keep tacking more and more and more things that we need into this prompt payload. What we are basically doing which is collecting information through different strategies through different processes to put inside our prompt to make sure that our prompt has everything it needs to fit into the model so that we can get the right response is called context engineering.

4:43 It's called context engineering. Yeah. What we're doing is context engineering. Of course, definitely trying to manage the context window is part of context engineering, too. But a lot of the work is in actually grabbing everything that is needed for the current request. Every single thing that is needed for the current request, we're grabbing it in because our model calls are stateless. We've learned that from the very beginning. Our model calls are stateless. So, it doesn't remember anything. Everything starts from nothing. Every model call starts from nothing. So we have to ensure that we are packaging everything that is required for the current step to be able to execute. So what context engineering simply does is that it converts the question of what should the model know right now because we know this from the statelessness of the model. It converts the question what should the model know right now to what do we put in the context. Yeah. What do we put in the context? Because that's what you're trying to say. Anytime you ask what should the model know right now for this particular step we want to take. Maybe we want to generate a particular API endpoint. Whether we want to generate a model that performs data analysis, we need to ask that question every time.

5:50 What should the model know right now? Because the model is stateless. What should the model know right now? And what context engineering translates that question to is what should go into the context? We need the system prompt. We most likely always need the system prompt. Do we need to retrieve some documents from our vector stores from our chunked knowledge base? Do we need some tool results whether from the current response or from some previous responses? Do we need to attach some documents? And that question has to be answered on every turn. That question needs to be answered on every turn. Like I said, more and more stuff can still go in here. More and more stuff, skills, agent MD, stuffs like that, all those stuffs, a lot of them can still go in here. So this is basically what the context part of a harness is all about.

6:43 Now let's talk about what a good design looks like. What a good design looks like. And first is the fact that you should always have a system prompt. Yeah, you have you should have a system prompt. You should always have a system prompt. You as the builder of the NS, you must have a system prompt. And what the system prompt does is that it tells the model who it is or it tells the agent who it is, what tools it has, and what good behavior looks like in the context of its role. It tells it who it is, what tool it has, and what good behavior looks like in the context of its role. Because good behavior when you're a coding agent is different from good behavior when you are a let's say travel agent or an healthcare agent.

7:25 So it defines what good behavior looks like based on his role. This is the control plane of you as the developer as the agent builder. This is your control plane. This is where you add code your instructions. This is not agents.m MD. This is not the agent.m MD. Even though agent MD can also go into your context. This is what you define as the builder. Agent MD is defined by the consumer of the agent. You as the builder, you you determine the system prompt. The user is the one that writes the agent.m MD or let's say soul.md when you talk about things like open claw or her agent. So those ones are defined by the user and they can also go into the context but your own hardcoded instructions go into the system prompt. Another thing you need to decide is your history strategy.

8:08 By the way I missed out that here and this is a very quite important part of the context and I missed it out. The conversation history. Yeah the conversation history. We saw it with the loop. We saw it in the gaps. the conversation history. So the conversation history also has to go with every turn. I think the system prompt and the conversation history, those two you can't you can't do without them in the context. So yeah, you have to pick what your history strategy would be. How do you pass the conversation history on every turn? Every turn of calling the model, how do you pass the conversation history? And to a lot of you, you might just want to say that yeah, just pass the entire thing. We need to remember everything. Let us pass the entire thing. And yeah, you're right. to an extent you're right in context in a context you're right because that is a strategy called full replay that strategy is called full replay that's the name of the strategy where you just pass the entire history but context windows are a limited resource so you can't always just pass everything and sometimes too much history is going to confuse the agent yeah the agent is going to be confused because it will be finding it out to focus on the specific parts of that history that are required for the current operation So to manage the context window and to also make sure that you're focusing on the main parts that the model requires to respond to its currently given task or the current step it is processing, you then have summarization strategies like compaction. Yeah, this where things like compaction come into play where you compress all your history into a summary. A summary that still makes sure that you are not missing the important part and the model can still be able to get the required context for the current step. it wants to perform. And that's that's a particular type of strategy.

9:52 And even from compaction, you can always have pointers that can point to a file in your file system where you actually dumped the entire conversation history. But more on that later when we get to the chapter on the context layer where we're managing the context and any chapter that has to do with that, we're going to look at these strategies and things like truncation and even how to structure our memory. Whatever strategy you are picking must provide the model enough history that will fit the current task it's about to perform. That is that is that is the main thing you have to keep in mind. You have to feed the model what it currently needs to perform the task it needs to perform at that point in time. Another important good practice we're going to be discussing is retrieval. Retrieval.

10:39 You need to bring in just enough outside knowledge that will be useful for the current task and not bury the model in details. Make sure you're not burying the model in details. Just retrieve what you need. The only thing you need is what should be retrieved. Don't say, "Yeah, let me just dump this entire book or Python into this model because I'm just I just want to write a simple observer pattern." No, retrieve only what you need because we need to manage the context window. Yeah, the context window is a very limited resource and every strategy that we are going to be using in performing context engineering is going to be anchored on the fact that we're trying to manage this limited space. Deciding what to include, what to compress, what to leave out, and when.

11:26 Yeah, that when is also important because it depends on the task that the model is to perform at that point in time. This practice used to be called prompt engineering back then but prompts are static and even if you look at all the prompt engineering strategies they are all static strategies. Context engineering is different because the context is a moving target. The context is a moving target. It's different on every turn. As we have explained it is different on every turn because the task you're about to perform is going to determine what is going to be loaded into the context. So the design of your context is an ongoing concern and not something you just write one time like prompt engineering. So now that we understand what the context is all about, let us do our case study and this time we're going to be using cursor.

12:13 Enough of cloud code, let us look at cursor. Now cursor's context engineering is what set it apart from charging the early days. I think Koso was the first time we saw a proper ANS in operation like Corso was just awesome. Everybody was copying and pasting code from charge. Then Cos came and like yeah you can do it here. You can do it here and we have the full context. We understand your entire environment. We can create the files for you. We can create the entire project for you and we are going to always understand what you're doing at every point in time.

12:44 Corso was the first earnest that we actually felt and honestly big ups to the guys that created Corso because they really really were ahead of time on a lot of things. So let us look at some of Cursor's context parts just some of them the ones that are very very obvious. And the first is cursor rules. Yeah, if you're very familiar with cursor, you know what cursor rules are all about. So the first is cursor rules. And this is the system prompt layer. Yeah, this is how cursor does system prompt cursor rules. That's the first layer in the context. Every single prompt to the model that cursor is running gets this cursor rules. That is system prompt layer. Next is cursor's codebase indexing. Yeah, how cursor index your entire codebase. And this is how cursor does retrieval. This is how it does retrieval. anytime it is trying to reply you in the chat window or it's trying to generate code when you are typing it has your entire codebase indexed and it can retrieve any piece of information it needs from your entire codebase in order to be able to know what next to do or how to reply you in the chat window so codebase indexing that is where retrieval comes in the next thing is your currently open file that's the file you're looking at the file that you are currently on its tab the currently open file is is your focus. That's where that's where you're currently working.

14:07 That is what cursor uses as the main point of focus to determine every other thing that is important. That is like his anchor point to really determine what needs to go in the context. So that's the focus. Then you have the file that you have at mentioned in your chat window. Yeah, those files that you hit and it brought out all the files and you picked one of them or you picked like three of them. Those atmentioned files, those are your attached files. Remember we have attached documents. Those are your attached files. They're different from the codebase indexing where that cursor is doing in the background. And you can always use retrieval to get what is important for the currently open file. But if you explicitly attach any file, it's going to take it as a full attachment. It's going to take it as a full attachment for the current task that you want to perform. And finally, we have the recent edits that you've made. the recent edits that you've made.

15:02 These represents your history. These edits represent your history. That is the history what it uses as the main history for what to do next based on the prompt you are giving it or based on the code you are trying to write. Every single time that you hit a keystroke and that keystroke is to trigger a completion, a code completion, every single thing we just described is going to be assembled into the context. Every single thing is going to be assembled into the context to determine what that completion should be or if you're chatting if you are in the chat window. Every single thing is going to be assembled to determine what the response is going to be. So like we said your cursor rules are your system prompt. Your codebase indexing is where you have your retrieval. This where retrieval takes place. Your open file is your current focus or in the case of code completion is your current prompt.

15:53 The files you mentioned with the art sign, these form your attached files and your recent edits make up your history. Yeah, they make up your history. So that's basically how cursor works. Cursor is basically just a wrapper. But that is a very very very oversimplified way to look at it. Of course, you have the model. They use the same model that we all use every day. GPT4, GPT5, claw this, oppus that. But for Koso, the model sure is the same, but the context is the product. The context that Koso is able to supply it. Koso being able to engineer the right context to go into every prompt. That is the product. That is the Coso product. And that is why Koser is currently valued at almost $30 billion just for this context. Just because of the brilliance of the engineering around this context. is basically using an IDE that they Yeah, they're using VS Code. They're using VS Code. The context is the product. The context is the product. And in the chapter dedicated to context, we're going to be looking at how we are going to be composing our context and even including things like agents.mmd so that users of the agent can also add their own explicit rules to the ars.