Day 7: Harness Engineering - Agent Memory

0:00 So the next component we're going to be looking at is the memory. The next RNS component that we're going to be looking at is the memory. Now natively LLMs don't have any concept of memory. Yeah, they don't have any concept of memory. If you send in an impute and that impute produces an output, that is all for that single model call. This model is not going to remember the impute. Neither is it going to remember the output it produced. This is all you get. And that's how it is for every single large language model. There's no large language model today that has memory built in. The LLM itself has no single memory. So if you want the LLM to remember its impute and its output, you have to build a memory system around it.

0:47 I know you must have used CH GBT or Gemini and you'll be like, "Yeah, it does remember what I say. We have conversations. We have full conversations. Claude Code remembers my code. Oh, he understands my context. All that is powered by wrapping a memory system around your LLM. Every single interaction that you've had with a chatbot, an LLM powered chatbot that remembers what you have discussed, what you're discussing, and you can even come back after a couple of days and it still remembers is as a result of a memory system that has been built around the LLM. And that memory system is not a model responsibility. It's a hardest responsibility. It is the earnest that is introducing that memory system. And this memory consists of two types. We have two types of this memory. And that's in form of a short-term and a long-term memory. These are the two flavors of memory that are built around LMS today. Short-term memory and long-term memory. You must have heard of different types of memory systems like episodic memory, periodic memory, and the likes. Everything can still be boiled down to these two main memory types. short-term and long-term. Now, what does short-term memory look like?

1:57 Short-term memory is memory within, let's call it a chat session. If we're going to be going with chat bots, so you open up a tab, a new chat tab in your favorite chatbot, let's say or Gemini, you open up a new tab and you start having conversations with the LLM. Start having conversations with the LLM. your entire conversation within that particular session, everything it holds in memory so that it can continue to give you coherent answers and follow along with the conversation is what is termed as short-term memory. Short-term memory, memory within a task like if you're going with CL code, you're working in a particular project. You have a particular session, the project, you're not starting a new session.

2:38 Memory within a specific session is what is called short-term memory. What is called short-term memory. These are your conversation history. scratch pads that your model uses to jot somewhere to result. When the model calls tools and it returns results, files that you mentioned within the conversation, everything is within your short-term memory. All that is within your short-term memory. Now, let us look at long-term. Long-term memory on the other end is memory that can be shared across different sessions. So you start this chat session today. You close it.

3:13 You have a couple of important items saved to the long-term memory. Probably a new user signed up and started a new session. So it saves the information about the new user into the long-term memory. Then when this user comes back tomorrow and starts a new session, it doesn't start to ask the user about their name, their email, and all those profile stuff again. It has saved it. It has recognized from this first conversation when the user signed up that this information is important. So it is saved into the long-term memory store. By the way, this arrow should have been double-edged because we can save stuff and we can retrieve stuff.

3:44 So, it is saved into this long-term memory store. And when this conversation kicks off, let's say the next day or even just an hour after, the LLM can understand or rather the earners can remember the user by passing the user's information from the long-term memory store into that session. And the same for any other session that is open after that as long as that particular information is relevant to that session. So this is where you store things to remember about the user. You can store the user profile. You can store the code base like CL code. You can store information about the code base there.

4:18 Every single information that needs to persist beyond a single session is stored in the long-term memory store and retrieved as needed. So that's how long-term memory works. Short-term is within a session. Long-term is across different sessions. Now let us discuss what a good design of this system looks like. There's still just like every other part there's a whole section chapter dedicated to memory. So we won't be going into serious details here. We're just going to be mentioning a few things so that you can always understand and look forward to them in the section where we focus on this and in other sections where we reference memory and we use memory. So what does a good design look like? The first thing is that the two flavors of memory have to be designed differently. Yeah, they have to be designed differently. You can't design short-term memory the same way you would design long-term memory. Most of the time, short-term memory is full fidelity, meaning that you keep all the actual messages, you keep all the two results, you don't leave anything out.

5:20 Unlike long-term memory whereby you just look at the entire conversation and you see the useful things that you can extract that are going to be needed for a different session. Long-term is usually compressed. You have things like facts, embeddings, and the budget for long-term memory is quite small. Yeah, the budget is quite small. There's a lot of things filling up the context window, and long-term memory is kind of like I won't call it low priority, but there isn't so much space reserved for long-term memory because most of the context window is already spoken for.

5:53 You have the system prompt. You have the active conversation. You have current tool results and you know the model within a session will most likely call a bunch of tools. You have also files that have been referenced rag stuff that has been pulled from the vector database. So whatever space is left for things to remember from the long-term store is small. It's quite small. So long-term memory is designed around fitting useful information into that small size of the context window. So you have to be saving things that you know are going to be very very important for another session.

6:28 For example, like I said, user profiles. User profiles are quite important. Then some preferences, some user preferences. Probably they like to be replied in French for some reason. That might be an important preference for the user. Probably they don't like answers beyond three paragraphs. you've mentioned it explicitly. So you want to save that and wait for them to correct it before you know stop giving them their answers that way. You want to save things like their location if they allow you. You want to save their address core information. You don't want to be saving any tool result because you feel is important. You don't want to be saving any type of document that can be retrieved through rag or can be referenced by the user. You don't want to save all that. You can in you can index the entire codebase for something like CL code. can index the entire codebase so that you always have that in memory. You always have pointers to different parts of the codebase in memory. But like I said, the budget for long-term memory is quite small. So you want to ensure that you are storing what is necessary and also passing only what is necessary into the context window whenever the task at hand requires it.

7:35 Remember these are all single tones according to the LLM. The LLM does not have any concept of memory. So, anytime you are filling up that long-term memory part of your context window, make sure that you're passing what is important for the current request, not for the entire session. What is important for the current request. Another very important design decision is that right triggers are an explicit honest decision. Yeah, you don't want to delegate to your model when to save something in long-term memory. And this is for long-term memory. This not for short-term memory. This for long-term memory. Short-term memory is hot. It's full fidelity. You most likely going to be saving everything. But for long-term memory, you want to be in control of what you're writing into your long-term memory store. For example, you can have trigger points. You can say, okay, at the end of every conversation, it should save anything that is related to this.

8:27 you can tell it to save a particular information or to save to the long-term memory store when a tool is called or when a specific event happens. The most important thing is that you have to be in control of when you're writing to the store. Because if you just leave that to the model, it can just dump in a lot of crap based on what it thinks it is reasonable. So whichever your trigger rule is, whether when an event occurs when the tool is called or at the end of every session, this should live in the it should be a clear mechanism that the is handling and not the model itself.

8:58 Now this is not a rule that is not going to be broken at all. Sometimes we might inform the model on the type of things we want it to store, tell it when to store it and all that. But that we will see cases where we will need to make that decision going forward as we build the harness in this course. But it is important that you control the trigger points and exactly what you want the model to save, exactly what you want to be saved into your long-term memory store. If you leave the model to decide what to remember, get a memory system that saves different things on different runs and is impossible to debug. It's going to be very very difficult for you to debug what the model's thought processes when it was saving a bunch of stuff that you probably don't need. Like I said, the model should contribute the content while yours gets the right operations. Also, retriever for memory is bounded. Retriever for memory is bounded. Now, what does that mean? It means that we don't dump our entire memory from our long-term memory store into the context. We only pull what is relevant. As you can see, a lot of these things are for these three items are for long-term memory. So, I'm still talking about long-term memory here. So, you don't just say, "Oh, yeah, I saved this bunch of stuff. They all seem to be important. Let me dump it into my context window." No, only pull what is relevant. Use retrieval strategies to pull what is relevant from your long-term memory store into your current context window, your current LLM call.

10:30 Only what is important to the prompt that is currently going into your LLM call. So, always keep it bounded. Don't just dump everything from your store into the context window. You're going to end up confusing your model and that is not going to be seen as a model problem. You can't you don't you can't blame the model for that. Your harness design is the main reason why you are having crappy answers because you're dumping your entire long-term memory into your context window. So, now that we know how memory works in the harness and we've also looked at what the different flavors are and what a good design looks like, let us look at our harness in the wild and our case study is going to be Cursor. Curser also implements short-term and long-term memory. And as you may have guessed, causes short-term memory is simply the chat history within a conversation. The chat history within the conversation that you're having in the chatbot window of your cursor session, that is where all the short-term memory lives. However, cursor's long-term memory shows up in things like the art feature and cursor rules. In your cursor rules, when you tell cursor things like this codebase uses snake case, it will remember it across all your sessions. All your sessions. If you tell it that you want to use TypeScript, it will also make sure that it is not using vanilla JavaScript to write your JavaScript code. And you can also use the art feature to reference files within your codebase. So you can see that these two systems also inso are operating in a different manner. In your chat history, the conversation is live. The old history is being kept full replay to make sure that the model is remembering everything that you're discussing. But then you have things like your cursor rules that you've defined once and cursor understands that these are the important preferences in your project.

12:15 So it makes sure that it's passing this into every call of the model passing it into the context window fitting it into that long-term memory space. And also when you use the add feature to mention any important item within your project, it also bundles that along with that current model call. Two different systems operating differently even though they are both memory. But like I said, there's a whole section dedicated to memory. So, when we get there, we're going to introduce both short-term and long-term memory into our Annest