Why Your AI Agent Gets Brain Damage, and How to Fix It - Tyler Barnes, Mastra

0:00 you're working for a long time, a long session, then all of a sudden you hit this imaginary window that then now causes this massive compaction event and you lose context and you feel like you have to just like restart the session. >> A lot of people have hated compaction just because it's like your your agent it's like they get brain damage as soon as it happens. >> I'm with Tyler, founding engineer at MRA and we're going to be talking about memory, specifically observational memory. We're going to talk about agent signals. But first, you know, a lot of people probably seen you before, but it's been a while since you've been on the show. What were you doing before Ma?

0:33 And then what was, you know, what are some of the things you've worked on since you've joined Maestra, which, you know, was very early. I mean, you joined when we were in YC. >> I joined Mastra, like you said, early on. I had been working on like my own agents and stuff. And I think you and I would get on, you know, calls just to hang out sometimes and we kind of demo stuff to each other. And I was getting pretty pretty excited by the stuff you guys were working on. So, >> I I remember you you telling me it was very it was very early, but you had you basically wired up a a camera where if your cat came, it would detect that this is a cat, so feed it. And if your dog came, it would detect that it wasn't a cat. Don't feed it or something like that.

1:10 >> It would play like an alarm to scare the the dog away. So, it was always eating the cat food. >> So, you're using like some AI there to like detect. >> I think it was like GPT40 or something, you know. Yeah, I think I think it was before that actually, but yeah, but it was somewhere in that range of models. >> It was. Yeah. >> Yeah. >> We always talked you know just models and what we were building. But then you joined MRA and you've obviously worked on all you know all different areas for sure, but you've specialized in a couple.

1:37 >> Yeah. I guess memory was the first thing. >> So tell me a little bit about the first version of Monster memory because we've gone through some iterations now. There's still a lot of people that use the initial memory system that we built. Yeah, there's quite a few people I think. So, it started with there was three types. maybe we just started with like the message history very simple, you know, the last x number of messages like 10 or 20 or however many you want. And then we added working memory which is sort of like the agent can update a tool or use a tool to update a chunk of context. So over time it can kind of keep track of something.

2:08 >> And did that context basically just like sit in the message history or sit in the system prompt or how did that context actually what was the underlying mechanism to make that happen? It did sit in the system prompt which is not great for prompt caching. We've actually fixed that since with a newer version of memory but >> when we built that no one cared about prompt caching. >> That's true. It wasn't a thing people were talking about.

2:27 >> Yeah, I think it it existed but not all the providers even had automatic prompt caching at that time. It was like you're a lot of the time you were just paying uncashed prices all the time. So, >> and so you had message history, working memory, what else? >> and then the other one was semantic recall which is just rag. So every new user message and assistant message you would basically do a a rag query and then insert into the system prompt again some relevant context. At the time we ran long me eval on it and got like a really high score and it was it was somewhat controversial because it was like a oh you just use rag and you can score very highly on you know this benchmark. We've since gotten a much higher score but it was quite interesting that such a simple system could work so well.

3:08 >> Yeah. And I mean the interesting thing about it was you could basically take a huge message history and only insert the parts that mattered. So you'd actually have less context which at the time you know for folks that that was great. You didn't send as much context. It was cheaper. You had the extra lookup of course. So maybe you had some extra latency but you probably made up that latency because you were sending less tokens to the LLMs. But over time we've realized that maybe there's other approaches that could even be you know even be better especially as prompt caching became more prominent. Yeah, >> walk us through like how did you go from okay this first memory system which was pretty good and arguably there's still a lot of memory systems that use rag today but what was the the next iteration >> I guess we sort of had like quite a big jump into the next one which is observational memory I'd been doing a lot of experimenting with coding agents and prompt caching with coding agents is very important they just eat tokens like just non-stop calling tools and you know reading big files and things like that so the memory systems that we had really didn't worked very well for that use case. So through a lot of iteration, you know, just trying things out, I did I eventually came up with observational memory which is a prompt cachable system. We ran long eval on that as well and we got like state-of-the-art at the time. So that was like a very big jump as well.

4:25 >> Yeah. And can you tell people so we we talk a lot about benchmarks on the show, but can you tell people a little bit about longme eval? And I know we want to talk more about observational memory of course, but what is longme eval? And is it a good benchmark? Some people don't like it. I think it's an okay benchmark. We actually don't have a lot of great memory benchmarks, but I I think it's quite good at testing the recall for, you know, a single turn. For agentic use cases, you really want to be able to test across many turns, and you want to be able to test prompt caching how well the memory system can guide the trajectory of an agent as it's working on something. We're probably going to end up, you know, running some some more benchmarks in the future. Long VM eval conversation histories. So, you know, semi-real world.

5:08 >> Can you tell a little bit more about how does observational memory work? So, we have this great memory system, but how does it what's going on under the under the hood? How is it prompt cachable? How does the system actually work over long conversation periods? So you you know you don't lose context because I think before one of the frustrating things with coding agents and they've gotten a little better but some still suffer from this is this idea of you're working for a long time a long session and all of a sudden you hit this imaginary window that then now causes this massive compaction event and you lose context and you feel like you have to just like restart the session. But how does observational memory differ?

5:44 >> A lot of people have hated compaction just because it's like your your agent it's like they get brain damage as soon as it happens. I think with codecs it's gotten a lot better but it's still not quite as good as you know like a better memory system. Observational memory is sort of it's almost like a hybrid of compaction and a better memory system. So as your agent is working in the background there's an observer. So this is another agent which is taking in all of the the turns and it is creating condensed observations of what happened.

6:10 >> So this is running kind of in parallel to if I'm talking to an agent there's another agent that's just watching the conversation. >> Exactly. it's sort of buffering these observations in the background. So each chunk of observations maps to some set of messages in the conversation history and they just continually build up until you hit a certain threshold and then those messages get replaced with the observations. So your very you know tokenheavy tool calls and messages suddenly get replaced with a very dense representation where the information is not lost. what you lose is really the a lot of like the context rot, you know, the things that didn't matter contextually to the conversation.

6:48 >> And so you have this observer, right, as this observation agent that runs. >> And then what happens if it continues to grow even past that? Does it just continue to run? How does it know that it can basically go forever, right? I think that's that's part of that's one of the benefits of observational memory. You know, one example is I have a an email agent that has run through at this point tens of thousands of emails, right? and I >> I've kind of steered it to what I want and it keeps track of the decisions I've made in some of the context, but it doesn't I can just use that same memory, you know, I've been going on like 3 months in the same like never never changing. Does it run on a cron or something or is it when emails come in >> every day? I basically will just like run it and it'll just like run a script that processes all my emails. I I eventually I'll put it on we we have master schedules now. I'll put it on I'll put it on a schedule one of those.

7:34 but this is before schedules existed. So I just had a you know I just have a script that I run but it's uses the same thread. It's used the same thread for you know probably four months 5 months at this point. >> Yeah. So I think that is one of the the coolest parts of observational memory is that the chat just feels like it goes forever. This background buffering means you never need to like stop and wait for compaction to happen and then suddenly the chat is much worse. The quality just stays sort of like consistent and you never notice the memory system doing anything. it's just sort of happening in the background and then swapping out these big heavy you know tool calls and stuff for with observations and I guess yeah you just ask like how does it how does that actually like keep going forever like eventually you know these observations are going to get so long that they won't fit in the context window and that's where we have a second agent which is the reflection agent and at a certain you know token size of observations it will go ahead and look through all of them and figure out what are like the overall like themes and like the important parts of this and what are the things that didn't actually really matter that were observed and it will sort of condense that even further down. By default, it actually condenses the first 50% of observations which makes it, you know, even more sort of consistent feeling. So you get like condensed observations, raw observations, and then the raw conversation history.

8:42 >> So that's kind of how the context window stacks. You have condensed observations, then like another tier of the second 50% of observations that haven't been reflected, >> and then any new conversation history that comes in until it hits another level where observations run again. >> Yeah. it just goes in a loop forever like that basically like the the messages are getting observed and then the observations are being reflected at token thresholds essentially >> and so what are the big differences because it's still not completely lossless right there still is the chance you could lose some information >> but it does and if you use it you know and I've obviously used it extensively it does feel better than just you know early compaction right where it could get this huge context and then eventually just trim it way down and you felt like you lost a lot of information So what makes it what are the things that the characteristics that make it better?

9:31 >> We've sort of dog fooded it for like months basically before we released it or before we even benchmarked it or anything. We were just sort of tuning this prompt based on like vibes essentially, right? Like we're like manually evaling it through dog fooding it daily in master code actually which was originally the first versions of that were created to dog food observational memory. But, you know, we just sort of worked on it, iterated on it till till we got to the point where it's like, "This thing works really well. Let's benchmark it." And then it scored really high and we sort of released it at the same time that we released the benchmark results.

10:02 >> Yeah. Fun fact about master code. I remember like early versions of master code and obviously it's changed. So, it wasn't even called master code. It was something else ree or something that you were working. It was just like a markdown coding agent that you were using to test this and then those experimental. >> Yeah. those like early experiments turned into what became master code which was like because of all the context encoding agents really good way to test that memory was working well.

10:24 >> Yeah. Yeah. Cuz they just eat tokens right. >> Yeah. I know recently we've been improving observational memory a bit. You know we released this I think it was what February maybe or February March time frame when it first came out. What are some of the things you're excited about? What have we released since? And what's coming potentially coming next? A couple minutes ago, you asked a question which I didn't answer, which is like what are the downsides, I guess, of observational memory. And that sort of feeds into your question now. The new features that we've been releasing and that we're about to release sort of make up for any of those little downsides. A big one is, you know, over a very long period of time, eventually it is going to get so compressed that you'll lose details. So, one of the first things we added was recall. So, that's again that is just rag, right? You're actually doing rag against the observations rather than raw messages. So, you save quite a bit of space by doing that. But the agent actually has a tool that it can use to search for something. You know, if it has a hint in its observations that there's some something happened in the past, but it doesn't fully understand from those observations what happened, it can do a search intentionally, which retains the prompt cache. And then it has some tools to sort of page back and forward through like the raw messages at that point.

11:33 >> So it is it is searching through raw messages but based on context that it sees in its observations. >> Yep. And so then it tool call comes in, it gets stacked on the end of the message list. So it preserves prompt caching I'm assuming. Yep. >> And then the results get pulled in. So it's just like feeding it's it's like an agentic search, right? It's like feeding new results to the agent so it can kind of search >> its own history essentially.

11:56 >> Yep. That's exactly it. Well, that actually works quite well. We want to go even further than that. We want this thing to be like a perfect memory system. You know, you can just throw literally anything at it and it will always remember everything. That's the next thing that's coming. so we have two features. So one we did just release it's a lower level feature called observational memory extractors. And what this allows you to do is provide some kind of schema and then as the observer is running it can extract some structured information out and then you have a callback. You can do whatever you want with that. So we actually use that to fix working memory right which earlier we just we said that that was invalidating the prompt cache continually every time it get got updated. Now we can use the extractors.

12:33 There's another feature actually we were going to talk about agent signals. It uses that as well. There's a combination of them, but the gist of it is that, you know, you can extract some data and do something with it. So you you not only rely on, you know, you keep the prompt cache of the main agent, but you use that prompt cache of the observer to make a follow-up request and extract the data. You're sort of piggybacking on the observer's cached context.

12:56 >> Awesome. So this would allow you to basically define certain types of information that might be important for your application. Later, we're going to talk with Yan about video production. So, I'm building a video production agent. It might p need to pull out types of settings that I use or types of clips that I want to take and it would then be able to extract information if I message something around those parameters and then save that for later. So, it gets pulled back into context as needed.

13:23 Something like that. >> Exactly. Yeah. I think maybe like the simplest to understand example is that we have a thread title generation as a observational memory extractor. So the observer is continually checking does the thread tit title need to be updated and then it just updates it basically. So >> so yeah so if I start talking about one thing and then I steer it in a different direction it can update the title of the thread. So >> exactly >> if I have thread history in a chatbot I can know that it's actually correct.

13:49 >> Yep. Okay. >> That one it is a little more it's sort of hard to explain to people. It's not that you know complex but can be a little bit hard to you know kind of mentally understand when you would want to use it. But the reason that we're we've shipped that one is an upcoming feature which is subconscious observational memory. So this this one's going to be you know the the actual memory feature that pushes OM a little bit further. when you first told me about observational memory and I think the last time you were on the show talking about observational memory you said you kind of thought about like you know my brain seems to work very well like try to make it humanlike and subconscious OM sounds even more humanlike >> right yeah I mean yeah you're we didn't even mention that but that was like the original inspiration is just thinking like >> you know as as a human is working writing code or doing whatever you don't need to like choose to to remember things or or to save memories right it's just you have something in you that's observing and I guess what you're hinting at similarly right there's you have a subconscious behind that even right which is sort of probably like keeping track of things longer term and storing things in different ways and >> I've seen other people you know say that agents need to dream and I think it's kind of a similar vein of like something that's happening in the background at certain times right where you can like pull out the right information but tell me a little bit about subconscious OM >> it's going to use extractors so well you know every you know chunk of observations get that get made it will have extracted some information which gets sent to some background agents.

15:17 There will be some a few different types of them. Every chunk of observations there will be an agent which is taking that and converting it into a graph structure. Right. So you'll have some like entities with relationships between them like facts and things like that. It'll be stored in a database >> like a knowledge graph type thing. >> Yeah, a knowledge graph. Exactly. It'll be a knowledge graph as well as >> I knew we were going to have to talk about graphs.

15:39 >> Graphs. graphs always it everyone always wants graphs and now we're going to have them. Tell me a little bit more how does the graph work? H how will the agent interact with the graph? How does that process work? >> So at observation time we will have sort of like a more of like a stateless graph extraction happening. What that means is the agent that's doing the extracting isn't going to need to know all of the history going back. It'll just recognize entities and then relationships between them and just store those in the database. And when you get to reflection time, we'll have a more stateful agent that's able to go back and look through all the previous memory the graph structures that already exist and then it will take all of the you know sort of stateless graph structures that were created and rework those to fit into the existing structure. So sort of again like right observation and reflection taking that metaphor and continuing to go go forward with it basically. And then how does the agent then use that graph to you know potentially answer questions right assuming you want to keep that data and then recall it or use it when the agent needs to answer a question that might be stored in this like knowledge graph of sorts >> that actually goes into the other feature maybe a little bit agent signals >> that's something else that was released what tell me what are agent signals how do they work and maybe then we can connect the dots on on the graph side of things >> so there's a bunch of types of agent signals I'm not going to go into all them because there's some that are more advanced But essentially, it's just a way to decouple sending context into the running agent loop from needing to own that loop. So before agent signals, you would send a prompt and you would immediately get a stream back. So like the client or consumer or whoever sending the prompt essentially was the one owning the stream.

17:15 >> I kick it off, I own it, right? That's I think what most people expect when they talk to an agent is if I kick off the stream, I need to own the stream. >> Exactly. So with signals, we've decoupled that. So you can send any kind of context. There's multiple different kinds like notifications, regular messages and a few other things. You just send that into a specific thread, right? So like a conversation and then separately you're subscribed to the conversation and you're just listening for whatever is happening.

17:42 So many many clients could be subscribed and many clients could also be sending messages in. >> Okay. I think you're going to have to break that down because there's for me the the thing that made the most sense, the thing that allowed me to kind of get it is if I kick off a longunning task and the agent's doing some things, >> Mhm. >> I can essentially add a message or send a signal, you know, and then it'll just get inserted in the next at the at the exact right moment the next time the loop as soon as it's able to.

18:10 >> So, it doesn't actually interrupt the loop. It doesn't stop the loop. >> It just is sent as a signal. So you can essentially like steer an agent if you're, you know, if you think about a coding task, if you're watching the agent and you're wondering why the hell are you going that way, like don't do that. Go look at this part of the code because, you know, it you might send and steer the agent to the right thing or often what I do is I I send a long message. What I like to think is a carefully curated plan and then I realize, oh, there's one more thing.

18:36 >> There's always one thing >> also don't forget to do this and then it just gets inserted as as the agent's running. I think that's one example of a signal, but you said like external systems as well. What would be external signals or notifications that come in and how would those work? >> Yes. So maybe maybe your agent is working on like a pull request, right? And it needs to subscribe to any info from the pull request, right? Like CI is failing or you get new PR comments.

18:59 That would be a notification signal. Your agent would be subscribed and anytime this external notification happens, it'll just be dropped into context essentially in the same way as what you were just describing. but with a little bit of different formatting so the agent understands this is like a system sort of piece of context. >> So it's like an external web hook of sorts. >> Yeah, basically >> but it gets inserted into context. What does that context look like? You know, so how does the agent know that this is I received a notification from GitHub or whatever.

19:29 >> So we wrap each signal in some XML, right? Because agents are trained with XML like user tags, assistant tags. With this one, we'll wrap it in like a notification tag. Maybe it'll say like type GitHub or something like that, right? And then it'll have some info about, you know, oh, maybe Code Rabbit left some comments or or whatever. >> And then it just gets inserted as So this could happen the agent has stopped or this could actually happen while the agent is still working on something else.

19:54 >> It's both, right? So then that's actually a good point. That's that's a really nice part about it is if your agent goes idle, it can just wake it up again, right? So it can immediately begin addressing those pull request comments. So the agent is basically sitting there. CI fails, pull request comment comes in, whatever. GitHub sends a web hook to your agent, wakes it up, your agent keeps running, and then while it's running, I could then say, "Oh, actually, ignore that comment. I don't care about it." And I could steer it as it's as it's running as well. Right.

20:21 >> Exactly. Yep. Yeah. It's very flexible in that way. I guess where this ties back to observational memory is we have a type of signal called a state signal, which is just like a chunk of state that is in context, and anytime it drops out of context, it gets added back in. Typically people would be having a dynamic system prompt for that, right? Like we were talking about that earlier, >> but that invalidates the prompt cache.

20:41 So with this new state signal, since we can just inject context at any point, like you were just saying, we'll be able to surface pieces of relevant memory from the graph, maybe here's like the top, you know, five nodes in the graph that are updated or like were most recently updated or >> so. So that that's how it connects to the graph. So you have this the signals get sent in from your potential like nodes from the graph that might be >> useful for whatever the agent's trying to do and then it's at the bottom of the context and then when observations happen I'm assuming that information if it's needed just gets moved back into observations.

21:15 >> Yes, exactly. And if the observations remove the state signal from context, right? Like we're compressing the messages, it'll again just get injected. >> I feel like I'm learning more stuff today. I knew a lot of it but I didn't even know all this. Okay, so we're in London today. We're doing this live in London, some of you watching this live. This is the third conference we're doing. What is one of the your most fond memories of Maestra? You've been here a long time. Whether it's one of these events, whether it's, you know, even something we launched.

21:42 >> I think it was probably my first week, which would be we had like a meet up in SF, you know, when when Monster NYC just worked in I think it was the dungeon is what it was called, right? the dungeon, which was a two-bedroom apartment near in the dog patch of San Francisco. Yeah. And we had probably what, 10 people working in that office. >> Huddled in there like just grinding. >> Yeah. That, >> you know, getting done.

22:06 >> It was not it was not comfortable. It was definitely packed, but we got a lot done >> and it was a lot of fun still, you know. >> I don't know if it was that week, but a lot of the team was at hotels, but I was sleeping on like a fold out cot. I think Tony was sleeping on a mattress on the floor. I mean, it was just it was a fun week. >> Yep. All right. Well, thanks Tyler.

22:23 Thanks for coming on. thanks for talking about observational memory, subconscious observational memory, and telling us a little bit more about agent signals. We appreciate you, spreading some knowledge with with the rest of us. >> Well, yeah, thanks for having me. It's a lot of fun. >> Every week in AI, something insane happens >> and there's so much drama. Every Monday, we break it down live. >> We do the news. We bring on guests building in the space >> and we go deep into the stuff that actually matters.

22:47 >> Agents Hour, every Monday, noon Pacific. >> Follow. Don't miss it. Peace.