Doga Kerestecioglu - Mindful Agents: Human Inspired Memories for Long Horizon Tasks

0:02 Hello everyone. It's great to be here. I'm from the Microsoft Fabric AI team and I'm here to talk about the work we're doing in the real-time intelligence space on our longunning agents. where we take some inspiration from ourselves as humans to improve the memory so we can get better at accomplishing tasks. so in the space that we're working at we tend to have longunning agents that has access to data that's high volume streaming data that's high volume and to just like set up the problem that we're dealing with and the motivation for the approach that we're implementing here I kind of want to contrast it to how we as humans store memories and how like we tend to have the situation for our agents instead. So as humans we're not we have like storage constraints. So what's likely to happen is we because of that constraint we will tend to the limit makes us better at distilling memories. So for instance, recently I was on a train ride in Europe and you know as you're on the train you look outside the window and you pass by you know endless fields and you see some maybe windmills you see some animals grazing and we don't have the capacity to necessarily remember every strand of grass or every bit of thing over there. what we're good at is being able to distill that information and just remember interesting things that might be relevant to us. and this is just something that we through evolution are good at. whereas if you think about the agents, we don't have that problem in terms of the storage constraints.

1:50 we can log every bit of information somewhat like relatively cheap and we can log every trace and what we do is we capture retrieve and typically we summarize and retain the summary. so over time we are with summaries of summaries of summaries which typically works well until you know at the time of retrieval when you're looking for something very specific for most cases but for the case that we're working on we're interested in getting agents that work with these like big volume data sets and we want them to be proactive. so what we worked on is we are working on building a memory life cycle as a feedback loop for our agents where we have a system that starts with first ingestion where we basically ingest all of the data from the observability substrate and then what we do is consolidation which is kind of like similar to the idea of sleeping but what we do is we ddup merge and we try to consolidate all of the memories that come in to become candidates. And the idea in here is while this is similar to maybe humans, the consolidation cycles for the agents that we're working in are domain dependent and they might not always be temporal. so I don't know how many of you were are still in the room, but you were just building an agent that was like looking at F1 for example. So if you're a person like if the agent is working on Formula 1 races the telemetry that you take during the race will be very different from what happens after the race like during the race race days versus the off week and then it's going to be very different from the season. So consolidation doesn't necessarily mean it has to be daily but you need to process in some batch in some cadence. next we have a forgetting where there is decay and interference and the idea is we want to make sure that if we have all these consolidated memories that are stable candidates we want to make sure that we end up remembering the important things and all the unimportant stuff gets forgotten or most of it as best as we can and then the next stage is retrieval. so for the retrieval what we work on is a hybrid way where we have these memories stored in the cold storage where we know a domain and put them in a graph but we also have the live data coming in and we want to make sure that the agent can access both the streaming data and what's in the storage that's more stable and be able to determine what's the best memory to use at the time. so that gets us to the feedback cycle where at the time of the retrieval you will have usually conflicts. So something new might happen and it might conflict the memory that you have established that became stable in the cold storage.

4:40 that's where maturation and reconsolidation comes in. because we don't want to necessarily keep adding a new memory over and over into the graph that's very hard to retrieve from. So you need to determine whether you want to update an entity in the graph. whether you want to like add a new entity to the graph because there's some new information that's important or maybe discard what's the live information that's coming in because it's erroneous. So what like next I kind of want to talk about all of these bits have are there like each their own components with different optimizations that you need to make. So I want to chat a bit about how we approach our evals for them and how like our thinking around it.

5:24 so here is two cases where we evaluates kind of like the ceiling on the floor about the best case scenario and the a very tough scenario where on the left side we're having a retention benchmark which is more deterministic where in this case we are looking at a case where the system has clear labels of everything that's important. So what we were looking to optimize in here is like how often we want to consolidate and what's the kind of the cadence we want to use for it.

5:55 And for this case that we were looking at and this changes from domain to domain but it's things started stabilizing around 500 events at a time. sorry I think 200 events at a time. It's kind of small to see from here for me but let's see. Yeah 200 events at a time with a precision where if we like forgot over it that got better. So like we're checking it what's the best precision you get with how much you consolidate how often you consolidate and then how much for getting ads over it. in the next next case we were using a retrieval benchmark utilizing long map eval where we were looking at how much compression should we have?

6:32 What's the optimal storage size we should look for without being destructive. So this is an eval where it's conversations. So we have no labels about what might be important or not which is not the ideal case for the type of work we're doing because the like the idea that we have is these domain experts that do repeated tasks and there's an idea of how you can learn what's important but even then we can like optimize and there's a trade-off between how much you want to compress and what accuracy you get. So with that the kind of like approach in more general is for the type of memory work that we're doing we need three types of evaluations and they kind of have like different ideas about what they're looking for. So one of them is retention which can be a quick eval on deterministically if you're building a memory system and if you want to keep the important information can you just like make like can you make sure that what you're keeping is actually important versus not which is you need the labels for it and there's a lot of work that we're doing on like how we get better labels but then the next one is retrieval. So assuming that you built a good memory system, then you need to look at how well you're retrieving from it and are you getting the memories that you're looking for in that like great system you built because if it's hard to navigate your memory and it's how hard to do the retrieval part of things then that still like is necessary but not sufficient. So you want to have good retrieval and finally you really for the agents you want to look at your task completion evals which are the most important because it's closer to the user. If the you might give the best memory to the agent and depending on how you how it's exposed to the agent in your harness, the agent might choose to ignore it and still might not accomplish the task. and this is kind of like the most expensive one for because because for the longunning agents, it is kind of expensive to build and run the evals, but it's the most important. So like for each component, we're focusing on these three things. so to kind of just sum things up, logs are not memories. And I think like where things are going is like for longunning agents, you want to make sure that you have more than just the traces that's going to become available to them as memories.

8:46 And the the way like the volume works to be efficient, you'll have to figure out how you keep the important things or not. and then another important point is the domain is important and like for these cases it needs to be predeclared and depending on how well you can do it at the runtime you need to have it prior otherwise that affects the efficiency and accuracy of the agents that you're running. so in terms of kind the next steps we're working on we're working with the foundry Microsoft foundry team on end to end evals in state bunch and we're also doing some work on like better work on domain learning and graph ontology where the idea is what we found in a lot of these is the way you set your graph structure and the ontology for it has a big effect on your accuracy in addition to these components. So we're working on them.

9:36 so with that thank you very much and I look forward to meeting everyone after. Thank you.