Episode 01 Building Trustworthy AI Memory: The Foundation of Data Quality

0:03 Hi everyone. Welcome to the Cognitive Layer. Right now, everyone in AI is obsessed with giving agents memory. But here's the hard reality. You cannot build reliable agent memory on top of a broken data foundation. If your underlying data model and relationships are messy, memory just becomes a faster way to automate confusion. To understand how we actually achieve trustworthy, governed AI memory, we have to walk the stack from the ground up.

0:37 My guest today is Himanshu Singh, currently engineering manager at Netflix leading engineering on their entertainment knowledge craft. Himanshu has spent his career turning low-quality, scattered information into connected, trustworthy data assets at eBay and Microsoft as well. Today, we are walking the full path starting with data foundations and modeling, moving through entity resolution and adoption to see how it all comes together into a governed memory layer that teams can actually trust.

1:11 Himanshu, thank you so much for joining me. >> Thanks for having me, Vishakha. I'm excited about chatting about this particular topic. This has been top of mind for sure. >> All right, to get us started, real quick, coffee, cocktail, or dessert? >> dessert, I think without question, because like Okay, connecting this to our topic, memory is the dessert that everyone is trying to reach out for but now and the whole message is that to have that dessert, you have to eat your vegetables first, and those vegetables are the data foundation. If you skip them, then that dessert will make you sick. Yeah.

1:51 >> Love it. Love it. And plus, there is no wrong answer in those three choices, right? But and you know, what's one core memory from your career that taught you why data quality at the point of ingestion is non-negotiable? >> I think I have many memories. I'll I'll try to go back into time when I was just beginning to work with data foundations and knowledge graph. That was at Microsoft where I was working on Bing knowledge graph. And Bing is a search engine. It provides questions like it provides answer to the direct questions like from the knowledge graph apart from the search results. So, I particularly owned politics segment in that knowledge graph. And I remember it was election time and we were ingesting data about different politicians, their voting records, their opinions, their famous quotes, and many things. And there was an incident that we just prevented from happening somehow was that the data that we were getting for a very popular politician was trending at that time was sparse and not like it did not have enough metadata associated with that. And when we were merging that data from other sources or like from many sources, the data was getting merged, the we merged the wrong entities because we didn't have enough signal.

3:18 And this resulted in two different politicians having similar names merging together and bringing like giving a view which was not true. And imagine the consequences of this when it's an election time. And this kind of really taught me that merging problem happened only because the data upstream was not was not was not enough. And that's that's why the need to manage the data quality at the ingestion stage is a super critical. If the bad data fails quietly, then it never like stays for entire the impact will compound and downstream. So, you validate the point validated the point of injection and I would say that is the cheapest place on earth to really do it. After that, if you do it, you really end up paying a lot of cost.

4:09 >> I knew you were the right person to talk about all these topics from experience. That's that's really interesting how you got you had to learn by fire in that case, huh? So, you know, to build that bridge towards, you know, we will go step by step towards the whole AI memory cognition part, but like as you just pointed out, the step one is the data foundation. And you know, when we met at GraphConnect, your talk was it was it was such a well-done talk and you gave a piece of advice every engineering team needs to hear.

4:42 And it's hard. I mean, we are all graph enthusiasts and we will believe in the modeling that that model, right? But ask first if you need a graph. But, you know, and and then you talked us through once you've decided graph data model is the right thing, how you segregate the whole, you know, model from the database from all these things, right? So, when you're working with teams and you know, you've had exposure to a lot of different teams over your career, how do you help them break down the difference between a graph model, graph database, knowledge graph and well, I guess today it's so much about context graph before they start building?

5:19 >> Yeah. I think great and very relevant question as of now because most of the teams do escape it. I'll just reiterate very quickly what you said that step one is not how you build the system, it's about to what kind of system do I really need? >> Mhm. >> Do we do we really need a graph or not? Ask whether relationships are the product of your system. Do we even multi-hop connections that are evolving? Are you resolving the same entity across my many sources? A schema that is evolving.

5:55 So, all of those questions and maybe a few others will point that, okay, graph domain is something that you should go towards. And then, in this graph domain, there are multiple terms the ones which you mentioned. >> To untangle them, I would say a graph model is a model in >> choice. Like, there is nothing concrete or material associated with a graph model. It's about what are the entities, how are the relationships defined. Do we even have relationships between entities? And if you kind of model all of that in some format, that's the model.

6:35 >> Great. >> It would not which it it does not necessarily need to be a graph technology which is powering that graph model. A graph model can be modeled in kind of traditional or any general modeling technique as well. And you could still call it a graph model because you have entities and relationships which are connected together. Then comes the graph database. It is a storage engine that is used for efficient traversals. Many of the underlying technical challenges are resolved by an abstraction layer that this graph data database provides.

7:16 But, honestly, sometimes it is an skippable question at the beginning. You can run this like whatever graph model you have on the existing infrastructure in your team or in your company. Then comes the knowledge graph. And I guess this is the layer which adds the thing that matters most, in my opinion. This is the layer which is defining what is an ontology, what does it look like for your domain, how entities are merged and resolved together, like how do you define what is a coherent entity so that like you get the shared meaning from multiple sources, merged into thing which is accessible by different sort of users and personas.

8:03 So, this is the layer which is bringing lot of I would say standardization and a governance to that model once the data is associated with the model. And then comes the context. So, context graph is a I would say knowledge graph essentially that is put to work in put to work for grounding layer of AI and memory layer of AI as well. This is the I would say a I would say slight pivot from the knowledge graph where we are providing real-time context and retrieval that agents read and write from like into their systems. I think the key insight or the key thing that I want to mention here is that these are essentially a stack and not a I would say menu.

8:50 Everyone wants to start at the top which is the memory layer. But you are not thinking about the like layers underneath it which is like entities, how they are resolved, coherent then you're just kind of decorating or like building the top layer without building that solid foundation underneath. >> That is really great. Thank you. And you know, I've been trying to write blogs and blogs on this, but this is the best form of like succinct way of explaining to people how all of these connect. So, thanks so much for drawing that line.

9:26 Now, let's say, you know, the team settles on the data model, then we come to, you know, the what I would say are universal challenges because they are not unique to graph per se, the data drift and entity resolution. You know, in traditional setup bad data resolution will cause like a broken dashboard. But as soon as you stuff like multi-agent systems on top of it where there are dozens of agents reading, updating, and writing back to these, you know, these shared layers simultaneously, the data integrity problems can really like, you know, compound exponentially. So, how do you, like, you know, I know it's a really big topic, but like, you know, in in a sense, how do you solve entity resolution as a robust systems problem rather than, you know, just relying on LLMs to figure it out on the fly?

10:16 And, you know, how do we enforce that rule that you mentioned in terms of truth needs a receipt when multi-agent loops are generating facts on top of facts? >> Yeah, I think you definitely need the stakes that are there in this particular problem in a dashboard, a bad match is probably an embarrassing number, but in a multi-agent system when one bad entity matches with another, the impact cascade because every agent inherits that they hallucinate and sing, and more of them will agree on certain some wrong fact. So, yeah, I think in such a system, you do not generally ask an LLM to resolve entities on the fly as you said.

11:00 because that approach is non-deterministic, it is unorderable. It is like, it can drift from run to run, and you will not be building that confidence that you can kind of pass downstream to your systems. So, that's why I believe that entity resolution still in this new era in the context graph is still a systems problem and not a prompt problem or, I would say, LLM problem in general. >> Right. >> what we need to do is we need to build up infrastructure for a the entity resolution as well.

11:37 this could be offline. >> Okay. >> Like how do we merge and develop the algorithms offline so that the data that we have been kind of ingesting into our memory layer are we offline one and together? Even if let's say we need an online system, the auditing at least can be offline and little more deterministic. We need to apply techniques like blocking or pruning the candidates from thousands or millions of entities.

12:09 we need to have certain deterministic rules, certain machine learning algorithms which where like it owns its keeps and then we have let's say human as the top tier with for like very critical entities and entities which are very important from the business perspective. and all of this in my opinion still is behind a shared entity resolution service or a shared I would say router which decides what type of entities route to what kind of tier. Is it offline algorithm, online algorithm, human should be routed to a human for review or deterministic rule and so on. And it So yeah, this this is the basically intel- intelligence router which will probably route correct entity to the to the correct place.

12:55 And then the output of this entire entity resolution system is still I would say economical entity or canonical identity which is the contract that kind of keeps that fleet of agents from confidently agreeing to the wrong thing. Now, I think the second point you call you touched upon was the receipt or explainability. >> Right. >> comes in here. I believe that is also very critical in this agentic like in these agentic systems. And in like to solve that explainability, every fact should carry a provenance, like where does it come from? what was the rule? For example, when we did that entity resolution, which tier decided to pick like to do the entity merging and resolution in this case?

13:45 the confidence of our result or timestamp and many more things, like all the metadata that you can think that will be needed for explainability that should be there in this system as well. So that agents themselves can kind of reason about these things. when they for example, for example, when they look at some entity and they see that it comes from a source or provenance that is not highly confident, they will also try to I mean a good LLM model will try to look for another alternative source in that case.

14:21 So you are kind of explaining this to AI agent instead of human, but this explainability piece is still remains there. I think one very important thing in my opinion that is there in explainability part is that you need to keep the ground truth a separate agent generated facts. And that could be done by provenance. So for example, we should never we should clearly tag that this fact was observed or inferred by by an agent. And then we should like not auto promote promote an inferred fact without the same curation or let's say similar level of quality bar that we have for observed facts.

15:03 So provenance or this lineage is what is stopping you from laundering those fictitious things into empty facts and then stopping agents from merging fiction with facts. >> That's interesting. Okay, so would would you tie like all this information that you put in back into like you know what we were talking about context graph? Like is this part of how we are building up the context in terms of like you added okay this agent generated this this was the reliability of it. This is the time at which it was concluded. Is that or is that okay to think in in terms of the you know this is how we're building the context graph when you add all this?

15:44 >> Yeah. Yeah, I would put all of this data within the context graph. So, essentially when you have entities you have a lot of metadata associated with entities. Like how we reached this entities, which are the sources contributing to it, when was this created, which algorithm was used to merge this all the source entities into this canonical entity. All of that information does exist as part of a metadata which is like clearly tied to the entity. And when someone queries this entity, they can also like maybe there is an endpoint which queries the metadata as well to kind of reason on the explainability facts that we have.

16:23 Now, the question on where this metadata lives is it the same like context graph or is it a different endpoint? That is I think not that relevant. It could be either way. We can attach this metadata as like attributes of the same entity or we can keep the metadata separate in a different store and provide an endpoint or API endpoint where you pass the entity ID and get the metadata back. So, yeah, that question is really I think of less importance, but every entity has this metadata attached to it in some form within the context graph.

16:59 >> It's important to set up that aspect that you are actually noting all of these things down. And when you query, you do it in with that information. Whether it comes from the same database, another database, that's something that's under the scenes that agent shouldn't have to worry about. Got it. You know, this is the perfect segue into like that the top of the stack now, right? Like the living memory and cognition layer. and you know, you and I both talked about how this is like an active, evolving, and you know, unsolved sort of platform architecture question. Can there be you know, can there actually be like a horizontal memory substrate across an enterprise or does memory inevitably have to be built with domain-specific verticals? To put another way, how do you see central platform teams attempting to balance this? You know, is it possible to provide shared infrastructure and central cognition hooks while still giving distinct domains the freedom they need to own the business context or is it going to become very much like, you know, vertical-specific implementation?

18:04 >> Yeah, I genuinely think that this is an unsolved problem, so I will resist from giving a definitive answer and kind of share my candid and like open-ended perspective on this. I think a pure pure horizontal memory substrate across an enterprise is seductive, but in a way it tends to fail the way any horizontal infrastructure always fails, which is >> Mhm. >> it is probably too generic to serve any one domain's real needs on things like freshness, schema, latency, and it dies off like single owner fragility and like funding problems or other things.

18:44 Therefore, I guess I think a pure On the other side, if I think of a pure or vertical fragment or per vertical memory system, then every team will have to re-implement the entity resolution, provenance, governance, and you lose that cross-domain reasoning entirely. So, in my opinion, the pattern I would bet on is probably a mix where we centralize the hard universal primitives like identity and entity resolution, how does the provenance and lineage flow through the memory systems, the access control, the semantics registry and retrieval APIs, observability. All these things are taken care of the at the platform or horizontal layer, but then we federate the business context and domain memory to the different teams who actually work.

19:39 I'll I'll draw an analogy here. So, when when we think of human memory, >> Right. >> there pieces which are like common across all the humans. But, how does how does the brain work? What is a neuron? How does the information transfer from one neuron to another neuron in our brain, right? All of that is is theoretically like same thing. The concepts are same across all the humans. But, then every individual human has a different memory based on the experiences, based on the domain that they have been. And I would imagine a similar system where large chunk of I would say memory is in that substrate layer, but then still a lot of verticalization will will continue to happen.

20:27 So, this central thing provides, I would say, one more thing that I'll add is that when we are thinking of that horizontal layer, in my opinion, it is also important for that horizontal layer to provide very efficient hooks into different verticals. So, that when I am having my own memory based on my experiences, I can easily write it to my network system, my my neurological system. What does that path look like? That should also be defined by the horizontal layer. Same goes for, let's say, provenance, trust, governance. How that should look like, how business teams, how the vertical teams should think about it. That should also be defined.

21:11 At least defined by the horizontal layer, but then the horizontal layer should be flexible enough to a kind of accommodate multiple multiple verticals here. >> That's a great way to think about it. And you know, there are so many questions that I would like to ask, but in the interest of time, and perhaps we'll do another session when we start to have more of these answers too, right? For you know, just a closing thought for engineering leaders that are you know, working on these you know, trying to build these reliable AI memory systems today. What is the I guess the number one mistake they should avoid in their first year or like what is that you know, foundational thinking you would want them to keep with them as they start this journey.

21:56 >> Yeah, I think the one the number one mistake in my opinion is building that memory and agent layer before you you know that your data foundation is trustworthy. If your entities are not resolved, if your facts don't have that explainability receipt, then adding memory on top of it just automates the confusion. so you've made it like even more efficient to be wrong. So, I'd say resist the urge of just buying a vector database or a memory database and call it memory. Instead, earn a small governed slice of I would say truth first and then define those canonical identities, provenance. What does memory really mean for you? What are you going to store in this memory?

22:45 How will you expire that memory? What was the time to live? How will you make sure that the facts that remain in that memory are still I would say latest facts and are not overridden. So, all of it should be defined first and the method methodology how to how that will make into the system, memory system, and how it will go out of the system. All of it needs to be kind of understood first, in my opinion.

23:09 Then, understanding is one, but then proving is another fact. Like, after that once you have defined, how do you prove that this work? If you cannot prove it to yourself, if you cannot prove it to another human, it is really hard to kind of convey the similar messages to agents. And if humans are not able to understand that why something goes in, why this data that I'm getting out of the system is really trustworthy, is efficient, and trustworthy, then then agents who are retrieving this data back will definitely hallucinate. So, yeah, I think that's what I I think should we should start with.

23:48 >> Well, thank you so much. Thank you so much for walking us up the stack from the raw data foundations to how you think about building trustworthy agentic memory. And you know, it's an evolving space, so we'll hopefully talk again when we have more answers, and we build these and measure it for ourselves. So, thank you so much for joining me today. >> Thank you. Thanks for having me again, and this is an exciting space. let's yeah, definitely let's chat again when when we have more progress in this field.

24:17 >> Thank you.