Make Your AI Agents Predictable With Graph Engineering - Zachary Blumenfeld, Neo4j

0:12 we're going to talk about graph engineering today. How many people have heard about that term graph engineering? Okay. So so a fair amount of the room and then just so I understand how many people here are familiar with Neo forj and have used Neo forj. Okay. So we got a few people. So that's good for just letting me know who we're talking to today. so just some history on some of the verbiage that's being used. there was a tweet that went out by Peter Steinberger I think earlier this year in the summer that talked about graph engineering and it was almost like is this sort of a new discipline that's happening as this evolution from prompt engineering and context engineering and there's a lot of hype around this and the name for this is new but the mechanisms are not really new we've always known in computer science generally about state machines and workflow graphs and knowledge graphs and all these different things. What's changing is really how we think about them in reference to agents. and what I'm going to talk to you about today is just how we think about graph engineering. And I'm going to offer you a little bit of a new perspective on this. So if we actually try to define what graph engineering is, I think a good definition is it's just the practice of designing agent work as an explicit graph. So where nodes do work and edges effectively route state between them and I tend to think of it in terms of three different types of graphs. We have our general execution graphs which you're probably very familiar with. So these are the workflow graphs with routers and agents as actual nodes that happen with inside of an agent harness. But there's other ways that we can think about graph engineering too that overlap with this.

2:05 So that includes context graphs. These are decision traces, instances of what's happened and how it connects to other data inside of your domain. And then there's also something that I'd call more procedural graphs. So this is almost like having a skill or procedural knowledge as a graph where the steps might not be actual agents, but they might be steps inside of a way of doing something basically a procedure. So different calls, maybe a step for a decision model, steps for different agents, whether to escalate, validation steps, and these sorts of things. And I'm really going to focus on these last two because I think we don't talk about them enough. And I think by understanding these, you can actually get a little bit at what the first graph is trying to do as well, but in a very portable and efficient way.

2:57 So graph engineering lets agents really take on bigger more complex work and that's why we care about it. And really if you were to split it up I think in terms of the benefits that a graph offers agents in this context it would be predictability. So being able to constrain a path that an agent takes making sure that the the answer is basically accurate from a task that it's doing. and having an explicit graph, the ability to govern, control and guide what the agent does and audit it so you can understand what it's doing and make it better next time. And then the third one which is going to be interesting that we'll talk about is how do we actually make graph engineering portable and instead of thinking about a graph as something that happens in the runtime it could be something that you can store and actually move between different runtimes and have inside of a knowledge layer which gives you more interoperability and allows you to have a very simple stack. and in the runtime when we think about the graph it's really like a piece of orchestration. So we're probably all of us or a large portion of us are probably familiar with Langraph ADK and all these other agent frameworks even to a certain extent anthropics multi- aent research and and some of their stuff fits inside of graph engineering and what I'm going to talk about today is actually what happens when we take that sort of graph and we put it next to our data as context graphs and some of the things that I was talking about before. and really when you do this, as I was saying, you add this portability layer. You add the ability to take all of this knowledge and make it work basically with any type of runtime. So that's what I'm going to go over today. and this is what we're doing at Neo Forj is we're defining this knowledge layer. And this knowledge layer sits between systems of record that you would ordinarily retrieve data from. So, data bricks, bigquery, snowflake, these types of places. It could be documentation. It could be obsidian, these these markdown files.

5:00 and then can we create standard protocols and artifacts that live inside of that layer that then you can take a graph engineering process and make it available to any runtime or any agent. and the way our product works, just so you know, at Neo Forj, is we have our core database and then we have a ton of other AI stuff on top of it. so the database and the analytics layer. I'm going to be showing some of this throughout my talk. but what I'm really going to be focusing on is this top layer where we talk about sort of this agentic brain or these other memory tools that wrap around the database. So, just know that as I go through, I'm going to be showing you data being loaded into the database. That's going to constitute our knowledge layer. It's going to come from multiple different sources. but a lot of the sort of extensions of that tooling and some of the experimental things that I'm going to show you exists as sort of this graph AI that hooks into that and loads data in and out of that thing.

6:01 so what we're doing at Neo Forj and kind of how we're thinking about this, we start with this concept of a context graph which has different types of memory from short-term memory which is going to be like your conversation history, your long-term memory being entities that are extracted from that or from your data and then reasoning or decision traces inside of that. And one of the interesting things that we've been looking at from a research perspective and this really started out not so much at that graph but actually at individual skill level. There was a paper that we wrote earlier this year that we published in a workshop and we were really trying to ask the question can we take procedural knowledge whether it be from past decisions or even something like a skill like we have here and compile it down to a graph. And what we came up with was a process of taking written material, often a skill MD file and the scripts and the references and assets and moving that into an actual graph structure, which is what you see on that other side. And the result of doing this and giving the agent the access to that graph structure which has different nodes again representing inside of the scale decisions, different routing points, different scripts that have to be executed. taking pros and compiling them down to to deterministic scripts. When we do that and we force the agent to sort of run that graph, what we see is increases in task reward and performance. So this was from a while ago I am still working on updating it to the latest models. but we were able to use a smaller model in this case I think it was sonnet. and now we're we're on to haik coup as a matter of fact. we were seeing even for really challenging tasks over diverse fields in physics and computer science and various security physical training a whole a whole ton of things is this overall increase in task reward performance when we were able to switch over to this and our PMS Will Loin who works in our labs team actually took this idea and said well if we can do that with skills maybe we do it too with these memory graphs. So you can actually take that context graph that I was showing you earlier and distill a skill from that and have it still be connected in the same graph over to all of your past decision traces and data. And so then you get those performance benefits, but you also have all this provenence that goes back from every step. It's versioned, it's tracked, and you have governance over time where you can assess drift. you can understand where certain tool calls are coming from and their history. A human can go through actually inside of NAM's Neoj agent memory service which is our reference architecture for this. There's a QR code there and there will be another one at the end and be able to understand like exactly how this works, edit it, change it, do all sorts of cool stuff with it. so again, what we're talking about now are these artifacts that you can move around. I mean this thing makes a skill, right? And so any agent harness that can take that skill can now be running these graphs and doing all of this logic.

9:21 and we have various customers that are using us for graph engineering types of tasks. I'll go quickly because we're we're starting to run out of time. and even though they're not may not be using exactly the labs project that they that we have we have consultancies which are working with other asset managers for tough problems in compliance. so being able to write audit reports for example where you have data that might come from multiple sources. How do you automate that but do it in a way that's governed and trustworthy? So you have compliance documentation. You might have entities with different ticker symbols and names from different databases and those all need to be resolved. And so giving your agent these graphs that express the logic that you need to do on besides the data basically allows you to automate those things in a trustworthy way.

10:13 and we have other people such as Uber that do this with configuration graphs. so basically if you have you can imagine if you have drivers across the country compliance and regulations for different areas geographically vary greatly and they come from this sprawl of microservices that you need to pull data from to figure out like okay well what is it for each area for each component of a car for example and so being able to express that logic and have it sit aside the data also helps them operationalize some of these things effectively so I'm going I'm going to jump to a demo now. It I'm going to warn you in advance. It's going to be quite a lot and I'm going to jump through it kind of quickly. but it's very interesting.

10:56 And what I'm going to show you is how we can use that AIP library that I was talking about before to compile a skill and actually get a task that was not governed very well and took very long to run. Basically cut its cost by like 100% and then run it a lot lot faster. and of course it's a it's a demo. It's not a formal research proposal, but you can go there and look at the code. It's all it's all very transparent. so this is going to be an example for a loan a loan approval process. So inside of a loan approval process, there's lots of policies that have to be followed. Obviously, I'm not a financial expert, but many of you in the room are. and so there's probably going to be a policy dock. There's going to be a systems of steps that have to be followed exactly. And then you need your agent to be able to explain those steps very clearly, articulate them, and really get the answer right as if it was a human reviewing. and the first thing you can do, like I was saying before, and I don't have time to probably run this all live, and the internet's going to be slow, so luckily I I have this preloaded. and basically the question that was asked was, should we approve this new loan for this Rosa's Bakery business? And in this case, you know, we have a business plan that's in a PDF that was uploaded.

12:17 and that is, let's see, the business plan is the orange data. So, if I was to scroll out here, you can see it here in the graph that connects also to data that we pulled from data bricks, from past transactions, lending policy data, and past decisions that were actually made. And with that, the agent can crawl over all that information and basically realize that this person either had a loan before or there was a mistake before or something happened before and come up with an audit trail for that and basically post a solution very very quickly. and this is a great start just in terms of like resolving data and being able to follow policies and understand how things connect.

13:01 this is great. One of the issues with this is, and I can show you a graph kind of of what this looks like. This is an example of me running more of these policies. and if I zoom in, I have a conversation here and then I have my my reasoning trace that exists inside of this graph. And all these steps are timestamped and they have tool calls. what can happen is if I look at this happening twice, I think this is the right query.

13:32 this is this is two decision chains, so it's going to be a little bit hard to see. Let me zoom in. they they're of different lengths, you'll notice. So, one is longer than the other. in this case, it was cuz it found a president in one. basically something else that had happened before. But the the problem with this is that if you run these over and over again, there's nothing actually like enforcing that the agent follows an exact path.

13:59 you might have a skill or something that will say like do these things. but then there's other problems where it's like there's these decision steps where it's approve or deny is a simple one or approve it or send to a human to review and those decision steps as well come with confidence scores or you say like if you're confident about it if you're not confident about something but those decisions aren't calibrated. So it would be great to like even take some of this and send it to something like a type- safe, you know, Jev type of model as well, to be able to do some of those things. so what you can do is you can scrape all of this and you can compile it into a graphshaped skill, which is what we did here. and that graph shaped skill, if I go here, you're not going to be able to see it. but that's okay that you don't see a lot of the detail here. The important part is that we sort of compiled all the information down from those successful runs that we did, even though they were a little bit inconsistent, and we put them into this graph YAML shape inside of this skill. And this skill, even though you probably have trouble seeing, has scripts. It has assets here. It has sometimes other reference resources. It doesn't for this one. it also has source documentation around the policy that you're supposed to do for the loan. So there's so there's basically a ton of stuff in here for traceability.

15:18 but then also telling the agent how to run the skill. But the great thing is once a skill is in this YAML format, we can port it into all this different tooling including inside of a graph because it's all sort of this deterministic format. And in a graph it looks kind of like this thing here. So let me actually let me just run this run this sucker again. this is kind of what it looks like. So you have all of these steps out here and then basically what I just did is I wrapped a small CLI around this so the agent can say hey run the skill and instead of the agent deciding all the routing steps it's just done programmatically we took a lot of that policy down and we put a lot of it into these execution scripts. basically what's happening here if if I was just to sum it up, there's an execution that basically looks at, hey, did do we meet certain hard rules around policies? after that, it's routed if it's not auto approved or denied based on that, it's routed to a decision model like Jev or Typesafe. and I can even probably show you a little bit of a clearer view here. and it's given that that model is given like a risk tier choice.

16:37 let me make that a little bigger. Hold on. whether or not there's there's some exposure material and then like an outcome of whether or not you're approving or denying. And then the rest of this will basically follow down into writing a report, whether or not you need to escalate to a human review as this router. And then if not basically the agent's going to write a report and there's a circulation here to sort of check and lint that report.

17:05 so once you do this and you sort of get the model going where okay now we have this CLI that we can just run this thing in. well now I can go in here and then maybe I can probably don't have time to run this whole thing again but I will give it a shot. I'll see if it goes and picks this up. It takes about 2 minutes to run. but the idea I guess while this is running and it should pick up the AIP skill which which it is here. once you have it in this format, not only can you have the agent run this very quickly, you can track the runs. So for example, if I was to look at a single run, can run this again, I could basically persist the runs inside of the graph.

17:55 so this is a run of that skill with like individual steps that it ran. and then, I can just persist these over time. I can examine how well they work. So from a governance perspective, I have a little inspector here that I built really quickly. You can look at the status of all these. You can look at how a specific one ran, what was approved, what was denied inside of the process, what branches it took. and then once you have a graph, you can even do these cool observability things where you can ask, well, like what branches were taken and which branches weren't taken. which is very interesting from a governance perspective, failed ever. all these cool different things. And all of that is stored besides your data. and so in none of this, we're not like creating a loop right inside of our agent. All we're doing here is basically giving it a CLY to run this like programmatic skill that we compiled from documentation or in this case from our memory graph. and this might take a while to run just because of the the internet.

18:59 But what I will show you is I we ran this a few times. and what we saw is just running I was using Fable. So my organization already yells at me for this enough spending a bunch of money. but like just using Fable alone on the graph and it was correct you know pretty much every time but it took 2.7 minutes when we had the procedure it it almost cut in half just using AIP and then because we had the procedure we can use a simpler model like Haiku and then we can cut the model time down even more and then of course you know the cost just goes way down. So because you're taking all of that load, right, of having to figure out all the routing, all the stuff you're doing, you're putting into these execution scripts, these Jev models and everything.

19:43 which makes everything just much much more efficient. and this is going to take too long to run. it's mostly internet. You can see here about half a minute or 2 minutes wall clock on average down from like 7 and 1/2. All right. So anyway, I'll end it here. So, I want to give you some resources to check out if you're interested in this type of stuff. Aura is our cloud service offering where you can get the database that I was showing you before. You get $100 of free credits. We have nodes coming up which are is a speakers event.

20:16 There's a startup program and then all the code that I was referencing and the other projects that I was using to hook up to your agent memory. All that sort of stuff is down there. So, I'll leave this up for a second until someone kicks me off. And I guess that's it for me for now.