Building an OpenSearch Research Agent at Scale - Alfonso Graziano, Nearform

0:00 Today we are going to talk about how we built an OpenSearch research agent. The use case in this case is somewhat something that a lot of companies have, which is I need to have clever ways to talk with a lot of data That I ingested over a lot of years from a lot of different sources and expose everything through an agent or a multi-agentic system, as we will see. I'm Alfonso, I'm a tech leader at a company called Nearform.

0:29 We are an AI native services company. We work across the globe. We are more than 500 people. And we work in a lot of regulated industries, fintech, pharma, a lot of cool stuff. Everything that I'm gonna show today is done by the The amazing team that I've led in the last six months. I can see a few people here in the room which worked with us in the last six months. So thanks, thanks to the entire team for the hard work.

0:58 This, what I'm gonna show you today, it's a very small portion of a bigger system that we built In this case we built this multi-agentic system, which is again just it's just a small piece of the bigger software that we built. But Long story short, what we built is a main agent, which is an orchestrator, which allows all the users, all the clients, all the users of our client to just go through the front end and then chat with every type of data.

1:31 What we are gonna see today, what we're gonna focus today is this data retrieval subagent, which is then connected to an OpenSearch proxy, which we are gonna discuss in a minute But then we also have a lot more capabilities. So we for example built a full customized RAG document search. We have plenty of specialized subagents for like very narrow use cases. And then we do have a lot of management tools with connectors. Everything is through our observability platform.

2:01 And then of course As you may guess, we spent a lot of time building evals for this system. So ideally if we if we got time later we're gonna talk about the evals as well. But let's focus on the architecture of this piece of system. So, the problem as I was mentioning is a problem that a lot of companies have, which on paper sounds relatively simple to solve. So ideally, as a user I want a user which asks a natural language question and gets the answer.

2:34 Easy, right? That's what we told. So the flow is: the user asks a question about about whatever, everything that the company has access to then the agent somehow should be able to understand what's in OpenSearch, understand what's inside the entire knowledge of the company Build an effective query which works and which you know gives the right result, of course, with a time budget.

3:08 We don't want an OpenSearch query which destroys the cluster, right? And then the agent builds an answer. This seems trivial. In practice, we will see that there are at least three main issues The first one, which I'm assuming a few of you have seen in the past or are still or are working at the moment, is that when you work on OpenSearch or other systems You know you have the mappings, you have those fields, but those fields are not carrying a meaning.

3:40 I mean we know what they are because we built the system, right? But let's assume that I do have I don't know a field which is about wealth management. I have a field which says net worth. The only thing the agent has access to is the name net worth. It doesn't have access to the to the meaning of it, to how that data is computed, why that data is helpful. So we need a way to give a lot of semantic information to our agent to work correctly and to give us the right answers.

4:15 Then of course the other problem is okay I have an index Easy. I can just put the I can just put the mapping into the into the actual context window. It's gonna do something, it's gonna build the query. Easy. What happens if I do have tens, if not hundreds, of indexes, right? That makes things a bit more complicated. And what if The index itself is so complex that the mappings themselves are too big for the context for the context window.

4:45 So we need a clever way, especially because of course we pay by tokens, right? So we don't want to just put everything into the context window. So let's start to talk about the first issue. Which works for OpenSearch, but it's the same if you're using a database, it's the same if you're using pretty much everything else, which is a data model essentially So let me give you an example. Market value, right? I do have this type double.

5:11 The agent reads market value. Now If I don't tell exactly to the agent what this market value is, the agent is gonna infer, it's gonna think what this market value is. But when I build this market value, this actual field I put a lot of my domain knowledge about it. So in this case, this is month and snapshot, not live, whatever. Right? So we need a way to give all this information to the final agent How do we do that?

5:40 We did something which is basically building a meta index. So every single field has as we can see here as the field name, the type, and then a full description. And this description is injected at runtime into the agent every time we need to carry the meaning of a field And this is useful both for retrieval and then also for the generation of the answer because the agent now exactly knows what these holdings, for example, is Let's look at the other problems though.

6:14 Because this was relatively easy for us to solve. But this is a bit more interesting, which is we have just too much information. We cannot put everything into the context window. So That's the architecture that we built. A small warning before I tell you about the architecture. This is something that worked for our use case, for our client. And it worked very well, but you know, in a different scenario you might end up doing something different.

6:41 So the architecture that we came up with is a basically two-tier architecture So we have basically two subagents. The first one doing progressive discovery. The second one building the query, running it, and doing self-healing. And we will look at them in a minute. But so the first one is basically a search agent which is trying to figure out okay, I have this plethora of data.

7:12 What are the fields and the indexes that I'm interested in? So that's the first step. Second step is given the right information, give it the right indexes. I need to build the right query and if the query is wrong, if it doesn't work, I need a self-healing mechanism which is which is building the DSL query and it's fixing it. So for the progressive discovery, what we are doing, first of all, it's very similar to the concept of the skills.

7:41 So what we are doing here, first of all, I am listing the indexes. So I do have a tool which allows us to list Everything, see what it does with you know a brief summary of every index. It tells us what's inside of each one And then we can based on the real indexes that we really need, now we can run a describe index, which gives us a better description, some fields, you know, it gives us more information.

8:08 Once we have that, we want to understand the shape of the data, right? So we do something which is called describe fields. So We basically try to tell to the agent, okay, I'm interested in those fields. Can you give me some examples? Can you tell me more about those fields? And so it's gonna give us the semantic description of all those fields. Eventually, we are getting some field values because for example let's say let's say I have an enum with 50 possible items.

8:39 I wanna get some samples because I wanna understand the shape of the data that I need to query. After we do all this stuff, eventually We come up with something like this. So we start with a very broad search space. So how much of my portfolio is in crypto right now, for example I do this discovery and I found out that I have one target index in this case, but it can be more than one, of course.

9:05 I find all the fields that matter for my use case, I find some sample values And now I'm finally ready to go to the execution phase. This is an example of the discovery prompt, and that's how we built a very simple agent with Mastra, of course. This is a simplified version of what we have, but of course everything that we have built so far is built with Mastra. Now the execution phase, which has been the most tricky to play with and to set up So we do a few we do a few steps here.

9:38 So first of all, once we have all this information, once we know what the search space is, we start with a very broad query. So basically we give The target indexes, you know, all the fields. We give all this information to this execution agent, and we tell it to start to run a broad query. We understand if we have hits and then iteratively we add the filters. We basically narrow down the search and once the agent is happy then it's gonna return us the business answer plus every search note.

10:12 I just wanna stress here why the semantic meaning of each field mapping is important. Because we wanna give a business answer. We don't wanna just give a number. We don't wanna just give like a few fields information, we wanna Be able to tell the story to the final user of what this stuff is and why it matters to the user. Then of course things might break, things always break, right? So in case something breaks, we have a few possible error cases, right?

10:41 The response is too large. We don't wanna float the context window. the DSL the query itself is broken somehow there is a wrong value that we are sending right so this is the self-healing loop that we have And basically the agent is able to fix itself iteratively. This is an example of our prompt To actually build the DSL query. If you are an OpenSearch user, there is a beautiful prompt template for query planning, which is in Java.

11:18 So what we've basically done We have taken all this information from the from the Java template, we ported it into Mastra, and then we did a lot of refinement to make it work a bit better on our use case because we found out a few edge cases where it wasn't working for us. But eventually we are just giving as much information as possible to the execution agent on how to run and how to build a great OpenSearch query.

11:48 Now Permissions is a very big topic for us and I'm sure that everyone here struggles with permission. So what we usually do is instead of our agent going straight to the OpenSearch cluster, we have a proxy in the in between. And basically, thanks to the Mastra request context we send every information about the user to the OpenSearch proxy. The OpenSearch proxy is basically doing a rollback access control, rate limiting, it has a rules engine It does session restrictions, it can change values at runtime, so it can do a lot of stuff for us, and then we only focus about building the agent and ensuring that the agent works Let me give you a very quick example of why the OpenSearch proxy is very useful.

12:36 Let's assume that I do have a user. Our agent has access to the entire OpenSearch cluster. But if our user doesn't have access to maybe one feature or maybe doesn't have access to a few fields, I can just tell to the agent, hey, by the way, those are all the information that you asked me for Right? But please, dear user, be aware that at runtime I couldn't give you access to, for example, the market value or whatever or any other field Because you don't have the permission or there is another reason why at runtime I had to remove that value.

13:11 So thanks to this technique we can actually make a few changes to the response before we give it to the agent, which is way more secure. That's how it looks like. This is a real world example of what we built. So we have here the first discovery agent and then we have the execution agent. As we can see the discovery agent is doing the list available indexes, is describing the indexes and is understanding everything, is describing the fields, getting some field values And then it's running the execution step.

13:43 So it's building the DSL query, it's running the OpenSearch query, and it's just going deeper and deeper until it finds the actual solution. And yeah, this was our architecture. Thanks a lot for it for your time and attention. - You're wearing this open source shirt, and you've got some other folks at Nearform wearing it. - Absolutely. - What's the meaning behind that? - So actually Nearform has a very long history of contributing to open source.

14:15 We actually contributed to Mastra as well, with a with a few PRs, which is which is great So every time we do client work, every time we support our clients, we try to contribute back to the open source ecosystem. We have a very long history of contributions to Node.js Core. Some of our like some of our packages I guess that some of you might know them, like Pino, Fastify. They were born at Nearform, and then of course are free for everyone to use.

14:45 So we really up love open source. - That's very cool. I love open source and that's a beautiful t-shirt. Thank you again. Thanks. Thanks a lot.