Building and Testing Reliable Agents

0:00 hey this is Lance from Lang chain this is a talk I gave yesterday at the AI engineer World Fair and I wanted to get on here and just kind of give it again show all the slides and also show kind of a quick code walkr of everything I talked about and the theme is architecting and testing reliable agents so LM applications follow a control flow you have you typically start the application with some kind of user input

0:22 it follows some set of steps and it ends a chain you can think of is just a control flow set by the developer so it follow some set of steps the developer determines ahead of time and a rag chain is a good example of this question comes from the user you hit a vector store you retrieve documents related to the question you pass them to an LM the LM produces an answer but you return to the

0:45 user now an agent you can think of in simple terms as a control flow that is actually set or gated by an llm instead of the developer alone so in this kind of toy example you start your chain you pass from step one to an llm which then reasons about do I do step one again or do I move forward to step two so again chain is developer defined control flow that's fixed an agent is just an LM defined control

1:09 flow now typically when you talk about agents you kind of also hear about this idea of function calling or tool calling and really when you think about it how does an agent make a decision about what step to go to so we can use tool calling to actually select a step or tool that we want to use so here's an example an ele M receives a natural language input what is the output of step two and you

1:34 pass three to it we can actually bind step two as a tool to our llm and the LM can return then the payload necessary to actually run that tool called step two and again it pass it Returns the arguments for example from the user input the argument of fu is going to be three and the name of the tool is going to be step two so the way to think about tool calling or function calling is basically

1:58 you're giving the llm access to some set of tools allowing it to choose a tool to invoke along with the payload necessary to run that tool based on the user's input so that's really what time of tool calling is and a popular approach is just to run this as a for Loop so that as you give an LM access to some set of tools and you allow that llm to call Tools in this kind of cyclic fashion so

2:22 you take a user input the LM decides to call tool based on that input the tool runs you return the output of the tool back to the LM then the LM thinks about it and says okay I need to run another tool and this will continue until the LM finally says I don't need to run any more tool calls I'm just going to return a response directly to the user and then you end so this is the classic kind of

2:42 react style agent setup that runs more or less as a for Loop and the nice thing about this is extremely flexible so you can Implement many possible control flows with this type of agent it can run step one or step two only one two or 21 and flexible agents have a lot of promise for open eded tasks for example a lot of hype last year about different projects like autog GPT baby AGI and these were kind of open-ended agents

3:08 that had access to many tools and could perform kind of arbitrary control flows based upon those set of tools to solve open-ended problems but what we saw is and many people reported this mixed reliability with open-ended or flexible agents and the problem often boils down to llm non-determinism in particular problems with tool calling so in this particular toy example again this is basically passing a tool step two to our agent you can either get the wrong

3:39 payload so basically improperly format the input based upon the user's request to the tool so in this case the input is three and we pass four or you can call the wrong Tools in this case the user ask for step two and we call step three so these are some of the challenges with tool Clin that have been widely reported so if you think about it you can kind of Imagine a Continuum from on one hand you

4:04 have chains which are not flexible but they're very reliable on the other hand you have like an open-ended or like kind of reatile agent which is very flexible can Implement any control flow given the tools you provide but it's less reliable so can you have something in the middle that's both flexible and reliable that's kind of the problem statement and that's really the intuition for Lang graph and I'll kind of set set up in two different ways

4:30 so one part of the intuition is let the developer set parts of the control flow okay so let's say I always want to start with step one I always want to end with step two and let the developer interject the llm in certain places to make it an agent and make it flexible so in this particular case I want to interject and have the agent always run between steps one and two and line graph allows you to express

4:55 this as a graph so again you can express your steps as Gra nodes and you can ex these the the connection or potentially the conditional connection between those nodes as an edge and a lang graph agent just has a few components which is consistent with any kind of agent in terms of memory it has a shared state that lives across the lifetime of the agent and that any graph node can access tools can be called inside the nodes so the nodes can call

5:23 tools and importantly the nodes can modify the state so you're passing the state along to each node the node can call a tool and then you can write out the output of the tool to State and the edges can route the control flow based upon LM decisions to whatever node you want so in this case we can start with node one we do some tool call inside node one the llm then reasons about what to do next and it

5:47 decides either to go back to one or to proceed to two but of course you're getting that control flow you always want to start with one you want to reason about one versus two let L make just that decision and once you finish two you're going to end so we've seen this theme quite a bit papers have used this for routing so this is kind of a nice paper called corrective rag that does retrieval grades the documents for relevance to

6:12 based upon the question and if any documents are irrelevant kick out and do a web search to supplement your documents with more recent results from web search and then perform rag based upon those the mixed results we've seen other papers use the theme and self-reflection We've implemented this in L graph as well so this is kind of a case of a longer Lang graph that uses routing at the start to take a question route it between retrieval

6:37 or web search do retrieval do grading route do web search generate your answer based on the results of web search and your retrieval now we also incorporate grading of the answer relative to the documents for hallucinations and we can also do a sity check on the answer relative to our question so that's kind of the overall flow that we can Implement we've done this in Lang graph we shown that this works locally and another paper uses this same

7:07 theme in it it problem solving like coding the idea here is simply to produce a code solution run a set of unit tests if any unit tests fail pass those back to the llm and have it try again we've also implemented this in Lang graph this is a collaboration with mistol in this particular case we generate the code solution we do a number of unit tests we check for example like ports do they actually work we ran this on a bunch of questions

7:33 related to Lang chain expression language and we found we can significantly improve performance relative to just naive rag so again if we kind of step back a chain is the is not flexible but the most reliable a typical react agent runs as a for Loop most flexible but it's the least reliable and implementing an agent in Lang graph actually kind of balances the two in a pretty nice way it can be quite flexible and it can be reliable

8:00 we've shown that in a few different examples in particular we've shown you know in cases where you can even run with with we can run a relatively complex rag flow with local models you can run these kind of corrective rag flows that can run very reliably all the way down to running this with local models and you can run these kind of iterative code generation flows really well with Lang graph as well so I want to jump over and actually show this

8:25 what I want to show is let's try to build an agent in two different ways let's build an agent with both react and with Lang graph and let's compare them directly so I'm going to go over I have an empty notebook I've just done a few pip installs I've set a few environment variables and I can go over here and let's build a react agent first let's actually show in in detailed terms what I'm actually talking about here so first

8:50 I'm going to set up a few tools right you think about an agent agents very simply require some number of tools I'm just going to reset my kernel to make sure everything's fresh here so what I'm doing here let's just kick this off cool so I am defining a vector store I'm indexing four blog posts and there's my Vector store and here's my tool this is going to be my retrieve documents tool so I'm going to

9:19 retrieve documents from the vector store based on the query I'm going to have a greater tool which is basically going to grade the documents for relevance relative to my question I'm going have a web search tool and I'm going to have a final generation tool so this tool is basically going to generate my final answer so if you go back to my diagram that's exactly what we show here we're going to build an agent that has

9:40 these four tools so there we go now let's just build our assistant so this is like kind of the core part of the agent's planning so all I'm going to do is just copy this over so what's happening here is I'm setting a model so for now I'll use GPT 40 this we're going to see comes in a little bit later the assistant is really simply a class that's going to run my llm and here's kind of the main idea here I have my LM

10:09 I'm binding the set of tools I just created up above and here's my prompt your helpful assistant TK with answering questions provided to a vector store grade them to ensure that they're relevant before answering the question so that's really all we're instructing our agent to do and we're done so this is run now here's where we can just set up our agent this is just where we're setting up the connection between the components of our agent so I'm just

10:39 going to go ahead and run this I'm actually using Lang graph to implement this react agent and here's all it's going to go on I have my assistant which has the bound tools the the assistant will look at the input it'll make a decision to call a tool the tool will be called that'll go back to the assistant it'll make another decision to call a tool or not and this little Loop will continue until the assistant does not

10:59 make a tool call and then we'll end so that follows exactly what we talked about here it's just going to run this Loop until there's no more tool calls then we return the answer so it's a really simple little corrective rag agent and we expect it to follow the flow that we talked about in our slides where what's really going to happen is it's going to follow something like this we wanted to do Vector to retrieval

11:20 grade the documents if any are relevant kick out and do web search then finish that's really all we wanted to do so there's our agent we built it and we can go ahead and try to run it so here we go so now all I'm doing is I'm just writing a little function that'll basically wrap my agent I'll pass in an example it'll run basically run my react agent this react graph is my agent I pass in my example question

11:51 here's where I set my example question what are the types of agent memory so one of the my blog post was related to agent so I can just pass this in as a question and and we can see so here we go we that agent ran and here's our answer the types of agent memory include sensory memory short-term memory long-term memory and here are the tool calls it ran retrieve documents it graded them and it Jed the answer so

12:13 this looks really good we're happy we have a simple react agent we have a way to kind of log the tool calls it's using I can also open up lsmith here and I can go ahead and make sure I'm in the right tenant here let's open my personal project let's open up this and yeah this should be my L graph this just ran so I can even open this up and we can even have a look at the

12:36 trajectory so we ran our retriever that's our retrieve documents Tool we went ahead and did the grading we generated our answer so there we go we've logged our tool trajectory here and we also have the trace in lsmith so that's kind of step one we have a react agent we've run it now let's show a different way to do this I'm going to build this as a custom Lang graph agent so let's copy over a little bit more

13:01 code just grab this so This is actually where I'm going to lay out a prompt so this is going to be a prompt simply for my rag chain so it's just a rag prompt right here and here is just where I'm going to set up The Grater so I'm going to set up a really simple little retrieval grader right here and I'm going to use same logic we showed above I'm setting the site I'm setting up this

13:28 this up slightly differently and you'll see why in a minute so there we go now here's where I'm going to lay out my graph slightly differently let's copy this over nice run this so here's how I'm going to set my graph up more explicitly as a custom line graph agent so remember our call above with react all we are doing is cycling between two nodes basically we were we can actually go up and look at the diagram here so we

13:59 had an assistant the assistant makes a tool call and we go down we call the tool we return the tool call to the to the assistant and this goes back and forth now this relies if you go back to our slides this relies very heavily on high quality tool calling so this piece so again a simple react agent because we've just bounced some number of tools the tool calls need to be accurate and the agent needs to make the right

14:23 decision about what tool to call at each Loop okay now with this custom agent it's a little bit different if you go down here the the basic the sequence of calls is enumerated much more explicitly I'm always going to retrieve then I'm going to grade based on the grade I'm either going to do web search or generate directly and I'll pass whatever codes from web search into generate so you can see I'm laying out the control

14:47 flow more explicitly as a user I'm still letting the llm make a decision at this kind of Branch point or Edge based upon the grading so that's a subtle but important distinction that you're going to that we're going to see between the react agent and our custom agent so I've gone ahead and set my agent up here and let's go all the way down and let's go ahead and see it running so here we go we'll kick that

15:12 off just like before we the types of agent memory and hopefully we don't see too much of a difference in terms of the outputs okay so here we go this is the output so again we get a a sane response and we can see our steps logged here it goes to retrieve documents grade them it looks like it runs a web search because at least one of the documents was irrelevant and we get a generate answer

15:31 so that's cool so now we have two different agents we set up a custom Lang graph agent we set up a react agent now what I'm going to do is I'm going to show you the difference in reliability between these two and for this I'm going to do some very simple evaluation I'm going to show you how to do this exactly and we have a number of cookbooks available which I ran through at the workshop as well that'll show you

15:51 how to do this kind of testing yourself so first I'm going to do something really simple first I'm going build a data set of question answer pairs that are related to my agent so I want to do is I want to test the ability for the agent to retrieve from my Vector store and answer questions about my Vector store as well as answer questions just about current events so this corrective agent should have the ability to first check the vector store if any

16:17 of those are irrelevant kick back and do web search so two of my questions are just current events questions NBA Finals NFL draft three are related to my Vector store I'm going to create and save this as as a new data set called corrective rag agent QA now I'm going to evaluate the answers that come from each agent so this is basically using an LMS grader and you can kind of we have a bunch of videos talking about this explicitly so

16:45 you can kind of dig into those but here's what I'm also going to do I'm also going to grade the reasoning so basically these are the trajectories I expect my agent to take retrieve documents grade search generate or just retrieve grade and generate and I can basically just check do the agents follow the expected trajectories of tool calling so that's it I've set all those up and now what I can do is I can run my evaluation right here and I can

17:15 go ahead and kick that off great so those valuations both ran now I can see something really interesting right away so let's just kind of back up what did I do I have a little eval set I set up here with five questions inputs and outputs right three two are about current events three are about my Vector store and I tested both the react agent and our custom agent with gbd4 against

17:48 this little data set I measured two things I measured are the response is accurate so that's this answer versus reference score and I measured the tool calls are the tool calls in the exact order we expect so the second is like a a measurement of like the reasoning trace of the agent and the first is the measurement of like the output or like the endend performance of the agent so if you go back and you can see

18:13 something pretty interesting if you look here so basically I just really simply logged you can see in my in my evaluation function here I just loged the tool calls now let's look at react we can see here's something kind of interesting now a few of these look completely fine it'll retrieve the documents grade them generate an answer that's totally fine and one or two it retrieves web search gener answer no problem but look at this what some of

18:39 these other cases we retrieve then we jump right to web search we forget to grade again retrieve jump to web search forget to grade and generate right so you can see there's kind of variability in the reasoning trace of our agent with react if you look at our custom agent it's crisp every single time it follows the same trajectory now if we pull all the way back let's go back to original slides here what is the

19:04 intuition here the intuition is that if I kind of look at the the comparison here the tool calling react agent can follow an open-ended progression of tool calls given what's bound to it so it can set any control flow at wants given our tools it makes it flexible can be good for solving hard and open-ended problems s but if you want something very reliable and crisp and deterministic in production that can be a bad thing in

19:35 contrast kind of a lang graph custom agent allows you to set the control flow in a very specific way and that's what we see playing out right here so the control flow always follows the pattern that we want now of course I understand this is going to be less flexible if you have a really open-ended use case you need the agent to produce you know arbitrary kind of you know or like you know complex reasoning traces arbitrary

19:58 reason this might be a bad thing but for a lot of kind of production applications where the control flow can kind of be enumerated ahead of time this is actually this is actually a significant benefit in terms of productionizing we can actually see differences here our react agent scores 80% our custom agent scores 87% in the end to end performance and again this using the same llm same Vector store same set of tools just a different

20:28 configuration of those tools and in terms of tool calling as we saw before 87% for our react agent and 100% for the tool calling agent so again more liability in terms of the kind of control flow that's enacted or the reasoning Trace enacted by the agent and better endtoend performance using custom agent versus react so this is just kind of the highlight of what we talked through at the workshop and gives you some intuition as to why building agents

20:57 using Lang graph can can be a very nice way to balance between flexibility and reliability and the ability to lay out control flows kind of that combine The Best of Both Worlds so I can combine a lot of what the user wants in terms of reliability and like I want to be able to set a path that I want my agent to take and roughly guardrail it but allowing for some variability or flexibility so for example I want to

21:20 inject specific places in my control flare I want the llm to make different decisions and that's exactly what we show in our particular case of corrective rag where I do want that the agent to have the ability to kind of Reason about oh if my retrieval is bad I want to kick out and do web search and that's exactly what we see here where the agent is able to correct a number of cases or produce correct responses

21:44 number of cases where the answer question and answer are just current events not related to my Vector store so that's kind of an overview of what the the workshop discussed and hopefully it shows just like a nice demonstration of the differences between different kind of cognitive architectures for agents building a custom agent in Lang graph versus a classical kind of open-ended react agent and highlights some of the difference differences between the two ways of approaching the problem

22:14 thanks