Agent Response | LangSmith Evaluation - Part 24

0:00 hey this is Lance from Lang chain we're continuing our Lang Smith evaluation series now talking through agent evaluation this is one of the most requested topics that we've heard so I want to walk through this carefully explaining first what an agent actually is and then how to think about evaluating it and we'll probably walk through the different approaches for evaluation in three different videos so here's the starting point what is an agent there's a great blog post from

0:22 Lilian Wang that kind of breaks down the core components tool calling memory planning all right so that's step one now how to think about what is this thing called tool calling here's a really simple explanation of it all you're doing is you're having an llm basically return to you the payload of a tool that it can use okay so that's all it's going on let's give an example here I have an llm I'm going to define a

0:50 tool called Magic function and all it does is takes an input adds two okay so in L with this nice little decorator called tool and allows you to basically convert this into what we call structure tool and you can bind it to llms at support tool calling and here's the key point when you bind it to the LM the llm then given an input so what is Magic function 3 it can recognize hey I need to invoke this tool and here's the key

1:17 point this is often the most confusing part it just returns to you two things the tool to tool to call and the payload or the arguments to run the tool now it doesn't have the ability to match magically run that for you right again it's an llm it's string to string but what it gives you is a tool selection and a payload that's all you need to really know that's all that tool calling is so this could be really anything this

1:41 could be a really complex tool it's a simple tool that's a key Point you're getting the tool name and the tool arguments out and it's inferring those from the user input so that's like step one right and now agents use this particular tool calling step typically in a loop so Lang graph a really nice way to build agents not the only way but let me show you an example like here's how this whole thing would work here's

2:06 my agent I've stru I structure this in Lang graph in Lang graph you have nodes and edges so I have two different nodes my first node is my assistant that's my llm it sees my input just like before it has tools bound and it returns remember all it can return is just like a string in terms of like a raw response or a tool message which is essentially another string with like here's the tool I want to use here's the tool invocation

2:32 we have this we have basically what we call a tools condition node that will automatically look at the LM response for you and determine is it just a string response out or is it a tool call and if it's a tool call then basically all it happens is it takes that tool called payload it passes to this other node called The Tool node which basically has our two tools it looks up the right tool based on the specified

2:54 name it passes the specified input to the tool and you get the tool response out and we send that back to the LM now this keeps running until LM has decided okay I'm not going to call a tool anymore I'm just going to respond directly and then the tool conditions no just returns so in this particular case let's walk through it one Loop what's Magic function 3 llm says okay that looks like a tool call I need to use my

3:16 tool magic function Returns the tool call with the magic function name and the argument tool node gets that tool node says okay run magic function runs it with this input three you get the result five out it passes that back to LM as a tool message hey the tool output is five LM looks at that and says okay Returns the string the result is five that's all that happens super super simple basic agent explanation right so here's the interesting thing

3:45 how do you think about evaluating this thing well here's the way we've broken it down this is our conceptual guide and there's like kind of three different really intuitive ways to think about this first is final response this is just like end to end is it doing what we want to do so in this case the final response would be like looking at does it return five I don't care how many Loops it goes through does it return

4:06 five and you know if you think about that's just a string to string comparison typically we can use existing evaluators like we use for rag answer response easy stuff right so you're just looking at end to-end response you don't care anything about the agent process okay so that's kind of like one way to do it now another way to do evaluation is like digging in a little bit like looking at a single step of the agent

4:27 like here's a good example if I pass this input does it actually like you know want it does actually invoke the right tools it like make the right decision so for this my output would be like my evaluation output we'll talk about this in detail later will be like the tool name okay so you could just evaluate that you could say okay if I have this prompt I expect this tool to be called and just do an evaluation

4:51 there so that's like evaluating one step of an agent right and then you can also think about doing kind of the same idea but for many steps and so let's say this is more complex agent it has to invoke like both of these tools right you could have your reference be like you know fun magic function then web search Okay and you know and then in case your evaluation would basically look for there's a couple ways you could do

5:16 custom evaluator there you could look at the exact sequence of tool calls you could look at any sequence of tool calls you could look at like you know if it's close count it you know better versus far in terms of like the trajectory taken anyway we'll talk about all that later but the intuition is simply evaluating a whole selection of different tool calls so again evaluate final response evaluate a single tool call or evaluate many tool calls that's

5:39 like the simple minded way to think about at least three ways to look at agent evaluation now let's go ahead after that Preamble let's go to the first one so here's a notebook i' I've defined an agent so I'm using this chinuk DB this is a popular SQL database and so I'm going to build a SQL agent okay it's going to be using chinuk DB here's like the flow again just looks exactly like we just saw except in this

6:04 case my tool is SQL database so here's where I'm just basically going to pull in this is our existing SQL tool okay this's a whole bunch of SQL tools for working with SQL databases I'm going to find one or two custom tools this like check query tool this is going to basically check if the quer is correct I'm going to add one other like check result tool this will check if like the result from DB is like not empty does it

6:29 make sense so anyway there we go I I've defined some tools now with Lang graph you defined agent State and if you want more details on L graph I'm going to link a few videos to talk all about Lang graph agents that's kind of outside the scope of here but this is assuming you kind of know how to build an agent okay because you're doing evaluation so this is my agent State now here is just where I'm

6:51 like basically defining my agent what you might call the agent runnable or agent assistant this is basically my SQL prompt okay so this is basically telling telling the agent what you're going to do you're you know you're going to be interacting with sqle database you're going to be you know quering it looking at the response and then answering you know answering the user question and again there's a whole bunch of detail you can read The Notebook independently

7:15 a couple graph utilities don't worry too much about this and boom let's look at our graph cool so here's our agent graph just like we saw before and again this is just like a you know a line graph representation what we actually showed over here this EX exactly the same thing I have an assistant node I have a tool node I bounce between the tool and the loop until basically the assistant returns a string saying here's my answer that's

7:38 all we're doing Simple okay so here's a couple different questions let's just make sure I can invoke this thing and that it actually works and my config is not defined let's see yep so we're going to go ahead need to pull that up here boom let's try that and that'll work cool so we can see that our agent is running and it is running there we go so we get an answer and that's fine so the

8:09 agent works okay cool and we can also stream outputs but let's not worry about that for now let's move on to the eval piece so now we have an agent we build in L graph we know it works cool now let's talk through that respons eval again this is looking at the output or response of our agent no problem so first things first let's build data set just like we've done a million times I'm going to go open up lsmith boom and

8:37 let's log in here nice all right very good let's go ahead and open up let's make sure I'm the right tenant so I'm in my Lang chain tenant let's look at data set so actually I've actually already created this data set I don't want to create it again but I'll just show you what it looks like SQL agent response there's my data set here are my examples so basically input up a pair just like we've seen in the past right question

9:03 answer just like for rag eval same thing but in this case we need an agent's going to be doing all the work under the hood but again our evaluation approach can follow exactly what we've done with rag in the past right here's our examples so we created a data set with input output pairs question answer now these answers of course come from our SQL database so we need an agent to again interact with the database and

9:22 do all the work for us so we're just going to find a chain this is basically just going to invoke our graph with an example from our dat data set so again if you go look at our data set the data set is keyed with our inputs are this key input so again we grab our example key input that'll return basically the question Plum that into our agent no problem easy now okay now here's what's interesting our evaluator this is just

9:48 like we did in the past for a rag same thing we're going to be doing a string to- string comparison between a reference answer and our agent answer so this is literally the same thing we've done in the past now we kick our reel so again let's look what we're passing we're passing in our little function here that's basically just going to run our agent on each input okay so that's that our data set we defined our

10:13 evaluator we defined right here this is the same as basically a rag evaluator you know it's going to look at our reference answer relative to our agent answer cool so we kick off evaluation here that runs we can go over and look so I can go ahead and actually just I'll show you where I am so I'm in my our data set that's defined here I look at my most recent experiment so here we are so again we' kind of seen this

10:39 before this is basically a you know one zero is the answer correct or not scoring we can actually just look at some of these runs and kind of break them out so you know reference output Les has 14 albums the r zien has 14 albums this is obviously correct that's great we can look at some that are incorrect the most purchased track of 2013 was hot girl and this has some issue this is probably a problem

11:06 with the query so again this is what we could dig into and you can look through accordingly so in any case this is like the the simplest type of valuation you might think of it's it's actually the same as other types like rag evaluation we've already talked about where you're basically comparing the output or what's returned by the agent to a reference and you don't care anything about what's happening under the hood okay so this is just like an

11:27 example of endan eval on a simple test case on a SQL agent I'm just going to extend our response evaluation slightly here so initially remember our agent looks like this we start it goes to the assistant node the assistant picks one of several tools to use the tool is invoked we go back we continue that in a loop until we finish right now when we talk about laying out agents you can also St to think about laying out those steps you

11:55 want the agent to take or those tool nodes those tool calls very explicitly as independent nodes so here's actually a separate SQL agent that we've kind of just devised just kind of hacking on it a little bit and what we do here is instead of running this kind of just as a simple Loop where the assistant kind of makes the decision at each step as to which tool to use we encode the path of tools that we want the agent to

12:20 use very explicitly and so basically it follows what we did previously your first tool call which is basically list tables you get the scheme you generate a query you you check the query for correctness you execute it and you go back and if there's an error in execution then you try again so it's same kind of idea except we're just making a little bit more specific so what's cool is with our now eval set to find I can actually go and I've just run

12:48 this notebook and this is checked in I'll of course share the link here bunch of code this is again showing the the the kind of flow of the agent we going ahead and ran response evaluation on that same eval set again this is kind of the same eval we just went through with our updated agent so we can kick off evaluation here I'm going to name this evaluation SQL agent mult multi-step response so that runs and we can go over to L graph we can

13:17 look at our data set we can see here's our two experiments this is our initial agent it got around 53% correct I ran three repetitions and we can see that our newer agent SQL agent multi SE response gets to around 67% again across the three repetitions so this is pretty cool we can open this up we can go to comparison view and this is great so we can actually see you know we do two improvements and one regression

13:42 relative to the Baseline the Baseline being our initial model and our what we're comparing is of course the multi-step agent so in any case good example of how you can you know use evaluation to compare different agent architectures and in this particular case you can see that a more explicit architecture does a little bit better on EV set thanks