Online Evaluation (Guardrails) | LangSmith Evaluations - Part 21

0:01 hi this is Lance Lan chain we're continuing our Langs Smith evaluation series we're going to be digging into online evaluation here a bit more so if you recall this is kind of a framework of how to think about evaluation in General on the left here you see different types of data sets that you can work on so you can work on kind of manually curated data sets you can work on user logs different types of evaluators and what I really want to

0:22 draw your attention to over here is the differentiation in terms of how you can apply evals you can apply evals on production traffic so it's like I have an app in production and as my app is running I'm doing evaluations of some sort on it or of course I can also do evaluations on curated data sets or assertions right so online evaluations live in this top bucket and you can really see it here so basically we're

0:46 focused on production traffic so I have an app in production and I'm evaluating something about that app while it's running now common thing that people like to evaluate is something like toxicity so for example does the user input contain any kind of personally identifi information or toxic queries or something along those lines so toxicity is a thing that you know oftentimes you want to be able to detect within an application as as a developer so I'll

1:12 show an example of this and I have a project that I've Set called rag online evaluation and that's here a few traces here you can see and here's basically my rag app so with this app I'm basically simulating user questions here so here's like an input question here's an input question and those are all logged to my project so that's all it's going on now let's say I want to create a new rule that'll evaluate whether the input question contains any

1:39 toxic information or personal identif identifiable information so all I have to do is I go to rules and I basically say add rule so this is going to allow me to create a new online evaluator I just simply click here online evaluation create evaluator and I'm going to use one of the suggested prompts that we have already so right here you can see t LM Generations so this is a prompt that allows you to tag arbitrary inputs

2:06 with a a lab or a score based upon some specified criteria so you can see in the prompt template it already is set up and I can choose my model let's say I want to use gp4 turbo it already has this kind of preset for me so you're classifier they'll tag input text with a score based on Supply criteria and you're tagging based upon insults threats negative comments any personally identif reliable information right and you pass in the text now what's pretty

2:35 neat about this is I can actually test using this preview what this will look like on my particular chain so with this preview boom I can actually see the text it's going to get is for example my most recent Trace so in my case the most recent Trace was this test case how do I create a rag chain and I also plum in this open AI API key as like a test of an adversarial prompt or a prompt

3:02 that contains personal identifiable information like want a flag I have my score criteria down here so I'll basically return a score one or zero and I'll also return the explanation for the score so that's set I just hit continue here and then I'm going to name my I'll name this toxicity or I'll call this guard rails input input guard rails and I'll save that so now I have a new online evaluator called input

3:34 guardrails that'll run on the inputs to my chain so I go back to my app and I've passed in this open AI API key and I want to see whether or not my online evaluer can actually flag that so I go back and now my runs or my traces you can see here's my app here's the runs within my app so this is kind of the full Trace we're looking at we can see it's tagged with feedback so the

3:59 feedback I get is a score in this casee of one so if you look back at the prompt one means that indeed there is a personal identifiable information within this prompt so we can even look here we can look at the explanation for this open the evaluator run and we can go down and we can see yes the score it is scored as one the text contains personal aable information as includes potential API key which is sens of

4:23 information that should not be shared publicly so it's a good example of the ability to set an online evaluator against the simple project in this case a mock rag app it can run on the inputs of your app really easily and it can establish any customizable guard rails or things that you wanted to flag for in return kind of a very simple and customizable score yes no one Zer in order to identify that the the input contains information you want to

4:49 flag thanks