Evaluating LLMs with OpenEvals

0:01 how's it going everyone it's Jacob from linkchain evaluating your apps performance on a range of inputs commonly referred to as evals are a vital part of bringing llm apps from prototype to production that's why today I'm excited to show off a new open source package that makes getting started with this process faster and easier than ever open evals we're launching it centered around a specific type of evaluator called an l them as a judge this is where you have a

0:33 capable model grade the outputs of your app or some component of it assigning it a score which can be a past fail or something more granular we've seen this as a common starting point for many different teams and for these open evals include some pre-built prompts but makes them completely and transparently customizable so that you can use whatever criteria you'd like while judging and it allows you to customize the outputed scores to be either discret values or continuous range it also

1:02 handles parsing out the lm's results and the structured output that the model requires there's also convenient parameters for few shot examples which are example inputs outputs and scores that give the llm judge more clarity on what your preferences are it also supports a wide variety of models as the judge including open AIS anthropics Googles open source ones through AMA and more through lank chain or if you prefer you can bring your own

1:32 open AI client and it's available in both Python and JavaScript now let's see it in action okay so here we have a simple example llm app that's just a chatbot pretending to be a pirate and let's say you've been working hard on this you're ready to bring it to production for users who are not a little enthusiasts who really want to talk to a pirate and have that experience so they could ask a funny question like what is your

2:03 name and the app would respond with something like I be on far and wide is patchy and you know it sort of like satisfies what the user would expect when talking to a pirate it's in dialect character Etc so this is great and you can see the app itself is actually quite simple it's just a single model call to open AI with a simple system prompt saying you're a pirate name Pache respond in a certain way pass these messages in from

2:30 the front end into the client and then return the content this is great but let's say you know you're worried or you know you have some reports that users are trying to get the app to break character and one way they might do this is asking what model are you and you can see that it actually does leak that it's a model known as chat gbt a product of the illustrious open Ai and you know this is true but

2:59 but not really what we want our Persona pirate chatbot to be saying we want it to be taking on denying this and saying that it's a pirate named patchy and that you're really talking to a genuine pirate so the fix for this you might think is pretty simple it's just let's just update the system prompt as you can see it's pretty simple but you know if you had many people working on a team here or you know you wanted to make many

3:24 changes to this prompt at once you wouldn't want any regression you wouldn't want to remove a key B bit of prompt or like some other part of your app that was maybe blocking the app from responding and breaking character so a great way to this is where open evals comes in and a great way to do this is to write an eval that checks for this case so I've got a using a v test like Runner to run

3:51 my evals you just need to know that it can basically run this test case passes this input and reference output into my case here and I can pass that into my app and then my evaluator as well later on I'll leave a link to this in the description of this video if you're interested in learning more about this Runner but yeah I've just got a simple command set up here that just runs vest with this config and looks for anything that ends

4:17 in eval dots and we'll run that as inval eval so if I run this as is it's just going to log the response from my app let's quit the server and do yarn run EV and you can see it's just going to go through and it's going to spit out the response this time it says something slightly different but it thinks it's GPT 5 and you know was made by the fine folk at open AI which again we don't

4:45 want in this case so yeah how do we basically write a test that or like write an eval that will stop the that will flag when this app breaks character for this input and and yeah we can import the create llm as judge method from open evals here and the reason we're you know it's nice to use an llm as judge here is you know while this particular response may contain GPT 3.5 and various keywords we

5:15 could use to sort of detect whether the LM is breaking character let's say we were to switch models behind the scenes or you know use some other like yeah something like anthropic then our check would be pretty brittle and it would fail in many cases there may be other subtle outputs where we want to detect that the LM is breaking character so an llm is perfect for this sort of messy generative output and we're going to initialize our

5:41 evaluator like this Conant evaluator equals create llm is Judge we're going to use this correctness prompt that you can see is just an F string here and the open AI 03 Mini model and cursor is doing a great job with this but yeah oh it was okay so I forgot the outputs our Valor is going to take in the reference outputs from the from up here it's going to take the messages from here and then it's

6:05 going to take the actual output from the llm here and then we're going to log the result and we're also going to add an expect result score to be true because that's what we want we want our evalu to pass and we add the set Port here from vest this is a going to basically fail on the first attempt because or we expected to fail on the first attempt because as it currently stands our app is not very good at keeping in

6:35 character let's run the eval again and it's going to take a little bit longer this time because it is making that second llm call that sort of Judges whether our app is breaking character and you can see that it did indeed break character this time I thought it was gpt3 and our valuated return to score false and our expect assertion failed here okay and this is pretty easy to fix like I said in this particular case I can

7:05 just add a line to my system prompt never break character respond indignantly if someone asks if you are an AI model and deny let's say to cool so with this change let's go back to our eval script and run it again it's going to go through get the the initial model response you can see this time there's no mention of open AI or chat gbt anywhere in this response and our

7:37 evaluator this time score is true and our eval passes it doesn't break anything sorry it doesn't break any character so yeah last thing to emphasize here is this correctness prompt is really just an fstring so I could for example log it and I could very easily adapt it for different criteria or different things that I want to measure or care about so I'm just running this this time one more time to show you what that looks

8:03 like it's just this sort of longer like rubric and asking me to take on the Persona of an expert data labeler so I could add it as a new correctness prompt up here oops there we go and I could tweak this rubric to say let's say you know more heavily emphasize never Raks character or penalize different things when scoring and you know you could like add instructions and other reminders and then everything just gets formatted into

8:34 the prompt here so you have your inputs outputs and then reference outputs and for JavaScript these are stringified awesome so again all this is open source and I'll have a link to the open evals repo in the video description we're going to add a lot more do a lot more work on this in the next couple months and you know you can see expect to see more pre-builts for different use cases Beyond just llm as a judge and yeah if

8:59 you have also other you know evaluators or prompts that you find useful we would love contributions as well and PLL requests so yeah let us know what you think in the comments here or find us on X linkchain aai we love to hear your feedback and thoughts thanks so much for listening and watching and yeah hopefully this makes it easier for you to start evaluating your llm apps and yeah hope you give it a try take care bye-bye