Unit Tests | LangSmith Evaluations - Part 10

0:00 hi this is Lance from Lang chain this is the 10th video on our lsmith evaluation series focused on unit tests so unit tests are often simple assertions for example they can run as part of CI for app functionality now let me give a motivating example here so I've done some recent work on code generation and it follows a flow kind of like you see here so I have this code generation node that takes in some large set of

0:26 documentation like for example a bunch of line chain Docs takes in user questions and produces code Solutions based on the user question and the documentation now in this flow I typically do something to the output where I convert it into this pantic object that contains three things a preamble Imports and code and I do Downstream stuff with that which you don't have to worry about here the key point is I basically convert a natural language question into a structured

0:56 answer object that has these three things so how can instrument unit tests for example that check whether like Imports and the code are are executable that's like a very sane unit test you might want to do just to convince yourself that you're able to produce like a structured code object correctly and that the Imports and code actually work as expected right so that's like a pretty sane unit test that we we may want and again if we look at our overall

1:21 framing where does this sit so in this case we don't necessarily have like a standalone data set of examples we have you know a heuristic like a hard-coded decision or a hard-coded assertion and we have some reference that we expect for example in this particular case I expect that the code is executable correctly that's really it and of course unit tests they they can be run offline you know as part

1:51 of a CI flow or they can be run online and we're going to talk about the offline case and how you can instrument this with lsmith so what we're going to do is we're going to show the Langs Smith works with conventional Frameworks like Pi test for instrumenting tests and we're going to set up a few things we're just going to set up a very simple main.py file and I'll go over there and show you that right now which is

2:10 basically going to instrument my kind of generation logic and then we're going to set up a few different test as independent files in this test subdirectory that'll implement the various checks we want to do in this case to check that Imports and code are executed correctly so I'm going to hop over here to my VSS code so you can see it now and you can see here I'll move this over that I'm in this introduction folder so this is in Langs

2:39 withth cookbooks and I've set up an app so here's my app and main.py you can see right here now this is where the logic of my chain is going to be defined so in this particular case this generate code solution just takes in text and what's going to do is here's my generation prompt which basically says you're an expert in L expression language here's a document related to lch expression language here and produce an output that contains Imports a

3:09 functioning code block and a preamble which I will State somewhere in here yeah description of the code solution the Imports and then a functioning code block so those are the three pieces now here I'm actually going to Define data model so this is a pantic object that contains prefix Imports and code and I'm going to bind that to my llm using with with structured output method which is very convenient and this

3:42 code gen chain should output a solution object that has a prefix import and code so that's kind of step one now this is just a module so this is what I'm calling my app and that's all it has now if I look at my tests here I've defined two tests so one is test code so this is going to take in or it's going to it has one kind of piece of L transpression language documentation here so this is like an example input

4:13 to our function and this text execution simply will take that input right here it'll invoke our gen code solution which we defined over here so that's this guy right here there we go it will yep it will invoke that it will produce a solution and now we're going to test we're going to try to execute the Imports there and basically if that fails we're going to we're going to flag the py test. fail and we're going to

4:44 return the error so that's the setup here likewise with code execution same flow in this case though we're just going to grab both the Imports and the code itself we'll try to execute all of it and again we'll return an error so these are our two tests now this looks like standard unit testing there's nothing that's you know Lang Smith specific here but I want to call your attention to one thing this decorator unit which we import from

5:10 Langs smith. unit allows you then to log this unit test to Langs Smith so we have that in both cases so there's just a simple decorator on top of this function that's going to perform our unit test with from lsmith we import unit so we do that in both cases and all we have to do then is just say run P test so we're in our Dory we're in this introduction directory right here so I can just show you so we have my app and

5:35 my test here that's it so we can kick off we can run P test and this will kick off the unit test so this is just standard stuff there's nothing that's pretty that that's kind of very specific to lsmith here other than I've added this little decorator unit to my unit tests here that's all's going on so it's running both of them you can see this is kind of churning along so that's cool and here's my other one so I have

6:02 test Imports test code these are my two unit tests it looks like two PA so this is great this is just using p test and running unit test but there's one nice trick because I've run these with that Langs Smith decorator if I go over to my Langs Smith so if I go to data sets and testing so I can see now I have data sets for the directory name and then my unit test name test Imports test code if

6:25 I go in here I can actually see that the results of the unit test are logged and what's pretty nice is that you get all this metadata here which is pretty good and yep I can see the names of the various runs I can see the result so this is again pretty convenient basically it's allowing me to log the results of the unit test to Langs Smith and each test then is logged to an independent data set so I can keep them

6:54 compartmentalized which again is also quite nice let's actually click on one and kind of see actually let's go ahead and use comparison mode to look at a few of them so here we go this is pretty nice so I can look at things like latency I can look at the feedback score I can look at the tokens used again in each case I get logged input so this is the input that I passed to the unit test

7:19 and there we go so this is each one of them each each of the unit tests all pass and I can see latency and everything so anyway this is a pretty nice and easy way to use Langs Smith to log unit tests and this is a very convenient thing to to run as part of your CI thanks