How to Evaluate Document Extraction

0:00 hi I'm David from Lang train today I'll be walking through how you can evalate document extraction using Langs Smith's SDK before I taped this video I put together a simple pipeline that takes in a 10K report from a public company if you're not familiar with what a 10K is it's a report that every public company in the United States must file with the SEC every year inside of this report the company discloses its products and services financial performance risk

0:29 factors to its business anything material that investors want to know before getting involved with the company so this pipeline takes in a 10K in report parses its text and injects the text into a prompt this prompt is sent to an llm structured output and inside of this prompt we also instruct the LM to extract certain fields from the report and as you can see with the sketch on our right the fields that we'd like to

0:59 extract from the report are the company's products and services the company's risk factors the company's IRS employer ID number the company's address the company's earning earnings per share and the company's net income and because we're calling this LM structured output they'll actually return an object that has these Keys because these are the fields that we'd like to extract or fetch from the report and so let's use

1:29 apple as an example if we were to take Apple's 10K report and send it off to our llm structured output that again has that special system prompt instructing it to extract these fields from the report and the text from the report inside of the The Prompt itself we would hope that the LM correctly returns in in object with these keys and for the products and services key we'd like to see iPhone iPad MacBook iCloud in the value for the

2:03 risk factors key we'd like to see increased regulatory scrutiny supply chain disruptions in the in in the value and then for IRS employer ID number company address earnings per share net income we'd like to see that our llm Le structured output correctly took the fields from the annual report and place them in the object so for the company address we'd like to see the accurate address for Apple's headquarters that again were found in the report for

2:32 earnings per share we'd like to see it fetch the correct number from the report and same from that income awesome so we have a a basic pipeline that can perform document extraction when given a company's 10 care report now let's move on to how we can evaluate document extraction using lensi SDK so first I'm going to touch on why we're running the evaluation we're running the evaluation because we're not sure which model we should use and

2:58 there's two models we're considering for the extraction task there's gp40 and there's 01 both of these are models from open AI with support for structured output and when we're comparing these two models head-to-head there's three criteria we care about evaluating them on the first is latency so how long this model takes to perform the extraction task there is cost which is how expensive it is to use this model to perform the extraction

3:28 task and then there's the most important one which is accuracy so which model is performing best in producing the most high quality accurate outputs when it's extracting certain fields from that 10K report great so now that we've established our a rationale for running the evaluation let's touch on at a high level what running eval with the L withth SDK looks like so before you run in evaluation with lengths with SDK you must def you must Define three things

4:00 the first is a golden ground Troth data set and for this extraction evaluation the golden data set is just going to have some number of input output pairs inside of each input output pair the input is going to be an example query or prompt that we'd send off to our L and the structured output to perform extraction so this input again would just be that instruction that we give to the LM about what fields to extract and

4:27 the actual text from the report itself the output would be some response that we'd love to see our LM structured output return in production so in this case of extraction the golden gray output we'd love to see returned is an object inside of the object it'd be the keys or the fields that we that we wanted to extract from the 10K and corresponding to each key we'd like to see a value with the correct with the

4:56 correct text populated inside of it so again for the Apple example if the golden data set input had the 10K text from Apple's report the output would ideally have iPhone iPad Mac associated with the products and services key so on and so forth the next thing we need to Define is the application logic we're evaluating so because we're just evaluating the document extraction step or task the application logic we're

5:26 evaluating is just a structured output model call or model call structured output the third thing we need to Define is our evaluators so our vators are what's going to score the output produced by either the gepd 40 model or the 01 model for accuracy and again this is how we're going to determine which model we should use when performing extraction and all three of these things come together when we kick off an eval

5:58 using length F SD K when we run any EV Val lengths with SDK we take the application logic we're evaluating again in this case it's just an Alum call structured output and we run it over the inputs of our golden data set these runs produce new outputs and we score these new outputs using these evaluators relative to the outputs in our golden data set and these are displayed nicely on Lang's UI and we'll show what that looks

6:31 like shortly great and so this sketch belows just nice visualization of what Deval we're about to run in code looks like so what we're going to do is we're going to Define that golden ground Sho data set again the inputs are just going to be example prompts or queries that we'd send to an L on the structured output to perform extraction the reference output would just be a high quality response we'd love to see that Alum structured output

7:00 a respond with when given that that prompt in that 10K report text we're going to need to Define these evaluators that are going to basically assess and grade the outputs produced by GPT 40 or 01 relative to the reference output in our data set and that's that that's what this this EV Val thanks with SDK is going to look like at a high level so let's actually jump into the code and implement it great we're in the notebook before I

7:32 show off how you can implement the eval and code using the SDK I'm first going to touch on what the extraction pipeline looks like in code as well so the focus of this video is the evaluation so I'm not going to go through the code line by line I will attach a link to this notebook in the description of the video so you can dive deep into it on your own time so as you can see here the first

7:53 step of the pipeline is to load in a 10K and a report here I'm loading in Apple's 10K and I have this saved as a PDF in my code base the second step is to actually parse its text and inject it into a prompt down here and as you can see inside of this prompt we also have some pretty robust instructions for the llm that's ultimately going to be called a structured output and output in object with the

8:25 desired keys that we'd like to fetch from the report and so as you can see in the prompt we give the model certain instructions we ask it to avoid duplicates we ask it to be concise we ask it to make sure things are formatted correctly and validate that each piece of information it extracts from the report is actually explicitly stated in the text and we give this LM a few examples of good responses and bad

8:56 responses awesome so that's the second step is actually calling in our alignment structured output and so again the the two models we're experimenting with are 01 and gp40 just so happens that in this code I'm using 01 at this point in time and to lastly touch on this this is the output schema that we that we're going to call our LM with when we when we invoke it and so you can see here that when we

9:25 actually call this llm stru output it returns a a pretty great object in response so here the products and services look good iPhone Mac iPad advertising Apple Care the risk factors look good Di's employer ID number got truncated but should look good that income looks accurate now let's move into the evaluation portion of things so if you remember from that previous part of the video our first step is going to be

9:57 creating a ground truth golden data set and for the sake of time I did this before the taping of the video and let me go over to lenith now and show you what that golden ground shoot data set I manually created looks like awesome so I'm in lsmith I created a data set called 10K extraction and let's look at what each input in output looks like so my first input in output has the 10K from Apple and you

10:26 can see here that the text from the 10K report is is is related to Apple because if I scroll down here you'll see a whole bunch of information on Apple watch airpods Pro airpods Max Mac OS if I would scroll down a bit more you'd see detailed financial information earnings per share net income Etc and so that's the input right the 10K from Apple's or the text from Apple's 10K and the instructions that we give it in the prompt as

10:56 well and the output is just an example object that we'd really like to see in response and so I'll actually toggle Lang Smith to show the raw output instead of the rendered output so you can see it in a bit more detail here but ideally we'd like to see it return that income of 93 million we'd like to see it have these risk factors this correct company address for the headquarters of Apple this as the products and services

11:23 so you know this is just a ideal high quality output we'd love to see DL un structured output return and response to this prompt and we can click into a few others so this this this last input and output in my data set is the one that is that is associated with Nvidia you can see Nvidia is here if I scroll down you'll see some information about its financial performance Etc and again the output that we have here is just a high

11:53 quality accurate output that we'd love to see the Alum structured output return in production all right great so that's our golden data set it's called 10K extraction and we'll pull it in later when we actually kick off the experiment using linki SDK the second step is to define the application logic that we're evaluating and as I mentioned earlier because we're just testing the document extraction step all we need to do is Define a

12:23 structured output LM call as as our application logic and we'll do we'll actually how this application logic inside of a Target function that I've defined here and we're defining it inside of Target function because when we kick off this evaluation using length that's SDK the SDK will automatically pass in the inputs from our golden data set to the Target function and from there we're able to actually take in these inputs give them to our line instruction

12:53 output and return that response or output that our evaluators are going to score relative to the reference output for accuracy so that's our application logic that we're evaluating again just a structured output an L on the structured output and in this case it's it's 01 but in another experiment it'll be Jeep D40 now let's define our evaluator so what our evaluator is going to do is take in the outputs in the reference outputs

13:26 from our golden data set and from our Target function so similarly to how ddk passes in the inputs from the golden data set to the Target function when the experiment is kicked off both of these are passed to our evaluator automatically by the SDK when the experiment kicks off and what this evaluator will do is take both the output or whatever our Target function returned which is just a an L on the structured output call using either gd4

13:55 or 01 it'll take the output from our data set or our reference output and it'll format them and it'll format them because we're eventually going to want to pass these two objects into a prompt to be evaluated by an LM judge but we don't want to pass in the object to the prompt we want to clean it up and have it be returned in a string representation so with this helper function format format objects for alum

14:26 judge does it just takes in each of the objects the first is the one that's returned by our Target function the other is the one that's found in the outputs of our golden data set fetches the keys sorts them in alphabetical order and puts them in string form and then we call another helper function called run LM judge what run Alum judge does is it takes in both the two string representations of the output from our

14:53 Target function in the output interal data set and passes it into an LM judge and this LM judge is just cheap D40 with a special system prompt the system prompt instructs it to evaluate how well a model's extraction of a 10K document of a 10K document matches a reference ground shoes output and we give it some relevant evaluation guidelines and in context and again because of the helper function we're able to really neatly pass in a

15:21 string representation of the model output or what was returned by our Target function and a nice string representation of the object that's in our golden data sets output and you'll actually see that when we call this Allen judge we're calling it with structured output as well and we're asking it to return two Fields we're asking it to return a field called accuracy accuracy is just going to be a Boolean value and this Boolean value is going to indicate whether the model's

15:52 extraction adequately matches the reference ground truth and so we'll ask it to return one if the model's up butut captur captures the same essential thisinformation as the reference even if even if expressed differently and we'll also ask it to return reason or the rationale explanation of how well the model's extraction aligns with the reference ground truth so just two keys one that's a string that's a reason for why it either gave a zero or a one one being

16:23 the model's output was accurate zero being it wasn't accurate relative to the reference cont true truth great so we're actually going to kick off this this evaluator with the the running of the evaluate accuracy function and our last step now that we've defined our golden data set the application logic we evaluating and the evaluator which is just going to score the going to score the output of our Target function relative to the reference app in our in our golden data

16:54 set for accuracy our last our last step is to actually kick off the experiment using the DK and to do so we'll be using the valy function from Links of SDK and we'll have to just specify a few parameters the first is Target or the name of our Target function the second is the name of our golden data set which is just 10K extraction the third is an array of evaluators we're just using one evaluator which is evaluate accuracy

17:20 that we went through we just went through and this this third argument is the name of the experience and we'll see this on Lang Smith UI in a second so we kicked it off and we see that it ran correctly great let's hop back into Lang Smith and visualize our experiment results to do this let's toggle from the examples view to the experiments View and the first thing we can note in the experiments view is that we successfully

17:50 ran two experiments using L Smith's SDK the first experiment used GPT 40 as the model that performed extraction and the second experiment used 01 as the model that performed detraction task and as you can see in this column we get a high level view of which model produced more accurate outputs so here we can see that for all the runs in the experiment that used 01 as the model for extraction our Alm judge gave an average axis score

18:19 of 0.93 we can see in the experiment that used GD 40 to perform attraction our Alm judge gave an average ax score of .73 and let's actually click into the experiment used 01 and take a look at it in deeper detail the first thing that you can notice about this view is we get to see the input from our golden data set the reference output from our golden data set and the new output that was generated when we took the input from

18:47 our golden data set and sent it off to 01 with structured output all in one View and we can also see for each new output that was generated by 01 what our LM judge gave it in terms of an accuracy score so if I click into the accuracy score here we'll see that there's actually three repetitions in each repetition got an AC score from our LM judge that we defined in code and that's because when we ran

19:14 this evaluation with Langs Smith's SDK we actually ran it with three repetitions meaning for each input from our golden data set we ran it with our application logic again in this case is just A1 instruction output three times and we ran it three times and gave it an actually score three times to smooth out any variability in the scores given out by our LM judge so let me click in here again you can see for each repetition

19:42 our LM judge gauged at this new output produced by 01 structured output was accurate relative to the output in our golden data set and we can scroll down and see the new responses that were produced when 01 was output was run with the input from our golden data set and see what these other outputs got scored by our LM judge so here we can see we didn't get a full ax score for each each repetition here for repetition one we got graded as

20:15 was inaccurate but for the other two repetitions we found that our our model in this case o1 INF output was able to produce acur outputs one thing that's really helpful about this view is we can also click in and see what the evaluator run looks like under the hood if I click view evaluator run there and I scroll down to where the Alum judge step occurred now we can see in really deep detail what our Alm judge looks like under the hood

20:43 so here we can see we're giving an LM judge gepd 40 a special system prompt that asks it to evaluate how well a model's extraction of 10K document information matches a reference ground truth output we see the context and the evaluation guidelines that we defined in code and we see that we correctly pass in the model output or was generated by our Target function in code and the reference ground truth output which is just that that output in

21:13 our golden data set to The Prompt so both of those are correctly passed in and given to the LM judge and finally at the bottom we see what our LM judge actually returned which because we called our LM judge with structured output it returned to Fields the first field is just whether our new output generated by 01 was accurate relative to the ground truth reference output the second field reason explains the rationale why DM judge graded the output

21:41 as accurate or inaccurate what I think is often times more helpful than just viewing one experiment in isolation is to run multiple and pull them into the comparison view so what I'll do now is pull both of these experiments the one run with gp40 the one run with o1 In the comparison View and now I can still see the inputs for my golden data set the outputs for my golden data set but I can Al also now compare the

22:11 outputs generated by gp40 and the outputs generated by 01 in a side by-side comparison and in addition I can see how each one fared relative to the other in terms of that accuracy evaluator computed by our Alum judge and what's really helpful about this view is I can toggle by improvements and regression so if I click improvements this VI will automatically filter itself for all inputs that are o1 model performed our gp40 model on so let me actually click

22:40 that now when I click improvements now we're just going to see all the inputs from the golden data set that 01 did better than gp40 on in terms of that accuracy evalu that was computed by D LM judge we defined in code so here we can see those those two outputs that it outperformed g401 and there's no regressions meaning there is actually no inputs from the golden data set that 01 underperform ch40 on in terms of that accuracy evaluator so

23:06 there's no regressions to tole to otherwise we could click that button and the view would update automatically the last thing I'll point out about this view is we also provide really nice highlevel charts that compare the two experiments based on the evaluator you defined in code latency and cost and these are the three metrics that we mentioned are really important to us when deciding whether to use GPD 40 or 01 for the extraction task so as

23:33 we can see from this chart and from the high level metric shown in this accuracy column we know that 01 produces more accurate outputs than gp40 in terms of in terms of the subtraction task the average ACC score for 01 is 0.93 versus 73 for gp40 but when we used o1 we see that we introduce a significant amount of latency to the application so the p50 latency when we used 01 is around 35 seconds whereas the p50 latency when we

24:04 used GPD 40 was around 5 seconds so a really big difference there and same with P99 latency P99 latency when we used 01 for extraction is around 53 seconds almost a minute and when we use GD 40 the P99 latency was significantly less so around 12 seconds and we also get these nice charts that compare the two in terms of cost so we see that when we used 01 the total cost was around $22 and when we used GPD 40 the total cost

24:35 was around $3 to perform the extraction task and this is mainly because 01 used a lot more completion tokens because it did some reasoning in the background when it received a query or a prompt which led it to consume 26,000 completion tokens while gg40 just used 2,000 completion tokens so really big differen there in terms of both latency and cost and there's no right answer here you know every engineering team has its own

25:07 priorities so what Langs Smith does a good job of is providing you with the complete information that you need to make an informed decision so it'll give you detailed information how these two models fair in terms of accuracy how they fair in terms of latency and how they fair in terms of cost and then you can just make that decision with all the data with all the data available great and so that's the end of this video if you want to learn more

25:30 about Lang Smith and running offline evals with SDK feel free to check out Lang chain Academy Lang chain Academy has a great introduction to Lang Smith course where it goes over this online evaluations observability The Prompt playground and more so if you want to learn more check it out thanks for your time and hope to see you in the next one