LLM Eval Tools Compared: Braintrust

0:00 Welcome back to our series on eval tools. Today we're going to be taking a look at brain trust. So now we're going to go with Brain Trust and we're going to see what they got going on. And we got Wade, our friend Wade from Brain Trust, who is also a student in our class, the first cohort. So you might have a slight advantage, but we'll we will see how it pans out.

0:31 >> Hi, Wade Gilliam here from Brain Trust. Back again with another walkthrough. This time on how to accomplish homework assignments one and two with Brain Trust. And in this video, we'll look at homework assignment one, which is all about writing a good system prompt for our recipe chatbot. And then also providing it with 10 or so handcoded queries to test things out with and get things going. Uh, with that, let's go ahead and get into Brain Trust and see how to do this. The first thing I really wanted to do was define my intent for this iteration of recipebot. And I thought it would be interesting to attempt to build a recipe chatbot tailored specifically to the interest of my family. So I recruited my three sons and my one wife. I asked them to provide me 10 to 20 example queries that they would uh think of issuing to such a chatbot and I imported those into a data set to use. So you can see here's an example of one of them. We have the query and then we also have some metadata an ID. We have sources handcoded. This will be important when I start adding in uh synthetically generated user queries into this data set. And then I also thought it might be interesting to include who submitted it to further analyze if any problems were related to the queries of a particular person.

2:05 Creating a data set is actually super easy. Uh if you click this button and create data set, you'll be taken to this. >> I will say the the UX of the trace viewing for some reason I just like it more somehow. U I don't know why but we can reflect on why that might be as we keep seeing it. >> You can go ahead and use a CSV or JSON L file. Use a CSV and then you can drag and drop things wherever they're supposed to go. So the input we have uh is the actual query. We don't have any expected. And then what I did was include these in the metadata. Hit import and the data set was created.

2:48 Super easy. It is nice to have the metadata tags, I will say. >> Let's go ahead and get to actually working on the system prompt. Great place to work on prompts is the playground. And you can see that we have our system prompt uh in here. And I started this with coming up with the best guess uh following the recommended format for system prompts in the course. I didn't see him do the step where he used the app that you provided. I wonder if he's hooked that in at all or if he just went in and grabbed that prompt and pasted it in here.

3:30 >> Yeah, I don't know. >> Yeah, no idea how to >> in in a production use case, you probably have your real application and it can be really nice to bootstrap from what your real application is doing. So one thing I liked about the lang flow approach was sort of automatically started with like an existing application. >> Langid lang >> damn it. Uh it like automatically started from like an existing application and it like using the traces it was able to kind of load all that in and then you can fork from there. I kind of like that flow. I don't know what this does but at least it looks like maybe he just pasted it in. H it's a good question. I don't know either. We'll see >> the recommended format for system prompts in the course. And when I ran this against the queries for my actual uh users, it did pretty well, but I really wasn't sure.

4:30 >> Okay, so this this is a prompt playground. So it's like it has he has a system prompt in a playground and then the data and he's like running running that in a playground. So yeah, I guess it's like this is their version of like a similar playground. And to kind of piggyback off your comment, you know, they didn't like arrive at it through a trace. They arrived at it, you know, through this like other pre-baked kind of thing. Um, so it is interesting.

5:03 Um, there's this loop thing though. So that's I kind of Okay, let's see what this is. It's interesting. But what I did was ask Loop, our AI agent, uh, to help me create a recipebot relevant score because I wanted some means to assess how well the outputs were one following formatting instructions. Were they trying to recommend a recipe um, regardless of the input? Were were the outputs um indicative of including follow-up questions that would be required based on the user query? And so Luke created this recipe bot score, ran it with my system prompt, and like I said, it did all right, not great. So the next thing I did was ask Luke to >> So here he's asking the LM to generate evaluation rubric and then he's running that rubric with the LLM on the output.

6:05 Is that the correct? >> That's what it looks like. I'm looking at this and it's like okay I see I'm looking at the reasoning trace which I mean it looks like a reasoning trace anyway and it's like um created a code score ran an eval. So, he's kind of tried to do the whole eval loop, no pun intended, in one go. Um, which I'm not sure, you know, I mean, like I'm biased against it, let's say, you know, like of like as we talked about earlier.

6:39 Um, I'm just curious like how he's going to use it because it seems like a really large step at this point. Totally. >> Yeah. Generally, I'm fine for these kinds of like I would call them meaningless automations if they're not going to make a business decision for you or critical decision at this part of the loop. Like if he's running these emails and then he is aggregating the scores and then saying like, oh, like a certain kind of instruction is like kind of hard to follow. Okay, let me synthetically generate more queries of that nature. Like I'm super all for that, right? Like great job. But if he's trying to say that, oh, the specific instruction was only was followed like 2.2 out of 10 times, I'm like going to make a business decision on that and like not even have shipped this product.

7:29 I'm like super against using these kind of course >> gra how did you how did he come up with the rubric? You know, did he get it from error analysis? >> Um, >> yeah, there's also like the question of like stacking abstractions here. like where did he get the system prompt for loop itself and where is the where is the system prompt that is built into loop. So you've got like the user prompt that Wade has written to loop. There's the system prompt that is in loop. There's the system prompt that we never even actually addressed yet let alone look at the outputs here. So I think this is like stacked abstractions at least level three which starts to really make me sweat. Um >> I don't know. I think it's a little early for this big of a hammer. We'll say that.

8:15 >> Yeah. >> Yeah. Surprise. Yeah. We're all very aligned. Uh it seems like in this in our feeling, which is Yeah, it makes sense. >> Optimize the prompt based on the score in the data. >> Yeah. He said he said the magic he said the magic phrase. Optimize the prompt. >> All the context. I don't even remember. >> Right. Uh, okay. >> It made some changes uh to the prompt and then I went in there and added a few more modifications. For example, my family uh doesn't want to have recipes with harmful chemicals like red eye dye 5 or no.

8:55 >> Which one of us wants to say the phrase you can't optimize to an objective function that you don't know? >> You said it. >> Okay. Like was there not a diff that was applied? Like oh there's a diff button up there. Um you know there's a little diff slider on the top. >> So maybe there was a diff like possible but like still I don't >> I just have no idea what the objective function is.

9:21 >> I don't think this is going to this is going to lead to more consulting business for me. >> So I >> I'm serious. I think so. Like >> I think there's I don't think this practice is wrong of like trying to improve your prompts based on your evals. You should certainly use your evals to improve your prompts. I think we just like viscerally reacting to it's not done at the right time here. We've not reached that stage yet where we know what our failure modes are. Both the evaluators that we can trust and to Brian's point gotten an objective function.

9:57 >> Yeah, I agree. You know, and you know, just to be clear, Wade is my friend, by the way. So, I'm not picking on weight. We're just trying to be very >> love all the synthetic data generation that was done by his family and the subject matter expert family members. >> That is that is the best way. Yes, I agree with that. >> That's not even synthetic data. That's real data. >> Yeah, >> it's the most important data. Yeah, >> contain microplastics.

10:20 Also, another interesting thing looking at some of these queries is they weren't really uh re requests for a recipe recommendation in so much as they were actually questions about protein and how to get uh so many macronut nutrients uh into a meal or should do meal prep. >> Okay, but green flag here. >> So, I wanted to include >> he's now doing exactly what we want. He's reading the prompt. He's thinking through how it relates to his like family's needs. So like massive green flag here.

10:52 >> Definitely that that's Yeah. Yeah. He's doing the inherent product work that comes with error analysis that forces you to do some information about how to respond to those. And so with that in hand, I reran the evals and you can see things got substantially uh better. We are hitting an average score of 99%. I went through and spot checked. Everything looked great. Nice thing with loop when you create these uh scores, it's going to tell you how it actually created the criteria for the score.

11:29 >> 99%. Okay, sorry. Keep going. >> Yeah, I mean there's just so much I have to like >> reflect on what I'm seeing. >> Even give you examples to look at that uh might merit further inspection. And so it's really >> that's a green flag. >> Yeah. >> With that in hand, we now have a really good prompt get going with >> because being able to go from those examples to high scoring on different dimensions of the rubric, that's the whole point. Anyway, that's great.

11:57 That's like basically like sub subclustering and you know, >> slicing and dicing. >> Yeah, exactly. >> We love that. >> Do you think this is really interesting like do you think there's some good things in here, some things went too far, too big of a hammer. Is this like a necessary part of the journey of like getting to the right place of like people going too far and then zigzagging their way to the something that makes sense?

12:22 >> Here's what is the problem. And this is my experience from the teaching. Like you know when you're making this video, I'm sure Wade knows like he's not going to index too much on this 99.1%. Like that number is actually meaningless to him as it is to us. He's using all of that to get to like the slicing and dicing that Brian was talking about. What are the examples so I can go and start thinking about error analysis?

12:46 Someone else is going to watch this video and only focus on that 99.1% and then say they're done. That's like the biggest dissonance that I feel like is I see all the time with like AI evals >> or problematically some executive who's going to want the number to be higher than it is because if number not good number must be bad. >> Yeah. >> Yeah. I think those are good points. So I saved this prompt which we can see right here. There's our recipe bot prompt. And now we're ready to use it to start actually creating some traces. So that's homework one. The system prompt is good to go. And I'll see you in homework too.

13:35 >> Quick interjection on like design and UI and stuff. Generally, I found the brain trust like UI and design to be easier for me to guess where things were. Like I was finding myself looking in the right part of the app before he clicked into it. where as with the previous one, it was a little harder for me to like even even after he clicked it to understand how that was where I was trying to go. So, I think there's there's definitely like a little bit more um understandability in that UI.

14:08 Let me get homework too pulled up. Um yeah, I agree with you. I felt the same way. Something about the UI feels more intuitive to me as well. But there's some there's some places where I like the there's some places where I like the LSmith UI more. Like the annotation queue, I like that more. >> Um I like the I like being able to enter the playground from a trace. I think that's really powerful. >> I think the concrete things that I think were a little bit better here was less of like a border radius around everything, less jargon. So like everything in Lang Star has like a term for it that I cannot remember and I'm sure it's fine if you're like entrenched in that ecosystem, but I'm not. Um, and then there's a lot of padding in that UI versus in Brain Trust. It's like extremely compact.

15:11 I'll say roundedness I feel like is out of fashion now. I don't know why. Yeah. See, I have really bad design. I just know if it's bad, but it's good that you like can call out certain things because I just feel like it's off. I can't even like name why. Uh so it's good that you that you know why. Um okay. Shall we go? Should we do part two of brain trust? Homework two is all about error analysis and it's broken into two parts. Part one is all about the process required to generate good synthetic data that we could combine with our handcoded queries to run through our recipe chatbot and get traces for. With that in hand, we can move to the core bits of error analysis, which is engaging our subject matter experts in the process of open and axial coding to eventually arrive at a taxonomy definition of our failure modes. And with that information in place, we can prioritize where to put our efforts in terms of building evals.

16:20 So in this talk, I hope to go through some of the more interesting ways that we can use brain trust to Does Wade live in a rainforest? >> He lives in San Diego. I don't know exactly within San Diego where he lives, but yeah, >> I hear birds in the background. >> And it's very pleasing. Um, I think like, you know, bonus points for bird sounds. Um, yeah, that's that's really all I have to say on that.

16:51 >> Yeah, I'm I'm enjoying it, too. Actually, >> accomplish both parts of this homework. So, I'm back in Brain Trust and let's talk about how we can accomplish part one of the homework assignment, which is really all about getting some highquality synthetic data that we can combine with our handcoded data to then generate traces for. So just like I did in talk one which if you haven't watched I encourage you to do so because I go through this in a little bit more detail. We went through the process recommended in the course and we codify that in experiments. The first one is our dimension tpples. So we go through the process of generating a bunch of different tpple combinations that are candidates at this point because at the conclusion of each of these steps, human reviews required to go through and make sure that these tpples actually are representative of a realistic potential query. And so we can do that by clicking up here in review. And you can see that we've already gone through this process uh going through all these different unique combinations that were created and we have our two scores that we have configured for this.

18:09 >> Okay, I'm slightly confused. Um I think I know what's going on. It seems like he used a LLM to generate his dimensions and now he's going through and labeling which tuple like uh you know these dimension pairs or dimension tuples are good or bad. Is that your impression? >> That's what I think. So >> seems a bit over complicated, but let's keep watching. I guess >> terminology is overloaded. Like to call this a trace is going to confuse 90% of students in the class. However, let's carry on.

18:42 >> One great opportunity he had was he could have gone back to the subject matter expert with a couple of the generations and he could have been like, "What do you think of this recipe? What are some things that you don't like about this recipe?" And he could have used those to generate the initial like issues. >> Yeah. The like the exact thing around the plastics like that's not everybody cares so deeply about that. >> Totally. Totally. There could even be a failure funnel of like you know early stages of where it goes wrong versus later ones. But I don't know anything about that. Let's >> sniping us all >> comment and a is good score. The only one that we care about at this point is is good. And so we go through the subject matter expert and determine whether these tpples are good or bad.

19:30 >> This is extremely difficult to make sense of >> good tpples to generate some candidate user queries. And that's captured here in this experiment. And then we go through that same process of working with our subject matter experts and going through each of these and marking them as good or bad. Nice. >> And finally, we take only the good user queries, combine them with our handcoded queries, and update that original data set to include both of them. And at that point, now we're move ready. Why can't I have like a Tinder interface at that last stage? If you're already at the sort of like class of just quickly iterating through a bunch of examples with your subject matter expert, give them a Tinder interface plus+ like one that like shows them the card. Here's the sort of like request.

20:23 Here's the output. Here are the options for how you want to classify it. Like I think people under undervalue these like rapid iteration UX's. And I think that's a perfect use of it right there. Yeah, it would have been cool for sure. Even make it a mobile app. Why not? >> Yeah, absolutely. I actually think these apps could have sort of like a different subdomain that you send users to. It doesn't show them all the inards. It just shows them what they need to give you that data. That's what we've built internally at theory as a just like a quick internal app for ratings for that reason because they don't need to see all the the nasty bits. M >> you can move >> one comment that people may or may not care about but there is in UI there's a law called Fitz law. Have you guys heard of Fitz law?

21:14 >> No. Please tell us. >> Oh man. Okay. So the time it takes for your cursor to travel to the target is proportional to the size of the target and how far away your cursor is from the target. So, if you want to speed up button clicks or workflows involving button clicks, you need to make the button bigger or the distance from where your cursor is needs to be very close to that button. None of these interfaces know anything about Fitz law is my takeaway. But >> incredible. Incredible.

21:50 >> I forget that you are a you know human computer interface researcher as well. So, I mean that's why you know about these things. But it's like so applicable and I feel like my day-to-day life now. Anyways, >> another thing I want to note is that as you work through the code in a repo, you'll notice that we build the prompts for each of these steps in code and then upload them to Brain Trust. And one of the big reasons we do this is so that we can go ahead and play with these prompts and in particular use playgrounds just like we did in homework assignment one to optimize them. The next thing I want to show you are some changes that we needed to make to the recipe chatbot application in order to properly instrument it with brain trust. And so I copied the code into my repo and you can see here I'm in the utils.py PI file and the first thing I had to do was call init logger and this will initialize our brain trust logger so we can start sending information to brain trust and right afterwards I create this light LLM wrapper and so we have a method called w light llm that allows us to essentially create a drop-in replacement for light lm but importantly what it does is regardless of how deeply nested our code is and wherever we call light llm.comp completion for example it ensures that that information is properly nested in spans within the trace and additionally it ensures that any usage metrics are reported in brain trust as well.

23:39 With that in place we can go ahead and see some changes we made to the system prompt. We've set it up so that it attempts to load the prompt that we created in homework assignment one, which is the improved system prompt for a recipe chatbot. If it can't find that, it falls back on the default system prompt presented in the course repo. And now we can actually get to figuring out how to set methods up to log them to Brain Trust. And it's really simple. So here we have our get agent response method and all we have to do to get the this particular method call the inputs the messages metadata and the responses sent to brain trust is decorate it with our trace decorator and by default as I just mentioned what this will do is create a trace named get agent response it will set the input to messages and metadata and the output to whatever the method returns.

24:48 But because we want to exert a little bit more control over what that looks like so that it's more intuitive when we look at those traces, especially with our subject matter experts, we can go ahead and take full control over what traces look like by passing in this no trace io and setting it equal to true argument into traced. With this in place, we can use currentspan.log log to to specify exactly what we want set as input, output, metadata, or anything else. And that >> was a little too much for me. I preferred lang one decorator and I'm done.

25:27 >> But anyways, >> one thing I wonder is like current spin.log makes sense. the thing that is not I wonder what like I find the default surprising because I would have assumed that the default would be that you specify what these things are. So I don't I don't have any guesses to what this default behavior is going to be. >> I think it's like as weights and biases as you can get. >> You can't do better than that in terms of like there's a minimum amount of information you need to capture from the application, >> right?

26:01 >> You got to get it somehow. I don't know how Link Smith magically got it all. >> I wonder if it's maybe he just didn't show us this piece and it is similarly because this makes very sense to me. But also W&B. Um I'm a little biased. Um okay. >> I mean I think yeah I mean it makes some degree of sense like okay you want to know what's the input output and metadata and control that. Yeah. I I kind of think I would always I would always want to do this.

26:31 >> Yeah. So, I definitely feel you. >> Um, yeah, I think I always want to do this. Maybe it's the cut like Yeah. I don't I'm not really sure. >> I was going to ask Lang Lang Chain. >> Yeah, Langchain. I I don't know. I don't think he was using Lang Chain. We can definitely go look at the code if we wanted to, but like >> we can we can keep going. We don't have too much time left anyways. So, >> yeah. Okay, let's keep going. ahead and run through our synthetic user queries, our handcoded user queries, and look at those traces to perform the next step of error analysis, which is open coding.

27:13 Application traces can be viewed in the logs view. And so we go into logs, we can see we have a bunch of get agent response traces. And clicking into each of these, we can inspect the exact spans that make up each trace. So notice that for the inputs at each turn, we capture the system and user message there. And then the output is the AI assistant response.

27:43 Also, you can see that our calls to light LLM.comp completion are properly nested and the usage metrics reported back from light LLM captured here in brain trust as well. >> Really clean UI there like that trace. >> I don't like that there's so much space. There's like so much white space on the left though. >> Yeah, that's a tr it is in this right panel. It probably doesn't need to be all hugged over there. But these like collapsible uh what do you call it? Um spans or Yeah, spans basically. like they they're they're working for me.

28:20 >> Yeah, it's like a human readable sort of uh trace viewer. It looks like it's rendering markdown actually. >> Like if you look at you see like these heading one. >> Totally. So some you know it's trying to render it seems like. >> Which is a comment you made earlier in the lang review. >> Yeah. >> Brain trust as well. So with these traces in hand, I worked with one of the subject matter experts to work through about 50 or so traces and performed the open coding exercise.

28:53 And I want to show you a little bit about what that looks like. So of those 50 traces, we actually made comments on 21 of them. And if we go into the review UI, we can kind of look at some of the comments that we made. And in particular, we're trying to identify things that surprised us. Um, identify errors >> or identify ways that were maybe unexpected and can be improved in terms of how the AI responds to the queries that it was given. And once we had all of this in place, we were >> Yeah. I mean, just like having the subject matter expert do the labeling, that's just so be beautiful. You can't you get so many points for that, right?

29:43 Like >> y >> able to then do the axial coding exercise and come up with this failure mode taxonomy document. >> I wish they had showed how they used brain trrist to do the open coding like we kind of just took it for granted that somebody open coded it. in some magical way. But anyways, >> it looks like the comet UI is on the left hand side where that's where they're entering it. But >> yeah, but it's not clear to me like what are the how to like get into my flow state was not answered anyway.

30:14 >> Yeah, I mean I I know that there's hot keys here for everything. Like you press R, it goes into the notes and you you can go left and right key. I know that he didn't necessarily show that, but that's fine. Um but yeah that yeah I presume he just uh used these common fields >> is given. And once we had all of this in place we were able to then do the axial coding exercise and come up with this failure mode taxonomy document.

30:48 >> Actually let me just go back for a second. I do agree with you. Like I want to know how he set it up cuz like if you just push R, it's not going to have this comment thing, >> you know? It'll have just notes like we have we're trying to use in class and he had this like thumbs up, thumbs down. So it'll be good to see uh like yeah, we can dig into that later like how we set this up. It's interesting.

31:10 We were able to then do the axial coding exercise and come up with this failure mode taxonomy document. And again, all the code required to do this is in the repo. So you can walk through this and recreate these same artifacts yourself. And after that thought, hey, it'd be interesting to see if we can come up with a way uh to use AI to predict for each trace what failure modes were exemplified in in it in them and then kind of come up with a way to then take that information and make it available for human review and correction. So we can come up with a golden data set of failure modes.

31:58 And so in the repo you'll see how we actually uh created those predictions and saved them as a JSON file. And with that JSON file in hand, I was able to uh upload it as a data set here into brain trust and create this particular view which so I I think I heard that he downloaded something and then he uploaded it to the same tool which is kind of funny. Uh but okay.

32:32 Um, you know, maybe maybe you don't have to do that, but whatever. >> Which might look really interesting to you because it probably looks unlike any view that we've seen thus far in brain trust. And that is because I created a custom view just for failure modes called trace failure modes. And this >> this is such a beautiful table. It's crazy that they don't help you get here, >> right? Like this is the money table that you want for your AI application anyways.

33:08 >> Yep. >> You have your inputs and then you have your failure modes. Yeah. >> You have your like custom bespoke failure modes. >> Yeah. >> And labeled for each trace. Like what more could you want? >> This is the thing that like we were saying we wanted last time like to debug the axial codes. We want to see it like almost like a spreadsheet. >> Brian, you should build this for us. Thank you. >> I already have one.

33:31 I already built this >> important for us to do the review of the predicted failure modes for each trace and also the ability to add custom columns and we can go ahead and select from a field in our input or metadata or use BTQL if the source of those custom columns is deeply nested in our traces. So I just want to predict like what are you thinking Brian? It's like I would so use a notebook is what we feel what I feel like when anything to do with this like data manipulation pulling apart things blah blah blah you know create calculated fields that's when I'm like I'm out like totally I need to be in >> As soon as he said BTQL I find myself wondering how verbose is that how like what does that support um if it's if it's just duct DB then like why do I need it in this like custom UI like I definitely would rather just play with this in a notebook. Also, the more I get these metrics, the more I start wanting to make charts and slice and dice on different groupings. I want to do some pivot tables. So, yes, I definitely agree with you, HL. Um, I built some summary statistics into our app because there's some certain things I just want to quickly know, which all my summary statistics are pivots. They're just literally like you can add tags to the prompts that you are evaluating or the experiments that you're running or the prompt like or the examples. Those are all things you can tag. And I just do pivots on those different tags so that you can say, "Oh, in general, we suck at writing summaries for transcripts related to like pitches." Um, just to quickly do the rollup. But yeah, absolutely agree. Where's my notebook?

35:24 Where is my notebook? Yeah. And then like BTQL, history is riddled with all kinds of QL. Okay. GraphQL, whatever QL. I don't want to learn this other QL stuff because >> Right. I don't know. I haven't looked into it. >> Yeah, maybe it's good. I'm just being funny. Like just duct DB though cuz it's probably just DuckDB. >> Well, can you click on it in the video? Okay. Okay. >> That's so funny. Okay. All right. We'll play.

35:56 >> And with this in hand, we now have a way for us to again engage our subject matter experts and go through these item by item and make corrections as necessary to come up with a golden a golden data set of failure modes that we can then use to prioritize the evals that we end. read the course reader. Thank you. >> One brief thing I don't love in the previous uh oh notebooks mentioned I'm getting excited. Um can you go back like three seconds?

36:31 >> Yeah. >> Just one thing that I'm noticing here is he's got this like metadata and it's a YAML file and the YAML file is keys and values. I that just that doesn't appeal to me. I >> engineers versus data scientists. It's a very classic situation, right? It's like, "Oh, I have some thing to show you. I'm going to make a YAML file." And data science is like, "No." >> Yeah. I mean, I love YAML. I just I don't feel like for this purpose, YAML resonates with me. Like to me, this is like the most clear obso like obvious that should be a JSON blob.

37:10 >> You've got >> I think you can make it a JSON blob if I remember correctly. >> Presumably, it's already human values. No. Yeah, you can definitely, but you um I didn't think you would. You surprised me with saying you wanted things as JSON blah. I thought maybe you want to show it in like a table or something. Well, >> yes, >> certainly. I mean, I definitely think that like these are binary values like show them to me as like binaries in a list like as in the UI like duh. But it's weird that even the like let me give you an example of why I would expect this to be in like a structured like I don't know YAML also supports nesting. So I just it just feels like a weird choice to use YAML here. Um and definitely not display it.

37:53 A final thing I did was actually create these plots um in the notebooks to look at the predicted failure mode distribution for across all of our traces. >> Nice. and then also to understand that same distribution but only looking at the traces that we created comments for. So that's in the notebook as well. And with that, homework two is done. If you have any issues or questions, please feel free to ask in the Discord. Thanks. >> Well, >> all right. So, >> I love how at the end of both Harrison and WDE's error analysis, not one of the evaluators was like toxicity hallucination.

38:34 They were they were automatically generated though, which is >> Well, at least there was some attempt. Um, >> but it's makes me feel vindicated. >> Yeah, I feel like both left me cold in what they chose to to evaluate. Coming back to this point of like they're not using like off-the-shelf things, but none of them felt really like good evaluators, right? Like if you ask me like, "Oh, you want to evaluate like a recipe generator?" The things that are going to be in my like evaluator are going to be like, "How complex are these recipes? Are they too complex for me?

39:16 Are they too complex for like, you know, the ingredients I usually have around?" Uh, another one that I'm going to be looking at is sort of like, do they contain like certain allergens? Uh, I think that's something that's like highly relevant. He kind of talked about that with like the plastic thing. Um I just I felt like yeah, these evaluators didn't feel really respectful of the of the like task. They felt really obscure and canned.

39:48 I >> feel like it was a jagged competence between the two that we've seen so far. I I I don't feel like there's a landslide victory yet. I don't I have to like reflect on it honestly, but like >> it just like certain things were good about each thing and they went wrong in different ways in both places potentially. >> Yeah. Okay. Part of it is like, you know, both of them are doing a task where they like have absolutely no stakes and it's really hard to like for unless you have practice like in an educational setting, it's really hard to force yourself to be the person or like put yourself in that person's shoes, right? So, I do give them a little bit of credit for like trying at least. Um, but I understand that it's hard. You got to live this like teaching life every day to to really be that person.

40:45 Yeah, it's really interesting. I mean, I definitely give Wade credit for like the process thinking of like let me involve a subject matter expert. I love that narrative even though it's like outside the tools. Like who cares? Like it's still like an important idea. Um so ultimately like you know he's he's probably more likely to get a recipe bot that is useful. Um, you know, not that he was they're both optimizing for different things, like you know, he was maybe trying to make a recipe bot. Um, but yeah, I did find the part confusing about the the synthetic data generation was definitely something went wrong there uh to like debug with Wade at some point.