Section Insights
Introduction to Claude's Eval Design
What are the key principles of eval design discussed in the blog post?
The blog post outlines principles for effective eval design, emphasizing the need for challenging evaluations, low variance, and the importance of mirroring production scenarios.
- Eval tasks should reflect real-world scenarios to ensure relevance.
- Evaluations should be challenging to avoid saturation and provide meaningful insights.
- Low run-to-run variance is crucial for reliable evaluation results.
Testing Claude's Eval Skill
How can Claude's eval skill be applied to data without application code?
The speaker attempts to use Claude's eval skill on a dataset without application code, exploring whether it can still provide useful guidance.
- Claude's eval skill can potentially assist even without application code.
- The effectiveness of the skill in guiding the evaluation process is yet to be determined.
- Using traces from real scenarios can help in understanding the eval process.
Evaluating Data Before Building Evals
Should data be reviewed before constructing evals?
The speaker argues that reviewing data should precede building evals to ensure they are relevant and effective, rather than confirming what the model suggests.
- Data review is essential before constructing evals to identify potential issues.
- Building evals without prior data analysis may lead to overlooking critical problems.
- Human oversight is necessary to catch errors that AI might miss.
Challenges in Eval Interpretation
What are the difficulties in interpreting eval results?
The conversation highlights confusion over eval results when too many criteria are bundled together, making it hard to interpret outcomes.
- Bundling multiple criteria into one eval can lead to confusion and misinterpretation.
- Clear separation of eval criteria is necessary for effective assessment.
- Complexity in evals can obscure important insights.
Feedback on Transfer Protocols
What feedback is provided regarding the transfer protocols in the eval?
The speakers discuss the verbosity of transfer messages and suggest that they should be more concise while still providing necessary information.
- Transfer messages should be clear and concise to enhance user experience.
- Excessive verbosity can detract from the effectiveness of communication.
- Feedback on eval labeling indicates a need for clearer criteria.
Transcript
0:00 All right, so this is what we're going through. Anthropic released this thing today, automating eval design and hill climbing with Claude. Now, we did check out this thing previously, this plug-in eval thing, and it was a really bumpy ride, and I'm hoping that this one will be better. I did take a quick look at it. It does look a lot more promising, at least from the blog post. But let's like go through the blog post real fast, just like super fast. So, the name of this blog post is like automating eval design and hill climate with Claude.
0:28 Okay, great. And then it's like gives you a little bit of a intro about what eval are. We don't need to read that. And then they have bundled this skill under the cloud API skill. So there's a build eval and a hill climb. and it said like in this article we highlight the principles of good eval design and hill climbing first then show how cloud code with a cloud API skill applies those principles. Okay. It has eval design. They kind of had this eval design thing where like okay eval tasks mirror production. I agree with that.
0:58 Performance improves with stronger models and more thinking. Yep, agree with that. There is passable headroom at the frontier. So, what they're trying to say here is you should have evals that are hard. You shouldn't just score 100% all the time. I totally agree with that and I like that that you know you shouldn't be saturating your own emails and you know you have a okay they want you to say low runto run variance to make sure your evals don't have too much noise. So I like that actually did I did I mischaracterize there's a passable headroom at the frontier. Hold on a second. Let me see. I think I mischaracterized that. It says the most capable model at the highest effort should be well below 100% on evaluation.
1:41 Otherwise, you can't reliably judge how changes impact performance. Okay. Importantly, the gap should not be explained by impossible or ambiguous tasks. Common tell is that task fails on every evaluation run regardless of the number of replicates. A good task is one where two domain extras reach the same verdict. Okay. So, what it's saying is like you should not saturate the eval, but you shouldn't just have it be impossible either. It should be like a really hard task that you as a human could conquer, but the AI hasn't conquered it yet. So that's I agree with that. then they talk about adversarial sampling. It's like okay model capability is jagged and you should try to focus where like maybe if you there's a way to sample where the model is failing you know you can sort of that'll be more sample efficient like you will have higher information potential. So it's like okay model capability is is jagged. If you pick cases because today's models fails at them, you are sampling in the valleys of one of the models capability surface valuation can end up measuring that model's failure fingerprint rather than what's intrinsically hard. I was telling you about sampling you say so like pick hard cases because a human judged them to be hard. Okay, interesting. So pick cases because a human judged them hard. I don't dis I don't disagree with that. but I mean you they also have to like actually fail.
3:02 >> Yeah. Yeah. Yeah, you have to find I guess like before you judge anything, maybe you've already labeled the data of what's hard. I don't know. Like I guess >> Yeah. Well, and there's a lot of things that are extremely hard for a human that these models do just fine, you know, vice versa. So, I guess I don't know that I completely agree with that. >> Yeah, I'm not sure exactly. I would say like I definitely want to sample things that appear like failures. and I definitely want to sample diverse like sample across the space like diverse sort of user interactions or diverse sort of sessions that do completely different things so I can understand get a mental model of the kinds of different things people are doing but yeah I mean yeah I don't know what this means exactly I think I'll hold my judgment okay so they finally get into this skill so you have like called cloud API build eval the build eval command and the cloud API skill turns these principles into a guided workflow And then basically what it says is like Claude interviews you, builds the eval inside your codebase and pauses for approval at specific points. Claude helps you sample inputs to build evaluations in this order. So it like looks at production transcripts. then it looks for bugs reports. Then five to 10 cases you write by hand. Cases synthesized from your codebase. And then it says pri prioritizes production traffic but it can also generate synthetic data anchored to a few real examples. Skills instructs claude to generate a simple page that shows you every input and waits until you confirm them. It's like below we show you an example of of a sample set of inputs for an email router application. Okay. So like these are some inputs and then it's like you validating the greater after the inputs.
4:47 Cloud proposes cheapest grader that fits your application's output. So either they give you programmatic verification which is code or LLM as a judge if you can't use code that's great. Then it says Claude grades a handful of cases and asks whether you would have scored them any differently. That's interesting. It tries to it just goes we we'll see. I think we have to try it. It says like so Claude like grade it first and then ask you if you would have done anything different. and it says it's important to read a sample of scored transcripts before believing your evaluator. Okay, I mean speaking my language here. Scored failures are among the most common ways an evaluation is misconfigured. Okay, it says when you validated the grader, the skill tells you the size of the evaluation set runs the baseline and prints a score with a confidence interval. Curious what the baseline is, but we can maybe we'll find out when we run it. It says, "What you get back, the cases, the greater the runner, one JSON line, and one full transcript per case, and a plain page that lists each cases score with the link to his transcript." The writing could have been clear here. Feel like that's written by Claude, but it is a Claude. I guess they're allowed to write with Claude. I mean, if anyone's allowed to write with Claude, I don't know. It would be kind of weird if they didn't write with Claude, maybe. I don't know if you want.
6:06 >> I do think they've done a lot of editing on this piece, at least by hand. >> Yeah, it's not slopp. It's not a slop. I was able to read it just fine until like right now. >> Claude 5.5 does not write this good. >> Yeah, it it's definitely it's not it's not slop. If you want more than that that page shows it has claude and it'll build extra page next to it. So these are static HTML files. So they have like some kind of viewer. I'm not in love with this viewer, but then again it's illustrative and it's for a blog post.
6:36 So, how much can you possibly show on a page just cuz it's showing you like the raw trace is kind of I don't know. But then again, let's not nitpick it too much. This is the blog post only. Okay. Diagnostic checks during the baseline. >> Also kind of curious what this viewer is cuz it's not inline in the terminal. So, is it like writing an HTML file? Is it like building a new app or something? So, that'll be interesting to see.
7:00 >> Yeah, that'll be interesting. like does it wire it up to versel or is it doing like a SQLite database or just saving them back to JSON files maybe? I don't know. >> Yeah, we'll we'll find out. diagnostic checks. Okay, it says like during baseline runs mentioned above. Cloud checks a number of things. So cloud runs a greater twice on the same output and reports whether the verdict changed. plumbing claw checks for timeouts, API errors, and cutoff answers to ensure infrastructure noise doesn't pass as model variance. Okay, I guess this this is for LM judge then maybe or something. I'm not sure. Headroom. If the baseline already scores 95% or higher, the skill warrants the user if if it alerts the hill climb should aim to explore the cost or latency rather than quality.
7:46 >> Makes sense. >> They have this hill climbing section. Let's let's try it out. Let's try this like build the eval first. This build the eval thing. So, let's try it. So, let me share my screen here. I guess we probably don't have production traces or the production tickets to give it but we can work without it maybe. >> Yeah. So I actually have prepared something here cuz I know we're doing this live stream. So I have here a terminal these traces >> and I don't have a production application. I can't like share publicly this production application but I have a 100 traces from a real application. This like a real estate AI assistant thing. I can show you like what it looks like.
8:26 let me actually open it in my browser. should have shown it here. So, let me change the window back to the browser here. So, here's what the traces look like. There's a hundred of these. I don't know if you can see this. This is like from a real estate or this like apartment leasing assistant. It's like a real production system. And basically, this is what the traces look like. This is just a HTML viewer of it just so just so you know. Okay. So, it's like the user it has a system prompt which I hidden. It's like you are an AI assistant working as a leasing team member at some apartment complex. Your primary role is blah blah blah. Okay, I know you've seen this before, Isaac. I'm just doing this for to humor the audience. So, it's like, okay, so you have a user ask a question, you have maybe tool calls, you maybe have like the assistant gives you some information back, the user responds, blah blah blah.
9:13 And like, you know, we can look at these traces. some of them are different. Like this one's like, I'm running late for my tour appointment, blah blah blah. You get the idea. There's like hundreds of the there's 100 of these traces. Some of them have issues. I am very familiar with this data already because I've worked with it so much. It's like my teaching example for evals cuz it's like from me it's like from a messy like real situation and I know I happen to know like like where the errors are and like where the tricky ones. So you know in some ways but I will try to simulate like what it is like like let's say I haven't ever done evals before. Okay like can I point claude this skill at this data? So let's try that. What what do you say?
9:58 >> Yeah, I think that's good. >> Okay, >> we'll see if it if it guides us if we kind of follow it as a smart person would. If it kind of guides us to find all these issues, that'd be cool. >> Yeah. Okay. So let's see here. I'm going to so this is the directory where that stuff is. I'll just show you. so like yeah, it's in traces.json here. >> Okay, >> we were just looking at this viewer.html file. And basically what I'll do is I'll do like so the name of that command I think it's called and or no claude API and then what was it they want you to say build and I'll say like so I voice dictated a prompt into my terminal and I'm using like their build eval command and the the prompt says I'm trying to get started with eval. I don't have application code, but I have traces in the traces.json file, which we know that I have here. There. These are 100 traces from my apartment leasing assistant that So, this is this is the this is what I got.
10:53 >> And we got herder going. It's perfect. >> Yeah, it's probably overkill for the situation, but whatever. okay. So, I'm going to go with it. I'm going to see what it does. Let's see. Let's see what what it happens. I'll read the build eval guide and take a look at the traces. Let's check it out. Now, the way the blog post was written is I think it kind of expects application code somewhere, but I specifically told it I don't have application code. Like, can you still help me? And I feel like it should be able to. I reviewed the build eval guide and profile traces 100 conversations, 1,400 messages, 13 tools, 27 distinct community system prompts.
11:29 Okay. it says which which behavior of the leasing assistant do you want a number for first? I'll build one eval for one behavior. Oh, interesting. >> feel like we should do the recommended thing. Grounded answer call transfer tool scheduling, but find failures first. That's what it recommends. That seems reasonable. >> Sure. There is no application code here, only traces. What should the eval run against? Grade the recorded outputs. Yeah, I I think so. It says >> it's interesting that it had an option to try and like replay them all through Claude. That was interesting.
12:02 >> Yeah, I don't even know what that means. >> It's like I guess rebuild the application. Yeah, I don't know. I was just like, I don't know. That doesn't seem like a possibility. Okay. Do these traces run contain real caller data such as names, phone numbers, emails? Is it fine for them to live in this repo and be sent to cloud? Yes, it's fine because I've already anonymized them. This is totally fine. And submit answers. Oh, it says I will render the traces compactly.
12:26 System prompt shown once per distinct prompt. Tool results trimmed. So, I can read them all. Okay. I'm not quite sure. not quite sure what that means, but you know, then again, it's a it's a thinking trace, so I won't read into it too much, I guess. I wonder what model I'm using. I don't even know. Oh, I'm using 5.5 sonnet. That's fine. I'm not trying to grow broke here cuz I'm not really sure how much how many tokens we're going to turn. and plus, I'm not really testing model capabilities. I don't think I'm trying to test the UX and workflow of this. And also, you don't want it to run too slow because we're trying to just test this out. I haven't noticed this before. Deep in thought, of course, like I don't really use I don't stare at Clawude like this usually for this long, but it is thinking. It's doing some kind of thinking.
13:13 >> You just do it in async. You kind of kick it off and go to the next thing or >> Yeah. I mean, if it's doing something like this, like I'm going to do something else, maybe it's not a good thing cuz then I get distracted. But yeah, if it's thinking like this, I'm not going to sit here and necessarily stare at it. It looks like it's using some kind of scratch pad. Okay, it's I reviewed all 100 traces. Now I'll verify my notes with some scripted counts and check the text channel system prompt which differs from the voice one. Okay, that's that's interesting. It kind of knows that there's a different modality.
13:45 That's cool that it noticed that. It's doing some code calculation. What is this doing? Checking dollar and square foot. >> No idea. The scripted accounts confirm the transfer and placeholder issues. Okay, it caused some transfer and placeholder issues. Okay, great. But grounding needs a closer look since voice traces answer policy questions without any log tool call. Meaning the knowledgebased text isn't in the trace. Okay, I'm not sure about that. I'm now spot-ecking two suspected number mismatches along with the weekday date claims. Okay. So, there are definitely these issues in the data like there are the things that it found. I forgot what it where what it said but there are definitely yeah there's definitely problems like that. Okay, it says failure modes from least from most to least clearcut call transfer issues 14 great 14 traces called transfer call and 10 break the prompts protocol. Six re asks would you like me to connect to you after the caller already said yes. three talk around the transfer one transfer without no confirmation at all. Okay, I happen to be very familiar with this data set and I can tell you that yeah this is like one of the top problems and it actually actually nailed it. It nailed this specific error. It like knows like the nuance of the kinds of things that are annoying like the like the user keep has to keep asking to be transferred. So it picked up on that.
15:16 I like that. Okay, voice call hygiene trace 93 says triggering end call allowed and 56 says triggering transfer. Okay, I would want to see these traces to like know I want to review that. I'm a little bit suspicious of that. Tour scheduling 35 traces book tour 7 tell the caller that the tool was rescheduled but no trace has a cancel or rescheduled tool. So the old booking stays date errors 79 tray 79 gives Saturday January 27th. Okay, this is actually true. This is like one of the problems that are hidden in this data set. There is no rescheduling tool and this tour scheduling is actually gets it. So it picked up on this. It nailed this. Very impressive that it found this. I like that. Okay. Made up capabilities and facts. Virtual tools offer with no tool support. That is a problem because there's there's some apartment complexes that don't have virtual tour and then the agent just promises it. Bathroom layout claim with no data blah blah.
16:18 Email placeholder 12 replies across blah blah end with a little literal your name. says findings that didn't hold up. I suspected wrong prices in traces 10 and 94 blah blah blah. Okay. So, which Okay, so it's giving me some data caveats. Fine. Now it's saying which one becomes the eval. >> So I don't really like this. Like I feel like from here if I was doing this easy val this would be great but I feel like the next step would be to show me the traces so I can look at them.
16:50 >> >> yeah. I mean like I happen to know that these are reasonable but I would not trust it like this. Like I would want to like I would want to like verify that what it's talking about makes sense, you know? And some of these could actually be off. Like I still don't 100% trust it. I want to know like what it's seeing is right. So is there some other sentences in here that I'm not sure about like this date errors? I'm not 100% sure about that, but okay.
17:17 >> Yeah. So I guess I wish instead of it saying which one becomes the eval, I wish it was recommending it like creating a viewer of just all the failures to look at. Like I wish that was the step it was giving me. >> Yeah, I think so too. I think you need to I think this is the point where you need to look at the data. Like why haven't we looked at any data yet? But hopefully we will end up looking at some data at some point.
17:42 you know we'll see. Now it's saying which one becomes the eval? It's a little bit confusing wording like what do you mean one becomes the eval? Like can multiple of these become the eval or just choose one first? You know I don't know. One of the problems here is is like I guess this is like you should have a course so you learn about evals because it's like which one becomes the eval. It doesn't really tell you how to pick like should you pick the most prevalent one?
18:08 Should you pick one that is harmful to the user? I don't see how you know which one's harmful to the user if you don't see the trace. So it's like I it's like if you don't know much about eval I'm going to use this tool then it's like how how do you pick? I just like why is that like why is call transfer rules recommended? Is it just because it's like the most clear or is it like I don't see any reasoning about like this one is like super detrimental to the user or this one is just super common and it's a nice starting point because it's an easy fix. Like I don't I don't see any like why why would I pick one of these?
18:43 >> Yeah. I mean there's some stuff going on here like tour scheduling. They're saying that you need a judge in the codebased thing. I'm not entirely clear that that is this is like a judge covers the reschedule and date claims. I don't think you need a judge to cover the reschedule and date claims, but that's just because I know the data, but it's no but okay, let's just so let's humor it. Let's like, okay, let's go for what it recommends, the call transfer rules.
19:10 Like I I think this is the wrong time to be jumping into emails for sure, like you said, but whatever. Let's see. Hopefully, it's making us look at something. >> Yeah. >> Okay. So, I clicked on the call transfer rules and now >> I guess if it makes us look at data now, then I would be okay. I'd be like, "Oh, okay. It's going to focus us problem." >> This is the last chance. But if it's going straight into like building the evals, so I remember in the blog post they mentioned they build the evals and then they tell you to look at the data after building the evals and then they ask you whether you agree with it. But I think that's a little bit backwards. I think you need to like look at the data and like think about like should I build an eval for this? Does it even make sense? But I don't know. Let's see.
19:50 Let's see. Maybe I'm I'm open-minded. Maybe if you look at it, maybe it's fine. Like it might be okay to label them and confirm in some cases, but you I don't think that prevents you from looking at traces from scratch cuz you still need to look at you can't just confirm what the the judge sees cuz if there's a whole bunch of problems that the LLM didn't catch or is like a product problem that it doesn't even have context for because it's in your head about what you're hoping to release in the future.
20:23 like it can't like anything it misses the human has no ability to catch it, right? If you're only reviewing stuff that it found. >> Yeah, I think it's a little bit of a dark potentially a dark pattern. Okay, so let's see what it's saying. It says the inputs are in this folder claude hill climb transfer protocol inputs.md. I don't know why it's in a folder called hill climb because I'm not trying to hill climb anything right now with labels in cases.json.
20:50 It has all 33 conversations. So these are like 33 conversations about the voice transfer rules. It found like 11 should transfer, 19 should not transfer, three unclearars. Then it has like these like unclear things. It says, okay, >> so it labeled them. >> Yeah, it did some labeling. It says, "Please skim inputs.m tell me whether these look like the calls you care about." Okay, let me see what inputs MD looks like. >> I'm getting I'm getting more skeptical by the second telling me to look at the data in a markdown file. I'm a little skeptical.
21:22 >> I don't know. I really don't know about this. I really don't know about this. But let's see. let's see this. Okay. >> I've never I've never decided to look at, you know, traces and data by putting it all into markdown file. But >> okay, so this is the markdown file. and basically what you have here is like you have the trace number I guess the section, the tags, the turns. Okay. So you have some trace ID and what you have a trace. I don't know what any of that above means really. I'm like I don't know if that comes from the data. Like you probably do, but as someone who has not looked at this before, just starting emails, I'm like, did it lab did it say should not transfer and should transfer?
22:02 I think those are LLM labels. I don't like I guess the turns the number of turns. >> I would say this needs to be a HTML. Like come on, man. Like we have agents, you know, like we can do it. >> We can have something nice here if we wanted to. most of that top table doesn't help me do anything. So >> yeah, that top table I I agree. I'm like I don't know why this is Oh, it says open viewer HTML including the system prompt. Should I try that?
22:31 >> Yeah, let's do that because this looks horrible. I don't want to read a trace markdown. Like why would I? Okay, open viewer.html. >> Where is viewer.html? I actually don't I'm like really skeptical that there is Let me Let me go into this folder. What is in that? Oops. what's in that folder? There's no viewer.html in that folder. I think it's, you know, I think it's getting confused by the my viewer.ht. See, cuz I have a viewer.
22:57 HTML that I created. So, I don't think it's referring, it's not something that it did. It's conflating the two. So, okay, fine. It has this markdown file. Now, in this case, you could say, I mean, this kind of sucks to read. I wouldn't I wouldn't encourage reading data like this normally, but okay. Okay, it says like this call may be recorded for training quality purposes. How can I help you today? User says preview program. It looks like you might be interested in learning more about our community. and possibly scheduling a tour. Would you like to schedule a tour to see Mebo Gardens? No. let me just keep scrolling down. Says like, "Would you like me to It says seems like you might have a specific question or need assistance with something." I'm okay.
23:37 So, this there's a failure here. There's a different kind of failure here, which is the bot is being annoying as hell. Like it's saying it seems like you're asking about this. Like looks like Yeah, it's like you're getting therapy. I don't want Yeah, it just Yeah, you want to not talk like that to people. It gets really annoying. >> Oh, that next one is really bad. Great. I'll connect you to It says, "Do you want I'll connect you with a representative who can assist you? Do you want me to do that now?" Yes. Great.
24:04 I'll connect you to a represent who can assist you. One moment, please. Would you like me to connect you with someone who could help you? So, it asked again. So, that that was a bad failure. >> Yeah, it says I just want to talk. It says assistant. It finally at some point it did transfer the call. Now, I think like this specific trace, this is trace one. This is marked as should transfer. So, I'm thinking like it yeah, it marked this as a should transfer. And I guess we're supposed to look at these and decide whether or not we agree.
24:32 >> Oh god. Trace 18. that tool result in line is just like how am I supposed to that's good. Okay, >> you shouldn't you shouldn't have to read data like this. Again, this is not this is not 2004, you know. We should be able to make HTML apps. And why did it truncate the tool call tool result? It's completely wholly unnecessary, but fine. Said give me a moment. We have several onebedroom lots available here. A few options.
24:57 >> So, I'm I'm a little lost is what we're supposed to be doing with this. >> yeah, I'm a little bit lost. So, let's see. What did they ask us to do? It says, "Please skim the inputs. Tell me whether these look like the calls you care about, whether any label is wrong." Oh my god. So, we have to like tell it like label like no, you should again you need we need an HTML app that allows us to just like change the label. Like why are we editing like Neanderls? Like what's going on?
25:23 >> Yeah, I don't understand why you would Yeah, this is I guess what are you supposed to say? Like trace one? Yes. Trace 18? I disagree. Trace 14. >> Trace one is wrong. It should be I mean like no I'm not doing that. >> Give me a break. >> They're in this markdown block and it's like okay these are the turns. You can see the tool call result is truncated. It's kind of a mess to try to read this.
25:48 And what they want you to do is to manually decide which traces you don't that could be labeled incorrectly. And this is the label up here. and they just want you to like tell the agent what to do like hey trace 18 is should be doesn't need to be transferred and trace 19 is wrong too because and they want you to just act like you don't have any technology available to you like you're you're like HTML hasn't been invented yet or something. So like I'm being like overly harsh. I mean whatever.
26:19 >> No, I think it's fair. I think it's pretty fair. I you tell me we've only looked at a couple and it's painful. There's no way I would look at a 100 traces that way and then do this. >> So, we could like say, okay, you know, we so it's telling us to skim this inputs imd and say like whether the labels are right, whether these look like the calls I care about, whether anything is missing. I mean, how the hell am I supposed to know if anything is missing, right?
26:43 >> I guess it it assumed that I looked at the data. Okay, so what should we do from here? Should we just say like yes, it's fine and see what it does now? Yeah. I mean, I guess >> most people would do. So, let's see. >> I mean, I guess you looked, you didn't see any issues. So, yeah. >> I spot checked it. It seemed like it's probably a fine. So, I'll just say yes, it's fine.
27:03 >> I don't see what else you can do with a markdown file other than just spot check it. >> Yeah. Markdown file terminal. It's like we're on a Yeah. You know what you should be doing is to redirect this tool to be like, "Hey, I'm not looking at a markdown file. Like, create an HTML thing for me." And then you would have it do that and then have it do whatever it needs to do. I'm not trying to go down that rabbit hole because I don't want want this to become like 4hour exercise. Okay. So grade grading first. There is nothing to build here because traces carry no usage or latency. Okay. I mean now the grader and the runner. It reads the recorded traces grades each with code and writes results JSONL and percase traces and the reports usage. I'm not entirely sure what it's doing now.
27:50 >> I'm very confused. >> It wants to run some rmrf command. Fine. Whatever. >> Okay. You're very brave. I saw the rmrf. I thought you were going to kill it, but you just you just accept it. Okay. >> It's fine. We're We're good. We're good. >> Cuz I have that I have actually like the certain directories are blocked. So, it looks funny, but it's fine. So, what is it saying? Let's scroll up and see. Oh my god, I have to read all this. Okay, let's see. I built a code grader from the prompts transfer rules with no judge model on the 14 calls that transfer it checks one confirm exactly one. Would you like me to connect you before the transfer? I don't know about that, man. It's going to it's like checking that string. Would you like me to connect you? Really? Okay. That that's weird.
28:34 >> That seems like a really brittle test. >> Yeah, that's going too far in the codebased approach, I think. Then it says immediate. Nothing is said between the caller's yes and the transfer. Okay. But there's a lot of ways to say yes. >> Yeah. Checking explicitly for yes as well. I guess >> silent nothing is said or on or after the transfer. I don't think that's the right thing to say. If you transfer someone, you can say like, "Hey, I transferred you. Thank you." There's nothing wrong with that. You don't have to be overly like I don't know where I came up with that rule from.
29:06 >> The assistant never says triggering or other tool mechanics allowed. So, I guess it's that feels like >> that no leak though, it feels like it should be a separate eval to me rather than >> like I I would split out. I would say like, "Oh, well, let me make sure that let's do an eval on are we transferring calls properly?" And then that one feels more like a some sort of tone thing that I would split out.
29:29 >> Yeah, absolutely. Yeah, this should be two separate evals. I 100% agree. Okay. Protocol, okay, is the headline. This is It's like so hard to read these sometimes. A case passes only if all four checks pass on should not transfer calls protocol okay is one if no transfer happened baseline results should transfer to protocol to okay wait what does this mean again should transfer you have 11 cases and what is protocol okay >> protocol okay protocol okay is one if no transfer happened >> really >> okay that's not really >> on should not transfer calls protocol Okay. Is one if no transfer happened.
30:10 Okay. So, it's like Okay. I guess like two transferred happened when it shouldn't happen. >> Wait, two no transfers happened. >> I don't know what the hell is going on cuz it's like >> I don't know why it's making us like protocol. Okay. Is one means that there was no transfer. So, I don't know why. >> See, this is why you shouldn't bundle so much into one eval. Like, see how uninterpretable it gets like so fast cuz like we have all these rules. One confirm, immediate, silent, whatever.
30:37 I'm like, >> see, I feel like if protocol okay is one, if no transfer happened, >> they could just call the variable no transfer and then one is true and zero is false. Like I don't I don't get why it's protocol. >> It says protocol okay is the headline a case passes only if all four checks pass. But then see this is this sentence is confusing. It says on should not transfer calls. Protocol okay is one if no transfer happened. But I thought all four checks have to pass. Why is it a one only if no transfer happened? That's like see that's the problem with slop like you can't like that you can't read the reasoning because like it's so unclear and murky. You're like it's hard to understand what what the f is happening.
31:19 >> So I'm starting to feel like they wrote a good blog post and that's the takeaway. >> This is better than a lot of things I've seen. This is going this is getting better. This is way better than the last thing that we did. the last thing we got. That was a dumpster fire. >> Yeah, this actually like it actually like is getting it has some promise to it. Like I could see some life. >> it did try to get us to look at the data. It did. So that's good. Yeah, >> it wasn't it wasn't in a good format or a good way or good interface, but it it did try.
31:52 >> So it has like two Okay, two of the 14 transfers pass a trivial never answer transfer blah blah limits to keep in mind. Would you like would you have scored these any differently? I don't know if I would have scored these any differently. Like, what the hell are you talking about? Is there something you care about that isn't measuring? How am I supposed to know if there's something care about is not measuring? How am I supposed to know if I would have scored these differently?
32:12 >> I don't know which ones they are. Yeah. >> Yeah. I don't know what the hell is even going on here. >> What are some useful applications they probably had in mind for this? I mean, I think they were just trying to do general AI evals. >> No idea. so like would I have Yeah. So, we don't know the answer to this question. Is there something you care about? Okay, I'm just going to be truthful with it, okay? Be like, I have no idea what the you're talking about. I don't know what cases you scored specifically, and I have no way to look at them. And so, how am I supposed to know would I have scored any case differently cuz I don't even know how things were scored. You're just showing me aggregate statistics. I don't even understand what the aggregate statistics mean because the rules are confusing. Is there anything that you can do about this? Like you know what is what are the next steps here or what can you show me like a more detailed view of this or like a HTML file or some application that I can look at or something that help me figure out what's going on. And I, you know, I'm just humoring now. I'm like I'm not trying to act like a smart user who like is stuck cuz like I don't know how anybody would go beyond this point right now.
33:24 >> All right. I think this is all you can do. you can either kind of abandon it, but like this, it feels like it's already gone a bit too far without being grounded in what's actually happening in these traces in this data. And so it probably should have at first point where we talked about it would be nice to look at the data even before we chose this like one eval would have been a better place. And like you're familiar with the data and we're already getting lost. So, if you weren't here and you weren't familiar with the data, then like I I I don't see how I would even know like what call transfer they're talking about if this was a brand new data set and you weren't here to explain what some of this data is.
34:04 >> Okay, so this is where it gave me this it gave me this thing. It gave me this thing. So, like I complained. It gave me this like call transfer rules but was scored. Okay, >> better. Okay, >> it's better. this >> okay so the fail >> one trace one should transfer four confirmation questions before the transfer and needed exactly one so that's clear four assistant or I don't know what was at the top but okay that's interesting so what is it fail I agree with the verdict okay wait I clicked agree with the verdict and then it just took me to the top okay so we need to basically click on this next one so I said it says agree here this one says should transfer >> okay So, the bad the bad web design might be that you're using sonnet, I guess. I don't know.
34:50 >> Yeah. Yeah. I mean, it's it's not So, it says this one fails. Zero confirmation questions before the transfer. No, I don't agree with this. Like, it says you want to speak to a live person and they they just did it. So, I'm like I would score it differently and I would say like it transferred right away when asked. Okay. this one so like you know this is like backwards, right? I would have preferred to like go through this myself like at least show me the traces and have me open code the traces or have me make notes on the traces so I can understand like what is good looks like by having it score it is bias trying to bias the hell out of me and saying like you know so you have to fight it and say like okay yeah so like I agree with this verdict it's fine >> so that's so that's interesting let's look at those last two so trace 18 you said it was you disagreed it was a pass cuz they said they said speak me to live person and it just did it and that was good and then the next one they asked for a person right and it did a confirmation before it I guess either of those is fine whether they there's one confirmation or or none >> I missed that >> it said call a receptionist please would you like me to connect you with someone who can help you okay I guess like oh you're right >> it still feels okay but >> no it's like it should have just transferred it Actually, this feels okay. It's like I would say if you just say a receptionist, please. I think I would say I agree with the verdict.
36:21 Fine. Okay, this one fail. Would you like me to connect? Yes. Would you like me to connect? Yes. Yeah. I agree. This is fail. Okay. Would you like me make Yep. Yep. Yep. Okay. Yep. I agree with the verdict. >> So, if you refresh this page, does it lose Do you Do you have to hit copy all notes as JSON? >> I think you have to copy all notes JSON. Yeah, you have to copy all JSON. I'm pretty sure. I'm afraid to refresh and I'll lose my stuff.
36:44 >> Okay, >> fail. >> Fair enough. Okay, >> fail. And then why did it fail? It says, "Let's can I talk to customer service agent?" It didn't annotate that. You see, it annotated ask to connect. This should be annotate as connect. So, some issues with annotation here. Connect me to representative. Let me connect you to representative and assist you further. Please hold for a moment. Ooh, interesting. >> okay. So, something didn't seem right. caller. >> This is something else. This is interesting. It's like This is interesting. It failed, but it like >> So, it feels like it should There should have been a tool call right after that and there wasn't, right?
37:20 >> There wasn't. No, it just it just like duped the user. I was like, "Yeah, don't worry about it. I'll do it." It's like it basically lied to them. so, I would say like I want to fail. I want to fail, but I also want to note that this was a false confirmation. I would actually split this into I maybe have a different email for this cuz this is like a different kind of issue than the constant like badgering the agent to like transfer you. This is like you're telling me I'm transferred but it didn't get transferred or like something funny is going on here and I would want to I would want to dig into this particular thing. So maybe I'll just disagree cuz like I don't really know. I guess I'll just say disagree. Okay, we'll do like two more. This one traces. Okay, this one should transfer. Would you like me to connect you? Yes.
38:10 >> and then it doesn't that seems okay. >> No, but it did this thing afterward. Let me connect you to representative. It just >> so it did the tool call and then it said let me connect you triggering a representative. >> So I think I agree with the verdict there. Something, you know, >> well pass. >> So it when it transfers, it should transfer and then say nothing is what you're saying, right? >> It should say nothing or say something that makes sense. This says like it transferred them and then it said, "Okay, let me connect you." It already connected me. Like why are you why is it saying let me or maybe it >> Well, I guess there was it was a transfer success message.
38:47 >> Yeah, >> I don't know. I guess it depends how the system's working. But I guess there was a success message. So I would say if >> you know what I think you're right. I would say dis Yeah, it said it's basically saying like please hold on for a moment. Triggering transfer to representative. I'm forwarding your call to representative. >> It's a little bit too verbose here. Like what? Just say like, "Okay, I'm doing it." What What is all this stuff? so I would actually say like, it's maybe a bit of a tone thing, but I guess if if it transferred me and I got that message as a user, I don't think I would be super upset about it. I'd be like, "Okay, it's doing the thing. I'm getting a person. I'm getting what I want."
39:26 >> I would say like I Oh, yeah. the too verbose of a confirmation. Okay. So, let me copy the notes. Let me just say like, hey, it's a little >> I disagree with some of the ways you label things. I look through some things. Here are my notes. >> Yeah, it's a little interesting because you're saying I would score it differently, but it's like what you're saying here is like you would still fail it just for a different reason, you know?
39:53 >> >> yeah. And I don't know if this data that I pasted in actually like Okay, it only gave the ones I labeled. It said trace 18 caller said speak to live person >> your terminal. We can only see the >> Oh, got it. Thank you. Okay. So, I pasted that stuff into the terminal and now it's starting to ask me questions. So, it's like, see, I pasted in the terminal up here. Say, hey, I disagree with some of the ways you labeled things. I'll look through some things, blah blah. And then it's like okay the caller said speak to a live person and the assistant transferred at once with no confirmation question you mark that fine. So when a caller clearly asked for a human okay I would say >> you going to say okay >> I'm going to say type something >> it seems like >> if there's any chance of it being ambiguous then I would say ask a question. If it's unambiguous then don't ask a question. Okay it says traces both say both say while transferring 18 says one short line. I'm foring your call 56 several sentences. I say one short line is okay. Submit answer. I don't like this approach. You're like building the eval and then you're correcting the eval versus versus like looking at the data first. Building like a good building a mental model of like what good looks like for you and then going into the evals. Like this is you're trying to like create is basically like create some slop and then fix it.
41:17 >> I don't know. I like I think it's not I I would prefer to look at the data first. I like that it can help you find interesting data potentially. Like it did find interesting data to look at. It just didn't nudge you to look at it correctly. >> Yeah. Well, there was there were kind of like three different errors mixed into one eval which is going to make it, you know, kind of annoying to fix. And so I'm not sure if it's going to split them out into multiple evals or if it's going to keep them all as the same eval. And I guess it refreshed the app so you lost all your annotations.
41:58 >> Yeah. so the question remains like is this which way is more efficient? Is it like let's let's keep an open mind. Is it more efficient to create your email first and then like start fixing it? I feel like strongly like the answer is no. I feel like you don't know what you want until you look at some data. You don't know what good looks like and it's really helpful to look at some data before you jump into evals. Also, the way of going through it is has a lot of friction like the markdown file and then like the labeling it encourages you to be lazy because you're like I don't want to go tell the agent like all the instances of where it's wrong. Like you just so much friction, right? And so you have to have you have to know like hey this is not a good process and like build your own HTML apps and labeling apps annotation apps go through it. But if you're going to go through all that work to build annotation app, might as well just like forget the eval and just like look through some data and like introspect like what is wrong? Because like when it's trying to create an eval first, it might be doing it with a really bad design. Like like we saw, it might be trying to bundle in too many things at once and make it very confusing for us. even in our annotation where like we found like new things like that are wrong that we would actually split up into other kinds of emails or like something else. And so, I feel like it just nudges you in a bad direction.
43:24 >> Yeah, I agree. It just I think it comes down to like the way it organizes all the information makes it hard to understand what's going on. So, there's lots of examples of that. Like I don't know how many emails we have right now. I don't know if we still have one eval or if it's split it into like three evals. And then when I think about what are the problems that this product has, I guess there's the tone issue.
43:48 There's this transfer issue. I think there's actually two transfer issue. Like I don't know what the problems with this product really are cuz we kind of just started looking at the call transfer, but we don't actually know if that's like you might know, but anybody who's just starting to look at this data set and starting to do evals, like they just click this recommended thing. They don't know if this thing is important or not. >> Yeah, I found it very confusing and I'm I'm an I'm an expert in this data set like I've been looking at this data set for like more than a year like very closely. So >> I feel like if you had sent me if you had sent me this this this the the this data set and said and I didn't know anything about it and you were like tell me what I need to fix with my product I would probably just have immediately opened a data viewer just been used cloud code make a data viewer and just gone through and then I would have been like what are the important things cuz to me I'm like okay we're looking at these call transfer things but I don't actually know like I don't know how often they happen relative to the other issues. I'm still not clear like are there cases where a person never gets successfully transferred because that would be what I like if that's the case I would want to prioritize that over everything else like yeah sure asking twice is annoying but if every time it asks twice they do get connected eventually then but there's also you know a bunch of cases where they never successfully get transferred to a rep and then they fall off I'm like well I'm going to go focus on that first and maybe they're the same thing and maybe not and so I think it's a I think That's to me the problem is like we're already beyond the friction with the data. We're already going down and focusing on one particular issue without actually having any idea if this issue is the important issue to be working on cuz like from this it feels like oh I should just go keep working on this eval but I I don't know like there was other issues that it came up with. I don't even know what they were at the beginning.
45:36 >> You went into we start you go into like this deep rabbit hole in this one eval like you're right. It made you pick the most important eval like very early and then you just went straight into like polishing that one eval. but we didn't really know we didn't have enough evidence to know like that's the most important one that we should be focusing on. >> Yeah. We just like I don't know. Claude says it's important so I guess we'll click that. Like that's that's what I felt cuz I was like it just says recommended but I haven't seen anything so I don't know. Yeah. Okay. So, what's your verdict? Fail or pass?
46:08 >> I would fail it. Like to me, I I leave with this. I wouldn't use it myself. So, that would be the thing. I feel like if Yeah, I feel like you would get further if you didn't know if you didn't know much about evals, maybe someone watching's never done eval. I feel like someone who's never done evals before would get further if they just got the data, they made a data viewer, and they just went through a 100 traces and just took notes and that was all they did. I feel like they would be in a better place than we are in with this data set now. Even if they didn't do any fancy axial coding, they didn't do any grouping. They didn't do any, you know, automated testing. I feel like they have a much stronger grasp on what they should do if they just like went through 100 traces and just wrote some notes and that's it. That's my my take, I guess.
46:51 >> Yeah, I agree. I agree with this verdict. I'm Yeah, I don't really have anything to add to that. I think we said everything we wanted to say here. So, I think I think that's a wrap, man. >> Thank you for joining. All right, great. >> Take it easy. All right, bye. >> Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on eval to help you. It draws from more than a,000 hours of working with companies on building AI products and it's customized to what stage you are at. So, this is how you can use the guide. Go to bit.ly eval-guide. When you open the guide, it'll look like this.
47:26 And what you do is you select the stage that you're at. Maybe you're new to eval. You don't even know what eval are. Maybe you do have eval, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too timeconuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly Evails-guide and we'll send it to
Summary
- The new feature is part of Claude's API and focuses on automating evaluation design and hill climbing.
- Effective evaluations should reflect real-world tasks, maintain a balance between difficulty and feasibility, and minimize noise in results.
- Claude assists users by sampling inputs from production data, bug reports, and user-generated cases to build evaluations.
- The evaluation process includes diagnostic checks to ensure reliability and consistency in grading.
- Users are encouraged to review scored transcripts to validate the evaluation process and avoid misconfigurations.
- The system allows for iterative feedback, where users can adjust evaluations based on Claude's initial assessments.
- The overall user experience is mixed, with some concerns about the clarity of the process and the format of data presentation, suggesting room for improvement in usability.
Questions Answered
What are the key principles of eval design discussed in the blog post?
The blog post outlines principles for effective eval design, emphasizing the need for challenging evaluations, low variance, and the importance of mirroring production scenarios.
How can Claude's eval skill be applied to data without application code?
The speaker attempts to use Claude's eval skill on a dataset without application code, exploring whether it can still provide useful guidance.
Should data be reviewed before constructing evals?
The speaker argues that reviewing data should precede building evals to ensure they are relevant and effective, rather than confirming what the model suggests.
What are the difficulties in interpreting eval results?
The conversation highlights confusion over eval results when too many criteria are bundled together, making it hard to interpret outcomes.
What feedback is provided regarding the transfer protocols in the eval?
The speakers discuss the verbosity of transfer messages and suggest that they should be more concise while still providing necessary information.