Section Insights
Introduction to Evaluations and Raindrop
What is the purpose of this evaluation course and who is Ben?
The evaluation course aims to explore various aspects of evaluations, inviting experts like Ben from Raindrop to share insights. Raindrop focuses on identifying critical issues in production agents and simulating changes before they go live.
- The course invites industry experts to enhance learning.
- Raindrop's philosophy aligns with the course's teaching methods.
- The company specializes in real-time issue detection and simulation.
Navigating AI Trace Analysis
How can one effectively analyze AI traces without getting overwhelmed?
To avoid confusion in analyzing AI traces, it's crucial to utilize annotations and comments that guide users to relevant points. While catching issues quickly is important, preventing them is even more critical, especially in high-stakes environments.
- Use annotations to simplify trace analysis.
- Understanding the real-world implications of AI errors is vital.
- Preventative measures are essential in high-stakes applications.
Simulation Awareness in AI Agents
What is simulation awareness and why is it important for AI agents?
Simulation awareness refers to an AI agent's ability to recognize when it is being evaluated or tested. This awareness can lead to altered behavior, which is problematic. The goal is to ensure agents believe they are in a real-world scenario to accurately assess their performance.
- Agents should not recognize when they are being evaluated.
- Simulation quality is measured by how indistinguishable it is from reality.
- Self-awareness in models can lead to biased behavior during evaluations.
Scaling Evaluations with Traces
How can evaluations be scaled using traces?
Evaluations can be scaled by defining criteria that apply to any arbitrary trace, allowing for consistent assessment across different scenarios. This approach aligns organizational understanding of what constitutes good or bad performance.
- Define evaluation criteria that can be applied broadly.
- Align team understanding of performance standards.
- Utilize traces to ensure consistent evaluation across models.
Challenges in Simulating Complex Systems
What are the challenges in simulating complex AI systems?
Simulating complex AI systems is challenging due to the intricate infrastructure and tools involved. It requires creating accurate representations of the system's environment and ensuring that simulations faithfully replicate real-world conditions.
- Simulating complex systems is one of the hardest problems in AI.
- Utilizing sandboxes for testing can help replicate environments.
- Continuous testing and alignment are necessary for effective simulations.
Transcript
0:00 We've been running this eval course for a while now and every so often we bring people in who are working on evals and there's a lot of different vendors in the space and I have used every single vendor and I try not to have like pick favorites but I really do like raindrop you know I like their philosophy like how they go about things they tend to do things that are very aligned with the way that we teach in this class and so I think it's really helpful for you to see different aspects ects and things of evals. We thought, okay, let's invite Ben to give like more color around the way that he thinks about evals, what he's working on, etc. I think it would be really beneficial for y'all. And so, yeah, I'll just go ahead and kick it off to Ben.
0:43 >> So, talk today is called getting started with simulations. I'm the CTO and co-founder of Raindrop. This is both a product that we recently launched, also something that just in general everyone in the space is starting to do, labs are really starting to do, writing papers about. It's not not eval. We'll get into it in just a little bit. Our whole company is about finding critical issues in production agents. So, we find those issues in real time like minutes after they start, verify those fixes actually work without side effects, and then now we simulate those changes before they land in production. We also have this really cool open source tool, which honestly like especially if you're ever hacking on your own thing, it's a really nice like starting point. I use it even in like a lot of my own like side projects. I'll just be like, "Hey, take this UI like either use it, change it, whatever you want." It's kind of cool.
1:30 Of course, the tool that like our Fortune 100, Fortune 500 customers use is like what we call raindrop. And we also make this really cool guide, very very Hamill inspired called like how to eval.com and probably due for an update even even though we just published it like what like two or three months ago or something like that. But the space moves fast. Everything's changing which is why these these courses matter. Everything has changed to agents.
1:51 And this isn't just like a a word changing. It's actually kind of a big it's a big deal and I'll kind of explain why. So like in this kind of chatbot era the way that we would think about evals and again the word eval is very overloaded. So I don't mean to say that like oh like I'm not trying to say eval are dead or something like that. but just like what we used to consider the way to eval is the right way to put it. What we used to consider an eval like maybe a little over a year ago was like you have a prompt, it goes to the model, you have some sort of response that comes out and then you have some sort of like predefined way probably for that given input to know that that output is correct. Like a very contrived example, but an example that you'll still see on like vendors websites even now is like you know what is the capital of like the United States and then it's like oh well it's Washington DC. So it's like input output validator. The thing though is that like things have changed a lot since then. One is that like agents are a lot more spelled out, right? So it's not no longer that you have this like nice clean like, oh well, you just have to, you know, you change your prompt and then you rerun and you see what happened. Well, it's like, well, now it's like you have skills, you have the tool definitions themselves.
3:00 Like just adding or removing a tool can completely change the agent's behavior. Even just changing the input type to a tool can change or even the field name to a tool can change how the agent behaves right so it's like clearly it's a lot bigger than the prompt now you know users connecting MCPS you have of course you can change the model etc but then now you also have memory you have sub aents and then the prompts for those sub aents and a lot of these also increasingly are also outside of the repo right so you don't you no longer have all the context even in just the the repo which is really interesting.
3:35 And the other thing and kind of like which is related to that last point is like agents are very intertwined with the world now, right? Like they have all their little fingers and arms out touching the real world. Whether that's your own sort of internal production databases, whether that's other services and again no longer is all the state and all the context like the emails that the agent goes and reads are now part of the like that's part of the context. it's influencing the agent's behavior and what it's going to do next. That makes them a lot harder to eval. And also, they've become more and more capable, more and more complex. And what that means is that sort of like single input to output pairing really starts falling apart because most of what your agent is going to do is and most of where it's going to go wrong truthfully in practice are going to be things that you haven't really thought about yet. And that's what makes it really hard. And so I think that there was this like for some time in the eval world, there was this like more is more strategy with evals, which is like the more evals you have, like you have 4,000, 5,000, 6,000 evals, and then that is going to like that's your mode as a company, that's going to be the thing. And it's like, well, like, you know, there can be valid cases for that. like if you're planning on training your own like frontier model or your own model like there are cases and we can go into more details in the Q&A like where that really does matter but I think largely it's actually more about your strategy around evals than like the number of input and output pairs you can collect because again capabilities are increasing so essentially like your agent can do more things faster than you can make evals and the other problem is that like you start running into this problem where it's like okay so now I have 4,000 evals let's say if I add a tool am I let's say I add some sort of tool like am I actually going to run 4,000 emails in that PR like probably not right so it's like well how many are you going to run how do you find the right ones I think this is why like as a company like we actually like kind of stayed away from eval for a really long time we were kind of like we focused on the real world because that felt like the best agent thing to do like it let us catch issues you were never expecting it let us like have a lot of signal around those issues it's not like so for example one thing we saw with eval was that the harness would and the eval would break. Like the evals were expecting a specific tool to be called, but then you switch to a different harness and now everything breaks. So like the reality today, even some of the best companies in the world AI wise barely have functioning evals and I I've seen this time and time again. I won't call anyone out, but like even some of the poster child of EVALs companywise, it's you talk to them and they're like, "Yeah, well like all our evals broke when we switched to like you know quadcode as like the harness underneath or something like that." Like so it's actually really really interesting and so we really focused on again we really focused on production monitoring and we did that essentially we think of it and like we call it unsupervised real-time issue detection right so it's like there's three ingredients that go into that production monitoring the first is like your codebase your prompts right so it's like we can sort of understand like based on how you've prompted your agent based on the code base like what should be happening right so like that kind of gives you like one data point the next is like the baseline of behavior. So if something like you make a change and it it drifts like that's an interesting signal. And the last thing is like your feedback on the production data. So like looking at traces. I know HL like emphasizes a lot like look at your traces. Like we'll send issues, we'll send summaries, we'll send like alerts and based on how you interact with that we can learn like what you care about and what you don't care about. So again this is sort of like how we've figured out how to do that eval but in production, right? like how to like really really high signal when our almost every single issue we send a team they react to they open they do something with now which is something we're incredibly proud of and I think when you use other tools and you're just getting inundated by like random digests and summaries it's like well that's not useful right but this is just sort of like again we do these deviations on we figured out as a company how to do these deviations on not just like you know error rates and things but things that are like very semantic like understanding you know unusual patterns of the way that the agent's responding the trajectories it's taking etc This is what an issue looks like in Rachel as well. Not to hammer this home too much, but one thing that also I think teams really get wrong, somewhat of a tangent here with like issues, this sort of thing is like just show the traces. Find have tools that show you your traces.
8:01 Like do not drown in like AI slop summaries of what your agent's actually doing. Like so for example here is like we show these like little annotations these comments on the trace that kind of helps you jump to the right point which is important especially now that traces can be tens of minutes if not hours if not days but just like don't get lost in the sauce of like AI summaries what happened and what went wrong just like have ways that accelerate finding the right traces and help you understand the trace that you're looking at. So, I just talked a lot about production, but the thing is that like that is nice, but it doesn't prevent issues from happening.
8:35 Like it's good to catch an issue as soon as you can. It's good to know that your change actually did something well in the real world. And certainly the real world is like a really powerful arbiter of truth, right? Like it's again, it's not like, oh, your harness broke something and then your old eval stopped working. It's like, no, like people are complaining, people are mad or like no, it's really working. Look, you can see people have like, you know, recognized a difference somehow. But again it doesn't prevent issues from happening and especially as we have like Fortune 100 customers as we have like you know even like defense you know increasingly you know there was a story recently about the US misidentified a Chinese vessel as carrying nuclear weapons of some sort and like intercepted it based on bad information from from an AI we don't know which one but like so clearly like this is like getting higher and higher stakes right as we go into like healthcare defense finances and so as a company we're like how do we actually prevent these things from happening in a way that is you know, ridiculously fragile as well. And I think whenever you're faced with this sort of question, it's really useful to take a little bit of a step back and say like, what would OpenAI do? Like OpenAI, you know, or like insert foundation lab here, but like they're like a big company. They have a lot of users. They have a pretty good product that's like, you know, pretty pretty stable. And and so like it's worth like Codex, for example, I think is like one of the best AI products that exists in the world. So it's like worth asking yourself like, well, what what do they do, right? And this is one paper recently on Codex.
9:56 They've had a few around simulations. And essentially the answer I kind of gave it away actually is the answer is simulations. And so every time they make a change to a model, they have a whole rig that lets them simulate the past traffic on codecs and say like well like how would this new model change the previous codeex trajectories? And so the really powerful thing about this is that you're now sort of answering the question of like what would my change change, right? Like I think that's like a lot of times the the rut that we see teams get stuck in is that they have like a lot of times the fixes and getting to the point where you have a fix to some sort of problem with your agent or add some sort of capability. I think getting to that point is actually getting faster than ever. But then there's this whole like consequence of making that change and that's the thing where that really starts slowing teams down. So really simply again simulations are you replay real traffic real historical traffic against your new change and you see what's different. So that change could be anything right that could be a new model it could be a new tool. It could be removing a tool. It could be changing the prompt. It could be like you switched from AI SDK to PI.
11:04 It could really be you know as big as as you want here. And so you're sort of like again you have the old traces that you've collected and you find a way to just rerun your agent on those exact traces. So again what will my change change? Does it actually fix it? Does it cause other issues is kind of what what you're asking. Let's just take like one very specific example is like let's say you delete a tool. So what you can do is like find traces where before you made that change that tool was called like real user events where that tool was used to answer the question and then replay those and then say does the answer change right and then if the answer changes you know it flag those and those are the ones you want to review. So this again becomes a way that is like one extremely targeted to whatever change you're making to your PR or like whatever change you're making in your PR and like again it really really helps answer this question like what does my change change >> so when you replay traces from production let's say you know your agent is like has some multi-turn stuff going on user is asking follow-up question how do you deal with that >> this is a very very good question so this is I think one of the things that's actually very powerful as opposed to like the way that most people think about eval a lot of times we see teams do like these like single turn evals and I think the single turn versus multi-turn is actually a very like somewhat convoluted term right now. So like there's multi-turn as in you're like 20 messages into a conversation and then you ask the next question and then there's multi-turn as in you see the output and then do another question see the output. I think the former of those doesn't really work right now just to be like 100% honest like I think that there's people trying to work on user simulations like as in more like user prediction and I think that's really hard problem because you know you take something as simple as like a user looks at the UI and they're like oh this you know I don't like this design right it's like well like how would you like it starts becoming a pretty hard thing to simulate >> yeah it feels like a you like a whole human >> yeah exactly exactly We need to clone Ham in order for that to work. So what we're talking about here is a little bit different. So it's like how do you take you know we talked about earlier the agent's really sprawled out. The agent's really intertwined. It's intertwined with all these production databases all these services. It's like you know it can talk to Slack. It can talk to GitHub. It can talk you know all these internal databases. How do we say like you know you add a new tool or you move to change like we simulate that entire world around the agent that as best as we can that existed when that agent ran in the past all those services there's an illustration in a second that'll explain how it works but that lets you test those historical traces in a harness agnostic way and in a way that's not like I think the sort of naive way of trying to do this is like well you have the old trace why don't you try like cach it almost using those as like cached results, right? So, when the agent when you rerun it, the agent calls a tool and you just kind of give it the same results like, well, actually, if you try to do that in practice, the agent will instantly call your bluff. It literally like you can try this. It'll just be like, I'm in a simulation. This isn't right. And simulation awareness is one of the things we measure here. Does that answer your question? We'll talk a little bit more in detail.
14:09 >> So, were you describing with like simulating the tool calls and stuff like that? Is a good way to think about that like some kind of agentic mocks simulate a surf like is there like an LM behind like try to simulate the data that's coming back like that's really interesting. >> Yes, exactly. So we think about it in a few steps. The opening AI paper is also a good read here but there's really like two or three steps but what we think about is that it's a little bit like Swiss cheese. Okay. So like in the original trace there are tool calls and those tools had results right every tool call is almost like a little window from the agent into the state of the world that time. One thing that's really important to remember here is that the world from let's say I'm replaying a trace from a week ago. The world has changed since then, right? So I actually can't just like even if I could just connect my agent, you know, the current production and like use all those databases. Don't do that first of all, but second of all, even if I could do that, you wouldn't want to because the state has changed. Take something as simple as a customer support, you know, bot for a delivery app. It's like if the user asks, well, like what how far away is my order? And it says like, well, 15 minutes or something. If I try reimulating that and I it was hooked up to the current production database, it would give a different answer or no answer because nothing is pending. And so you really do have to understand the world. So we kind of take a two-prong approach. One is that we try to upfront sim like generate both using the previous tool called results and any sort of transformation we have to do for the current harness and like you sort of want to pad those holes, right? So that if it's like you know if it's like slightly off or like round like it it'll get the right results. And then also at the same time we're evaling the quality of those responses and the agents awareness that it's being simulated. And you can also do that once it's complete.
15:55 The opening IP paper talks how they they will essentially take the original trace and the new trace and then ask the agent which one is like most likely to have been simulated. And if you do that at population level, you can sort of detect it that way. And so essentially it's kind of a nice hybrid between like having enough data up front where you're not like slowing everything down, but also you have an LM that's always there to spit out the correct data when possible.
16:16 >> Can you tell me a little bit more about the simulation awareness? So are you saying like if there's a situation where your agent will just refuse? Like it'll just say hey like I think we're in a simulation like haha nice try. So refusing is one, but there's more where which is like the agent can be like and I think one of the insidious ones is like and to be clear this applies to all sorts of evals, right? Like I think eval awareness in general is a very like big topic right now because if you're like imagine you're doing like safety evals like and if the agent realizes it's being evaled, it's like oh well like I'm being evalued like I should probably not do this or I probably should do this or I should probably like behave in this way and it'll learn to do that. There's a similar thing with simulations which is like you want to make sure that the agent does not understand but believes that it is in the real world and that there's nothing like really differentiating in that way and that shows you how good of a job you did simulating. And so I think in general that's this is something that has changed a lot year or two is that models now are trained to understand what eval are, how to do them, how to detect them, how to write them, like all these sort of things which also means they're like self-aware and better at finding out when they're being evalued, when they're being tested and finding creative ways.
17:23 I mean the extreme example of this is the hugging face incident, right? Which is like I don't think a lot of people realize that like what was going on at that time was like they were running cyber security evals, right? like they were just evaling their agent and it was trying to find the answer key to the evals that it was being run, right? So, it's like that's kind of the most extreme example of this. >> And do you see this happening in production like at these companies you work with like the sometimes you find that there's eval awareness?
17:53 >> Yeah, definitely. Definitely. I again I think it's not usually like the hugging face incident at least that like usually but like it's more benign. It's like, "Oh, I see. Like, this thing wasn't working because it's a simulation. I should probably answer like this." And now you've just completely lost the whole point of the email or like, "Oh, the agent or like the user is probably like based on this user question, the user is probably testing to see how polite of a response they would get in this case. Like, I should be courteous of my tone and answer like this." And so, it's like, well, you've just lost the whole thing. And to be honest, I think that this is actually much more of a problem in the absence of simulations in many way like team. You think about all of these tools your agent has access to to search git, to search issues, to search previous conversations, all this sort of stuff. If all that's empty when the agent tries to run, it'll instantly know that it's in a simulation. It's like, oh, I'm in a fresh environment.
18:48 I'm in a empty sandbox. I know that like this is just all fake. And the eval themselves end up being a lot less useful. And so when you go to simulate this environment like okay you're not simulating the human but you're simulating like sort of the infrastructure are you recreating the infrastructure are you creating like to what or is it like kind of like an LLM computer that's like getting a request saying okay I'm going to mock this response based on this and my understanding of the world or like do you actually create all this code and all this like infrastructure to what extent or is it in the middle somewhere or like >> it's in the middle it's in the middle we try to recreate as much infrastructure as we can and the reason for that is that there's a lot of things that are somewhat deterministic, right? Like imagine, you know, like a common one is like teams will have some sort of like SQL dialect or something like that they have internally for fetching data or something like that. And it's like there's a parser there. So like the agent's generating something and then it's getting parsed. As much as we can use the actual code to validate what the agent just did, like that's what we want to do. And so we try to push that boundary as much as possible all the way out to like the to network calls essentially and then simulate that.
19:56 >> Interesting. That's really fascinating. >> That's the really I think magical thing about simulations. And again like I think that like there are ways to get like started with this like just on your own and depending on how complicated your your infra is. But like it really goes back to the looking at your traces thing. Like you want to see your actual real agent traces, right? like not like some like random AI slot thing of like something it's like no you want to see that your actual agent ran the code for your agent ran and this is what happened but ideally again you want to do that before production is also good right testing new changes yeah so we talked a little bit about I'm back like you don't need thousands of simulations open air we'll talk about doing thousands of simulations you really don't need thousands of simulations every single change I think how many simulations you need is really a factor of like how big of a change you're making if you're changing like let's say you remove one tool. It's like well like maybe a little bit more than five but like probably on the order of like five to 10 of like traces that called those tool just like you're sort of just sanity checking that like the agent still like does the same thing. You know if you're rewriting the entire harness or like something like that like you might want to do 50, you might want to do more than that. It really again it's both how big the changes and how well you pick the scenarios. You can imagine that like you know a thousand wouldn't be enough if I removed a single tool if I never call if I never reimulated a trace that actually used that tool. And so again it's really about how well you select those scenarios and how big of a change it is.
21:17 And some of the things you're looking for when we do simulations and these are sort of just like again we'll talk a little bit more about evaling simulations in a second but like things we're looking for is like is the final response different from what it was before? Are there new tool errors? Are there new tool usage essentially? Did the trajectory of the response change? Is the step count different? Like is it now taking many many more steps to do the same thing? Cost and like latency are the other big ones like essentially is there a huge cost increase now or did the cost actually decrease as a result of this change? These are some of like the diffs that we're doing based on old traces. So one thing we think about are like there's essentially diffs and then there's evals. So diffs are kind of what we were just talking about which is like kind of like what changed and you can also use an LM to decide if the change is good or bad. But then there's eval which are more like scores that you want to like push up and target over time. I say score not all not just in like a you know one to three sort of thing but like pass fail and like the number that pass and fail. And so back to evals. So we sort of think about it like simulations are this sort of like base infra in some way. On top of that you have anomaly detection. So that's kind of like did something weird happen. And then eval are what we define I think increasingly are being defined as like pure functions that run over traces. So as opposed to these sort of input output pair evals which still exist they're more like pure functions. And when I say pure function what I mean is it takes an input and it returns an output. So like that input output could be like a pass or fail.
22:39 Also the input can be both the new trace and the trace before the change. So just sort of defining this really quickly like we talked before how there two years ago when we talked about evals I think people mostly talked about like input output pairs. So people talk about like oh generate this react component and then it should generate code that's like this right that doesn't scale well when we talk about evals now most of the time what we're talking about is again you you might hear the word judge you might hear the word score you might hear like something essentially it's like something that takes an input which is a trace and then decides it's good or bad for example let's say verbosity that's one that we we see teams use like you know you don't want the agent to like you know have these super long rambling responses so your team might create a new eval which is verbosity and maybe even either it's too verbose or not or like say a score of 1 to three or one to five like how verbose it is. I generally prefer the kind of binary checks but like we can talk about that at a different time >> and is the right way to to understand the one on the right is like underneath like inside this function f ofxy you are doing you're computing all these metrics like the diff the length all that stuff you showed on the previous slide and that's informing the passive and fail.
23:50 >> Yeah. Yeah. So for example, you could create an eval that's like it's not 20 it's within 20% of the number of steps from before, right? Or it's within 20% of the cost or it's like the output length is within 20%. You know, something like that. And so that's like you can start or like you can create all kinds of cool things like this like you can say is the answer the same is the answer more concise and have an LM decide like is it more concise, right?
24:15 And so the key thing as opposed to the input output pairs is that this eval once you define it should scale to any arbitrary trace right so now it's like you this is something you can just run over traces versus like it being something that you're kind of accumulating these like input output pairs. The way to really think about it how we think about it is like you are aligning your organization the humans in your organization to a definition of what good or bad is >> just to make sure other people are following. So like y in this situation is a trace that you've decided is good correct like from the past like you said hey like and you want to make sure x is still good is that how to understand >> it can be but it doesn't have to be this is one of the really interesting so like certainly this could be a data set of good traces like you could you could have a you could have a whole data set and you could run and say like essentially like well if I replay this trace I want the same answer there's also often value if you think about like the output length one it doesn't really need to be something that you know is good you want to say like, oh, I don't want it to get like more than I want the output length to be like within 20% of what it was before. Like I kind of know the state of my agent today. So like for example, let's say I'm like trying out a new model. I'm like, "Oh, cool. GPT 6 just dropped. Let's plug it in and as long as it's within like 20% of the length of what it was before, then like I'm good. I'm good to ship it." Or like create an eval that's like is it more verbose? Yes or no? And it's like as long as like mostly like my pass rate is like you know 80% of like it not being more verbose then like I'm fine. Or if it's like maybe it's above 50% like you know you can kind of decide. And so it can be good traces but it doesn't necessarily have to >> makes sense. Okay. It's like your one of the things you may want to check essentially is consistency like and so this is a yeah I like it. I like this is a nice way to >> is the answer the same is a good one.
25:58 Right. And then it also helps you narrow down the traces you should look at. Let's say that you create an email that is like, is the answer the same? Run it over a thousand traces and look at the ones where it's they're not the same, right? Those are the traces you should be looking at and it's either better or it's not. But again, creating something to score like it is a better answer. Well, that starts getting a little bit trickier. You could still do it, but like really easy one is like is the answer the same? Like you could do that with Jev really cheaply, right? Like is the final answer the same answer? Is it easier eval to create? Yes. So this is again easy one like to your point actually so around the the input data it could actually be bad traces actually it's like one that is like that is one data set we'll see a lot like imagine like very easy like loop we'll see in range office is you get an issue notification like oh bunch of like users are you know complaining about this thing they're not able to like download a file or something like that you push a fix and you're essentially comparing the bad traces and saying like does the new output exhibit the same issue right the judge If any of them return yes, then like now you know that you like your fix didn't work. So, so sometimes these are actually like bad input traces.
27:08 >> I like it. Yeah, it's pretty smart. It's like cuz this consistency thing is super important because this is a question that people want to know when they switch models or make a change and stuff like that. And I like the fact that there's a lot of metrics like you don't necessarily need to call an LM like you highlighted the fact that you can compute all these metrics which are super useful and you might you might want to do that first before reaching for an LLM cuz those are super high value kind of things that are not expensive. Yes, I think there's a it's exactly what you just said. I think there's a gradient of like really deterministic things to like you start introducing like really deterministic classifiers to like your you know if you if you think on the complete opposite side of the spectrum like is the writing good or something. It's like well man it's a really hard and very tricky thing to define which is why models are still really bad at writing and so it's like you can do it it's possible to get an LM judge to like you break out like ex what your organization means by bad, which is some mix of like using specific phrases, using m dashes, but it's like it's really hard and so definitely like you want to start with things as deterministic as you can and then start moving up the gradient of like consistency like you said and and then go from there.
28:21 >> Yeah, I'm going to go ahead and frontr run this question because I know it's going to come up. You mentioned classifier. Have you tried gem in this? Yeah. >> So, we actually we published this blog post like two months ago where we were talking about like classifiers, right? and call like our pipeline and like I think we realized like years ago like oh you could actually just train really good classifiers and make it you know easy etc et that's not our business but like we just needed to do it to make our product and we've been evaling Jeff a lot recently it's very good our our pipeline is still a like I don't know it's still like 50 to 100x cheaper or something like that and the accuracy is the same so it's like but there's a lot of things that we're like now >> so so you did eval pipeline >> we did eval we have so many evals I couldn't even start like we have so many emails now so many online like everything some that would be a really fun one is actually like how to be clear like this is mostly like what I'm talking about here is not different from how we think about it right which is sort of the magic of the company that we're building is like we try to just like be really upfront with ourselves of like how we would do evals and like a lot of it is for example like historical data like does it agree with what we've labeled before and like if no like well like let's look at like where it disagrees and have even another classifier decide if that those disagreements are like right or wrong.
29:34 So, we can like, you know, then escalate that to opus, right? And so, you can start building those sort of chains where it's like I only need to have opus decide on the ones where it's like it disagreed or something like that. >> Another thing I like about what you're showing about this consistency thing is one of the things we talked about in the class is like how to sample data and you want to be very sample efficient after you learn a little bit about evals. And so, I really like the fact that okay, you're have this in your pipeline and you're trying to show these examples.
29:58 These are like one of the kinds of things you're trying to surface. people say like, "Hey, this is probably something you want to look at." >> Totally. Yes. I think about sample efficiency both in like obviously LMS and cost and all that sort of stuff, but also like humans looking at traces. Like we talk a lot about like, you know, look at traces, look at traces, but it's like, well, if you have like 100,000 events a day, like you're not going to look at all of them. And so what are the ones you should look at? It's not even just the ones where things went wrong because if you're looking at the same ones, you'll often have a few error modes like that dominate that you're not that interested in. Like if I think about a lot of consumer apps that is like people complaining that the product isn't free. It's like well like you know that might not be the number one thing you want to be looking at every single day, right? Yeah. Very very important from both perspectives. So again just some more examples of like you know easy things here like you know is the tool call count after a model? Is it less or is it more? Super easy eval to define really easy to run. You know we talked about verbosity a little bit like you know rating it between one and three.
30:54 And this is one again where like human alignment does really start to start to matter. Yeah, this is the loop is like you know you log, you detect issues, you simulate fixing those issues and you ship and you ship knowing that like at least having a little bit more signal that you actually fix the thing before you ship which is great and then you kind of just can just do this loop over and over again. So just like common foot guns with simulations in general, this is like outside the scope of our product. Like our product exists to like solve a lot of these things but like replay against live systems. We talked a little bit about like where that starts to hit problems and like where the agent will start like when you if you try to do this the agent will often start to like realize it's being simulated. Like the easy one is like there will be data from like a week after you know when it actually was ran in your production systems. The agent will see the fact that there's later data that disagrees with the time that was passed in and it'll be like oh I'm being like this is no longer the right date. Just trying to serve cache tool responses doesn't really work. Only replaying failures like good but like you can overfit really easily. So you also want to be including like a random sample when you replay important to replay like like take into account these things are nondeterminism or nondeterministic. You want to replay it a few times usually not looking at the traces is like the big thing but like not looking at the traces is like really really critical like when you simulate you want to actually look at at least a few of those traces and that that'll actually tell you more than like any your evalu will will actually tell you. I have a question about this. The okay so we kind of we talked about simulation and so the nagging question I have in my mind is like okay you have this agent it might have complicated infrastructure it might have tools there might be data databases you know all kinds of all kinds of stuff it might be very complex and so how do you like it seems like a non-trivial task to simulate all of that like >> it's like what do you do like you take the person's code and you like >> yes >> fork it or something and do something and then like the question that enters my mind is like how do we know the simulation is faithfully simulating the thing you know because it seems like could be a lot of surface area so if you could give any thoughts on that cuz like you've obviously thought about it way more than I have about how to simulate these things >> I have thought about it a lot and you're right it is not trivial it is actually pretty hard it's actually I think one of the hardest problems in the world right now so yeah a couple things like one yes our customer is connected to GitHub and like we are checking out their entire their agent into a sandbox. We have other sandboxes where the agents are actually executing keeping state like if their agent has sandboxes we're also spinning up sandboxes to you know replicate those sandboxes and there's a whole flow of like essentially like almost like a smoke testing or like alignment flow of like we're trying we're doing a loop of like well like let's take like again the simulations for the simulations and like eval to like take production traces from main rerun them they should equal Maine and So before you're running on any PRs or anything, you're just trying to align with production as it is. And >> that's really funny. Like the answer to my question is >> simulations. I know. I know. I know.
33:53 Question. That is actually how you do it, right? It's like you're trying to align to something. You know what you should be aligned to. So you just keep doing that and you know, keep picking fresh things to try. So that's the first step is that like we call it like the world building phase. And and then there's as far as like the faithful simulation. And that's that's what I was talking about earlier is like there's a few ways you can eval for faithfulness.
34:14 One probably more obvious one that you can do after it runs is to have an LLM, you know, blindly decide look at all say say you run 20 simulations. You have an LLM blindly decide essentially take the original trace, take the new trace. Is it a simulation? Yes or no? If an LLM can correctly guess more than you know like would be statistically probable that it is a simulation, it's a pretty good sign that something's wrong. And so there's a few tricks like that that you can use.
34:39 >> And then what about data? So okay, you connect to your GitHub. Maybe they have some databases and stuff like that. >> Yes. >> How does that work? >> Good question. So two two things. So there's sort of this like this is where like the Swiss cheese analogy comes in, which is like the previous run of that agent probably has some good signs of like what the underlying databases etc. had and what you'll need to see for that question. B is that like on the fly we can like let's say you add a new tool for some sort of new database that was like never there before we can simulate and guess like what that database would have also like we have the ability now for people to connect like MCPS to the world building process so like if people want more realistic data they can either drop in you know JSON files or whatever or they can like connect an MCP that can be used in the world building process and that'll like you know actually pull that production data in and the MC CP is like meant to impersonate the thing.
35:34 >> It's actually easier than that. It's like okay, let's say you have like some new database. You could connect like a readonly, you know, MCP just like let's say you're using Superbase. You could, you know, do a readonly Superbase MCP just so the agent that builds the world for that run can pull the data it needs and understand the schemas from Superbase and truthfully the schema can be understood from the codebase, but the data it allows it to populate that data from production. that is really we've only really seen that necessary if you're adding like a whole new data source internally that like the agent has just never seen before in its life.
36:11 It's like if you really want that simulated faithfully then then that's something you should do. >> Interesting. I think this is really fascinating. Yeah, thank you for sharing this one question. Okay, so a lot of people are learning evals in this class and a question always comes up what are good companies to work for if you are into evals. Do you have any thoughts? well raindrop as you just heard we have a lot of evals I mean I think there's a range I think a lot of our customers are pretty good signs of people that like care a lot about eval whether that's Verscell whether that's Poland is like there's a lot of people in the consumer space obviously there's like you know bigger companies as well but I would be remiss not to not to mention ourselves here I think ultimately you want to look for companies where the real world cost of failures is like high and I think that it's more than you'd expect actually like anything that touches the real world in a meaningful way or a lot of people probably the cost of failure is pretty high.
37:01 >> Cool. What kind of roles are you hiring hiring for? >> We're currently hiring most roles. so it's like sales head of sales AES and then on the engineering side we're hiring for FTEEs, we're hiring for product engineers, and we're hiring for MLES. So those are the three roles. >> All right, cool. I think a lot of people in this class would be interested in checking it out. That's why I asked. >> Yes, you can DM me or email me. It's benraindrop.ai AI drop in the chat too.
37:26 >> Probably have some time for some questions. Is there any situation where simulations might be overkill? Like how do you know you need a simulation? When should you go for it? Should you go for it right away? Should you start something else first? >> What do you think? >> Very good question. And so I mean the very first place that you start is just trying it yourself, right? Like you know you have it running locally and just try it. Take the traces, ask an agent what went wrong in the traces and for it to like show you the parts that went wrong.
37:51 Some way of like annotating and seeing those traces. So like that's honestly why we built like Rindrop workshop open source tool runs locally streams all the traces directly into it. Like do look at your traces locally. Like it's very very important. I think it's like by far probably the most important thing you can do. Go through every tool call understand and ask an agent like how could I improve this? How could I optimize this? Like it really does work.
38:13 So that's where you should start. I think as soon as you're trying to start running evals now that you have agents like you really kind of need to find some way to simulate it now like depending on how complex your agent is like you may be able to just do this yourself like if you only have like a few tools or like the data sources are pretty straightforward or maybe you have a case where like you can just hook it up into like a readonly version of production and like limit the data it's reading or something like that like definitely like try I think that like you should always try building out your own solutions here because you'll at least understand the problem that we're solving or any other, you know, vendor is solving. But especially if you're doing like classification, there's a lot of things where it's like just run them both in parallel. The problem is that like agents are really expensive. So if you're going to do a shadow deployment of your agent across all traffic, now you're doubling your costs and then now you also have to look at those traces still. So like it's not totally solving the problem, right? And so I would think about it a little bit like simulations are the way to have an action. Like also the iteration loop there is ridiculously slow, right? It's like push to production, monitor for some time, find the things, then like go make another change, push to production. So it's like I think you want to try to make that feedback loop as tight as possible on simulations versus experiments and like AB test. So Raintop actually has like we have an experiments feature that like is probably one of the most heavily used things in our entire app right after issue detection. It's really good. It's really important. Like I don't want to like deemphasize the importance of production. It is like production is ultimately like the source of truth of like what is actually happening in your agent, how it's actually behaving. You just like want to do something before you get to production to make sure that like people aren't, you know, bad things aren't happening.
39:58 >> All right. Amazing. Thank you so much.
Summary
- Raindrop focuses on real-time issue detection in production AI agents, using simulations to test changes before they go live.
- Traditional eval methods are becoming inadequate due to the complexity of modern AI agents, which interact with various tools and real-world data.
- Simulations allow teams to replay historical data against new changes to see how those changes affect agent behavior.
- The effectiveness of simulations relies on creating a faithful representation of the agent's environment and ensuring the agent does not recognize it is in a simulated context.
- Evaluations should be based on pure functions that assess the quality of responses rather than simple input-output pairs, allowing for more scalable and meaningful assessments.
- Companies should prioritize understanding and analyzing traces of agent interactions to improve performance and identify issues effectively.
- Raindrop is hiring for various roles, including engineering and sales, and emphasizes the importance of working in environments where the cost of failure is high.
Questions Answered
What is the purpose of this evaluation course and who is Ben?
The evaluation course aims to explore various aspects of evaluations, inviting experts like Ben from Raindrop to share insights. Raindrop focuses on identifying critical issues in production agents and simulating changes before they go live.
How can one effectively analyze AI traces without getting overwhelmed?
To avoid confusion in analyzing AI traces, it's crucial to utilize annotations and comments that guide users to relevant points. While catching issues quickly is important, preventing them is even more critical, especially in high-stakes environments.
What is simulation awareness and why is it important for AI agents?
Simulation awareness refers to an AI agent's ability to recognize when it is being evaluated or tested. This awareness can lead to altered behavior, which is problematic. The goal is to ensure agents believe they are in a real-world scenario to accurately assess their performance.
How can evaluations be scaled using traces?
Evaluations can be scaled by defining criteria that apply to any arbitrary trace, allowing for consistent assessment across different scenarios. This approach aligns organizational understanding of what constitutes good or bad performance.
What are the challenges in simulating complex AI systems?
Simulating complex AI systems is challenging due to the intricate infrastructure and tools involved. It requires creating accurate representations of the system's environment and ensuring that simulations faithfully replicate real-world conditions.