transcribe

How To Test AI Agents With Simulations

Hamel Husain · 40m · transcribed Oct 2026
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Evaluations and Raindrop

What is the purpose of the eval course and who is Ben?

The eval course invites various vendors to discuss evaluations, with a focus on Raindrop and its approach to finding critical issues in production agents. Ben, the CTO and co-founder of Raindrop, introduces the company's philosophy and recent product launch related to simulations.

  • The eval course aims to provide insights into different evaluation methodologies.
  • Raindrop is recognized for its alignment with educational philosophies in evaluations.
  • Ben's talk focuses on simulations and their relevance in the evaluation space.
# 8:02

Importance of Trace Analysis

How can one effectively analyze traces in AI evaluations?

Effective trace analysis requires tools that highlight important annotations to avoid confusion in lengthy AI traces. While catching issues in production is crucial, it does not prevent them from occurring, especially in high-stakes environments.

  • Utilizing tools that show traces can help identify issues quickly.
  • AI evaluations must focus on understanding the context of traces to avoid misinterpretation.
  • High-stakes applications require careful monitoring to prevent critical errors.
# 16:04

Simulation Awareness in AI Agents

What is simulation awareness and why is it important?

Simulation awareness refers to an agent's ability to recognize when it is being evaluated or simulated. This awareness can lead to biased behavior, making it crucial for agents to operate under the assumption that they are in the real world to ensure accurate evaluations.

  • Agents must not recognize they are in a simulation to provide genuine responses.
  • Simulation awareness is a growing concern in the field of evaluations.
  • Understanding how agents react to evaluations can influence their performance.
# 24:06

Defining Good and Bad Traces

How can organizations define what constitutes a good or bad trace?

Organizations can define good and bad traces by establishing criteria such as output length and consistency with past performance. This allows for scalable evaluations across various traces without relying solely on historical input-output pairs.

  • Defining evaluation criteria helps align organizational understanding of performance.
  • Evaluations should be adaptable to different traces to ensure comprehensive assessments.
  • Output consistency is a key metric in evaluating agent performance.
# 32:08

Challenges in Simulating Complex Systems

What are the challenges in simulating complex AI infrastructures?

Simulating complex AI infrastructures is challenging due to the need to replicate various components accurately. This involves sandboxing the agent's code and maintaining state across different environments to ensure faithful simulations.

  • Simulating complex systems requires careful replication of the original environment.
  • Sandboxing is essential for testing and validating agent behavior in simulations.
  • The complexity of simulations is one of the significant challenges in AI evaluations.

Transcript

0:00 We've been running this eval course for a while now and every so often we bring people in who are working on evals and there's a lot of different vendors in the space and I have used every single vendor and I try not to have like pick favorites but I really do like raindrop you know I like their philosophy like how they go about things they tend to do things that are very aligned with the way that we teach in this class and so I think it's really helpful for you to see different aspects ects and things of evals. We thought, okay, let's invite Ben to give like more color around the way that he thinks about evals, what he's working on, etc. I think it would be really beneficial for y'all. And so, yeah, I'll just go ahead and kick it off to Ben. So, talk today is called getting started with simulations. I'm the CTO and co-founder of Raindrop. This is both a product that we recently launched, also something that just in general everyone in the space is starting to do, labs are really starting to do, writing papers about. It's not not eval. We'll get into it in just a little bit. Our whole company is about finding critical issues in production agents. So we find those issues in real time like minutes after they start. Verify those fixes actually work without side effects. And then now we simulate those changes before they land in production. We're used by some really amazing AI companies. Everyone from like Verscell, Speak, Framer, Clay, Door Dash, and many others. So some really awesome customers here. We also have this really cool open source tool which honestly like especially if you're ever hacking on your own thing. It's a really nice like starting point. I use it even in like a lot of my own like side projects. I'll just be like, "Hey, take this UI like either use it, change it, whatever you want." But it's kind of cool. Of course, the tool that like our Fortune 100, Fortune 500 customers use is like what we call raindrop. And we also make this really cool guide, very very Hamill inspired called like how to eval.com and probably due for an update even even though we just published it like what like 2 or 3 months ago or something like that. But the space moves fast.

1:55 Everything's changing which is why these these courses matter. Everything has changed to agents and this isn't just like a a word changing. It's actually kind of a big it's a big deal and I'll kind of explain why. So like in this kind of chatbot era the way that we would think about evals and again the word eval is very overloaded. So I don't mean to say that like oh eval like this I'm not trying to say eval are dead or something like that. but just like what we used to consider the way to eval is the right way to put it. What we used to consider an eval like maybe a little over a year ago was like you have a prompt it goes to the model you have some sort of response that comes out and then you have some sort of like predefined way probably for that given input to know that that output is correct. a very contrived example, but an example that you'll still see on like vendors websites even now is like you know what is the capital of like the United States and then it's like oh well it's Washington DC so it's like input output validator the thing though is that like things have changed a lot since then one is that like agents are a lot more spelled out right so it's not no longer that you have this like nice clean like oh well you just have to you know you change your prompt and then you rerun and you see what happened it's like well now it's like you have skills, you have the tool definitions themselves. Like just adding or removing a tool can completely change the agent's behavior. Even just changing the input type to a tool can change or even the field name to a tool can change how the agent behaves, right? So it's like clearly it's a lot bigger than the prompt. Now you know users connecting MCPs, you have of course you can change model etc. But then now you also have memory, you have sub aents and then the prompts for those sub aents. And a lot of these also increasingly are also outside of the repo, right? So you don't you no longer have all the context even in just the the repo which is really interesting. And the other thing and kind of like which is related to that last point is like agents are very intertwined with the world now, right?

3:51 Like they have all their little fingers and arms out touching the real world. Whether that's your own sort of internal production databases, whether that's other services and again no longer is all the state and all the context like the emails that the agent goes and reads are now part of the like that's part of the context. It's influencing the agent's behavior and what it's going to do next. That makes them a lot harder to eval. And also they've become more and more capable, more and more complex. And what that means is that sort of like single input to output pairing really starts falling apart because most of what your agent is going to do is and most of where it's going to go wrong truthfully in practice are going to be things that you haven't really thought about yet and that's what makes it really hard. And so I think that there was this like for some time in the eval world there was this like more is more strategy with evals which is like the more evals you have like you have 4,000 5,000 6,000 evals and then that is going to like that's your mode as a company that's going to be the thing and it's like well like you know there can be valid cases for that like if you're planning on training your own like frontier model or your own like there are cases and we can go into more details in Q&A like where that really does matter But I think largely it's actually more about your strategy around eval like the number of input and output pairs you can collect because again capabilities are increasing. So essentially like your agent can do more things faster than you can make evals.

5:23 And the other problem is that like you start running into this problem where it's like okay so now I have 4,000 evals. Let's say if I add a tool am I let's say I add some sort of tool like am I actually going to run 4,000 eval in that PR? Like probably not right? So it's like well how many are you going to run? How do you find the right ones? I think this is why like as a company like we actually like kind of stayed away from evals for a really long time. We were kind of like we focused on the real world because that felt like the best agent thing to do. Like it let us catch issues you were never expecting. It let us like have a lot of signal around those issues. It's not like so for example, one thing we saw with evals was that the harness would change and the eval would break. Like the eval were expecting a specific tool to be called but then you switch to a different harness and now everything breaks. So like the reality today, even some of the best companies in the world AI wise barely have functioning evals. And I I've seen this time and time again. I won't call anyone out, but like even some of the poster child of Evals companywise, it's you talk to them and they're like, "Yeah, well like all our evals broke when we switched to like, you know, Claude Code as like the harness underneath or something like that." Like, so it's actually really really interesting. And so we really focused on again we really focused on production monitoring and we did that essentially we think of it and like we call it unsupervised real-time issue detection right so it's like there's three ingredients that go into that production monitoring the first is like your codebase your prompts right so it's like we can sort of understand like based on how you've prompted your agent based on the codebase like what should be happening right so like that kind of gives you like one data point the next is like the baseline of behavior. So if something like you make a change and it it drifts like that's an interesting signal. And the last thing is like your feedback on the production data. So like looking at traces, I know HL like emphasizes a lot like look at your traces like we'll send issues, we'll send summaries, we'll send like alerts and based on how you interact with that we can learn like what you care about and what you don't care about. So again this is sort of like how we've figured out how to do that eval but in production, right? like how to like really really high signal when our almost every single issue we send a team they react to they open they do something with now which is something we're incredibly proud of and I think when you use other tools and you're just getting inundated by like random digests and summaries it's like well that's not useful right but this is just sort of like again we do these deviations on we figured out as a company how to do these deviations on not just like you know error rates and things but things that are like very semantic like understanding you know unusual patterns of the way that the agent is responding the trajectories it's taking etc. This is what an issue looks like in rangeop as well. Not to hammer this home too much, but one thing that also I think teams really get wrong, somewhat of a tangent here with like issues, this sort of thing is like just show the traces.

8:07 Find have tools that show you your traces. Like do not drown in like AI slop summaries of what your agent's actually doing. Like so for example here is like we show these like little annotations these comments on the trace that kind of helps you jump to the right point which is important especially now that traces can be tens of minutes if not hours if not days but just like don't get lost in the sauce of like AI summaries what happened what went wrong just like have ways that accelerate finding the right traces and help you understand the trace that you're looking at. So, I just talked a lot about production, but the thing is that like that is nice, but it doesn't prevent issues from happening. Like, it's good to catch an issue as soon as you can.

8:46 It's good to know that your change actually did something well in the real world. And certainly the real world is like a really powerful arbiter of truth, right? Like it's again, it's not like, oh, your harness broke something and then your old evalu stopped working. It's like, no, like people are complaining, people are mad or like, no, it's really working. Look, you can see people have like, you know, recognized a difference somehow. But again, it doesn't prevent issues from happening and especially as we have like Fortune 100 customers as we have like you know even like defense you know increasingly you know there was a story recently about the US misidentified a Chinese vessel as carrying nuclear weapons of some sort and like intercepted it based on bad information from from an AI we don't know which one but like so clearly like this is like getting higher and higher stakes right as we go into like healthcare defense finances and so as a company we're like how do we actually prevent these things from happening in a way that is you know, ridiculously fragile as well. And I think whenever you're faced with this sort of question, it's really useful to take a little bit of a step back and say like, what would OpenAI do? Like OpenAI, you know, or like insert foundation lab here, but like they're like a big company. They have a lot of users. They have a pretty good product that's like, you know, pretty pretty stable. And and so like it's worth like Codex, for example, I think is like one of the best AI products that exists in the world. So it's like worth asking yourself like, well, what what do they do? Right? And this is one paper recently on Codex, they've had a few around simulations.

10:06 And essentially the answer I kind of gave it away actually is the answer is simulations. And so every time they make a change to a model, they have a whole rig that lets them simulate the past traffic on codecs and say like well like how would this new model change the previous codeex trajectories? And so the really powerful thing about this is that you're now sort of answering the question of like what would my change change, right? Like I think that's like a lot of times the the rut that we see teams get stuck in is that they have like a lot of times the fixes and getting to the point where you have a fix to some sort of problem with your agent or add some sort of capability. I think getting to that point is actually getting faster than ever. But then there's this whole like consequence of making that change and that's the thing where that really starts slowing teams down. So really simply again simulations are you replay real traffic real historical traffic against your new change and you see what's different. So that change could be anything right that could be a new model. It could be a new tool. It could be removing a tool. It could be changing the prompt. It could be like you switched from AIS SDK to PI.

11:13 It could really be you know as big as as you want here. And so you're sort of like again you have the old traces that you've collected and you find a way to just rerun your agent on those exact traces. So again what will my change change? Does it actually fix it? Does it cause other issues is kind of what what you're asking. Let's just take like one very specific example is like let's say you delete a tool. So what you can do is like find traces where before you made that change that tool was called like real user events where that tool was used to answer their question and then replay those and then say does the answer change right and then if the answer changes you know it flag those and those are the ones you want to review. So this again becomes a way that is like one extremely targeted to whatever change you're making to your PR or like whatever change you're making in your PR and like again really really helps answer this question like what does my change change?

12:02 >> So when you replay traces from production let's say you know your agent is like has some multi-turn stuff going on user asking follow question how do you deal with that? >> This is a very very good question. So this is I think one of the things that's actually very powerful as opposed to like the way that most people think about evals. a lot of times we see teams do like these like single turn evals and I think the single turn versus multi-turn is actually a very like somewhat convoluted term right now. So like there's multi-turn as in you're like 20 messages into a conversation and then you ask the next question and then there's multi-turn as in you see the output and then do another question see the output do I think the former of those doesn't really work right now just to be like 100% honest like I think that there's people trying to work on user simulations like as in more like user prediction and I think that's really hard problem because you know you take something as simple as like a user looks to the UI and they're like, "Oh, this, you know, I don't like this design, right?" It's like, "Well, like, how would you like it starts becoming a pretty hard thing to simulate?"

13:04 >> Yeah. It feels like AGI. You have to bore it's like a whole human. >> Yeah. >> Yeah. Exactly. Exactly. And we needed to clone Hamill in order for that to work. So, what we're talking about here is a little bit different. >> So, it's like how do you take, you know, we we talked about earlier the agents really sprawled out, the agent's really intertwined. It's intertwined with all these production databases, all these services. It's like you know it can talk to Slack, it can talk to GitHub, it can talk, you know, all these internal databases. How do we say like you know you add a new tool or you move to change like we simulate that entire world around the agent that as best as we can that existed when that agent ran in the past all those services. There's an illustration in a second that'll explain how it works. But that lets you test those historical traces in a harness agnostic way and in a way that's not like I think the sort of naive way of trying to do this is like well you have the old trace why don't you try like cache it almost using those as like cache results right so when the agent when you rerun it the agent calls the tool and you just kind of give it the same results like well actually if you try to do that in practice the agent will instantly call your bluff it literally like you can try this it'll just be like I'm in a simulation this isn't right and simulation awareness is one of the things we measure here. Does that answer your question? We'll talk a little bit more in detail.

14:18 >> So, were you describing with like simulating the tool calls and stuff like that? Is a good way to think about that like some kind of agentic mocks simulate a surf like is there like an LM behind like try to simulate the data that's coming back? Like that's really interesting. >> Yes, exactly. So, we think about it in a few steps. The opening paper is also a good read here, but there's really like two or three steps. But what we think about is that it's a little bit like Swiss cheese. Okay. So like in the original trace there are tool calls and those tools had results right every tool call is almost like a little window from the agent into the state of the world that time. One thing that's really important to remember here is that the world from let's say I'm replaying a trace from a week ago. The world has changed since then right? So I actually can't just like even if I could just connect my agent you know the current production and like use all those databases. Don't do that first of all, but second of all, even if I could do that, you wouldn't want to because the state has changed. Take something as simple as a customer support, you know, bot for a delivery app. It's like if the user asks, well, like what how far away is my order? And it says like, well, 15 minutes or something. If I try reimulating that and I it was hooked up to the current production database, it would give a different answer or no answer because nothing is pending. And so, you really do have to understand the world. So, we kind of take a two-prong approach. One is that we try to upfront sim like generate both using the previous tool call results and any sort of transformation we have to do for the current harness and like you sort of want to pad those holes right so that if it's like you know if it's like slightly off or like round like it'll it'll get the right results and then also at the same time we're evaling the quality of those responses and the agents awareness that it's being simulated and and you can also do that once it's complete. The opening eye paper talks how they they will essentially take the original trace and the new trace and then ask the agent which one is like most likely to have been simulated. And if you do that at population level, you can sort of detect it that way. And so essentially it's kind of a nice hybrid between like having enough data up front where you're not like slowing everything down, but also you have an LM that's always there to put out the correct data when possible.

16:25 >> Can you tell me a little bit more about the simulation awareness? So are you saying like if there's a situation where your agent will just refuse? like it'll just say, "Hey, like I think we're in a simulation like haha, nice try." >> So refusing is one, but there's more where which is like the agent can be like and I think one of the insidious ones is like and to be clear this applies to all sorts of evals, right?

16:45 Like I think eval awareness in general is a very like big topic right now because if you're like imagine you're doing like safety evals like and if the agent realizes it's being evaled it's like oh well like I'm being eval like I should probably not do this or I probably should do this or I should probably like behave in this way and it'll learn to do that. There's a similar thing with simulations which is like you want to make sure that the agent does not understand but believes that it is in the real world and that there's nothing like really differentiating in that way and that shows you how good of a job you did simulating. And so I think in general that's this is something that has changed a lot year or two is that models now are trained to understand what eval are how to do them how to detect them how to write them like all these sort of things which also means they're like self-aware and better at finding out when they're being evalued when they're being tested and finding creative ways.

17:32 I mean the extreme example of this is the hugging face incident right which is like I don't think a lot of people realize that like what was going on at that time was like they were running cyber security evals right like they were just evaling their agent and it was trying to find the answer key to the evals that it was being run right so it's like that's kind of the most extreme example of this >> and do you see this happening in production like at these companies you work with like the sometimes you find that there's eval eval awareness Yeah, definitely. Definitely. Again, I think it's not usually like the hugging face incident at least that like usually, but like it's more benign. It's like, oh, I see like this thing wasn't working because it's a simulation. I should probably answer like this and now you've just completely lost the whole point of the email or like, oh, the agent or like the user is probably like based on this user question, the user is probably testing to see how polite of a response they would get in this case. like I should be courteous of my tone and answer like this and so it's like well you've just lost the whole thing and to be honest I think that this is actually much more of a problem in the absence of simulations in many way like team you think about all of these tools your agent has access to to search git to search issues to search previous conversations all this sort of stuff if all that's empty when the agent tries to run it'll instantly know that it's in a simulation it's like oh I'm in a fresh environment I'm in a empty sandbox I know that like this is just all fake and the evals themselves end up being a lot less useful. And so when you go to simulate this environment like okay you're not simulating the human but you're simulating like sort of the infrastructure are you recreating the infrastructure are you creating like to what or is it like kind of like an LLM computer that's like getting a request saying okay I'm going to mock this response based on this and my understanding of the world or like do you actually create all this code and all this like infrastructure to what extent or is it in the middle somewhere or like >> it's in the middle it's in the middle we try to recreate as much infrastructure as we can and the reason for that is that there's a lot of things that are somewhat deterministic, right? Like imagine, you know, like a common one is like teams will have some sort of like SQL dialect or something like that they have internally for fetching data or something like that. And it's like there's a parser there. So like the agent's generating something and then it's getting parsed. As much as we can use the actual code to validate what the agent just did, like that's what we want to do. And so we try to push that boundary as much as possible all the way out to like the to network calls essentially and then simulate that.

20:04 >> Interesting. That's really fascinating. >> That's the really I think magical thing about simulations. And again like I think that like there are ways to get like started with this like just on your own and depending on how complicated your your infra is. But like it really goes back to the looking at your traces thing. Like you want to see your actual real agent traces, right? like not like some like random AI slot thing of like something it's like no you want to see that your actual agent ran the code for your agent ran and this is what happened but ideally again you want to do that before production is also good right testing new changes yeah so we talked a little bit about come back like you don't need thousands of simulations open air we'll talk about doing thousands of simulations you really don't need thousands of simulations every single change I think how many simulations you need is really a factor of like how big of a change you're making if you're changing like say you remove one tool.

20:52 It's like well like maybe a little bit more than five but like probably on the order of like five to 10 of like traces that called those tool just like you're sort of just sanity checking that like the agent still like does the same thing. You know if you're rewriting the entire harness or like something like that like you might want to do 50, you might want to do more than that. It really again it's both how big the change is and how well you pick the scenarios. You can imagine that like you know a thousand wouldn't be enough if I removed a single tool if I never call if I never reimulated a trace that actually used that tool. And so again it's really about how well you select those scenarios and how big of a change it is.

21:25 And some of the things you're looking for when we do simulations and these are sort of just like again we'll talk a little bit more about evaling simulations in a second but like things we're looking for is like is the final response different from what it was before? Are there new tool errors? Are there new tool usage essentially? Did the trajectory of the response change? Is the step count different? Like is it now taking many many more steps to do the same thing? Cost and like latency are the other big ones like essentially is there a huge cost increase now or did the cost actually decrease as a result of this change? These are some of like the diffs that we're doing based on old traces. So one thing we think about are like there's essentially diffs and then there's evals. So diffs are kind of what we were just talking about which is like kind of like what changed and you can also use an LM to decide if the change is good or bad. But then there's eval which are more like scores that you want to like push up and target over time. I say score not all not just like a you know one to three sort of thing but like pass fail and like the number that pass and fail. And so back to evals. So we sort of think about it like simulations are this sort of like base infra in some way. On top of that you have anomaly detection. So that's kind of like did something weird happen. And then eval are what we define I think increasingly are being defined as like pure functions that run over traces. So as opposed to these sort of input output pair evals which still exist they're more like pure functions. And when I say pure function what I mean is it takes an input and it returns an output. So like that input output could be like a pass or fail.

22:48 Also the input can be both the new trace and the trace before the change. So just sort of defining this really quickly like we talked before how there two years ago when we talked about eval talked about like input output pairs. So people talk about like oh generate this react component and then it should generate code that's like this right that doesn't scale well when we talk about evals now most of the time what we're talking about is again you you might hear the word judge you might hear the word score you might hear like something essentially it's like something that takes an input which is a trace and then decides if it's bad for example let's say verbosity that's one that we we see teams use like you know you don't want the agent to like you know have these super long rambling responses so your team might create a new eval which is verbosity and maybe even either it's too verbose or not or like say a score of 1 to three or 1 to five like how verbose it is. I generally prefer the kind of binary checks but like we can talk about that at a different time >> and is the right way to to understand the one on the right is like underneath like inside this function f ofxy you are doing you're computing all these metrics like the diff the length the all that stuff you showed on the previous slide and that's informing the passive and fail.

23:58 >> Yeah. Yeah. Okay, so for example, you could create an eval that's like it's not 20 it's within 20% of the number of steps from before, right? Or it's within 20% of the cost or it's like the output length is within 20%. You know, something like that. And so that's like you can start do or like you can create all kinds of cool things like this like you can say is the answer the same is the answer more concise and have an LM decide like is it more concise, right?

24:23 And so the key thing as opposed to the input output pairs is that this eval once you define it should scale to any arbitrary trace, right? So now it's like you this is something you can just run over traces versus like it being something that you're kind of accumulating these like input output pairs. The way to really think about it and how we think about it is like you are aligning your organization the humans in your organization to a definition of what good or bad is >> just to make sure other people are following. So like why in this situation is a trace that you've decided is good.

24:52 Correct? like from the past like you said hey like and you want to make sure X is still good is that what how to understand >> it can be but it doesn't have to be this is one of the really interesting so like certainly this could be a data set of good traces like you could you could have a you could have a whole data set and you could run and say like essentially like well if I replay this trace I want the same answer there's also often value if you think about like the output length one it doesn't really need to be something that you know is good you want to say like oh I don't want it to get like more than I want the output length to be within 20% of what it was before. Like I kind of know the state of my agent today. So like for example, let's say I'm like trying out a new model. I'm like, "Oh, cool. GPT 6 just dropped. Let's plug it in." And as long as it's within like 20% of the length of what it was before, then like I'm good. I'm good to ship it. Or like create an eval that's like, is it more verbose? Yes or no. And it's like as long as like mostly like my pass rate is like, you know, 80% of like it not being more verbose, then like I'm fine. or if it's like maybe it's above 50% like you know you can kind of decide and so it can be good traces but it doesn't necessarily have to be.

25:57 >> Makes sense. Okay. It's like your one of the things you may want to check essentially is consistency like and so this is a yeah I like it. I like this is a nice way to >> is the answer the same is a good one right and then it also helps you narrow down the traces you should look at. Let's say that you create an email that is like is the answer the same? Run it over a thousand traces and look at the ones where it's they're not the same.

26:19 Right? those are the traces you should be looking at and it's either better or it's not but again creating something to score like it is a better answer well that starts getting a little bit trickier you could still do it but like really easy one is like is the answer the same like you could do that with Jev really cheaply right like is the final answer the same answer is it easier eval to create yes so this is again easy one like to your point actually so around the the input data it could actually be bad traces actually it's like one that is like that is one data set we'll see a lot like imagine like very easy like loop we'll see in range is you get an issue notification like oh bunch of like users are you know complaining about this thing they're not able to like download a file or something like that you push a fix and you're essentially comparing the bad traces and saying like does the new output exhibit the same issue right the judge if any of them return yes then like now you know that like your fix didn't work so so sometimes these are actually like bad input traces >> I like it yeah it's pretty smart. It's like cuz this consistency thing is super important because this is a question that people want to know when they switch models or make a change and stuff like that and I like the fact that there's a lot of metrics like you don't necessarily need to call an LM like you highlighted the fact that you can compute all these metrics which are super useful and you might you might want to do that first before reaching for an LLM because those are super high value kind of things that are not expensive.

27:45 >> Yes, I think there's a exactly what you just said. I think there's a gradient of like really deterministic things to like you start introducing like really deterministic classifiers to like your you know if you if you think on the complete opposite side of the spectrum like is the writing good or something it's like well man it's a really hard and very tricky thing to define which is why models are still really bad at writing and so it's like you can do it it's possible to get an LM judge to like you break out like ex what your organization means by bad which is a mix of like using specific phrases, using am dashes, but it's like really hard and so definitely like you want to start with things as deterministic as you can and then start moving up the gradient of like consistency like you said and and then go from there.

28:30 >> Yeah, I'm going to go ahead and front run this question cuz I know it's going to come up. You mentioned classifier. Have you tried gem in this? Yeah. >> So, we actually we published this blog post like two months ago where we were talking about like classifiers, right? and call like our pipeline and like I think we realized like years ago like oh you could actually just train really good classifiers and make it you know easy etc etc that's not our business but like we just needed to do it to make our product and we've been evaling Jeff a lot recently it's very good our our pipeline is still a like I don't know it's still like 50 to 100x cheaper or something like that and the accuracy is the same so it's like but there's a lot of things that were like now >> so so you did eval pipeline >> we did eval so many evals I couldn't even start like we have so so many emails now, so many online like everything. some that would be a really fun one is actually like how to be clear like this is mostly like what I'm talking about here is not different from how we think about it, right? which is sort of the magic of the company that we're building is like we try to just like be really upfront with ourselves of like how we would do evals and like a lot of it is for example like historical data like does it agree with what we've labeled before and like if no like well like let's look at like where it disagrees and have even another classifier decide if like that those disagreements are like right or wrong so we can like you know then escalate that to opus right and so you can start building those sort of chains where it's like I only need to have opus decide on the ones where it's it disagreed or something like that.

29:51 >> Another thing I like about what you're showing about this consistency thing is one of the things we talked about in the class is like how to sample data and you want to be very sample efficient after you learn a little bit about evals. And so I really like the fact that okay you're have this in your pipeline and you're trying to show these examples. These are like one of the kinds of things you're trying to surface people say like hey this is probably something you want to look at.

30:12 >> Totally. Yes. I think about sample efficiency both in like obviously LMS and cost and all that sort of stuff but also like humans looking at traces like we talk a lot about like you know look at traces look at traces but it's like well if you have like 100 thousand events a day like you're not going to look at all of them and so what are the ones you should look at it's not even just the ones where things went wrong because if you're looking at the same ones you'll often have a few error modes like that dominate that you're not that interested in like if I think about a lot of consumer apps that is like people complaining that the product isn't free it's like well like you know that might not be the number one thing you want to be looking at every single day, right?

30:46 Yeah. Very, very important from both perspectives. So, again, just some more examples of like, you know, easy things here, like you know, is the tool call count after a model, is it less or is it more? Super easy eval to define really easy to run. You know, we talked about verbosity a little bit like, you know, rating it between one and three. And this is one again where like human alignment does really start to start to matter. Yeah, this is the loop is like, you know, you log, you detect issues, you simulate fixing those issues and you ship and you ship knowing that like at least having a little bit more signal than you actually fix the thing before you ship, which is great. And then you kind of just can just do this loop over and over again. So just like common foot guns with simulations in general, this is like outside the scope of our product. like our product exists to like solve a lot of these things, but like replay against live systems. We talked a little bit about like where that starts to hit problems and like where the agent will start like when if you try to do this, the agent will often start to like realize it's being simulated. Like the easy one is like there will be data from like a week after you know when it actually was ran in your production systems, the agent will see the fact that there's later data that disagrees with the time that was passed in and it'll be like oh I'm being like this is no longer the right date. Just trying to serve cache tool responses doesn't really work. only replaying failures like good but like you can overfit really easily. So you also want to be including like a random sample when you replay. Important to replay like like take into account these things are net determinism or non-deterministic. You want to replay it a few times usually.

32:06 Not looking at the traces is like the big thing. But like not looking at the traces is like really really critical like when you simulate you want to actually look at at least a few of those traces and that that'll actually tell you more than like any your evalu will will actually tell you. >> I have a question about this. The okay so we kind of we talked about simulation and so the nagging question I have in my mind is like okay you have this agent it might have complicated infrastructure it might have tools there might be data databases you know all kinds of all kinds of stuff it might be very complex and so how do you like it seems like a non-trivial task to simulate all of that like >> it's like what do you do like you take the person's code and you like >> yes >> fork it or something and do something and then like the question that enters my mind is like how do we know the simulation is faithfully simulating the thing you know because it seems like could be a lot of surface area so if you could give any thoughts on that cuz like you've obviously thought about it way more than I have about how to simulate these things >> I have thought about it a lot and you're right it is not trivial it is actually pretty hard it's actually I think one of the hardest problems in the world right now so yeah a couple things like one yes our customer is connect to GitHub and like we are checking out their entire their agent into a sandbox. We have other sandboxes where the agents are actually executing keeping state like if their agent has sandboxes we're also spinning up sandboxes to you know replicate those sandboxes and there's a whole flow of like essentially like almost like a smoke testing or like alignment flow of like we're trying we're doing a loop of like well like let's take like again the simulations for the simulations and like eval to like take production traces from main rerun them they should equal Maine and So before you're running on any PRs or anything, you're just trying to align with production as it is. And >> that's really funny. Like the answer to my question is >> simulations. I know. I know. I know.

34:02 Question. That is actually how you do it, right? Like you're trying to align to something. You know what you should be aligned to. So you just keep doing that and you know keep picking fresh things to try. So that's the first step is that like we call it like the world building phase. And and then there's as far as like the faithful simulation. That's that's what I was talking about earlier is like there's a few ways you can eval for faithfulness.

34:22 One probably more obvious one that you can do after it runs is to have an LM you know blindly decide look at all say say you run 20 simulations you have an LM blindly decide essentially take the original trace take the new trace is it a simulation yes or no if an LLM can correctly guess more than you know like would be statistically probable that it is a simulation that's a pretty good sign that something's wrong and so there's a few tricks like that that you can use >> and then what about data so okay you connect your GitHub maybe that you have some databases and stuff like that.

34:53 >> Yes. >> How does that work? >> Good question. So, two two things. So, there's sort of this like this is where like the Swiss cheese analogy comes in, which is like the previous run of that agent probably has some good signs of like what the underlying databases etc. had and what you'll need to see for that question. B is that like on the fly we can like let's say you add a new tool for some sort of new database that was like never there before. we can simulate and guess like what that database would have. Also like we have the ability now for people to connect like MCPS to the world building process. So like if people want more realistic data they can either drop in you know JSON files or whatever or they can like connect an MCP that can be used in the world building process and that'll like you know actually pull that production data in >> and the MCP is like meant to impersonate the thing.

35:43 >> It's actually easier than that. It's like okay let's say you have like some new database you could connect like a readonly you know MCP just like let's say you're using superbase you could you know do a readonly superbase MCP just so the agent that builds the world for that run can pull the data it needs and understand the schemas from superbase and oh truthfully the schema can be understood from the codebase but the data it allows it to populate that data from production that is really we've only really seen that necessary if you're adding like a whole new data source internally that like the agent has just never seen before in its life.

36:19 It's like if you really want that simulated faithfully then then that's something you should do. >> Interesting. I think this is really fascinating. Yeah, thank you for sharing this one question. Okay, so a lot of people are learning evals in this class and a question always comes up what are good companies to work for if you are into evals. Do you have any thoughts? well raindrop as you just heard we have a lot of eval I mean I think there's a range I think a lot of our customers are pretty good signs of people that like care a lot about eval whether that's versell whether that's holen is like there's a lot of people in the consumer space obviously there's like you know bigger companies as well but I would be remiss not to not to mention ourselves here I think ultimately you want to look for companies where the real world cost of failures is like high and I think that it's more than you'd expect actually like anything that touches the real world in a meaningful way or a lot of people probably the cost of failure is pretty high.

37:10 >> Cool. What kind of roles are you hiring hiring for? >> We're currently hiring most roles. so it's like sales head of sales AES and then on the engineering side we're hiring for FTEEs, we're hiring for product engineers, and we're hiring for MLES. So those are the three roles. >> All right, cool. I think a lot of people in this class would be interested in checking it out. That's why I asked. >> Yes, you can DM me or email me. It's benraindrop.ai. AI drop in the chat too.

37:35 >> Probably have some time for some questions. Is there any situation where simulations might be overkill? Like how do you know you need a simulation? When should you go for it? Should you go for it right away? Should you start something else first? >> What do you think? >> Very good question. And so I mean the very first place that you start is just trying it yourself, right? Like you know you have it running locally and just try it. Take the traces, ask an agent what went wrong in the traces and for it to like show you the parts that went wrong.

38:00 Some way of like annotating and seeing those traces. Like that's honestly why we built like Rindrop workshop open source tool runs locally streams all the traces directly into it. Like do look at your traces locally. Like it's very very important. I think it's like by far probably the most important thing you can do. Go through every tool call understand and ask an agent like how could I improve this? How could I optimize this? Like it really does work.

38:21 So that's where you should start. I think as soon as you're trying to start running evals now that you have agents like you really kind of need to find some way to simulate it now like depending on how complex your agent is like you may be able to just do this yourself like if you only have like a few tools or like the data sources are pretty straightforward or maybe you have a case where like you can just hook it up into like a readonly version of production and like limit the data it's reading or something like that like definitely like try I think that like you should always try building out your own solutions here because you'll at least understand the problem that we're solving or any other, you know, vendor is solving. But especially if you're doing like classification, there's a lot of things where it's like just run them both in parallel. The problem is that like agents are really expensive. So if you're going to do a shadow deployment of your agent across all traffic, now you're doubling your costs and then now you also have to look at those traces still. So like it's not totally solving the problem, right? And so I would think about it a little bit like simulations are the way to have an action. Like also the iteration loop there is ridiculously slow, right? It's like push to production, monitor for some time, find the things, then like go make another change, push to production. So it's like I think you want to try to make that feedback loop as tight as possible on simulations versus experiments and like AB tests. So Raintop actually has like we have an experiment feature that like is probably one of the most heavily used things in our entire app right after issue detection. It's really good. It's really important. Like I don't want to like deemphasize the importance of production. It is like production is ultimately like the source of truth of like what is actually happening in your agent, how it's actually behaving. You just like want to do something before you get to production to make sure that like people aren't, you know, bad things aren't happening.

40:07 >> All right. Amazing. Thank you so much.

Summary

Ben, the CTO and co-founder of Raindrop, discusses the evolution of evaluation (eval) methods in AI, particularly in the context of increasingly complex agent systems. He emphasizes the importance of simulations in understanding how changes to AI models affect their performance in real-world scenarios, advocating for a shift from traditional input-output evaluations to more dynamic simulation-based approaches.

- Raindrop focuses on identifying critical issues in production agents in real-time and simulating changes before they are deployed.
- The traditional eval model, which relied on simple input-output pairs, is becoming obsolete due to the complexity of modern AI agents.
- Simulations allow teams to replay historical traffic against new changes, helping to identify potential issues before they reach production.
- The importance of trace analysis is highlighted; teams should closely examine traces to understand agent behavior and catch issues early.
- Simulations can be tailored to specific changes, allowing for targeted evaluations rather than overwhelming numbers of evals.
- The concept of "simulation awareness" is introduced, where agents can recognize when they are being evaluated, potentially skewing results.
- Raindrop encourages organizations to connect their systems for more realistic simulations, enhancing the fidelity of the testing environment.
- The discussion emphasizes the need for a tight feedback loop between simulations and real-world performance to ensure reliability and effectiveness in AI systems.

Questions Answered

What is the purpose of the eval course and who is Ben?

The eval course invites various vendors to discuss evaluations, with a focus on Raindrop and its approach to finding critical issues in production agents. Ben, the CTO and co-founder of Raindrop, introduces the company's philosophy and recent product launch related to simulations.

How can one effectively analyze traces in AI evaluations?

Effective trace analysis requires tools that highlight important annotations to avoid confusion in lengthy AI traces. While catching issues in production is crucial, it does not prevent them from occurring, especially in high-stakes environments.

What is simulation awareness and why is it important?

Simulation awareness refers to an agent's ability to recognize when it is being evaluated or simulated. This awareness can lead to biased behavior, making it crucial for agents to operate under the assumption that they are in the real world to ensure accurate evaluations.

How can organizations define what constitutes a good or bad trace?

Organizations can define good and bad traces by establishing criteria such as output length and consistency with past performance. This allows for scalable evaluations across various traces without relying solely on historical input-output pairs.

What are the challenges in simulating complex AI infrastructures?

Simulating complex AI infrastructures is challenging due to the need to replicate various components accurately. This involves sandboxing the agent's code and maintaining state across different environments to ensure faithful simulations.

© transcribe · For agents Built with care and craft by Gokul Rajaram