You Can't Test What You Can't See - Rachel Lee Nabors, Arize

0:00 Good morning everyone. How are you doing today? Yes, that's what we like to hear. Has this ever happened to you? Your agent is working perfectly in production, or so it seems. The types they pass the demo, it works. You've shipped it. But hold up, look closely. What is that? All is not well. The agent has recovered, but there's a silent failure that's happening. If you blink, you might miss it.

0:33 We We actually solved this exact problem for regular code about 20 years ago. We called it test-driven development or TDD. but agents don't work quite the same way as as code. They're non-deterministic, et cetera. Agents don't have tests. They have evals. And for agents, we need EDD or eval-driven development, the third member of Ed, Edd, and Eddy. yes, I am Rachel Lee Neighbors. You might remember me from such awesome places as the MDN Web Docs and React documentation. Any React people in the house?

1:11 Hello. Oh my gosh. And TypeScript's my community. I still I I'm working with a bunch of Python engineers, but I still very much love to work in TypeScript. I love the tooling story that comes with TypeScript. I truly do believe it's got one of the best tooling stories in the industry. And today I work with Arize's Phoenix team. Arize Phoenix is written in Python, but but it has TypeScript SDKs, which we'll be using in this example.

1:38 Phoenix is an open-source platform for agent development and evaluation. how many people here are familiar with test-driven development? Oh jeez. how many of you have agents or LLMs in production right now? Very nice. And how many of you have evals set up? Okay, we have some work to do. And how many of you don't have LLMs or agents in production, but it's going to happen. You're here to make that happen today.

2:10 Yeah, awesome. Love it. Okay, well then this this is the talk for you. We have a shared vocabulary of concepts for all of you who have used test-driven development and are curious about evals, I'm going to give you a little one-to-one here. unit tests? That's just an eval. When you think unit tests, just think eval. test suites? We would have reams and reams of twist test suites in test-driven development. With evals, we have what's called a golden data set. It's a a set of inputs and outputs that you've curated and you want to make sure that every time an agent or an LLM receives a certain input, it's going to give a certain output.

2:49 a text fixture is called a trace. And assertion is an evaluator. And instead of passing and failing, we use score thresholds cuz nondeterminism, you know, it's not always going to pass in some search circumstances. You're just trying to get like over the 85% hump. and instead of CI gates, we have regression evals. That's to make sure you you don't fall backwards. All right. Okay, I'm going to pause on that one while everyone takes photos. We will be making the slides available later, too. Just I know this is this is fun stuff here. So, let's go back to the silent failure, where we can see that the agent just said, "Let me find the right table to search from." Well, that's that's kind of You don't want people to see that. That implies that the agent doesn't know where it's searching from.

3:40 Let's go dive into the back here. First off, you can't test what you can't see. Thankfully, agents and LLMs are inherently observable. They leave a loggable record of every step they take. if you've ever, you know, looked at the logs of your chat GPT conversations, there's the user input, the LLM's output, back and forth, and you continue to pass them back and forth. You just need to capture those logs. This is, you know, you need to implement observability, as we like to say.

4:12 these logs are are called called spans. They're a recorded unit of work that your agent did. It includes an input, outputs, timing errors. It's nice, you know, you get enough of these, you can mine them for gold. Now, I'll be using Phoenix for the demos today, obviously. but you have so many options for implementing observability, there's really no reason not to. Phoenix is small and it's open source, but if you're working at a big burly enterprise that needs R back and all that stuff, you can use Arize AI X, or maybe you could use a competitor like Braintrust. You know, even Mastra ships with something called Scores today.

4:49 There's no reason not to turn this on and start gathering those logs, because then one day you're going to have a failure, but you're going to have all the logs for where it happened. All right. set it up today, collecting those traces is key. A trace, that's just the full call stack for one request. let's take a look. So, you can think of like a trace is like a collection of spans to accomplish something. Here's what's going on behind the scenes at the store. And you can see we've got a collection of different traces there.

5:20 I have sorted them by errors and went into the the spans. So, this here is one collection of of traces, and these are each individual row is a span. If you go over here on this one failing span, it's got a little exclamation mark there, it's logged that there's an exception from the tool it was calling, no such table products. This agent is trying to write its own SQL queries, but it doesn't know what the shape of the database is. So, it's guessing. And if you look I'm going to I'm going to make you play that again.

5:55 No, not that way. The other way. Yes, like that. But if you take a close look, that one little issue, when you go in and you take a look at all of the different spans inside this trace, this tool call failing triggered another tool call to go and query the entire database and another tool call after that. So, one failed tool call meant that we ended up doing three round trips to Claude Sonnet instead of just one.

6:24 this is the kind of it looks like it works, but it didn't work failure that slowly eats up all your inference budget and gives your users a terrible experience. Agents are smart, but they're still expensive. That's not one of the catastrophic failures, either. I'm giving you an easy example today. So, once you find what's breaking, you want to save all those broken broken error logs into a data set. Now, the data set is your test suite.

6:52 Well, you can think of it as one. It's a curated collection of examples, each one an input, optionally including the expected output, and you run your agents against it about every time you change something. There are only three things you can change: user inputs, the model, and the the prompt. I guess you could change the harness, too. And anytime you change the infrastructure, you want to rerun that rerun your golden data set. So, we take these broken spans and we save them as a data set.

7:20 I'm just going to call it failure. there you go. And boom. Now, you go check out that that wonderful data set. And there they all are. You got your inputs, no real expected output, but I did I did include some metadata there. That's not so hard. And the good news about testing agents is you don't have to imagine all the ways they fail. You've ever stayed up late at night going, "Oh my god, I thought of another edge case. Hold on, let me go add an assertation."

7:48 With agents, we learn from failure. We harvest the error logs and turn them into that golden data set. You can't predict all the ways in which a system will fail, but you monitor it and it becomes a part of your weekly practice. There's actually a really good article on the Phoenix docs and the cookbooks called finding high signal traces about the practices of going through your evals once a week and looking for the errors and harvesting the ones that really make the difference in the long run.

8:18 So, you'll keep acquiring these failures as a golden data set that you can use as part of your eval suite. this is important especially as you introduce new harnesses, models, and tools. So, next up, we're going to need an evaluator. evaluators, they validate the outputs. There are two kinds of evaluators. You could argue that humans are an additional kind and I've heard of agent as judge, but it boils down to this. You want to check to see when I give my when I give my agent this particular command, does it return this particular output or something close to it, something acceptable? And if it's something like I don't know, a you know, you you you want to get like properly formed SQL queries out of an agent, that's very easy to validate with code. You can simply run the SQL query or you can look at the SQL database and see if it's the right shape. so, code-based evaluators are cheap, they're fast, they're easy, they're practically free compared to LLM as judge. This is where you take the output and you say, "Hey, Claude Opus."

9:24 It's always a bigger model and usually one from someone else. If you're using like chat GPT as your agent, probably want like Gemini or Claude as a judge of its outputs because I can tell you they do tend to agree with people who come from their own family a little too much. They favor their younger sisters. So, anyway, LLM's as judge come in where judgement would be required. It's a little fuzzy, like was this answer helpful? Did the Did the the respondent seem frustrated? Were you able to stay on topic? is the tone correct? These are things that are much harder to rejects.

10:05 In this case, I made a programmatic evaluator to make sure that the SQL queries match the shape of the database. And I wrote it in TypeScript. It's actually probably much more complicated than it needs to be. I'll be honest, I let Pixie, the little agent down here that Phoenix ships with, I was like, Pixie, just make me a an evaluator for this, please. And it did. It was very nice. >> >> I I am a lazy coder, but I'm not a lazy tester.

10:35 Now, the next the the fun stuff here, this is the part I like is the experiment. You got to test. You You You found a way to check that the outputs are correct, but now how do you get the outputs to be correct? Well, if you saw the original, you're thinking, well, maybe you could just give the agent the shape of the database, or you could, you know, also make it a tool for accessing the database instead of letting it write random SQL queries left and right. I'm sure none of us have ever done that before.

11:05 there are a number of different ways, and you can set them all up inside an observability platform like this. Now, you can modify the three variables. You can modify the prompt. That includes the system prompt. That could be a prompt that's actually being sent to the agent with each user's each user's message. You could change the model. Some models are just smarter than others. In this case, I don't think you're going to get a better result with a different model. They're all going to pick the wrong table because you didn't pick like a a non-deterministically appropriate name for it.

11:42 And you can change the input. Although this is less possible when you're dealing with user-generated inputs. You can't really control what users are going to say. They say the darndest things. So, this is how you run an experiment. I chose to update the system prompt. I just included the shape of the the shape of the the database in the system prompt and you can see them running against each other here. And this one, look at that, average one, all green. Over here, the original zeroed out. The So, each time that was running, it was getting three calls instead of instead of one and now with the updated prompt, it'll knock it out of the park single shot every time.

12:25 Same model, same snapshotted use cases, same evaluator. Old prompt versus new, new prompt wins, let's put it into production. Testing is great. It's still rather contrived. We are testing We're We're isolating all those variables and putting them into a a little terrarium together. We try to make it look like the real system, but let's be honest, nothing beats introducing it into the complex system that is production. You want to make sure that you're verifying that the agent works within its natural environment, of course. And here it is actually working. It'd be a great toy for a toddler. It does not bork out and and have to go and query the database a bunch of times to figure it out.

13:09 Good job. Thank you, little agent. And if you go and you check the traces, look at that, so much shorter. We like it. That's probably saving us like 15 cents a run. Ooh. All right. And step zero is repeat. It's the cycle of eval-driven design. You don't do this once, you do it over and over again. You find what's breaking, you take the traces, you add them to that golden data set, and you make them a part of your regression evals. these, you know, they're how you keep your CTO from blowing away your agentic experience by accident. That has happened to some founders I know. I was like, "Did you not have regression evals before he went in and changed the model and the prompt and the harness?" No, we didn't have that. And it's like, "Well, that's why all of your customers are calling you very upset right now."

13:57 If you are a CTO, this is how to keep your job. >> >> All right. But, the neat thing is about this technique that I just showed you is that you can use it to evaluate more than just prompts. I used it to actually get rid of Claude Sonnet entirely from something that I build on the side. I was once in production, this would have cost me about a dollar per user per week. But, using using test-driven development, pardon, using evals, I was actually able to switch over to a local model and shave a full 1 and 1/2 seconds off the latency. So, it was a win for me cuz I completely externalized the inference cost to user. And also win for the user cuz they don't have the latency anymore. And, you know, honestly, Claude Sonnet like this was probably beneath it. All right, the cycle of evals. Cycle of evals. You collect traces. You take snapshots of the ones that fail. You evaluate them to figure out what it would what you'd need to see for them to be not failing. You experiment, try out different prompts, maybe different models, and even different inputs, and then you verify, and then you repeat.

15:08 If you'd like to talk more about evals and developing with them, I'll be here all day. You can check out Phoenix and her big bigger sister AX at following QR codes. If you do one thing today, I ask you that you turn on observability and you start collecting traces. If you're using agents in production, do it today. Phoenix is free and open source. Just instrument it this afternoon. In 1 week, you can start building your golden data set. You can even use those traces with a different product if you like. But what matters is that you start tracing now.

15:42 Thank you very much. I look forward to seeing you at the rest of the conference. >>