transcribe

How to Evaluate Coding Agents

Hamel Husain · 24m · transcribed Oct 2026
More from Hamel Husain Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Section Insights

# 0:00

Introduction to Evaluating Coding Agents

What is the focus of this session on coding agents?

The session discusses the evaluation of coding agents, which differ from traditional coding practices and are more aligned with foundation model benchmarks. The speaker emphasizes the importance of humility in developing evaluation systems for these agents.

  • Coding agents are evaluated differently than traditional coding tasks.
  • Humility is crucial in developing evaluation systems to avoid overconfidence.
  • The speaker has experience in both scientific and distributed systems, providing a unique perspective on evaluation.
# 4:53

Selecting Tasks for Evaluation

How should tasks be selected from a repository for evaluating coding agents?

Tasks can be selected from the repository by examining the pull request history, focusing on measurable outcomes like fixing bugs. The choice between representative tasks and measurable tasks is critical for effective evaluation.

  • Evaluating coding agents requires selecting tasks that are either measurable or representative.
  • Using past pull requests can help identify suitable tasks for evaluation.
  • Clear definitions of success criteria are essential for effective evaluation.
# 9:46

Understanding Model Limitations

What challenges do coding agents face during evaluation?

Coding agents may struggle with longer contexts and complex reasoning tasks. The evaluation process must consider how these factors affect the model's performance and understanding.

  • Longer contexts can dilute a model's attention and focus.
  • Models may excel at algorithmic problems but struggle with context-heavy tasks.
  • Understanding the strengths and weaknesses of models is key to effective evaluation.
# 14:39

Grading Criteria and Evaluation Environment

How should grading criteria be established for coding agents?

Grading criteria must be clear and should not give agents hints about success. The evaluation environment should be representative, minimizing external influences and ensuring that agents are not aware they are being tested.

  • Grading criteria should be specific to avoid giving agents clues about success.
  • Creating a controlled evaluation environment is crucial for accurate assessments.
  • Minimizing external influences helps in obtaining unbiased results.
# 19:32

Measuring Agent Performance

What factors should be considered when measuring the performance of coding agents?

Performance measurement should focus on the ability to detect task completion, repeatability of results, and understanding variability in outcomes. Key dimensions for evaluation include speed, correctness, cost, and policy adherence.

  • Detecting task completion accurately is vital for evaluating coding agents.
  • Repeatability and understanding variability are essential for reliable assessments.
  • Performance should be measured across multiple dimensions for a comprehensive evaluation.

Transcript

0:00 Welcome everybody. This session is about evaluating coding agents. Now, coding agents are a little bit different than what we teach in this class. Coding agents tend to be more closer to foundation model benchmarks in the way you might want to think about evaluating them. Less like product specific emails. There is like a lot of interest in this class about coding agents. It comes up all the time. My friend, he has a company called Momento, and I'll let him introduce what Momento is. But, yeah, it's an honor to have him in this class.

0:29 And yeah, thank you for coming and showing us what you're working on and how you're evaluating it. >> All right, thank you, Hamel. I Thanks for that intro. My name is Khwaja. I started my career working on cameras on board the Mars rovers and and started really working with scientists. And then I kind of converted into a distributed systems person working on high-performance things. And eventually made my way back into the inference era back, you know, 15 years after I was initially working on that space.

1:00 And it's really brought back the memories of what it's like to be a scientist. And today, I'll talk to you a little bit about what it takes to build a an eval system for coding agents. And I just want to preface this with we started going down this path a little over a month ago, and it has been nothing but a humbling experience on how difficult it is to build an eval system, let alone an eval system for a coding agent. And the that humility is really, really important because the moment you lose the humility is the moment where you get you get overconfident and you stop diving deep into your entire eval setup. So, I wish I could come here and tell you that I have all the answers on how to build a eval system for coding agents, but I can share a lot of things that we did wrong along the way and and what we're learning.

2:02 There's still a whole bunch of open problems that, you know, we're we're trying to trying to address. And when I started working on evals for for coding agents, my initial instinct was, well, we we built a bunch of stuff and we got a bunch of really really cool sound bites. Like, GLM 5.3 flash does the same task in the same amount of time with this provider for 20 cents, whereas Astra is taking $9.28.

2:33 Sense. job done. Like, eval system is is complete and and we're we're ready to go. And as tempting as that may be, you know, we we could just end the talk there and say we're we're done or we can kind of keep diving deeper. And this is kind of like thing the repeated theme of this talk is when we kept diving deeper and deeper. We started thinking about, well, what does it take to build a rigorous eval system? And you're you're trying to create a set of tasks. And it all centers around the tasks that we are trying to you know, use to assess the efficacy of a particular agent.

3:12 And within that task harness, like as we as we come up with this eval system, you got to generate tasks. You got to think about how am I going to grade those tasks? you got to do sandbox execution with a lot of repetition and all of that feeds into a score. And the score the punchline there is like the score in our experience so far, like we we haven't converged yet. Every time we see scores, we are ending up with more questions than answers.

3:47 and I'll I'll talk you through why what some of that looks like. So, so let's say you started by thinking about okay, I'm going to go eval this entire setup. And that entire setup has a you know, there's a human. There's the prompt. There's the actual coding hardness. There is the model. And then there's the various reasoning levels for a for a given model. Each one of those things is a pretty big variable and that entropy kind of just multiplies along the way.

4:17 And any one tiny tweak to to one of these things can make make a big difference. But let's talk about how do you start by selecting what tasks you are going to evaluate the entire setup on? And you can start by SweeBench or one of the existing kind of benchmarks that are out there. And you know, if you're an enterprise looking at a variety of coding setups, the SweeBench may not be or or whatever benchmark you find may not be representative. You may not be a Django shop or you may not be a Go shop. It's not representative of your style, your frameworks, the types of problems that are happening in your repository. It's been public for years, so guess what?

5:02 All the models are already training on cuz the criteria is out there. and it has somebody else's definition. Like you may have a very different style and a different definition of what done means inside of your organization versus versus what the SweeBench and existing other benchmarks are doing. So, okay. Now we say, well, let's go look at our own repository. But how do you select the tasks from from your own repo? The most obvious thing is you say, well, I'm going to go find I'm going to go through my PR history and I'm going to find a place where I fixed a bug. And you know, you find something that's super easy to measure.

5:43 There's a unit test that's failing and I want to see which setup can fix that bug. And I'll just evaluate did it flip the test? And how long did it take? How much it cost? And you know, so you can do that. It's just roll back a version. And let's see if the model can come up with an answer. And and by model here I really mean the whole you know, end-to-end setup. And it it'll give you an initial set of tasks that that you can you can do that that you can test against.

6:17 And then there's a design fork here where you really have to decide if you want the tasks that you're generating to be representative of what you're doing or do you want them to be measurable? If like the the example I gave is incredibly measurable, right? Did my unit test pass or not? Like it was failing before, it passes now. The definition of done is very very crisp. You can also go a step further and say, "Well, I want to make sure that my other unit test didn't didn't fail." but that is extremely measurable, but it's not necessarily representative of a day-to-day coding exercise that developers might be going through. The other tasks like represent like make building representative task requires a lot more non-determinism and a lot more uncertainty. And those are actually harder and harder to to measure. So these things are like attention with each other in terms of whether you make your tasks representative, whether you make them measurable, or you or you do both. And and it's it's a spectrum. Like there isn't really a right answer, and you really need a full kind of combination that is specific enough to grade and real enough to matter. And as you're going through that loop of identifying the task, grading them, running them, diving deep into the traces, you're constantly thinking about well, what what do I need to do to enrich my enrich my eval setup? And our initial prompt, so the other thing here is that you can constantly have to just go back and keep digging in. What happened? Why did it happen? And we that that big headline number, that was generated. It was a you know, we we just rolled back a particular fix on the Valkyrie code base open source repo, lots of tests, easy to find issues that are that are fixed and the unit test that that went with it.

8:05 And and as we went to look through the prompts that was generated to to kind of solve this problem, the prompt looked like this. It's like, you know, big headings. Here's your task. Here's what you need to implement. Here's the source, you know, the the file that you need to go change and here's the test that you need to to go fix. And there's a whole lot of problems in in how representative this is. So, first of all, it names This might be a little hard to read, but the implementation file is named for you. Like it's like go fix This is like it's like being in college. Like if you went to like a code school or or or like a CS 101 where like the teacher has like created method stubs for you and you just have to go fill them in. It's much much easier, right? Cuz it's telling you here's the file, here's the test you need to fix, here's the grading command.

8:50 There's like minimal ambiguity. And the other reason why it's not representative, it's like very few people in the real world. Like I'm pretty OCD about this, but most people end up running very very long sessions. So, the session history is like diluted over time and the context is diluted. You don't show up in a cold session with a with a well-defined prompt like this that's like really really well laid out for the agent. There's all kinds of crap beforehand that may or may not be relevant and they may actually distract from the from the solution. And no human like or you know, got to be a pretty maniacal human that writes prompts like this.

9:30 Right? Like it's we we just talk to it. We just talk to the coding agent. So, what as we started got getting into this, we then started thinking about well, what is what is hard mean? We want it easy task, we want it hard task. What does hard really mean? And you know, there's somebody on my team for whom for him running a 5-minute mile is very very easy. And for me, that's very very hard. Right? And but there are problems that that I can solve in in ways that he can't solve and there's problems he can solve that I can't solve. Each model, each human is optimized for a specific set of things.

10:06 And what looks hard to you as a human may not be hard for the model. And it's really important to to to understand that particular delta there. But what what really some things that make life harder for a model is always longer context. all you need is attention and you know, like you your attention gets diluted over over time as the longer the context gets. And and that the longer the context is, the less focused it's going to be. But if you have a longer context, that often kind of conflicts with each other with with itself. Cuz that that happens. Like if you you know, and you can go out of your way to put traps in the context itself. One example would be if you have like a team meeting notes summary where you where you went on a digression and had like a completely wrong way of doing it.

10:58 Like that could trip up a model, especially if it's on high high reasoning. and if you need to make a lot of tool calls. Those are the kinds of things that like make life a little bit harder for a model. It's not about solving like you can give it a very hard algorithmic problem. Most of these models are actually quite good at like lead code style problems that may appear hard for for humans. So, now let's get back into the grading side. The super measurable tasks are are easy because you have you can run tests on the the whole repo, you can make sure that the test flips. but you can also This is a hairy one. I know it's one of Homel's favorites. So, you can just lean on a an LLM judge. And the the problem here is that the more measurable a task is, the easier it is to not rely on an LLM. But as the tasks as you start adding entropy, you might end up writing prompts five different ways because humans aren't going to write the prompt the exact same way. Like I won't write the same prompt the same way five times if you ask me different times of the day. As you start adding more entropy and as you start adding more variability into the task, a an LLM judge may be required to augment the the specific measurements that that you may have.

12:15 now this is where getting into how do you how do you rely on this judge? How do you, you know, do inter-rater agreements to make sure that the judge is actually agreeing with you. There's a whole lot of can of worms here that that open up as you start to rely on on the LLM judges. And the other part of it is at least we found that we we we're trying really hard to not let the agent detect that it is being judged, that it's it's undergoing a test as well. So, if you the moment you give it a very crisp grading criteria, you've kind of leaked that, you know, it's under under a test.

12:55 But that said, you you can when you're when you're doing a bug fix or you're making a code change or you're making a performance improvement, you can detect the bugs that you thought about, but you can't really prove the absence of the bugs that you didn't think about. and and that again, like some of this stuff like just doesn't show up. Like even when humans are coding, when you build mission-critical systems and deploy them into production, no amount of testing catches every problem. And there are there are controls that you have in place that get fed in and then you can do rollbacks and so forth. So, you can never prove the absence of the bugs or the completion of the of the of the task that's bug free, but you can detect when there are very obvious bugs that you have you have thought about.

13:43 so, we started going down this path of well, for the more the less specific, less measurable tasks, the the what we call the more representative task, we needed ways to to judge the the completion and and assess the the the setups efficacy on them. That requires us to build an eval for for our eval. It's it's quite meta to to make sure that our judgment criteria are reproducible. And then, when we run these tests, we realized coding agents specifically, they they stress the models in ways where the models are forced to get very very creative. And we learned the hard way that the level of sandboxes that we needed was always more than what we thought it was. So, when we initially started doing this, we rolled back it, we we gave it, you know, gave the the setup a task to go complete.

14:39 we observe in some cases it just rolled the history forward and found the answer. And we're like, wait, >> >> you can't do that to me again. I'll do a get snapshot and give you only you know, the point in time where you don't have you can't roll forward anymore. So, the second time in the early days where this happened, it actually found a like a a completed answer from from another agent on the local file system.

15:08 Harnesses have like built-in things to like remember. and especially some of these coding harnesses. So, like they have like ways of recollecting. And it was kind of spooky when when it went and like found another answer. And you wouldn't have known. Like initially, like you can just say, "Oh, look, it did it for cheaper and it did it so much faster." But nope. Like what happened was it it it went and started looking at other sources that had been done.

15:35 so the even the grading criteria, even when you tell it like, "Here's how I'm going to assess whether you're done or not." The definition of done gives it hints as to like what's going to be what it's going to take to succeed at this task. And it is very different than how human coders are prompting their agents. They're not that specific on success criteria. They're usually like, "Go fix this bug." Or "Go add this Go move this button." Or or change the layout here.

16:03 so agents should not know that it's being tested. We we try to make the environment as representative as possible. We we try to minimize any score files in the workspace. So we we run it in a sandbox, it's in a Docker. We we contain that Docker with minimal network access. We're kind of instrumenting every network call and tool call that's being made. got to make sure that session memories aren't are feeding in or memory breadcrumbs aren't there. And prompts have to read like normal human human prompts.

16:37 We kept learning that the agents were very creative and they kept breaking our isolation every time. And and while we we kept getting better at it, we also realized that you cannot, in fact, sandbox everything. But as we went down this this notion of sandboxing and finding ways to isolate, what we learned is that there are things that you just cannot sandbox. So like, if you have deterministic tools like your compilers, your your test runners, and so forth, those are easy to to sandbox and and and isolate.

17:11 But coding agents often require you to make network-facing tool calls. Like, sometimes they'll do a web search. Sometimes, if they're connected onto your wiki, you know, on your confluence, like they will make calls that are that are really, really hard to contain. And Google might give better answers this week than next. Or it, you know, it may give better answers next week. And that entropy makes things really, really difficult. And And again, it kind of goes back to the measurable versus representative task. Like, coding agents inherently end up doing a fair amount of web tool calls. And constraining that environment becomes really, really challenging.

17:53 so, the there's two ways that you can kind of get around this problem. the the first one is just repetition and aggregation. You just keep going and you you keep running the test over and over and over again, and you kind of understand what's happening along the way. and the other one is you can Like, so repetition is one thing. Just get as much data as you can, and then going deeper into traces. Like, continuously looking into your traces is another really crucial thing. Like, what happened? Why did it happen? Having that intimate knowledge of how things are shaping, like why models are failing. Often times, we kept getting really, really bad signals. Like, we would think that the test at like a particular model had failed, but we realized that we were just being throttled by the inference provider.

18:50 Right? Like, if you don't like dive in to see that's what happened, or you got an error on the gateway, the or or a call just hung. Have it like, unless you dive into, you know, why why the tests are failing, and unless you have enough repetition to to kind of facilitate that, you're you're not going to get an answer. So, these are like they're like basic things, like the whole like pass at K and and and pass to the carrot K like it's essentially knowing how reliable and how repeatable the the agents are becomes really really crucial. And with coding agents like it really comes down to what what is your environment like? How well can you detect when the when the task isn't actually done?

19:37 And because if you can detect when the task is actually done really really well then even if a if a really really cheap agent can solve it one out of 10 times, you can just run it 10 times and pick the answer. Right? But if you if there's a lot more ambiguity, then you need more more repeatability. You need you need to to be able to consistently pass the test over and over and over again. and you need to understand why why things happen. And we we constantly kept learning that a single run is not a measurement. There were scenarios where we you know, we we did this experiment where we were like, okay, let's run this particular task across two agents and pick the best for one agent and the worst for another agent. And there it looks like the good agent did, you know, 6.6 times better, whereas if you aggregate it, they were only like a 16% delta. So, look, marketing wise, it's really easy to come up with numbers that look flashy without really being scientific about how your your kind of getting getting into the numbers. And then as you look at the aggregates instead of just saying that here's the variability that you saw, just constantly diving into the traces to make sure you understand why and and can attribute it becomes important. The other part, we when we measure our our agents and our setups, we're looking at as I mentioned the the the prompt, the harness, the the model, and the and the reasoning level.

21:11 and and we're we're looking at dimensions along the lines of speed, correctness, cost, and policy adherence. Those are the four dimensions that we that we really tested on. And we fell into this trap where we were looking at how long like when we're just measuring the the latency. How long did it take an agent to complete this task? And a lot of it was diluted because the grading criteria was taking like the the the running all the tests was taking 20 minutes.

21:43 And the task itself was taking less than a minute. So, it looks like all the agents are performing at about the same speed because well, 19 of the 20 minutes are going towards the grading. And until you dive into that, you really don't know, and your results can just remain diluted inside of inside of that. the other lesson that we learned along the way was your harness needs to have observability built in. I've been a pretty big fan of the Strands agent harness that Amazon just released, but one of my pet peeves on it so far and and we might just go add this is like it doesn't have the level of observability that we need to grade how it's doing. It doesn't show you the cost or how many turns like we have to instrument a lot of it outside of the harness. And my advice here would be just like make sure that your harness is emitting the right metrics, emitting the right traces to make your debuggability and your evals a lot better. And these investments go go a long way. All right. So, just keep in mind again while you keep diving deeper, every little component makes a huge difference, and the entropy is just like very very multiplicative here.

22:55 For us, the model matters, the harness matters, the provider or or gateway matters, the policy and the reasoning levels matter and like any one factor changing makes a makes a pretty big difference in your your end-to-end results along the way. >> Well, yeah, thank you so much. I think everyone really enjoyed this. To like hear your experiences with the creating evals. Especially for something as difficult as a coding agent. I hope people give more evals a try. Thanks so much for coming.

23:29 >> Thank you, guys. >> Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than a thousand hours of working with companies on building AI products and it's customized to what stage you are at. So, this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are.

23:57 Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval. Or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.

Summary

The session focuses on the complexities of building an evaluation system for coding agents, emphasizing the challenges and nuances involved in creating a rigorous and representative assessment framework. The speaker shares insights from their experience, highlighting the importance of humility, iterative learning, and the need for a deep understanding of the evaluation setup.

- Coding agents require a different evaluation approach compared to traditional models, focusing on foundation model benchmarks.
- Building an evaluation system is challenging, often leading to more questions than answers as one dives deeper into the setup.
- Task selection is crucial; tasks must be both representative of real-world scenarios and measurable, which can create tension between the two.
- The evaluation process involves grading tasks, sandbox execution, and understanding the model's performance through repetition and trace analysis.
- Coding agents often exhibit creative problem-solving, making it essential to design prompts that mimic real human interactions rather than structured test scenarios.
- The evaluation must account for various factors, including speed, correctness, cost, and policy adherence, with careful attention to how grading criteria can influence outcomes.
- Observability in the evaluation harness is critical for understanding performance metrics and debugging.
- Continuous iteration and learning from failures are key to refining the evaluation process and achieving reliable results.

Questions Answered

What is the focus of this session on coding agents?

The session discusses the evaluation of coding agents, which differ from traditional coding practices and are more aligned with foundation model benchmarks. The speaker emphasizes the importance of humility in developing evaluation systems for these agents.

How should tasks be selected from a repository for evaluating coding agents?

Tasks can be selected from the repository by examining the pull request history, focusing on measurable outcomes like fixing bugs. The choice between representative tasks and measurable tasks is critical for effective evaluation.

What challenges do coding agents face during evaluation?

Coding agents may struggle with longer contexts and complex reasoning tasks. The evaluation process must consider how these factors affect the model's performance and understanding.

How should grading criteria be established for coding agents?

Grading criteria must be clear and should not give agents hints about success. The evaluation environment should be representative, minimizing external influences and ensuring that agents are not aware they are being tested.

What factors should be considered when measuring the performance of coding agents?

Performance measurement should focus on the ability to detect task completion, repeatability of results, and understanding variability in outcomes. Key dimensions for evaluation include speed, correctness, cost, and policy adherence.

© transcribe · For agents Built with care and craft by Gokul Rajaram