CMU AI Agents 2026: 6. Agents for Coding and Software Development

0:00 Okay. today I'll I'll be talking about agents for coding and software development. So, presumably many or mo most or all people here are using coding agents. So, I I barely need to describe what their underlying functionality is. but I would like to talk about how we can build coding agents and what the necessary things we need to have in order to build coding agents are. And actually I just realized that the title of this slide is maybe a little bit of a misnomer. The first thing that I'd like to talk about is what it means to have a language model. Not a not an agent, but a language model that's able to write code. and so there's different things you can do.

0:51 The kind of obvious thing is you say in English I want you to write a program that visualizes this data for me. and you give it like a CSV file in your prompt and creates a bar chart for this data. how many people do this sort of query like frequently? Doesn't necessarily need to be a bar chart, but like write write this program for me to do this task.

1:24 Probably a lot of people and you write it in like chat GPT or a coding agent or something like that. however, there's a lot of other tasks that involve generating code that's not just like a oneoff like prompt, but you can also do you can have it modify repositories. So, you can have it modify like multiple files. and so this would be something more like in this repository fix this issue or make this improvement or something like that.

1:58 And then finally, actually developing software is not just writing code and implementing fixing issues and implementing features, but there's also a lot of other stuff involved. So I'd like to talk about some of that other stuff as well. And of course like all of our other agents, this will involve us writing prompts, providing tools to the agent, and having a good LM that's able to solve these tasks. I'm mostly in this lecture going to focus on the second and third parts of this. And the reason why is because we spent the last two lectures talking about like skills and memory and you know how we incorporate long context. And so for the prompting I I think many of the things were already covered in the previous lecture.

2:53 So the first element that we have here is a large language model that has the ability to write code. And so the example is writing code. in order to write code from a prompt, you essentially need to be able to cover a few things. and sorry, just to clarify, this is this is single step at the moment. It's not agentic codew writing. It's singlestep language models. And so in order to have the ability to write code in a single step, you need to be able to understand the language the programming language that you're going to be writing in.

3:31 Another very very important topic is editing because usually we don't write code once and then throw it away although we do do that sometimes. Usually we like iteratively improve and that is not something that comes for free from language models. you need to train it into the model. And then the final thing is reasoning. And coding has been one of the more successful applications of reasoning models just because like math which is another successful application of reasoning models. You can easily verify or at least somewhat easily verify that you did a good job of it. So these are the the three things that I'd like to cover.

4:17 So when you're training recently every language model is a coding model. so there there are no language models that don't know how to write code. And the reason why is because coding is one of the most valuable and like widely used use cases for language models, but it's not something you get for free. so may maybe before I go into the like the actual content, what sorts of things do you think are necessary to train a model that is good for generating code? Not necessarily for agentic tasks first, but like what what sorts of things do you think you might need to do?

4:55 >> Any ideas? Yeah. >> First of all, tokens must >> Okay. Yeah. all the tokens must exist. so what what are what are the some tokens that must exist for code that wouldn't necessarily need to exist for non-code models.

5:27 >> Yeah. So there's I'll give a few examples of of what these tokens look any any other things that you might need to do? but like sorry just to to summarize. So you need to make sure your tokens have the tokens that are going to appear in code. Yeah. >> You just need like a lot of coding. >> You need a lot of coding data to train on. Yeah. and so where where would you get that?

5:50 >> GitHub. >> GitHub. Sure. Okay. Cool. any any other ideas? Yeah. For coding there's like special tools like all the OS related >> for coding there's special tools like GP and OS related tools. So yes this is very important. so that's more that's maybe more important on the agentic side once you start creating an agent that actually like moves around the codebase. But if you were writing bash scripts for instance, you definitely need to know know about those even as in a single step setting. So yeah, these these are all great points.

6:30 so pre-training you need to pre-train on a large corpus. It needs to include code, but it can't be only code because you also need it to understand English or whatever language you're quering in it it in. As well, you need to make sure the tokenization's right, which I'm going to talk about in a bit. the next thing is mid-training. And I'm going to talk about mid-training and supervised fine-tuning as kind of being the the same or mid-training or continual pre-training or supervised fine-tuning is all being part of this step. in here you need to train on code mixtures.

7:08 Another really important thing for coding models is to be able to handle long contexts because you want to pull in lots of context from throughout the codebase. And then another thing is RL post training. And of course we're going to be talking about each of these in more detail later with respect to the specific mechanics of how we train these models. So I want to mostly focus on kind of like the coding specific details in this task and leave the overall overview of how you do supervised fine-tuning or RL postraining for later.

7:42 So pre-training this is basically we need to learn from code and text and of course we're just optimizing next token prediction at this stage typically as is shown in the equation. So when you're training a coding model in contrast to training a model that only needs to answer questions you want to mix web text of course because every model learns from web text but also technical pros. So this could be explanations of how code works.

8:15 things like math for improving logical reasoning abilities, source code, documentation, many many sources. Another thing, this is before probably any of you started working on language models, but there was a time when we normalized whites space. so basically we would take all of the spaces and like drop them down to a single space or something like that. That would be really bad if you're programming in Python, right? Because whites space is meaning bearing. So you need to be careful about these sorts of things. You also need to be careful about file boundaries and how you express file boundaries. You need to be able to have APIs and you need to be able to preserve text and code relationships.

9:03 Like if you have a markdown block within text, you want to make sure it remains in a markdown block. And of course, you will be evaluating while you're doing training. Typically, you'll do this just on a held out loss on coding tasks, but you need to shape your evaluation so that it covers the sorts of things that you want to cover. So choosing and cleaning data is hard when you're training a textbased model.

9:36 It's even harder when you're training a coding model because code is very unique in its distribution. So the first thing you need to do is filter by language and source. There is a lot of code online that's completely autogenerated and so it's like less interesting from the point of view of you wanting to train on it. So you can remove that sort of code. you can remove things that are in wrong encodings and low value repeated codes. So one example of this is forks or vendored libraries or nearly identical files or copied solutions. So if you look online on GitHub, there are thousands to tens of thousands, hundreds of thousands of forks. And so if you just download them and use them as is, then you'll have lots of lots of trouble. Oh, sorry. also mentioning this is tiny down here, but I'm taking a lot of this from a paper called Start Coder, which was one of the first coders on one of the first papers on pre-training models to be good at coding tasks. So if you're interested in all of these details, you can you can take a closer look at that. And I believe Daniel you were involved in this. So you can ask Daniel for more more details.

10:58 another interesting thing that people think about in some cases don't think about things in other cases is recording licenses and source dates and optouts and sensitive data filtering. So just to give some examples, if you train on code that's in a proprietary license and then release the model and then make a billion dollars on this model, the people who have the proprietary license are going to come around two years later and say, "Hey, you trained on my code. Give me a hundred million because my code is contributing to your model." And you might not want that to happen. or you might not care. But you need to decide u actually US copyright law is a little bit vague about you know whe whether you should care about that or not.

11:54 source states so if you don't want a whole bunch of old code you might want only new code. you can allow people the ability to opt out. also sensitive data. people upload all kinds of things they should not be uploading like their API keys which nobody in this class will do, right? please please do not push your API keys to the repository but you know people do this. and I there are tools that allow you to do this sensitive data filtering like I I and hugging face use one called truffle hog which is one that basically identifies all these API keys and then if you want to build a a training set or a test set you'll want to build a training set or a test set but this is also very difficult because of forks and all of the other problems you deal with. So moving on to the next thing which is tokens. this is something that you know kind of needed to evolve as code became more important. But basically here I wrote bite level BP on code and pros. But BP is basically just any subword tokenization method. I think maybe most people would be familiar with with BPE but you can substitute in anyone you want. the thing that you need to be careful about is preserving whites space and also compressing very frequent things. So up until we started working on code a lot whites space might be like one token. So that would mean that you would if you're generating 16 indented you know Python code you would have 16 spaces or something like that.

13:44 And so typically what it looks like is it would be you know compressed into like three spaces and a a return in like a single token. So you start tokenizing specifically in a way that's useful for code. Another thing you can do but don't necessarily need to do is like normalize the code so it becomes in a more normal more standard setting but you you don't necessarily need to do that.

14:16 So another thing you need to think about is the languages that you want to handle. And so Starcoder they took the approach of modeling all the languages or modeling not all the languages but a lot of very popular languages and kind of necessarily the ones that had more training data ended up being better at particular tasks. Another other things that you need to be aware of are you know like what languages will the users want to be using and also what languages are easier to write or supported by you know whatever agentic framework you're using.

14:58 So for instance, if you ask a model to do something and like you don't specify the language, what language will it it use? Like have you ever said code me up an app to do X? What language will it use for that? Yeah, >> mostly Python. Yeah, I think mostly Python. What has anybody seen any other contrary things? typically react.

15:31 >> Yeah, exactly. So, if you if you ask for like a dashboard, it will do React. If you ask for just about anything else, it will do Python. and of course, you can specify something else. I I had a heavy data processing pipeline that I originally wrote in Python and I asked the model to convert it to Rust to make it faster. And so the model did that but then every time it ran another program it ran it again in Python. So like I had to say in rust in rust in rust in rust every time until finally it did it. So they have these very strong biases. I think the biases are built in for two for two reasons. One is ease of writing the language. So, Python is like relatively concise and easy to write. Popularity and then also like downstream what was the model RL trained on? And it was probably RL trained with like a Python interpreter.

16:27 So, it it's more favoring that language. Cool. Any questions about pre-training? >> Yeah. >> Is there a situation where you want to augment your tokenizer specified like space for example? a separate token so that it's easier for the model to internally separate like language tokens and coding tokens even though they're like the same. >> Yeah. Yeah. So this is a great question.

16:58 I mean like it it does have return space face in here in the example. But one one interesting thing is you actually don't need to treat whites space any differently than every other token. bite pair encoding is like literally meant to merge together bytes. and so however in almost all of the cases people do treat whites space differently. and so the reason why is because like let's say the model is performing a little bit poorly. It's very weird for it to put out a partial word. And so they want to like essentially bias the model so that it's not outputting like partial words or partial word tokens. And so typically they will tokenize whites space separately and non-white space separately. But there are also kind of exceptions to that.

17:51 >> Yeah. >> Yeah. You can get a more efficient like representation in particular for code if you like tokens across space. So in the encoder paper we did that you can get like 40% tokens but it can also break things in weird ways like said so like import nump numpy as np would be one token but like the model wouldn't be able to stop like import numpy as you have to like back off and do like and you also have to be really careful if you change your tokenizer can break at scale and you have to be a little bit smart about how you do it. There's some papers on that we can share.

18:37 >> Yeah. So, so just to repeat basically you can the two of the main points were like you can get a much more compact tokenization if you actually allow it to just tokenize indiscriminately regardless of whites space or not. but it can break things in weird ways. So like if you write import numpy is np that could become like a single token in your your code model but then you'd have trouble like doing autocomplete or something like that because it's too long. So, cool. Any any other things?

19:15 Okay. So, now I'd like to go into mid training. And, I have a few examples here. One is from Code Lama. Another one is from, Quen Coder. And so, these are, public papers that explain how they take a previously trained non-code model and adapt it to being a coding model. So this kind of is good for highlighting the the difference between a co a non-coding model and a coding model. So the first thing that you can do is you can increase code exposure. So if you start off with a much you know more heterogeneous data set you can increase the exposure of code. you could also choose the mixture empirically to try to get code gains and maintain general capability and so for example in code llama they took llama 2 and then did 500 billion additional tokens for the the additional models and there context extension is also important here. So here's one empirical result from Quen 2.5 coder where basically they said this is a coding model and we're going to make it really good at code and so we'll just add 100% code and and train on 100% code afterwards. But the problem was it got quite poor scores on other important tasks like math or question answering. but if they continued training with not 100% code but but also text and math they were able to have like a tiny drop in the coding performance while maintaining other performance. And this is obviously possible or important because you know like nobody's going to use a model like just for coding nowadays.

21:06 So another thing is long context handling. And I already kind of talked about long context handling a lot. So I'm just going to go into this in a previous class, but I'm just going to go into this in a little bit of detail. But you basically want to come up with coherent sequences of to train models on. So a very naive way to create data for a coding model is to say I'm just going to take this repo and I'm going to concatenate all the files together. But concatenating all the files together actually very often will exceed the long context of even the longest context models. Like you only train up to typically 1 million tokens and there's many code bases or like I'd say most reasonable code bases are more than 1 million tokens. like the Google monor repo is allegedly like billions of lines long, right? So that's not even on the same order of magnitude. And so but you would like to have a coherent file understanding. So you can do things like take everything in a directory or you can parse the dependency graph and follow the dependencies and concatenate together things that depend on each other. that sort of thing.

22:26 so Quen 2.5 coder just as an example it started training into 8K and then it extended up to 128K with yarn like like I talked about a couple classes ago. So this is also really important and this sort of context manipulation allows you to do that and you can validate it through tasks like cross file completion and infilling and short context retention which I'm going to talk about in a bit.

22:57 So next I'll talk about infilling. I feel like actually I shouldn't talk about this because Daniel's the expert and this is mostly about his paper but I I will talk about it anyway and people can ask him questions. but he had a paper called encoder. it was one of the first papers on training strong coding models with an infilling objective so that they would be able to perform tasks that are not just code completion but also kind of predicting the inner part of code.

23:27 And so what this means is we get something like this is positive and we want to predict its return type would be just one example of this. And so the problem is if you just look to the body to the the left in a normal auto reggressive model, you actually have no idea, right? Like there's no information about is positive. I mean, well, maybe you could guess bool from is positive, but it it could also be a string or whatever else or an enum. And so then if you look at this, you could you know, tell that it's actually the return type.

24:03 So how do we do this? the answer is surprisingly is that a question or no? okay. So I think the answer is surprisingly you know you can do surprisingly simple things to make this work which is basically you turn an infilling task into an auto reggressive task. And so you have like the prefix, you have the suffix and then you infill infill the span here. And so this m0 is essentially the the first like the first span that you infill and then this is that comes after m0 here. and so then at inference time you just do you put m0 here between the prefix and the suffix and then you put m0 again and then it generates a span.

24:54 So this also generalizes to one or more holes. So like if you have multiple holes, you can have an M0 and an M1 and then generate M0 at first and then M1 second. So this is a a general way that you can do any sort of infilling task. And this allows you to do all kinds of useful things from the point of view of software engineering. So you can complete code in the middle of a large file. You can infer types like I did for return types. You can generate comments, rename variables. All of these things are like instances of an infilling task.

25:33 And here are some of the experimental results from the paper. So there are basically left to right inference on a number of tasks that they evaluated on single lines and multiple lines was like 48 to 24 and 25. if you did left to right with 10 candidates and reranked them using the context from the whole file, you would get do better but still would not achieve nearly the improvement of training a model to do this. And this is you know not requiring any reranking at all. That's just using the the infilling based inference and you can demonstrate that you can train with infilling but still maintain standard code completion.

26:22 And so if you did causal masking where you basically you know masked the the left to right they still were able to maintain performance. So it's not like really a trade-off between the two. any any questions here or any follow-up comments? >> Yeah. >> Yeah. You apply like a a random selection of like kind of like a long tail distribution.

26:58 So the majority of the time you're just doing one, but you could do up to I think like 100 chance. And then for each one of those you like sample and those are like you just replace the model >> and so the model doesn't predict the mass. >> Okay. So I I'll repeat that for the back but basically the question was how do you decide how many spans and how big the spans are to sample and the answer was basically you use a longtail distribution so that you're doing one most of the time but then you do two some of the time three less of the time and and similarly for the the length of the outputs and you could probably model that after the infills that you actually see in real life if like you wanted to do that. So >> yeah let me just count on scale. Yeah, cool. Okay, so this is one example of an important thing that you can do for code in filling. but you can actually become very creative in learning from code.

28:08 So there's a bunch of different sources. So, you can use commit diffs. And so, if you have the before, if you have the files before the diff and a message, like from a a pull request or something like that, you can then generate the diff or the after output. So, this would allow you to basically learn good code editing models. another thing is learning from diagnostics.

28:40 So, you can get broken code plus an error and generate, repairs. So, does anyone have an idea of where you could get like a ton of this data, nearly for free? >> Stack overflow. >> Stack Overflow is is one good one. That's not the one I was thinking of, though. issues. >> GitHub issues maybe a little bit more >> yeah, GitHub pull requests is is good, but like if you're a software developer where where do you see code break? Maybe in in GitHub or something tests. Yeah.

29:16 So I I specifically was going to talk about continuous integration tests, which are the tests that run when you make a push to a repository or something like this. All of this history is included in GitHub and it's public and so you can see that a particular version of the code ran and generated an error and then a later version of the code eventually passed and got merged in. So this is a very rich version of this sort of test data that you don't need to like generate yourself because generating it yourself would be would be painful.

29:50 of course, if you start running an agent, you can run an agent and generate the data. which is what I'll be talking about later. So, yeah, the tests is related to that. another thing is execution traces. And so, this is kind of interesting. It's a little bit niche, but there you know it it's pretty cool, which is like you can see how the program actually acts by instrumenting it and running it at test time. And you could for example ask the language model to generate this the stack at a particular point in in program execution or something like that. or generate the outputs at a particular point in program execution.

30:34 And I'm going to talk a little bit about code world models at the very end of this talk which kind of relies on this. Cool. any any questions? Good. Okay. So next I'd like to talk about evaluating generated code. there's a few paradigms to do this. one of the most obvious ways to do this, which actually, we we did in a paper in 2017, I guess, is comparing with a reference solution. And so, what this means is you have a human generated output and you compare not the execution results, but you just say, is the machine generated code similar to the human generated code? And so a long time ago, there was a famous metric called blue. Has anybody actually used this anymore?

31:30 Oh, wow. I'm actually surprised by how many people still know still know of this, but there's a metric called blue, which is basically measuring token and engram overlap between the the output code. And so this is this would allow you to find code that looks similar on the surface, but the problem with this is that it if you have like more than or equal, this is only one token different, but it's actually very different from the point of view of the behavior of the program. and on the other hand, there's things with different expressions that look very different from the point of view of token overlap, but actually are are not different.

32:17 So there's been a few ways to essentially account for this. So one way of doing it is this metric called codeblue. And basically what it does is instead of comparing the surface level similarity from the point of view of tokens it compares the abstract syntax tree between the programs. So one feature of programs is every program can be turned into a tree that essentially specifies how the model you know exe or how the program will be executed according to the logic of the programming language. And so they they compared based on the syntax tree and they also compared based on a syntactic data flow match or semantic data flow match which is like which variables are kind of come from which place and and stuff like that.

33:13 another way that you can compare code similarity is using embedding based methods. And so this is something that we created in 2023 called code bert score. And the basic idea is that you would take code and this would be one example of code. This would be another example of code and you embed each token in the code. And then you check to make sure that each token has a good other token that corresponds to it in the in the other set of code. And this this allows you to tell like that all of the code is all of the code is essentially reflected in the other implementation.

34:04 So these are good in the case or actually sorry I'm going to talk about the trade-offs in a few slides. So may maybe I'll just switch to the other paradigm of evaluating code and this is by far more popular nowadays. so nowadays the typical way of evaluating code is by actually running it. And so what you do is you have a candidate a candidate output and then you have a bunch of test cases and based on the candidate output you run the test cases and you evaluate and if it passes all the test cases you get a score of one.

34:42 If it doesn't pass all the test cases you you get a score of zero. so each of these have benefits and disadvantages. So if you look at the execution checks the benefit of execution executionbased evaluation is that this can accept different correct implementations. The disadvantage is this is really hard to create really good unit tests. And for every famous benchmark that you will find, there is a paper complaining about how bad the unit tests are and saying that they either have false positives from the point of view of allowing incorrect implementations because the tests are not rigorous enough or false negatives which is that the input to the model is the input given to the agent is not enough to pass all all of the unit tests. And just to give an example of this, it might say I want a I want a nice button on the top right of my interface that says learn more or something like that. And that might be great, but the test might also be checking the background color of the the button or something. because that's a requirement that wasn't obvious right at the very beginning of the test, but became obvious later. So th this is like the problem it's like one of the biggest problems in training coding agents nowadays because we move we're moving to more and more difficult tasks and creating tests that both have low false positives and low false negatives is is very hard.

36:33 so the the advantage another advantage is detecting semantic errors through execution is good but this needs a essentially good tests dependencies in an isolated environment and so for instance if you need to run the tests it's easy if you're just using the Python standard library but if not you need to install all the dependencies beforehand. and so the typical way we do this is we have something like a docker container which installs all of the dependencies.

37:13 but you might think you have installed all the dependencies but then you didn't think about the versions of the libraries that you're using. And so the versions get upgraded the tests die. and you're you're giving a newer model a lower score just because the tests are like out of date. So this is another another problem we have to deal with. And then the final thing is you need a stable execution environment in time. And so running the tests for a very big repository can actually be very time consuming. And in a lot of the benchmarks that we deal with including Sweetbench which I'll talk about in a moment they you know have come up with ways to fix this but it it's definitely a a difficulty.

38:02 Cool. so ju just to give some ideas of benchmarks that are popular in like singlestep code generation. human develop string and you need to generate a completion and there are hidden unit tests. So, this was actually, I think, just manually created by people at OpenAI. so they came up with a bunch of like kind of tricky Python, functions, to implement. And so it's not necessarily reflective of real world usage.

38:50 one that's a little bit harder is one called code contests. And basically, this is like code forces style like more competitive programming things. And there's a bunch of a bunch of other evaluation benchmarks, but they go from kind of like simple to to more complex ones. on the fir on their first attempt can solve a code forces problem with no mistakes and get it 100% correct. But that's kind of what we're asking singlestep language models to do here.

39:30 And so this is actually a pretty pretty difficult task and it requires extensive reasoning. so this is an example of something from a paper from 2021 that introduced humanl and everybody if you know what codeex is nowadays you'll be very confused by the fact that this is also called codeex but this was the original code completion model that was implemented for github in 2021 and basically here what they're doing is they are evaluating how how much basically the model passes and so you can see that as they scaled up the model the model passed more at one and what pass one means is you get to generate a single output and that passes but it's also common to report like pass k which is like if you generate if you generate five outputs. is any one of those correct? And so that's like a slightly more lenient metric that you can evaluate as well.

40:46 >> Cool. any questions here? >> Yeah. >> How is code efficiency or complexity factored into training? You read my mind. thank you. Thank you for introducing my next slide for me. So the answer is coding efficiency is for non-agentic tasks was mostly not factored into training. they mostly did not did not consider whether it was faster code or slower code. They mostly just considered whether it was right code. there are a few exceptions to this. one of them is this paper no fun of and the the existing basically the idea here is that you don't just evaluate like functional requirements which are like that it does the thing it's supposed to do correctly. But you also do runtime things like runtime efficiency latency or resource use and maintainability and security based on things like static checks. So there is work to evaluate this other stuff as well. For agentic tasks it's more common to consider efficiency and I I'll talk about that in a bit.

42:15 Cool. Okay. So for singlestep code generation models a typical way to learn the models is through RL over the test results. So this is pretty straightforward from the point of view of RL tasks which is you just sample responses you run the tests over them and then you update the model. So nothing nothing too surprising here. but one really important thing is this allows you to train large language mo train large reasoning models. So as I mentioned solving code forces in you know a single step is quite difficult and so it requires you to really think carefully about the response and reasoning models are very very helpful in doing this and so this is possible possible to do. So like one hint of worrying about efficiency is like if you're doing a code forces thing a lot of competitive coding tasks give you a execution timeout and so like if you go over the execution timeout basically you won't get you won't get any score for it but yeah that that's basically the method that you can use here and you you can show that this reinforcement learning style thing after mid-training can you know further improve your further improve your results. This is just one example from a paper called deep coder that you can look up. But basically you know they trained for a while at 16k context trained for longer at 32k context and eventually reported on like 64k context here.

44:11 Cool. so yeah any any questions? >> Yeah >> is that to provide the single step model context to like code more efficiently for code or is that the reason because it was language model and then >> yeah that's a a great question. So to repeat the question for the comments that are included in code why do I get you know comments in code when it's actually not necessary you know to like solve the task.

44:48 I only have hypothesis here because I I don't I'm not 100% sure about like the models that I use the most. You know what what GPD is doing or what what Claude is doing. so I'd like to go back and and read a little bit more about this, but my guess is that number one, the models are trained on existing code and existing code has comments, so they'll fill them out. Number two, they are trained to do reasoning. And I feel like there's probably some implicit incentive to add more comments in your code to buy more tokens so that you can think about what to do next while you're generating the code. so I think there might be an implicit incentive to get models to do comments. And then I'm certain now now nowadays maybe not like six months ago but now there's also some like non-functional you know classifier or like code quality model that they're incorporating that says write comments write an appropriate number of comments like not too many not too few because one of the big complaints about previous coding models was that they would generate like a million comments that nobody would ever read and be super verbose. So, I remember when the more recent version of Codex was released, one of the things they stressed was we will not give you a whole bunch of comment spam and we'll just do it concisely. So, yeah.

46:22 >> Is there specific training methods for code quality that make it human readable? I mean like I think the the obvious way of doing this is to come up with some model that you know judges code quality and add it to your reward during RL. I don't know of any details of this. Do do you know of any >> I think like when people do post training they'll often like collect readability as one particular judgement that >> so yeah okay >> so to to repeat for the back like when you're doing post training there's often a kind of like RLHF step where you train a reward model that predicts how good the output is and readability is one of the things that you can collect from human annotator ators and then that will be boosted in like indirectly through the reward.

47:16 Cool. okay so I definitely like to cover aentic coding. So this is the big difference between singlestep coding and agentic coding is that we now are doing iterative tool calls like you know all agents and let let's first talk about the most common setting of either fixing bugs or implementing new features in GitHub repositories or in code repositories when you want to do something like this typically you you have three steps.

47:54 the first one is localize. so this is like if we are talking about sorry I think my my running example is now a long time ago but basically we had we had an example of fixing some code so that the return value would be would be appropriate. And so localize essentially here is trying to find the place that needs to be modified.

48:24 then edit is changing the place that needs to be modified. So for example, if we're setting a default only when the value is none. and then verify is checking that you actually did a good job. And then typically you will do this in a loop. So if you if you find that verification fails, you'll go back and iterate until everything passes. So yeah, here here's the the running example. so if you if your job is to find the place where retries is not defaulting appropriately when the value is zero, you will first localize the implementation and reproduce the behavior.

49:13 and so then you edit and so to fix the issues you will then go in and edit the code and then you verify by running all of the tests and if the tests fail then you use the failures to guide the next step that you're performing here. So this is agentic coding. and you basically put this in a loop using the tools that I'll talk about in a bit.

49:45 I'd just like to point out there's one alternative way that you could do repo level coding without any iterative agentic workflow. there was a paper called or there is a paper called agent list which is actually by Steven Sha a like new assistant professor here. But basically what they did is instead of having the agent be able to call call tools to do localization. It basically split it into a localization step where they would first localize they first predict the files that need to be edited then predict the classes and functions that need to be edited and then pick specifically the lines that need to be edited. Then they would generate a patch just using a singlestep language model. So this is an alternative. This was like for a short time like frustratingly effective because I I at this time was building coding agents and we saw quite a few language models where this workflow actually worked better than using a coding agent. And the reason why is because the language models were trained very heavily to be good at like singlestep code resolution and they were not trained very heavily to be good at like agentic tool calling and and verification loops.

51:15 and so yeah anyway this is an alternative way that you could solve repo level code generation but typically now everybody everybody uses agents. So how do you apply agents to this? one answer is you give the agent a single tool. the one tool that you give to an agent is the ability to run bash commands. And so this goes back to, you know, the comment before that being able to use bash commands is really important. And a mini suite agent is something that was created by the people who created SWUB which is essentially an agent that only uses bash commands to to solve problems. And so this allows you to localize. So the way you can localize is using things like gp or rip gp which is kind of a a more expressive version of gp cat and find. if you want to edit files you can use things like said or you can run a python oneoff script you can use patch and then to execute checks you can use piest or build commands. So at least in theory this is all that you need to implement a coding agent. can anyone think of some reasons why this might not be might not be ideal?

52:41 >> Yeah. So, so the answer was there's only a limited number of tools like GDB and other and there's other tools like GDB, but you could just run GDB through the bash commands, right? So, like it just just to clarify, I'm not saying you only provide these tools to the agent. I'm saying you provide only one tool to the agent and it's a bash tool.

53:14 >> Yeah. >> 85 90%. So if you make those explicit just >> Yeah, great great point. So just to reiterate what the point was is these will be most of the tool calls and if these are most of the tool calls if you encapsulate them into a more efficient tool the agent should be able to do a good job here. And so just to give an example if I want to edit files the the agent needs to think every time about whether it uses said or python or patch. whereas if you make a standard file editing tool that's specifically tailored to agents doing a good job at it then it will just mostly rely on that file editing tool and it will get the file edits more correct. And so just to give an example this is what it would look like if you were running with said and so that will you know match the line match the lines and stuff like this. But what if you needed to replace a thing that had lots of backslashes in it or something like that. Then you would be like backslashbackslash all the time.

54:38 You need to get the number of backslashes exactly correct. and so there's just a number of of things that make bash not necessarily the best possible interface for working with files for instance. And so the biggest change that most or maybe even all you know competitive coding agents use is they also add at least one tool for file editing. And so you can either write a whole file or you can search and replace within the file. And typically these are implemented from the point of view of like having a tool that allows you to to perform these operations in a like fixed format that the this tool uses.

55:26 So one one way that has been examined is a unified diff format. So basically you have like the lines and the change that needs to be made. But the problem with this is if you ask the agent to get the lines, it will mess up the lines all the time and like not apply properly. And so instead, the most typical way of doing these things is, you have something that looks a lot like a diff, but it's not, it's not purely a diff or it doesn't specify line numbers. So, this is probably the most common way of doing it. It's done in a lot of, it's done in a lot of, coding agents. Can anyone think of a downside of doing the right one instead of the left one? Any ideas of where this might fall apart?

56:37 Yeah, exactly. So if there's multiple versions of retries equals 3 anywhere in the file, it won't match and then you you won't be able to disambiguate which one should be fixed. So basically what the coding agent needs to do in that case is it needs to write more lines until it becomes unambiguous. But the advantage of doing the thing on the right is much larger than that disadvantage. So almost all coding agents use this. so diff format can make a big difference and file editing tool format can make a big difference. this is an example from something called the the Ader code edit benchmark or code editing benchmark and basically it evaluated GPT4 Turbo which was a model at the time on comparing like search and replace and unified diff format and GPT4 Turbo was just like a lot better at unified diff format because it had seen more of that in its training data.

57:35 nowadays it's a little bit less prominent but it still is a thing. Like until very recently Gemini performed quite poorly on a number of harnesses because Gemini like Gemini was trained on its own edit format that was different than everybody else's and so it was not working well. I think they fixed it in the past like few months or something like this. But this can be a major major major bottleneck to getting agents to work well if it's not done properly.

58:06 Cool. any questions here? Okay. so the next thing is finding relevant code. So you start with searching for you know whatever information you have. So rip gpap is the most you know popular tool for for doing this but you can also search with grap or you know a dedicated code search tool and then you follow the values and the agent goes and fixes things.

58:44 There's also more complex methods that people have examined with respect to code localization. And so just to give one example, this is a paper called LOK agent where LOK agent what it did is it provided the agent with a special tool that allowed it to follow dependency paths. So if you found the issue, you found config.py and then once you found config.py, you could automatically jump with this tool to other related files if you wanted to.

59:20 So this is interesting. what I will say is from my experience it's hard to make this work this sort of approach work better than the simpler stuff. And I'll I'll give some actual examples of that later. So for coding agent evaluation and training the most popular benchmark for evaluating coding agents is bench. Probably many people have heard of this. Mo most people here. Okay, cool. but anyway, like because I assume that not everybody has heard of it. Basically, the way this benchmark was created was it was created from GitHub issues where they take the GitHub issue which is the problem. They take the codebase state at the time the GitHub issue was resolved. So, it's basically what happened before the pull request. And they also find the tests that were associated with the GitHub issue. And so the tests are because they were introduced together with this GitHub issue, these are presumably testing the functionality that needed to be fixed in the GitHub issue. So then based on that they you know have an agent that generates a patch. based on the patch they apply the patch they apply the tests and they generate results and if all of the relevant tests pass then you get a score of one. If all the relevance tests don't pass they they get a score of zero. harness here means actually something different than you know what an agentic harness means. It means like the test harness.

60:59 so it's just running the test. But one important thing to mention here is that running the whole test suite on some of these GitHub repos would be really expensive. So like some of these repos are Django in NumPy and other things like this and running the test harness would be like 5 minutes or 10 minutes for each example that you wanted to evaluate possibly even more. And so what they did here is they came up with a smaller subset of tests that need to pass in order for this issue to be considered resolved. And these fall into some categories. The first one is failed to failed to pass. And so these are things that failed before the PR and pass after the PR. So they're testing the functionality that was actually implemented in the PR. Then there's also pass to pass ones and these are related tests that passed before the the PR was filed and then also need to pass after to make sure you didn't break anything and didn't have any regressions.

62:00 Cool. So to build runnable benchmark tasks essentially what and you'll do more of this in the evaluation evaluation benchmark place assignment but run runnable tasks are basically you need a starting state. So you need the files you might need the dependencies and you need the test command and the issue is the requested behavior and the checks are the the tests.

62:32 So the good news about agents nowadays is actually evaluation is one of the biggest problems in making better agents because if you can generate lots of evaluations you can actually perform reinforcement learning on these evaluation benchmarks and train an agent to do a good job. And so if you can go through this process of generating many many of these outputs, you can actually train a model. And the way you do this is basically you just run the model to run the model to generate an output and calculate the reward if you know the it did a good job or not.

63:19 And one of our one of our first papers on doing this was something called sugeim. So the thing with sui bench is that swbench is only an evaluation data set or they only have configured training environments for the evaluation data set. So for Swedge we basically created additional training environments and trained a model on it and demonstrated that a model that was you know quite poor before with about 11% accuracy with even 500 examples could scale up to being quite a bit better. And I'm not going to talk about this a whole lot yet because we're going to be talking about this in a later class. But we also demonstrated that if you generate lots of outputs and rerank them you can do inference time scaling and do even better.

64:10 so the biggest effort on this which I actually just realized I don't have it in the slides but is worth knowing is something called SU rebench and SU rebench basically what this is is it's constantly creating more SWEBench environments and so they've been they've created them from PRs from December 2024 to June June 30 2026 And I actually just discussed with the author recently and I think they're going to up update this in October or so. so this is just creating more and more of these and they have a training set and a development set. So this is probably the best example of a large scale large scale publicly available coding coding agent data in the format of SWEBench.

65:03 But there's also more lightweight ways to create this data. So one of the problems with this is you need a single pull request for each one of these training examples that you create. And so this is an example from Swissmith which is not relying on pull requests but it's instead relying on artificially injected bugs. And what you do is you prepare an executable baseline and you mutate the code but keep the tests. And so essentially how many people have heard of like mutation testing?

65:48 A few people. Okay. This is this is actually a really interesting idea from kind of like software engineering or software engineering practice which is if you are not sure if your tests are good like if you have unit tests but you're not sure if your unit test suite is very comprehensive. You can go in and make small mutations to the code that should break the code and then you see if the tests pass or not. And if the tests still pass those tests are like not good tests. they they aren't covering like real breakages. So you should augment your test suite. so I think this work was inspired by that idea, but it's not exactly the same in that they mutate the code in the repository and then they try to find tests that fail afterwards. And so what this essentially allows you to do is it allows you to create synthetic buggy data that you can then use to train models. just by just with the existing tests in in codebase.

66:53 so here's a example. So if you have code where all four of the existing tests pass, you then go in and mutate the code into something like return three if not value instead of if value is none. And then one of the tests may fail now. And so this is a good example where you essentially are able to come up with something that says please fix this.

67:25 cool. Any any questions here? So the good thing about this is this is very lightweight and you can generate a nearly infinite amount of data by doing this. The bad thing about this is these are not necessarily like natural training examples that you might want to tackle for real. so you can scale out to different programming languages with this framework as well. So there's something called multisbench and another thing called SWEBench multilingual which basically scale out to you know a number of different programming languages.

68:04 The only logistic difficulty is that the conventions are different between the languages and you need to handle them. the good thing is now coding agents can do this very easily. So it's not that not that hard to change the conventions. a final thing I'd like to talk about is multi-harness training. And so as I mentioned before, if the edit format is very different between what a language model expects and what the harness expects, most most language models will perform worse on the harnesses that they're not trained on. So now it is very common for language model providers to train on multiple harnesses. So they'll not just train on, you know, a single harness, but they'll train on like open hands and open code and codecs and you know, all of these other harnesses. And to give one example of this, you can read the reports of Neotron by Nvidia where they intentionally trained on I think five or more five or more harnesses to provide this to provide a model that works with a lot of different harnesses.

69:19 Cool. we don't have a whole lot of time left. I'm maybe we'll just go quickly through a number of different a number of different tasks other than just like straight up issue resolution. These are less these are less mainstream in research, but they're very important if you want to build a functional model. So, front-end development of one of is one of them. there's a front-end development benchmark called Swebench Multimodal. this is a benchmark that focuses only on fixing bugs in frontend fixing bugs in front-end tasks.

70:03 And so it is not it's not focusing on developing like new frontends. It's more kind of on the issue resolution thing. And so there's two differences in the in creating a multimodal model. The first difference is typically you would think that you would need a a multimodal model to do to like take in the input and understand the input. So you would need a language model that was trained to be able to ingest images.

70:38 the other thing is you need to verify and so you need to there's a couple ways to verify in the browser. One way is to write tests that work in the browser. another way is to look at the the output. so there's kind of two ways that people do this nowadays. one common way of doing this is to take a guey based agent and use it as a component of your coding agent or have a have an agent that can handle both guey based tasks and coding tasks. Next class is going to be about guey agents. So I'm not going to explain at all about how we build guey agents. another way you can do it is by writing a script that tests the guey. And so for instance, Playright is a common guey testing library that a lot of people use to do this sort of thing.

71:30 and so basically, you know, you might if you have an issue where you're able to submit with a length of zero input, you would then be able to verify that visually that this happens. or you would be able to write a a playright script that makes sure that after you add zero it enter an unblink name will appear. so yeah and then su bench multimodal is is a way to evaluate that one very shocking result from a couple months ago was there's a popular model called GLM 5 GLM and GLM 5.2 was a model that came out. It was not a multimodal model, but it got like number one on front-end design arena. regardless of this, so that kind of shook my assumptions about what you need in order to be a good front-end developing code coding agent. But you know, I I still think multimodal behavior is probably pretty important.

72:35 so there's a famous idea of inner loops and outer loops in development. And the inner loop of development is kind of what I've been talking up about up until this point. And it's essentially the iterative process of doing bug fixes, testing, and these sorts of things. Then the outer loop of development is everything that's not this. And there's a lot of stuff involved in software development that's not this. So for example, code review is one example.

73:04 deployment and monitoring is another example. Maintenance and also planning and coming up with requirements are are other examples. And ideally we'd like to have agents that can help us with all of these things. but the hard thing is all of these things are kind of harder to define and also harder to find training data for. so this is an interesting survey from Microsoft in 2019. So this might be different than it is now but in 2019 they did a survey of their software developers and asked how much of your time do you spend coding and the answer was 15%. and this is like the the new issue development and and feature new feature development and issue resolution. They spent 25% of their time on meetings and email 14% debugging. 8% running tests etc etc. So like actually this is a big part of the software development work as well.

74:10 So what we can do is we can compare tasks and so you know if we have repair issue repair I already talked about that but there's also creating things from scratch creating good tests for your code fixing failing deployments or something and developing you know apps over a long period of time. And so a first example is you can train specifically on localization. so you can have models that are are particularly good at identifying you know like where to where to start working on a coding task.

74:54 We have a RL approach to doing this called code scout. this might be interesting to you if you want to do an RL related thing for your project because it's like coding agents but it's a little bit more tractable and easier to do with smaller models. So if that sounds interesting to you, you can check out our code scout paper. a second task is creating apps from scratch. there's something called viench which is evaluating agents for how good they are at kind of vibe coding in general. this is very hard to do because you need to you need to be able to evaluate them properly and so you need to come up with good verifiers.

75:44 Another thing is imp implementing full kind of like backend libraries. there's a few benchmarks for this. One older one called commit zero and one newly popular one called program bench. In program bench essentially what you need to do is you need to re reverse engineer a binary and reimplement it from scratch. And so here what you can do is you can run the binary and probe based on inputs and outputs and then you need to create a new implementation. And this is interesting because it's very long horizon and you know with shorter horizon tasks becoming easier and easier it's now like one of the new challenges.

76:22 Another example if you're interested in benchmarks is the sui milestone benchmark. And the basic idea here is that you don't want to implement a single issue. You want to implement many issues in kind of an epoch. And make sure that as you implement the issues one by one, you're not making a huge mess of the codebase and making it harder to maintain because that's a big problem with code some coding agents. And so this is a benchmark that's meant to evaluate that.

76:53 Another really big problem nowadays is generating tests that allow you to validate whether you did a good job of fixing an issue. Because I feel like now with coding agents, if you can write really good tests to validate that the issue is solved, you've kind of solved half of the battle, right? you've like specified what task needs to be done and then agents are just really good at iterating on them. So, SWTB is an example of a benchmark that that allows you to do this and it's based on the mutation testing that I mentioned before.

77:27 A final one is kind of like more operational like tasks like deployment and I don't think there's a really good there there are really really good benchmarks for this yet and it's kind of harder because you need actual infrastructure to run it but I have a reference for an example that was created by Jet Brains that you can take a look at if you're interested so yeah, we're we're at 50 we're at 50 past.

78:02 I'll I'll maybe just cover this one very very briefly. So there's also some some work by people here at CMU including me and Daniel and for the other one called SU playground and SU playground and hybrid gym which basically are automatically creating coding benchmarks for things other than just coding for other things in software engineering. So, sorry I ran out of time and I'm not going to be able to cover code world models, but I'm happy to, talk about that in our, in office hours or something like that. So, yeah, thank you everyone.