Transcript
0:00 Okay. hi everyone. Welcome to 11768, AI agents. We're very excited to be teaching this new course, which is timely as, a lot of people are starting to use agents. my name is Graham Nubig. I'm an associate professor in the school of computer science here. I've been working on agents for maybe several years now. coding agents, web browsing agents, those sorts of things. and I'll be co-eing together with Daniel who will be kicking off this class. So without further ado, I'll turn it over to him and he can start this and I'll I'll do the second part of the class.
0:40 >> Thanks Graeme. How's this? Cool. Yeah. So hi everyone. Really excited that you're here. My name is Daniel Freed. my research group also works on agents. so grounded agents and also interaction between people and agents and these days interaction between agent systems. So excited to talk about all these topics with you and yeah we have a really great core staff too who we will introduce in in a few slides. But yeah so agents are are really timely right now and the capabilities of agents are advancing pretty quickly. So probably a lot of you are using coding agents in your you know day-to-day work and we've seen some big success cases of agents. So one example is there was this nice experiment by Carlini at anthropic that saw could we develop a really complex piece of software using a multi- aent configuration of a a recent coding model. And this was pretty successful.
1:50 So 16 agents collaborated together over two weeks and constructed a rustbased C compiler which was able to compile the Linux kernel. And you've probably you know been using agents effectively in your own work and and seen how much better they've gotten especially in the last like year or so. There's limitations of current agents too. So how many folks have tried using an agent that controls your computer like controls applications or Okay. So, a lot of folks have How many folks have used OpenClaw?
2:22 Yeah. So, we have a number of folks, too. there was this pretty high-profile incident involving OpenClaw which got posted on Twitter by the the person who ran into this where she was asking the model to organize her inbox and the model kind of went out of control and started deleting old emails. And so she's trying to intervene. She says, "What's going on? Can you describe what you're doing?" But the model says, "I'm taking the nuclear option. I'm going to trash everything in the inbox that isn't already in my keep list." And the user is trying to intervene, saying, you know, do not do that. but the model goes on to delete a lot of this really valuable information. And at the end, it acknowledges that, you know, I learned my lesson. I'm not going to do this in the future. And I do remember that she told me this, but I violated it. So, what happened behind the scenes here? Well, it seems that it was the model compacted its past context, which is a way that we'll cover in this class of of reducing the past history so that the model is able to take in more interactions with the environment, but that caused it to lose this instruction to not delete emails.
3:33 So, models are getting much more capable, but they also still have really rough edges. And a lot of this class is going to be about how do you develop these capabilities and models starting from base language models but also what are the research problems and what are the remaining gaps that we need to fill. So let's just try to get a sense of what people think agents are are good at these days. So we have like six different tasks that you might want to try to use an agent for. And these are pretty subjective. I just want to get a sense of like what might you be comfortable with. so let's do a show of hands for a task like say you have an online store like some code application that hosts a store for you and it started failing. You want to use an agent to try to diagnose that. How many people would be comfortable with having an agent carry this task out like fully on its own? Can we get a show of hands? Okay. Yeah. how about having the agent like ask first as it's carrying out this task? Maybe asking you for help. How many people think that might be necessary?
4:37 Okay. And how many folks think that this would just not be something that you would ever want an agent to touch at all? Okay. So, for this one, I think we had more autonomous. How about drafting and sending a product launch email to 50,000 customers? So, how many people are okay with this being fully autonomous? Okay, we got some brave souls. how about asking first? Okay. And how about an agent shouldn't do this at all. I want to be responsible for it. Okay, so I think this one was was asked first. How about collecting your tax forms and preparing and filing your 2025 tax return? Fully autonomous.
5:19 Okay. Yeah. How about ask first? And how about never? Okay. Yeah. So I think this had the most never of anybody but ask first probably one migrating a payments API from Python to Rust. Say maybe you have some test cases that were written for the application in Python that you can apply to the Rust version. How many people would be okay with this being autonomous? Okay, how about ask first?
5:50 Yeah, that was very close. I don't know which one was more and never. Okay. So, we'll let's have this be one for the agents. Autonomous. Buy concert tickets to my favorite band if they set up a show in an area in my area. Autonomous. So, you'd be okay with the agent buying the tickets, too, if it finds good ones. I think that one probably won. And how about this last one? Adjusting an insulin dose after a week of glucose readings. So, an agent like, you know, has all the like the capabilities to do this. Like if you're feeding it your health information, it could monitor that. If you give it access to like something that can change a prescribed dosage, it could do that too. How many people would be comfortable with this autonomously?
6:38 Okay, asking first. And how about never? Okay, so yeah, so that one was never, although I was thinking maybe it would be ask first. So cool. So I think you're all you all kind of have a sense of some of the boundaries that agents have. Although you know there's a lot of these tasks where we might just not be sure. but you also have a sense that like trust is important and later on in the class when we talk about like sandboxing and security and interaction and oversight by people we'll get into these a little bit more.
7:15 Okay. So just to give like a very brief kind of tour of how agents have extremely brief. It's just two slides. how agents have progressed over the past few years. and also to motivate some of the things that we'll see later on in the course. so we've been doing some work on guey agents, like agents that can use computer applications here at CMU over the past few years. This is work from a lot of faculty. and here's a demo that was made by one of our students, JY Co, two years ago showing an agent in action controlling the browser.
7:50 And so you give it a task like navigate to the page of a good Thai restaurant in Pittsburgh. It should have at least 200 reviews and 4.3 stars. And on the right here you can see, sorry it's a little fuzzy, but you can see the interactions with the language model like the chains of thought that are being produced. And on the left here, the model is controlling the browser typing in the search box on Yelp and then navigating to this page. Fusidities is a really good Thai restaurant. and so that's a success and we've seen you know frameworks and agents like this really taking off over the past few years with Manis and OpenAI's operator cloud computer use agent and so on.
8:29 And we've also done a lot of work here at CMU on building coding agents. Graham's group in particular has done a lot on this and also through open hands which develops open- source agents. and here's a demo video from Graham which I'll play showing the agent like both writing code but then also interacting with the application that it's built in the browser to test it out. >> To-do list app in list.
9:01 >> So this will take a little while for the agent to work. we can watch it work over here and see that it has made a directory for static files and templates and it has created the main app file. We can look at the changes it made to the app file by clicking over here in the changes tab which shows us all of the things it implemented. So that's kind of nice feature. So you can kind of watch what the agent is doing while it's working. and it's editing the index file, the style file to make the app look nice.
9:34 requirements.ext to make it easy to install and adding a read me. So you can see it's kind of following good coding practices. And now it will start up the app. And so it's running very very quick. And what I can say is let's run in the background and then test out with your browser. And so if I do this, it will run the app in the background.
10:09 And it checked if the app is running. And now it should navigate there with its browser. it's checking the log and saw the port was already in use and it's trying it out. So that that's kind of the nice thing of agents. They can debug their own problems and and make sure that things work. So, okay, it should be working now.
10:41 And you see it browsed to the app. And we can see our tuning list app over here. And it sees that it's running, but now it wants to test the functionality. So, what we can do is we can see that it went through and it filled in buy groceries and it clicked. And you can see that in real time it's testing out your app and then it also is testing the delete functionality and other stuff like this. So I find this really exciting. I'm not, you know, the best front-end developer, but it's able to kind of try out the front end, see if the functionality is working and fix any issues on the fly. And I even had one time where the app crashed and the agent realized that the app crashed even before I did and it was able to go in and fix the problems.
11:26 So this is as simple as just saying, you know, go and develop an app and then test out if it works and you don't really need to do anything else. So now we're done. let me stop this. In a real scenario, what I would do next is I'd ask it to push it to GitHub and you know finish it up so that I can continue working on it. >> Yeah. So we're excited to give you hands-on experience building agents in this class. And in one of the lectures later on in the course, we'll cover open hands and its SDK and how you can develop agents of your own like this.
12:03 So we have a really great core staff. We have our our core staff all does research in these areas. wrote some of the foundational papers and we'd love to have you all come up and introduce yourselves and at the end of the course you'll be getting to work on a mini research project of your own and we'll be getting guidance from the entire core staff helping you out with that. So yeah, if we can just go through and have each of you maybe just project and introduce yourselves.
12:39 >> Yeah. So the yeah say it so the people in the back can hear like what what your research is on because then the people here can like essentially know what you're working on so they know which TA to talk to when you're working on like a project in that area for instance. >> Yeah. >> Hey everyone I'm a third year PhD student. I'm advised by Grant and Kamar. my research areas are focused on training agents to better leverage interest and compute and also organized in multi- aent systems.
13:11 >> Hi everyone. my name is Adita. I'm a first year Ph student working in the professor program. my main research interests are like designing reward functions for like agent reinforcement learning. I particularly work on code generation for for now. hi everyone my name is and I'm a second year PhD student working with professor Anna Yeah and my current research focus is learning and research >> hi everyone I'm here working with the previous of working on mobility agents and recently I've been working on Hey I'm Sus. I'm a PhD student working with Daniel. I work on how we can train agents to communicate more efficiently with people.
14:08 >> Hi, I'm Lily. I'm a Syria business student by EMU. So I'm work on training on agents on authorizing coffee and battery. >> hey everyone, I'm Andy. I'm a fourth year PhD. I work with Daniel and Monop. I mainly work on alignment post training and also multi-agent introductions. >> Thanks everyone. >> Yeah. So like Graham said, we definitely encourage you as you're starting to think about your projects later on in the course, we'll talk about the course structure towards the end, but as you're starting to think about the projects that you want to do, TAS are going to be a really great resource for you to go to in office hours and get their expertise.
14:52 So, we're also really grateful to several really awesome startups for donating compute which you will be able to use in the assignments and projects for the course. and without their support we wouldn't be able to run this at the scale that we would like to. So, we're really grateful to fireworks AI to modal to prime intellect and to sale for helping to support this semester. Okay. So today we'll give a brief overview of what an agent is and then we'll talk about the plan for the class and how things will be structured.
15:33 So an agent agents have been around for a long time. The concept of an agent and work developing agentive capabilities isn't just you know a two or three year old thing. So, Russell and Norvig in their foundational textbook on AI, define an agent as anything that can be viewed as, perceiving its environment and acting upon that environment. And that's really true for the agents that we have now. And a lot of the techniques that were developed for agents in the past, not all, but a lot of them will apply to current agents too, especially things like search reinforcement learning. So, how does this pan out for our current agents? So, agents are going to be situated in an environment, which could be something like a code repository, a website, some other application on your computer, and they'll be interacting with that environment by getting observations of the current state that they're in in the environment. So that's something like messages from the user they're interacting with, the contents of a file that they're investigating, the current web page that they're on, screenshots, or as we'll focus on in a few slides, the results of tools which are you like programmatic functions that they can execute in the environment to interact with it.
16:58 And they're taking actions at each point in time. which is going to these actions will update the environment and then change the state that the agent is in. So that could be something like replying to the user or editing a file or running a command in the shell or calling an API that backs the web pages that they're interacting with. And finally, there's also this notion of reward. So how do we define whether the agent was successful or not using a numeric score? And we could define this as like if there are some test cases for a codebase, do those pass? If they do, then you get a one, otherwise you get a zero. for other tasks, it can be difficult to like have sort of a programmatic reward. So we might need to rely on an LLM as a judge to evaluate whether this was successful or not. In your second assignment, you'll be developing LLM as a judge based evaluation approaches among other eval.
17:56 And maybe we're ultimately interested in did we, you know, make the user happy with the task that they asked the agent to do. So we could also use feedback from the user for that. And I think in our human interaction lecture that Valerie Chen is going to be giving a guest lecture later on, we'll talk about this. So we asked all of you to have prior experience training language models and given that experience you should all be familiar with language models doing next token prediction right so at each step the agent is going to predict a distribution over the next token we'll sample or in some other way choose one of those tokens from that distribution insert it into the context window of the model and repeat this over over and over again. And recently we've seen a lot of a lot of improvements on complex reasoning tasks by introducing chain of thought to these models. So as as you should remember a chain of thought is predicting a sequence of tokens which aren't the final answer that we want the model to output but do allow it to sort of have a scratch pad for its intermediate reasoning steps which it can then use and condition on to predict the final answer. And so this is like a non-aggentive setting, right?
19:20 This is just the agent interacting with prompts that you give to it. So how do we go from this to having models that can take actions in the world? So the main way that we do this is through tools. So a tool is something that's situated in the environment which will provide an interface for the model to interact with that environment. And you could think of this as being like an API for example like if we want the model to be able to find out what the weather is so that it could respond to a user asking about that maybe we have an API for like you know weather.gov which is wrapped in an interface that allows the model to call it. So there's a bunch of different ways to represent this interface. Graham will be covering them in more detail I think in the next lecture but here's one example. This is the open AI tool specification where for a given tool aka function that the model can call.
20:20 You'll have a name for that tool a natural language description of what it does. So for example, read file might be a tool that a coding agent uses to look at files in the repo that it's acting in. We'll also need to tell the model what it can pass to this function. So we'll have a description of the parameters that it takes like maybe this function just takes a path which is a string you know for the file to open. So we have to so we have the name and description we have some structured representation of the interface to the function which here is JSON and this also needs to get fed into the model in some way. So we'll have this you know underlying specification of the tool and then the model will view it by applying this template which turns it into a sequence of text which will then be tokenized and read by the OM which might look like this and the model will have to be sort of either trained or with few examples know how to use these tools.
21:28 So this is how the model becomes aware of the tools that it can use to interact with the environment. But the model will also be needing to use those tools to interact and it does this through tool calls. So for example generating text like this like tool call and then you specify the name of the function that you want to call and the arguments as well if you're using sort of a JSONbased representation. Can anybody make a suggestion for another way that you might have the model call tools?
22:04 Anyone? Let's Yeah, in the back. >> Yeah, you could just write code directly. for example, bash scripts or if you represented this as a Python function, the model could just, you know, call the function using Python syntax. And we'll see in a couple lectures that that's actually a very effective and often more effective way than generating JSON. but there's some trade-offs, too. So, when the model calls the tool, it'll get back results from executing that tool in the environment. So for example, if we're reading this test file, maybe that test file has this code for an add function in it and the environment will return you know the results of that tool in this content field and then that'll be rendered to the model using a template which the model will then read in as tokens and get to continue producing output. Any questions about this?
23:07 Cool. So now that we kind of have like the basic tool interface, we're able to have the language model use these tools to act in the environment. So the way it does this is just by reading in these token sequences that represent what tools are available that represent the results of tools and then generating token sequences naming the tools which then invokes them. And so this we have this distinction between the language model itself which is just you know interacting with tokens later on we'll talk about multimodal language models which maybe also take in images or other modalities. so the language models just tokens in and tokens out maybe multimodal tokens but then there's this harness which is responsible for actually coordinating these tool calls using the language model as an engine for it.
24:03 And so the agentive loop basically and tool former was a really impactful paper that showed that you can train models to be able to use tool calls and then condition on the results of those to make future text more likely and showed an early way that you could train these models to use tools that generalized prior LLM training procedures. So once you have this basic setup that you then can implement an agent by running a loop and one of the influential papers here was called react from 2023 which stands for reasoning and acting because the model's producing chains of thought and then taking actions afterwards and the way it works is you'll have some context which you know includes maybe a definition of the general task of solving pull requests on GitHub or solve issues on GitHub. We'll see some examples of this in a bit. Then you have a task which is maybe an initial query from the user. you'll also have a list of all the tools that are available for the agent to call which will be represented using one of those formats that we showed. And you'll also have a history of the past observations and actions that the agent has produced and observed as it's been interacting.
25:19 And in each at each time step so for a given particular state the agent's in it's going to condition on all of that produce a reasoning chain you know chain of thought with a scratch pad about what it should do and then produce one or more tool calls that will be used to interact with the environment or with the user. So for example reading the file. So that tool call will get executed in the environment. We get the results. We add those results to the history and the environment is also getting updated as a result of our tool call and then we repeat the process again. So the model's conditioning on this updated history and you know the observations that it gets from this time step and then issues a new tool call and it repeats over and over. We could also have tool calls that end the task that for example send a message to the user or otherwise signal completion. and so this is a way that we can you know signal that we are finished. And in the planning lecture we'll talk a little bit about like how might a model determine that it's done.
26:25 So I won't get into the details of this too much. you'll get into it when you do assignment one which asks you to implement a react style loop to implement a small coding agent and use it to solve problems in a repo a a chess engine repo actually. So there's some fun demos that go with it too. so we won't get too much into the details of this and the code that you will write will look different than this but you know at a high level this is a very simple implementation. This is an excerpt from this nice minimal repo called mini sui Asian which actually gets very high performance on the bench benchmark which we'll talk about in a few lectures. It's a very standard benchmark for coding agents. but it's a really nice repo that I'd encourage you to take a look at because it's effective with recent models, but it's also pretty simple and really kind of distills you know the the distills the basics down. but you'll start off by having some messages which give the model context about it being an agent and about the task. And then you'll just run this loop over and over again that implements what we had on the past slide where the agent will issue a query to a language model using all of the past u messages which contain the history and then it'll add that message to the history and execute the execute the actions in the environment. And we've left this off, but you can take a look at the code repo if you're interested.
28:08 Execute the actions, get the observations, and return them. So, I think it's also instructive to take a look at example agent trajectories. And one nice thing about the Swebench data set or Swebench evaluation task and leaderboard is that they have some example trajectories here. so, let's take a look at maybe GPTO OSS.
28:38 And this can give you a feel for what tool calls can look like and what the model is actually seeing as it's carrying out this task. So let's see if I can zoom in a bit. Yeah. So the model gets some context in a system message which says you are a helpful assistant that can interact with a computer shell. So this is like very generic but then we'll also have a user message which gives some context about a particular task solving this particular poll request.
29:19 and so the model conditions on all of that and then it needs to issue a first action and you can see here the chain of thought trace that it generates. So this blue is generated by the model. So you have the chain of thought here and it produces a tool call. This agent is using the the tool format that you suggested which is just issuing bash commands by you know writing the command that should be run that gets handed off to the environment. The environment executes it and here the result is just an exit code exit code zero showing it succeeded but then we go on to the next turn and the agent you know issues a new thought and a new bash command and so on.
30:02 So hopefully that gives you a sense of you know one possible way of instantiating tools and instantiating prompts and you'll experiment with this in your first assignment. Cool. So now I'll hand over to Graham. Thanks a lot. Okay. Hi.
30:36 I guess everyone can hear me. So next I'd like to talk about agent capabilities. And what I mean by this is like what makes a good agent. because as Daniel pointed out it's not very hard to make an agent. especially nowadays that there are libraries that you can call to you know do inference with a language model or something like this. The hard part is making one that actually works. so I'd like to think a little bit about the capabilities that you need to have in an agent. So a fairly large number of people said they use agents in some way you know every day like openclaw or coding agents. what are the things that frustrate you when you use an agent? like what what has gone wrong that was a problem? We saw an example of deleting all your email files. That would probably frustrate you, but any any other things?
31:29 >> They're too slow. >> Okay. Yeah, that that's a good one. in the backy wordy and verbose. That's a good one. I saw a hand. Yeah. >> so they forget things. Yeah. Okay. >> They might misunderstand what you said and make mistakes due to that. Yeah.
32:03 >> Do doing what was it? >> Like accessing things they shouldn't be accessing. Yeah. Okay. That that's a good one. Yeah. random knowledge gaps for very common or normal things. Yeah, that that's a good one. Yeah. >> Yeah. Writing a thousand lines of code when two would do. Yes, that that's a good one. That's kind of a different variety of verbose, but it's also Yeah. verbose. Yeah.
32:37 >> Yeah. Forgetting previous session interactions. So, yeah. >> Not knowing how to push back when you say something, unreasonable. Yeah. >> They may lie to you. Yes, that's a bad one. >> A lot of implicit assumptions. Yeah. So, these are all great points. all things that frustrate me, too. So they're they don't exactly align with what I had on the slide but you know that's a a great a great thing. So a first capability that you need to have in agents is accurate tool calling.
33:18 This is something that actually nobody mentioned because a lot of people take it for granted nowadays but this is not something you can take for granted if you're starting this class and you're in charge of training the model. so you know it's kind of bread and butter. If you can't call tools accurately you're going to fail at any task you do. But it's something that doesn't come for free. Another thing is coherence over long context. And so somebody said it annoys me when they forget things or it annoys me when they forget the previous sessions or something like this. So that that's one example of it. Another example of it is what the example that Daniel gave where previously the model was told to not do something but then it ended up doing it anyway. So that that's another example.
34:09 Another thing is customizability. And so you want the agent to do things your way. everybody has different ways of doing things different requirements. So our class might have a different requirement than another class. And so you need to make sure that if you're using an agent it's going to follow those requirements. complex task management. So breaking down tasks being able to handle very long horizon things is a very big very big issue.
34:40 environment understanding and what I mean by this is there were like one good example of this is making a thousand line change in a codebase where two lines would do that's an implicit failure of environment understanding to know that you know this is actually a really simple change but it didn't realize that and it goes and does a bunch of other things instead and then a final thing is safety and a lot of people separate safety and capabilities they say I'm a safety researcher or I'm a capabilities researcher. But from my point of view, capability safety is a capability. It's something you need to bake into the model. And if it's not baked into the model, you won't feel like you won't feel comfortable using the model because it might go off and do something that you don't want it to do. And so from my point of view, it's a a failure of the model or the agent more broadly.
35:35 And so there's two ways to build these capabilities. And this is something that's really really important when you're working on agents. the first one is training in LLM. The second one is engineering the harness so that you put structure around the LLM to get it to you know solve like solve the problems you want in the way that you want to. So question which one do you think is more important or effective here? So does anyone have an a strong opinion that LLM training is the way to to get your agents to work well?
36:20 Okay, actually surprisingly large small number of people. What about train engineering the harness around the agents? Okay, that that's a pretty large number of people. I saw Daniel raised his hand twice, so he's not allowed to do that, but he's the instructor. So, of course, both of these are important. my my personal opinion or well, first I'll I'll you know talk a little bit about what they entail. So, the first one is changing the model's behavior through pre-training supervised fine-tuning or reinforcement learning. and then you can teach reusable patterns for reasoning, tool use and recovery.
37:03 The important thing is the capabilities become part of the learn model or the learned policy. And so in harness engineering, you change the system around the model. You give it prompts, tools, memories and control flow provide context, validation, retries and safety boundaries. And the capabilities emerge not just from the model, but they emerge from the combination of the harness and the model. So my my answer to this is they're they're maybe both important but typically what happens is you identify a problem and you solve it here first.
37:40 Then the people who are training the models catch up and realize that this is a big problem. It's a big enough problem that it warrants them training the model in a particular way to solve the problem. it gets solved and you don't need to solve it in the harness side anymore. So that this is a typical flow. so from my point of view this is kind of very often the more fundamental solution on the left side but it takes a lot of time and so you eventually you need to start out by solving it on the right side because you don't have the time when you're improving your models. So I I personally favor LLM training as a fundamental solution to problems if you're in a situation where you're able to do that. Yeah.
38:35 >> So, this is a great question. So, what are the capabilities that will survive LLM training? one I can definitively say is long context in like maintaining all of your your memories in context because there that is not a that's not a model accuracy problem only. It's also a model efficiency problem. So you need to be able to model very long sequences. And sure you might be able to come up with a model that doesn't have like n squ complexity like the transformer does but you know that's you know that's kind of a bigger a bigger discussion assuming we're staying in our our current modeling paradigm I'd say most of the other stuff I have on here most of the other stuff I have on here.
39:36 Accurate tool calling, coherence within your context window, complex task management, environment understanding, mostly from a capabilities perspective can be and maybe even safety mostly from a capabilities perspective could be theoretically solved through the model, but they're not solved yet. and so we still need we still need harness engineering. Customizability is a very interesting one because you could train the model on the fly. but there are also cases where you kind of want to give it like a script or you want to give it separate instructions based on the situation that you're in and they're like maybe training it would not be the best solution. But we can talk a lot more about you know all of these details later. Yeah.
40:28 >> Yeah. So a follow-up question on when inference latency is very important is engineering the harness to fit the problem a good solution. So what what I want to point out is all of these capabilities can be handled either through training or through the harness. and so I I wouldn't say like one is necessarily always better than the other. They're both both solutions. The interesting thing about like inference time efficiency is you need often need to use a weaker model or you need to use less compute and because that's the case you often need to put more guard rails around it because you're going to have less su success or you need to do like adaptation to the particular task you're interested in. So the the smaller the model you're using the more careful you need to be because the very big models can often generalize better. they're not perfect but they can generalize better.
41:22 Cool. any others? Okay. So moving on to onto this. So one of the big goals of this class is to get people to be able get all of you to be able to train a model for agentic tasks. And this is a big point in that not that many people can do this well. And so we'd like the people in this room to be not that many people, right? We we'd like you to be good at at something that that's hard to do, but hard to do, right? But, you know, like it's a good thing to know how to do if you do know how to do it. we're not going to be handling pre-training because pre-training you know, it's on the scale of the entire internet and that's unfortunately not something our our modal credits will support.
42:19 but we will be thinking about mid-training and supervised fine-tuning and reinforcement learning. And the points here are mid-training and supervised fine-tuning is where you already have example demonstrations of how an agent did a task and you're training on those example demonstrations usually by optimizing maximum likelihood. Hopefully everybody in this class has done that before because it's a prerequisite for the class. So we'll only be covering that briefly and be focusing more on how you create the data for doing this in indogentic setting. and then we're going to focus more on reinforcement learning because reinforcement learning for like agents is is pretty tricky. So how many people have done reinforcement learning with language models before? I guess maybe not that many people. Yeah.
43:08 Okay. How many have done it for reasoning reasoning tasks? How many have done it for agents? Okay, maybe about half and half and it's only like maybe 10 10% 10% and 80% have not done it. So yeah, we're going to try to put a lot of effort into making sure that everybody is able to do this by the end of the class. Cool. so looking at all the capabilities one by one. accurate tool calling is an important capability to have. we do it through harness engineering and LLM training.
43:42 some of the things we're going to talk about in class are grammar constraint decoding, which allows you to make sure that your tool calls are wellformed and match the specification that you have. for LLM training, this is first done through SFT or through through mid training or or whatever you want to call it. and this is done by training on tool calling tool calling data tool calling traces. for coherence over long context this again can be done through harness engineering or or LLM training. For harness engineering, we do this through context compression or compaction which are kind of synonyms where you take all of the context and you summarize it down for the agent to continue working. It also can be done through dynamic memory lookup where you have memory over all the context that you want to be handling but you don't pull it all in immediately you pull it in you know on demand. and it can also be done through sub aent delegation. So you delegate some parts of a very long task to an agent. It keeps that context in memory, but then it drops it out of memory when it's done performing that.
44:58 for LLM training, you can do that through long context training. We're not going to handle it a lot, but we'll talk about it maybe a little bit. then for customizability, this is actually one of the best use cases for harness engineering right now, I think. And there's a bunch of methods that you can do this. one is through agent memory. So making sure that the agent continues to learn as you interact with it more through skills which are kind of prompts and sometimes scripts about the sort of thing that you want to do that you pull in at each time peri time time point when you want to use those things. also possibly custom tools for a particular task.
45:44 for LM training this is a nent area. I don't think there's a lot of people who are doing this in in a lot of detail. but there are methods where you can learn from user feedback. And so like if a user is is using a tool and they give a thumbs up or a thumbs down, you can learn specifically from them to adapt to them. Complex task management. ways you can do this are through harness engineering. So you can provide planning or decomposition tools. You can have a plan mode for the model where it it gets to plan. I this is a little thing that I I learned but there was a popular coding harness I think it was codeex no sorry maybe it was clog code where they had a plan mode and the only thing when people pressed the plan mode button was that it added an extra thing to the prompt that said please plan do not do anything.
46:40 but everybody wanted a plan mode. They wanted a button. So, they made sure that it was done entirely through prompts. But there's also more more complex ways to do this as well. again, you can do this through sub aent delegation by like breaking down the task and delegating it to other agents. And a very good way to do this in training is to train on complex long horizon tasks. So for environment understanding, what I mean by this is whatever data format or environment that your agent is interacting with, it needs to be able to understand it. And just to give one very good example of this in computer use agents, you need to be able to understand web pages and or you know guey interfaces or something like this.
47:29 This is not something you get for free. In fact, like agents are actually pretty bad at it right now. even the strongest agents are maybe not, but like many of the open source models don't even support multimodal data and like the ones that do are not perfect at understanding it. They make lots of mistakes, many more than when they're understanding text. So this is a failure of understanding the environment because they need to understand the multi- modal data in their environment.
47:58 But now let's put them in a different environment where they need to trade stocks or something like that. they're not very good at understanding time series. They make a lot of mistakes on time series. Or you put them in an environment where they need to understand a picture of a, you know, a petri dish with lots of bacteria in it or something like that. They fail to do that as well. So you kind of need to be able to understand whatever environment you want the agents to you know, interact with. So there's ways you can do this through hardness engineering by like giving skills that correspond to domain knowledge but you can also train on data with the expected observation shape or train in domain specific environments.
48:41 So right now actually like there's kind of this thing that people say which is like oh the models are going to get better and then we won't need to worry about that anymore. the thing is models don't get better. people make models better. And so one of the things we'd like to teach in the class is like what's actually going on under the hood when suddenly Claude goes from 4.7 to 4.8 and it can suddenly like compose music better. and the there's a few things that go into this, but one of the really big ones is they make an environment to train the model on a like compose music environment and suddenly you know they they train on it for you know they add it to the training mix and then the model gets better at composing music or something. So a lot of the a lot of the work here is creating domain specific environments that models can work in.
49:38 then the final thing is safety. And we're going to talk about it for like a bit because if you don't have a safe model, you're not going to be able to use it in real consequential settings. and there's a lot of things with respect to harness engineering that you can do here. You can give it a sandbox. you can limit access to cred credentials. You can also monitor the agent as it works. and you can also do safetyaware reinforcement learning.
50:06 I I think probably a fair number of people heard about this, but who heard about the OpenAI incident where it hacked into hugging face? Okay. Well, so I think most people heard about it, but recently there was a there was an example where the newest open AAI model was in an agentic harness and it was working on a cyber security benchmark and so its instruction was hack into this system and so it was going to hack into a system and that was its task and it wasn't able to hack into the system. So instead it basically hacked into the hugging face website and and got the answers from the hugging face website and and did it there. And so there were a bunch of failures that h that happened here. the first failure was a failure in sandboxing because they didn't properly contain the agent when it was doing this benchmark.
51:02 they they did have limited access to credentials and so it worked around that problem by by hacking into something it didn't have credentials for. and they also didn't have sufficient monitoring so they weren't able to tell that this happened. they didn't have sufficient safety guardrails on this model because this model was being trained to test you know like whether it could hack into systems. So presumably the models that they eventually released to the public did have better guardrails of this model.
51:33 But like these are all ways that you can handle safety and we'll be talking about this sort of thing as well. So the final thing and any questions about the capabilities part. We're going to go into a lot more detail of course. So okay, sounds good. so another thing that I we really want to reinforce here is agents are systems. they're not just models and they are far more complex than most of the things you've dealt with in any machine learning class that you've taken any you know class here that you've taken before. So you're going to have to be able to handle a bunch of different things. The first thing is a harness.
52:19 which is a moderate to complex piece of software. and it has a lot of moving parts like you need to deal with the context, you need to deal with the the tools, the guardrails, manage workflows, stuff like that. you need to have a sandbox. So, you need to make sure the agent is working within a contained environment. And, there are different ways you can do that. it needs to work with a model and you need to do inference. But if you've done inference with language models before, you might have dealt with contexts of like 16 tokens or 32 16k tokens or 32k tokens. Here you need like 256 at least they have kind of like a state-of-the-art coding agent for instance. And so the inference problems are harder. You need to deal with caching your inference other things like that. and you need monitoring and training.
53:18 So from the point of view of this what we're going to teach in the class is harness engineering. So how to manage state tools memory control flow how to validate actions, handle errors, enforce permission and safety boundaries. and we're in for each of these systems, we're also going to try to give examples of software that you can look at. for harnesses, there's kind of two big varieties of harness that you use for coding agents. some popular ones right now are cloud code, codecs, open hands, open code, and pi.
53:54 probably you know you're you have used or have heard of at least one of these. I develop open hands so I know it very well. and so I'll be talking about talking about this. I'd also like to give a little bit of my experience and some of the lessons we learned in in developing it. Separately from this there are orchestrators. I I wrote lang chain but maybe I should have written lang graph instead in crew AI and kind of the they have a very different philosophy between these. the the coding agents are are generally simpler in terms of how different intera agents interact with each other but they're more complex in the action space that you give to the agent. So the agent can write code, it can interact with a website, it can do other stuff like this. orchestrators are less complex with respect to the individual agent that they have. So the agent might just be able to answer customer service requests or look something up in a database, but it couldn't write an arbitrary Python program. But on the other hand, you get like these declarative workflows of what the agent workflow is able to do. And so that's like a a different way of building things. It's more guardrail but less expressive. And they both have their place in different settings.
55:12 For sandboxing to prevent your agents from sharing your API keys without your permission or hacking into other websites. we need to isolate code and tool execution and limit the compute network and file system access that agents have access to. another big thing is for evaluation and training. It's a very good way to create reproducible environments. so like for SWEBench for instance each problem has a each problem has a sandbox and you start in the sandbox and that's the state of the environment before the agent starts working and so you can run these locally. some of the technologies that we use for this are Docker and Appainer. or you can run them on the cloud and two of our kind of compute partners for this class Modal and sale are like cloud in providers and actually prime intellect for that matter.
56:13 So the next thing is language model inference. And so I I think you know given your previous experience in an in an NLP or like large language model class presumably most people have encountered this before but basically they are required to serve model gener generations batch requests. So if you have like multiple agents hitting your language model at once, it's more efficient but also you need to batch the requests together.
56:43 An extremely extremely important part of agents is caching previous requests because agents take more and more actions and so the more and more actions they take the more kind of context they build up and you need to make sure that you reuse that to do it properly and you know other systems considerations. some example software that we're going to talk about here is things like VLM and SGLANG. These are the two most popular ones. And then there is also tons and tons of inference providers. if people have not seen open router before, open router is kind of a a general thing that gathers together all the inference providers.
57:21 And for any model, you'll see like 20 or 30 different providers serving the models. So there's lots and lots of them. one of our compute sponsors fireworks is an example of that and sale training systems. So here what they do is basically they need to update the models weights and so they prepare data, collect rollouts they coordinate a whole bunch of workers who are working together to to train the models and they can checkpoint evaluate and reproduce runs. some example pieces of software here are Sky RL and Miles. and so the these can be used for for training.
58:04 And finally observability and monitoring. so this is to understand how your agents are working. So these capture the traces or trajectories of the agent, what the agent did and they gather metrics around them such as like how much time it takes, how much cost it was track the quality, the cost and the failures. and they allow you to compare trajectories and evaluations. And so I'm I'm pretty familiar with one called Laminer because I use it a lot. MLFlow is another example. I I just realized that Daniel showed one called transluce which is not on my slide here but that's another another popular one.
58:45 So by the end of the class what I hope everybody in the class would be able to do is implement an agent from scratch on top of an open source LLM. So this is implement a harness yourself design evaluations for multi-step tasks. And so basically we'll give you a task that you need to evaluate and you need to create an evaluation for it. This is super super important even if you're not interested in evaluation but you're also interested in training because one of the best ways to do training is to create an evaluation and scale it so that you can do reinforcement learning based training. So very important skill to have train agents to improve their capabilities. So we actually run the RL loop, handle all the systems problems and be able to to solve this reason about safety and reliability trade-offs.
59:36 So we'll we'll be talking about safety and finally we will have a project which I'll I'll talk about on the next slide. So these are the assignments. the first assignments are basically going to be implementation assignments. So you'll be given something to do and be asked to do it. And the first one is create a harness. the second one is evaluation as I measured as I I read measure as I mentioned. and then training where we train with RL. And so these are approximately the first half of the course. and then the second half of the course will be to do a project using these skills to you know do something kind of interesting or novel in whatever area you choose. And the project will be group based. So we're going to be asking you to form groups of two or three and and create a project. And the general schedule is we're going to talk about agent capabilities first.
60:39 we're then going to jump into some specific domains like coding and guey agents. do training and including SFT and RL. we're going to be talking about frameworks and safety. And then we're going to be talking about interaction and then we have some project hours where everybody discusses their projects. And then on the end of the class we're going to try to get some domain experts from each of the various domains to give a guest lecture which maybe combines a little bit of overview but also their research. and so hopefully you'll be able to learn from some pretty exciting people. And I Daniel and I tried to brainstorm the best people in the in the world to talk about their things and I don't know if anybody said no.
61:29 so I think everybody said yes. So we we got our first choice for every every topic. So I'm pretty excited about that. so before we begin we have some prerequisites and this is a heavy course in terms of implementation in machine learning. we want we want everybody to be able to train an agent with RL by the end of it. And so our prerequisite is also we require prior serious experience training a language model. the LTI courses on language models, language model systems, NLP would all qualify for this prerequisite.
62:08 So if you took one of those, that's okay. some deep learning courses might qualify for it. if you're not sure, feel free to ask us. but we are going to require this and we're going to have everybody have to submit a form for it. So please please do do that. if there's like 30 people who want to know whether their course qualifies, please don't ask me that specific question after the class here because it'd be better to do it through email, but other questions we'll be happy to answer after the class.
62:39 we'll share a Google form on Patza and please submit the form so we can finalize enrollment. and yeah, I I already I already covered it. If you haven't taken a course, if you've worked in industry and you've trained a model in industry, it should be a real one like at least 4 to 7B size, not not 100 million parameters. but we will also accept that. and so this is the grading policy.
63:12 so assignments one to three will be 40% total completed individually. we're going to have lecture highlights and so basically the idea here is that we want people to come to the lecture and give a highlight about you know something you learned in the lecture. The idea around this is not to be, you know, not to make people do like undo work or something like this, but if you watch through the entire lecture, you hopefully you came up with something that's interesting. If you said if you could say nothing was interesting, I knew all of this already. I learned it here in this previous class that's already in the prerequisite list. That's also useful information for us. So share it with us. But hopefully we'll say at least one interesting thing in every lecture. So you can you can follow up with that. the research project is 50% of the total and this will be in teams of two to three because it's half of the class assignments.
64:18 the proposal is 5%, the check-in is 5%, presentation is 10% and final report is 30%. And if you've taken an LTI project based class before, grading will be similar to the LTI project based class requirements. if you have not then we'll we'll be sharing details about this. so here's the question you've all been asking for. This is an AI agent class. and we discussed this very very thoroughly about what we wanted to do here.
64:58 I'd like to preface this by saying I use agents all the time for nearly everything I do. but at the same time, it's kind of dangerous to use agents for everything because you lose like sight of of what you want to like of what you're learning. You miss some of the details and other things like this because the goal of this class is learning. but we also want everybody to be familiar with the standard practice. we we want you to keep this in mind and AI tools will be permitted for basically everything in the course unless we say otherwise and we probably won't be saying otherwise very often. so lecture highlights must be written by you not generated by AI. and so these do not need to be long. they can be short, they can be a few sentences, but please write them yourself. And you are responsible for every submitted claim, citation, result in line of code.
66:02 And we will create quizzes for the assignments so that you will you may need to explain parts of your code. And how will we do this? we also know how to use agents. So we will We we are considering having agents read read your code and come up with quiz questions tailored to your code. so we're we're serious about this. and that means that the thousand lines of slop that your agent generated will be a bad idea for you to submit. so it's better to submit something concise that you really understand well than submitting you know more basically.
66:46 so yeah, we're almost done. deadlines, submissions in Slack. So, all the assignments, include two 24-hour Slack days. they cannot be transferred or shared. After the Slack days are used, the penalty is 5% of the assignment score per additional day or part day. So, what this means is the deadline is the deadline. And, but we're being nice and if you miss the deadline, we'll give you some time to make up for it. if you miss the deadline by more than two days then we're not going to take most excuses.
67:23 If you you were in the hospital or something like that, we might ask you to get appropriate acknowledgement of that, but that would be an example reason why. but other than that, please try to meet the deadline because we're not going to make many exceptions to this. cool, that's all. we're at 441. so which means we have nine minutes for questions if people have questions. otherwise looking forward to having everybody in the class. Any Yes.