DSPy vs. LangGraph: AI Agent Frameworks Explained | Stanford CS329Z Lecture 5 with Michael Ryan

0:05 Today's lecture is on frameworks and orchestration. It's a very very practical lecture, so I'm I'm excited for you all to We're going to get our hands dirty with some programming today. So I'm going to start with a quick poll. By a show of hands, who in the room has has used PyTorch before? You want to raise your hand? Awesome. So almost almost everybody. It's fine if you haven't. We're not using PyTorch today, but then you'll be familiar with what an abstraction means in a framework.

0:33 What I'm showing here on this slide is what it would take to write back propagation by hand if you were going to write it in just Python and NumPy. You have to do all of this math to to calculate the derivative and then figure out how we're going to get the derivative with respect to each of the weights in the model and apply them in back In PyTorch, all you have to do to write back propagation is you write your forward pass where you say what needs to be multiplied together and and produce the and then you just call loss.backward. This is nice because there was a lot of underlying math that's still going on when we write things in PyTorch and that it's still good to understand, but we don't want to do this all by hand every single time.

1:12 And so frameworks abstract away some of the underlying system details. In the rest of the course, our goal is to implement and understand these system details. That's what we're doing in most of the lectures in this course, but our goal for today is to understand these abstractions and the frameworks that make writing code for agents a lot easier. So the learning objectives today are going to be to identify common frameworks used in agent system engineering, relate these abstractions introduced by the frameworks back to the underlying system designs that we've been discussing throughout the course and will continue to discuss.

1:45 And assessing the usefulness of different tools for agent orchestration and reliability. So hopefully you'll get a taste of what these different tools can do. We're going to start by talking about what does it mean to engineer agents. We're going to talk about frameworks for writing workflows and agent programs, open source agent harnesses, that'll be a kind of brief summary of like what exists in the space. Then we'll talk about different libraries for orchestration, telemetry, and observation of agent workflows. And then finally, we'll conclude with some best practices and ongoing challenges in the space. So, one thing I'll leave this up for a second. So, if you want to take a picture or type this into your terminal, we're going to have code today. And if you want to like scroll through the code locally, I've I've posted the slides on the website, they should be up. And then also the you can use this GitHub repository. We have an Jupiter notebook. If you put your open router key in that we've sent to you all, it should cost less than $1 to run this. So, you can follow along with the code. But that's that's up to you if you want to do that.

2:50 All right, so let's going to start with what does it mean to engineer agents? I would say there these three pillars of AI systems. We talked about this a little bit in the first lecture, but there are three things. There's the system itself, so the language model or the compound AI system or the agent is something that you have to build. You also have your data. This could be like training data, rollouts, demonstrations, human feedback, all of these things that are passed into your system to make it better or actually produced by your system in the case of rollouts. You also have your evaluation and metric which you have to define for your system. So, it could be things like pass rate or we have like exact match if it's like an information extraction task. The LLM as a judge could be your metric. And so, building an AI system, all that you need is you can you can throw a system together. You can you know, build your agent. I'll show you today how you can build an agent without any data or any evals. But if you want to engineer an AI system, then you need all three. And I'm going to talk about how all of these three are built into the frameworks. So, we talk about these throughout the course just to give you a sense of like where we are coming from, where we're headed.

3:58 These are all the lectures that we will be covering in in course. And we're still in the system part of the course, but soon enough we'll be talking about data, metrics, and and later emerging topics. So, I want this to be clear that when you're engineering AI systems, it's not just about building the system, which is still what we're talking about right now, but you really do have to think about your data and your evaluations. So, I'm going to talk about the framework landscape. So, here, let me get my animations up. So, basically, what we're going to look at here is a spectrum of agency. So, on the left, we have models. These are small parts of a larger system.

4:38 Then, we have workflows or compound AI systems. And so, this is like you have multiple model calls strung together to complete a task. There could be some sort of looping and and control flow. The main thing is that the control flow lies in the program. So, you are writing what goes into the logic. So, like, we need to call the model three times. We need to route to this model for this case, this model for the other case. So, you're writing a program that gets stepped through, and the model is just a piece of that program. And then, finally, all the way to the right, we have agents. And the here, the control flow is in the AI model itself. And so, here, you might specify the task, but it's actually the model that's deciding I need to use this tool to complete this task, or I need to, you know, make this function call, and I need to call another model. So, where the control flow lies is kind of what changes on this spectrum. And we have different frameworks for all of these things. So, we have OpenAI ways you can call a model.

5:40 If you want to build these workflows where you chain together multiple calls of models to achieve some goal, then you could use something like DSPy or LangGraph. And if you want to build an agent, now, these things do blend together cuz I'll show you how you can build agents with DSPy and LangGraph also, but there's specific libraries that are just about building agents. and then, we also care about the reliability of these systems. So, there's tools like Temporal that make sure that the system can run over long periods of time and stay functional.

6:10 And then there's observability tools. So, we want to see what is actually happening under the system under the hood when we're running it. Are there errors? What does the trace look like? That's what these LangFuse MLflow type tools are for. And so, in this lecture, we're going to start by talking about the workflows of compound AI systems. I guess I'm making a somewhat of an assumption that you can figure out how to use the OpenAI API for yourself. But, we will talk about these workflows and compound systems. Then we're going to talk about these open source agent harnesses.

6:39 And finally, we're going to conclude by talking about these like reliability and observation observability tools. So, any questions about the roadmap for this lecture? This slide will will come back as I remind you of where we are on this spectrum. But, yeah, any questions here? Cool. Then, I think one question I've gotten before is like, "Well, okay, like, are you just grabbing a bunch of libraries and like these are like random projects on GitHub and and just showing these to us?"

7:09 I would say no, and like a quick slide to justify this is like you can see these are very popular repositories, so they have a lot of stars on GitHub. But, actually, if you look for jobs in agent engineering, and just Google it, you'll see like DSPy and LangGraph are like on some job listings they're like required skills or like recommended skills. So, these are things that are actually practically used like in the industry if you're going to be an agent engineer.

7:32 So, to recap on what we've talked about so far, engineering AI systems requires the three core components, the data, the system, and the evals. The spectrum of agency ranges from the models to the workflows to the agents, and that's where where does the agency lie? Does it lie in your code or does it lie in the system deciding what code to run? And then there's frameworks for each level of this spectrum. And then there's also frameworks dedicated to the observability and reliability of these systems.

8:00 So, now we're going to actually talk about some of the frameworks for for workflows and programs. But, before I jump into it, Any questions so far? >> Yes, the interesting between the observability observability and and the reliability. >> Yeah, yeah, that's a good question. So, the question is what is the difference between observability and reliability? In practice, a lot of these things blend together. So, I'm I'm creating a bit of a taxonomy where there isn't necessarily one. But, the way I would draw the distinction is that the observability is meant to be about looking at what is happening and traces and if there is errors and stuff like that, like it's surfacing it to you to go and fix it. Whereas, the reliability tools, they do all of those things, but on top of that, it's like I need to make sure that if this crashes, it comes back online. And like it's automatically healing the system and stuff like that to make sure it continues running. So, the difference is like is it surfacing it to you or is it automatically handling it and then you look at it later, basically. Yeah, good question.

9:02 Any other questions? Cool. Then, we are going to start here on our framework landscape. So, we are going to talk about these workflows in compound AI systems. And first, I'm going to talk about DSPy. Now, full disclosure, I'm biased because I worked on DSPy for a time a few years ago. But, I think this is like a really really good library. So, you know I'm coming with a bit of bias, but I think it's really great. So, we're going to start by talking about what are the what are the three core ideas that you need to know about in DSPy. So, the first thing is signatures. And don't worry, I'm going to show you code all of this in a second, so it's not like you have to internalize this and fully understand it right on this slide.

9:40 But, the signatures are kind of like what the language model sees. So, it's strictly defining the inputs and outputs to the language model. Then, we have modules and these define how the model is going to actually do the inference on these signatures. So, you can compose signatures and program logic into these callable functions and that that's a module. And then, we have optimizers which will automatically improve the prompts and the weights in your system. So, we're going to do now the live programming demo. So, if you have cloned the repo, you can follow along. If not, no worries. We're going to be doing it up here.

10:16 Okay, I have a notebook here. So, we are going to start I'm just going to run some setup code that just like loads all of the libraries. And it's also going to open up something here that I want to show you all. I had Claude build a little calendar application that we're going to refer back to today. So, this is going to be our external like tool calling environment. So, the tools will be things like adding to the calendar, deleting from the calendar, moving events around and stuff. So, this is just to make things a little bit more visually interpretable instead of just like all text today. But, we'll keep come back to this library. Sorry, the the calendar here.

10:56 So, we're going to start with DSPy. And we within DSPy we're going to start with signatures. So, right here this is a signature. And it's very simple. I mean, I'll show you a class-based signature in a bit. But, this is extremely simple. We're just defining the inputs and outputs. We're saying the input to the model is going to be a message. And the output will be a response. And so, this is like you can imagine like a little chat that we've just built here. We specify predict this this is we'll get into in a moment but this is a module here. And this is telling us how we should process the signature. But, predict is the simplest one. So, we're just basically giving the model a prompt that's framed framed around this. And so, I'm going to send I have built my chat module. So, I pass a message to it and I say, "Hello, how are you?" And I ask for the response. So, notice that the response here matches this. The message here matches this. And if I run this, we should see Well, there we go.

11:53 Hello, I'm doing well. Thanks for asking. So, I'm talking to I believe I set this up with a GPT 5.4 nano or something like that. And so, now the thing that's great about signatures is it's useful for structured input and output. So, you can imagine before I would have to write a prompt if I want to do like sentiment classification, I'd have to write a prompt that describes answering this format and make sure you respond in a specific way. But if I just say I want message and I want to get sentiment, I can run this on I'm happy, I'm angry, I'm confused I'm confused and we get positive, negative, negative. But something's actually bad here. I'll answer your question in a second. I just want to point out what's bad is like look we've got positive lowercase, negative uppercase, negative. Like we didn't actually do a very strict way of defining what we wanted the output to look like. I'm going to show you in a second how we can make this more strict.

12:42 But there was a question. That's a good question. So the question for anyone who didn't hear is like how do you actually enforce that it follows the signature? So there's something in DSPy that I'm not going to talk about today in depth, but it's called adapters. Adapters are what turn your signature into a prompt that allows it to be processed and strictly adhered to, but it follows basically the adapter is and I'll I'll I'll show you what a prompt actually looks like.

13:17 But it enforces either JSON following or like following a certain format and then DSPy itself does the validation to make sure that it hit the format correctly. Good question. Is there another question? I saw I maybe saw a hand. Yes. >> Yes. It's like you're using this message just sentiment. We're just relying on We're just So the LM is just deciding what it thinks sentiment means. Like those auto >> In this case So I will show you in just a moment Oh the question was like is the LM just deciding what sentiment means?

13:47 Yes, but I'll show you in a second how we can do type enforcement on the outputs. Good question. And in fact that's the next thing. So we can apply types. So here's a type signature. I I feel that the string signatures are starting to get a little messy here. So no we're in a second we're going to switch this to to be something a little more class based, but here we say the message is a string and the sentiment is a literal which is positive, negative, neutral. And here I'm actually also adding comma here which says like we want another output and we want the a confidence which is a float.

14:22 And you could also do things to specify that I want a float between like X and Y, you know, and stuff like that. But now if I run this, we get these predictions which says sentiment positive with confidence 0.98, sentiment negative 0.9. So you get this structured output here. And now to to the question of like how does this actually get enforced? We can look at how the prompt is actually written. So you can see the adapter converted your prompt into something like this which is your input fields are this, this, and this. All interactions we structured like this. You may have message then the sentiment this is where you put your sentiment.

15:00 And then you can see respond with the following output fields and then the response is in this format and then DSPy parses this back into the format the the prediction that you can actually use. Any any questions about how this works or or what's going on here? Yes. >> So if the LLM does not like follow the prompt, does will it catch that and then like force it like repeat until it gets something valid? >> Yeah, the question is like if it the LLM does not follow the prompt, will it catch it and repeat and how does it handle that? Yes, so the adapters built into DSPy have fallbacks. So the chat is the first thing it tries. This is considered the chat adapter.

15:40 If that fails, it has a JSON adapter. You can also add custom adapters. Eventually, if it just can't do it, it will like error. But yeah, it it tries several things to get the output. Any other questions? Cool. Then all right. So now how about when your signature gets more complicated? Maybe let's stop using a string to define our signature. You can do this. Now this is a signature class and we're saying the email is a DSPy.input of type string.

16:14 And so this is going to be about extracting event details from an email. So like let's say you get an email saying like, "There's a talk at 5:00 p.m." then what this can do is take that email, it will extract what is the event name, what is the date. this is just coming from like this date time package, and you can see like we can do somewhat arbitrary types, and and DSPy will figure out how to format the model outputs to adhere to these types. So, we say the date, we say the time.

16:40 here we can give it a little hint like use a 24-hour clock. So, this this also goes into the prompt and gets formatted by DSPy. and then we say duration in minutes, and how long the event is if if it's in the email. with all of these types you'll also see I did string or none. And this is like if it's not included, it can also just output none. So, there's like a little bit of a nice complicated type handling here. yes, question.

17:03 >> Is the name of the variable the only thing that gives it description as to what to look for? Like, what if there were like two dates >> that's a good question. So, the the question is like, is the name of the variable all it uses to to know what to look for, or like, what if there are two dates? Now, the the answer to that is like, yes, you're you're limited to the signature. So, if there was like, "Oh, it would have been a really nice to tell the user like this other thing about the email, but it's not in the signature." It can't do that. And the other thing is the only thing it knows that it should put here is like this word date. Like, this this is what decides what goes here. Now, if you add a description, it gets other descriptions of like what should go into this slot, but otherwise it's just literally using the word date and and its type. It knows its type, too, to decide what goes here. So, this is like maybe the difference between an agentic thing where it's like making a lot of decisions about what it should tell you about versus like we've told it very strictly this is what we want as inputs and outputs. It's a good question.

18:01 Another question? Yeah, yeah. The question is like, could you have a system prompt that tells you some of these details about how to do this? so, yes, and in fact, this is the equivalent of a system prompt in DSPy. it's This is a little bit of a weird like a hack, but we take the docstring from the the function the class and use that as a system prompt. So, if I was like if there are multiple dates, pick the first one. Like, I could put that here or something like that. So, that's how you can kind of do those things. Yeah.

18:38 Another question? >> Is it better to be like less descriptive with that string or is it like does that help it? >> Good question. The question is like is it better to be less descriptive or should you put a lot of detail into that string? I think it depends on your task. Another thing I'll show you in a moment is optimizers in DSPy and what optimizers do is take this string and optimize it for you. So, I think the most important thing is to put every like thing that's key and essential to your task here and then the like little details, let the optimizer figure it out. Good question.

19:06 I think I'm going to move on just in the interest of time, but save your questions and and I'll I'll leave a spot to ask questions again in a moment. Oh, yeah. So, I want to actually run this. So, you can see here I had we have a meeting scheduled for this Friday, October 9th at 10:00 a.m. and it extracted it. It said the event name is meeting. Here's the the date. Here's the time and we don't know the duration. So, so that's that how that worked.

19:29 And then you can see it did the same thing where it formatted this into a prompt where this is what the prompt looked like and then we got the output from the model for that. Cool. I guess maybe if there's if there is one question about signatures, just like anything that's confusing, anyone have like a final question about signatures before I move on? Cool. Okay. Oh, yes. Last question. >> Does DSPy handle signatures in an LLM or like in a deterministic >> Okay, the question is like is the maybe I'm understanding correctly the question is like is the conversion from the signature code to the prompt done by an LLM or is it done deterministically?

20:11 The answer is that it's done deterministically. Yeah. Cool. Okay, so modules. So, sometimes you might want to different inference strategy than the default predict that I showed you. So, here I have a chain of thought. and so all this does, I think it's easiest if I just show you. This is the same signature, the extract event one that we just defined, right? And we're going to run this with a chain of thought instead of predict. And you'll see that it added this reasoning.

20:38 The order here kind of matters actually. So the prediction, the first thing it produced was reasoning to say, "Okay, the email stated this, this, this, this." All of these details. And now it's going to put these other fields. So what chain of thought does is it just simply actually adds a reasoning output to make sure that the model reasons through what it's going to output before it says the the rest of the output. and then I'm going to show you this is this is not to demonstrate anything about modules, but I just want to have two signatures for the next part where I can post stuff. So here's another thing.

21:11 This is a signature I wrote to pick one emoji that best just represents a calendar event. So we'll say the event name as input and the emoji is output. So I'm going to say dentist, and you can see it decided that a tooth emoji would be like something that relates nicely to that. Yeah, we have a quick question. >> Is the prompt as like a comment rather than like another variable that's like prompt? What's the advantage?

21:40 >> Yeah, the question is like why have it as a system prompt instead of another variable? I think that's a reasonable question. I So my role in the library was about the optimizer, so I didn't have a ton of sway over here. I think this is like somewhat I do think like calling out the system prompt, there's only one system prompt per module, and calling it out is a very easy to read thing for someone scrolling through. Maybe it's intuitively nice. So like when you look at the code, oh I maybe I'll just finish the thing I was going to say and then can share. But it's like nice that it's one thing. It's when you're looking at the code, you can read like this is exactly what this signature is supposed to do. So there's some some properties there, but I guess maybe you have a two >> I'm going to adapt this to be a constant.

22:23 So what a constant does is that they each have their own fields and do various things with them. you know notice that each of the fields are now some type of output. And imagine the evaluator would then say, "Oh, okay, you know, we're going to look at the inputs, look at the outputs." And you know, there could be an arbitrary number of them. Suppose you wanted to design a scheme for which system prompt, you know, you would have like saying colon string is equal to dspy.system_prompt. That's just great.

22:47 But then maybe a different user showed up and say, "You know what? What we're going to write is user is going to write system prompt colon float is equal to dspy.system_prompt. And suddenly that's a type error. And then to be able to guard against this, then the doctor has to spend all of this work and try to like scan across every one of their fields, check the types, check if it's a string. Okay, did the user use this correctly? Did the user only give one system prompt instead of many? Whereas like this structurally makes it so that a mistake is literally impossible about how to write this system prompt. And so it like makes more structural sense to have a user.

23:20 >> Yeah, that makes sense. It's just an interesting like it blurs the line between >> executable and not executable. >> Yeah, yeah. I think it's it is confusing a little bit that it blurs the line between like you're using documentation as an executable part of code. I think that's a totally fair point. It is kind of an interesting decision. Cool. all right, so the next thing is that we can compose modules and include Python logic. So here we have a dspy.module and so I'm calling it update calendar from email. So you can imagine what I'm going to try to accomplish here is we're going to call some Python code to update our calendar based on the email. so I'm going to initialize it with two two modules here, self.extract and self.pick_emoji, which is going to be a chain of thought for extracting the event and a chain of thought for picking the emoji. so hopefully everyone understands like everything that built up to here so that we can get to this point. we have the signatures, we have the modules and now we're storing them within this module.

24:15 and now we're going to define a forward pass, which is it's important that you call it a forward pass. this is kind of similar, inspired by PyTorch. and so we're going to say the input is going to be an email and the output will be a ds.py prediction. And so first thing we're going to do is extract from the email to get these details. And so we call this module here self.extract with an email. And remember this will return the date, the time, the name.

24:43 And so what we're going to say is if the event name is none or the details or the date is none or we don't have a start time, we'll just say we'll this is like you could you could probably poke holes in my code here. But this is not something about the library. This is like I've made the decision if I don't have the date, time, and name, I can't put it on my calendar. And so we're going to say we're going to return event is none and the status was no event was found in the email.

25:08 if there was so this is just Python, right? So if there was an event found in the email, now I'm going to pick an emoji for it. And I will use this tiny calendar API SDK thing that I that I had Claude make for me to say add event, which is the title, date, start, duration minutes, and an emoji. And we'll assume 1 hour if it wasn't specified in the email. And here we'll return this event, which is just like some event type returned by tinycal. I guess the it's a dict.

25:40 And then we'll say that we added it to the calendar. So let me run this to define the module. And now you can see we create the module and now we can pass remember the input was email. So we're just going to pass an email here. We have a meeting scheduled for this Friday at 10:00 a.m. So look quickly at the calendar here. You can see Friday there is no events. But now I will run this and you can see and the second thing was like an email that doesn't have a an event. So we ran it. The first one said that it added to the calendar with some details about the event. And the other one says none no event found in email. And now if I look back at my calendar, you can see we have a meeting scheduled here. So any questions about this?

26:28 Awesome. Okay. Now we're going to talk about optimizers which is the part of the DS Pi library that I'm actually most familiar with. This is what I did some research on two or so years ago. So the question that I want to motivate you in this section is like, okay, I just did this but I used GPT 5.4 nano. Is there like maybe small is a weird thing to say to compare against nano cuz we don't I don't know how big it is and stuff. But maybe let's say like kind of kind of local or like open weights model do this task too. And let's say it's like something that would be efficient enough to be able to run on like reasonable hardware. So what we're going to do is I've created 50 synthetic emails that have these that are somewhat realistic that have these gold labels.

27:13 So I have a folder of all these emails. So if you have the repo locally you can look at this but I'll show you an example here which is like okay, we have this like dear professor my thesis committee meeting is on this is here at this time and and it will last one one hour. And so the gold label for this is thesis committee meeting. The name doesn't matter. I'll show you in a second what our metric is but we we ignore the name but we want to make sure that we get the date, the start time and the duration correct for every email in our data set here. So if you think back to remember I had the three pillars. I had the data, the system and the metric.

27:53 This is this is the data in this case. So so we have data. We have all these emails that are labeled. And now we're going to turn them into DS Pi dot examples. This is just necessary for like making sure that it's the right input format so that DS Pi knows how to operate on it. And so we're going to say the inputs are the email field and then the rest we aren't labeling so they're going to be our outputs. And so what we just do is split this up into a train and test set. So 30 examples in the train set, 20 in the test set.

28:22 And now I'm going to define a metric. And the metric has to have this particular signature, okay? So So this is actually important for using DS Pi metrics. You say what the you take the gold as input, the prediction as input. Some of these are like you can optionally use them or not, like trace would show you like the actual calls to the model. and like these you don't like you could have actually like ended this here cuz those are the default. So, you really just need to care about gold and predicted.

28:52 but I'll I'll leave it just because stuff is cached in this notebook, so I don't want to I don't want to change things right now. and then we'll say these are the checks. So, the first check is going to be that you don't have to write This part is just Python code, okay? So, you don't have to write this exactly how I wrote it, but we said the date we want to check that the prediction date matches the gold date.

29:09 We want to check that the start time matches the gold start time and the duration of the event matches. so, we want to just make This is like an exact match metric. We're trying to make sure that all the details in the email are the same. And so, we will just do this all function which is built into Python that returns true if every if all the things are true in it. And it's just going to loop over these three criteria and say are these all true and turn it into a float, which would just be zero or one.

29:35 if it's wrong, then we can actually format So, this this is an an optional thing in the DS Pi metrics, but you can actually format it into plain text feedback. And so, here this is just doing all this formatting to say like what is wrong. you don't have to do that. You just have to return a score, but if you return the feedback, it's going to really help the optimizer and and we'll have a lecture specifically talking about how the optimizers work.

29:58 but then you can run this DS Pi dot evaluate on the test set on this event metric. you can just say how many threads and if you want a progress bar. And so, I'll run this and so, now we've built our evaluator. but you actually you have to call the evaluator. So, that's why nothing happened there. We just initialized the evaluator. And so, what I'm going to do is take GPT 5.4 Nano and show that if you evaluate it on our test set, we get 100% accuracy. So, nice.

30:26 Our our big quote unquote in this analogy big model works great. But let's see if we can get a small model to do the same. so, we have Gemma 312B which I chose because it was one of the smaller available models on open router that seemed like it could at least do the task somewhat out of the box but needs some pushing. So, all we do is we set a extract small we make a extract small thing on this extract event. We set the LM for it and now we score before optimization. So, I will run this and you see we got 55% so 11 out of the 20 emails we got the correct details on.

31:05 And so again just to show you like instructions before it was just extract event details from the email which was that docstring remember that we specifically typed out we had a little discussion about it the the exact thing in quotes. So, this was our instruction and then we had the full prompt here. So, now what we're going to do is optimize with the Jeppa optimizer. I'm not going to tell you guys how this works because in two weeks we'll have a lecture in-depth talking about optimizer algorithms but just know that it's going to roll out a bunch of times reflect and try to update the prompt. So, I'll run this. Now if you're following along locally this is not going to just take 0.6 seconds. It's going to take a lot longer but I have everything cached so this going to be an efficient lecture.

31:47 This might take like five minutes and now it rewrote the prompt and so I run this and you can see our small model now is getting 100% accuracy. So, and what changed? It's just the instruction. So, this is the new instruction that it came up with. So, it said you're given a single email your job is to extract events. This is like how you should format it. These are the output fields. So, like you could have written this all by hand and put all the details into your docstring there but instead we let the optimizer figure it out and it wrote a prompt that gets us like a lot better accuracy on our on our on our test set. Or I guess it optimized on the train set but on the test set.

32:28 Yes. Sorry, how many tokens does it take? So, yeah this is a good question. Off the top of my head I don't know how many tokens this run took but in general it takes it has to actually run your thing a couple hundred times. So, it probably would take on the order it could take on the order of a million or so tokens. So, you do want to make sure or like depends on what you set your budget to, but yes, you want to be careful about this because this is like a cost for sure.

32:56 But, it's something you run once and then now your system is optimized. So, I think there's a trade-off to it. Good question, yeah. yes, another question. >> Are these overfit evaluations? Because I believe it's taking the use of context learning? >> yeah, it can. Well, actually in this case it's not doing like the the prompt that I came up with here did not put any examples in it, but some DSPy optimizers will do in-context learning. Sorry, to recap the question was is this overfit to your to your set?

33:26 Train dev test split all still applies just like you're optimizing like you would do if you were training the weights of a model. So, this only looked at the train set. So, that the 100% accuracy is on the test set. So, this is an important thing to be thoughtful of though is like designing this making sure you have a train dev test split and all of the same practices that apply to any sort of ML engineering. Question. yes.

33:50 >> Can you speak optimized prompt like the prompt be stored? Like if you run it again, it's going to generate the same thing? >> it's So, the question is how is it stored? Like if you run it again, will it generate the same thing? so, DSPy has a cache, which I won't talk about too many details about how it works, but essentially every call to a language model, if you use the exact same input, exact same hyper parameters, it will get the exact same output cuz it saves it to your disk. If you did not save that cache, it would not be deterministic and you would get a different prompt. Yeah.

34:18 >> Good question. >> cool. yes, last question. >> Random words like some sort of jailbreaking thing? >> good question. So, the question is like have you seen the prompt just become random word words like it's like some sort of jailbreaking thing? In the past, yes, I have seen weird things. So, like an example I can remember is one time I was opti- or I I do this, but somebody else used my the optimizer that I wrote to optimize a model to be good at math, and it wasn't random words per se, but it made the model it wrote something like you are a captain on a Starfleet like you it was like it wrote like this really really wacky scenario. It it was like based on Star Trek, and it did make the model better at math, but it is weird that that works. this is no longer as much of a problem. Models have gotten better. This was like 2 years ago. I don't see this as much, but it's it's certainly feasible that you could get a weird prompt out of this.

35:18 well, we do care about like held out test accuracy and stuff. So, I think it's like you you should just be careful that your metric and your data is measuring what you care about. and then a weird prompt that does better on a good evaluation maybe is a good prompt. But, if your evaluation is hackable, then you might get a bad prompt that isn't doing what you want it to do. So, I think that's the that part to be careful about. Good question. I'm going to move on cuz I think I'm spending a lot of time on DSPy because I like it so much, but we do have other libraries we want to talk about.

35:48 So, now just want to show quickly how you build an agent in DSPy. So, remember this breakdown of data system evals, there's also the spectrum that I was talking about model workflow agent. So, I just showed you how to build a workflow, which is where all of the orchestration logic is in your Python code that described this is the inputs and outputs and this is how I'm going to call the model. But, what if we wanted the model to actually make decisions about how it calls tools and everything itself? Well, DSPy has a solution for that. You might remember in lecture one D introduced the react architecture for agents. so, this is built into DSPy. If you don't remember, it's okay because on Monday I'm going to teach it again in more depth, but react is an agent architecture. We can it's also built into DSPy as a module. So, now we say I want to go from a request to a summary, and the way at least that I interpreted it and it seems like DSPy understood what I meant and the model understood what I meant is like I give a request, you do all of the things I put in the request and produce a summary at the end. But you can imagine probably doing a better signature. This is a little hand-wavy. and then I specify my tools. so this is the tinycal.tools and these are all the tools that are built into tinycal which is like the the calendar thing. and essentially we have like these are just functions.

37:07 So these are all callable functions. They all have their own signatures, they all have their own inputs and outputs. And essentially this one line is all that it takes to build an agent in DSPy. So it's it's very simple. and we can say, "Okay, I want to schedule a 30-minute coffee with John on October 7th in the afternoon after my lectures and put a a put a book emoji on all my lectures, too." so I'm going to run this and let's see. It'll take a little bit of time but not very long cuz it's just making a few calls here.

37:39 And there we go. So okay, so it here's the summary that I wrote. Remember I I wanted a summary. This is the output as I requested it. So it said I added a book emoji to the two lectures and I scheduled coffee with John. So now if I go back to my calendar, you can see we now have a book emoji and coffee with John. So we have we built a little agent here. and then one other quick thing I'll mention before moving on is recursive language models which I talked about on the language model lecture can handle really long context and this is also built into DSPy. So here I'm loading all of the emails in my data set. I'm just joining them together into a really long string and I'm saying I want to go from my full inbox and a question to a list of event names as an answer to the question. And so my question is which events are held in the Gates building and on what dates?

38:30 and now I'm putting in you know 50 emails or something like that to the context which wouldn't saturate the context of a small model but just like imagine like it could have we could have done like a million emails or something like that. and if I run this it will tell me that there's a thesis defense on this date. So, it was able to to process a really long input by I I I don't expect you to remember how this works, but it was basically like it built a repple environment where it could treat the prompt as a variable and then call tools on the prompt itself.

39:01 Anyway, I want to move on because we've spoken about DSPy for quite a while, but any final questions about DSPy? okay, we have two and then I'll move on. So, first you. >> So, for that last thing where you were reading the emails, was that like were you playing all the emails in the >> So, you can it's like I'm treating context as a fuzzy thing here. the model itself does not read all the emails, actually. The model gets a variable that says inbox like you have an inbox variable and then it has a code execution environment where it calls the inbox variable. So, it might have like done some sort of like grep or something on that to look for the word gates. and so that's that's what it's actually doing in that last example. Every other example the context is what the language model sees.

39:46 Yeah, good question. other question? >> I have two quick questions. And the first one is what did you specify in like your model? Like did you specify >> that was in the setup section. I put it at the beginning. There was like DSPy.set_lm or something like that. Yeah, yeah, yeah. >> Like you never really like wrote your own prompt, right? Everything was like Python code and then you run this optimizer that like creates like a system prompt or like >> Yes.

40:10 >> Like can you like create your own prompts as well and like run them with the >> Good question is can you write your own prompt and still use DSPy and the answer is yes. You would have to put it in the docstring thing. I'll try to find one of these. Yeah, this is where you would write your own prompt if you didn't want to use an optimizer. you would just write it yourself here. I I recommend using the optimizers, but like there's there's reasons why you wouldn't. So, yeah. Good question.

40:37 >> If you write your own prompt but you don't use the optimizer, >> Yes, it'll start from your prompt. If if you you write a prompt and then use the optimizer, it'll start from your prompt and optimize on top of it. Cool. All right, let's move on in the interest of time. I think the other libraries will be quicker though because I'm less knowledgeable and less passionate about it. So, we'll but let's let's me skip to I just wanted to show I have some slides. So, this is the code we just went through.

41:04 But but Okay. So, react just to remind you what it was, it was doing a thought action observation format. So, this is what happened when we use dspy.react. The rlm was something like this where it was like we put the prompt into a variable and then let the model execute code on it. So, just just so you know what was happening under the hood, you can find this in lecture two, you can find this again in lecture one, and I'll talk about it again on Monday.

41:31 And so, the dspy summary is that signatures define the inputs and output contracts. The modules compose and specify how the signature contract should be fulfilled. Three examples are dspy.predict, chain of thought, best of n. I didn't show you best of n, but there's documentation online if you want to learn about it. Optimizers can improve the prompts and weights of the program given data and metric. I showed you a prompt optimizer. There's also weight optimizers if you have access to the model weights.

41:56 And then dspy has built-in modules to enable agents. So, dspy.react, dspy.rlm. Cool. Any any final questions about dspy? >> Yes. For the optimizer, what is the model being used? >> you specify it. You specify a teacher model for the optimizer. In this case, I used GPT-3.5-turbo. Cool. So, we're going to talk about a langgraph now. So, this is an outline of what a langgraph workflow looks like. So, langgraph also does something Langgraph I would say is somewhere between dspy and some of the orchestration libraries we'll talk about. But with langgraph, the state is a shared workspace for the graph, and you can think of it as an object that the graph will modify. So, So, we define state, and maybe it's like a a set of variables that that are fixed throughout this whole workflow. And then you need a start and end node for your graph and then you'll define nodes which are functions that read to and write read from and write to the shared state.

43:00 So let's imagine the state is a list of variables that matter to us then a node will update one of the variables in the state or read from that state to decide how it takes actions. And then you have edges that define control flow and in this figure I've put dashed edges as conditional. So like maybe we end up in let me do this with the right cursor so online people can see we may start here go to this node and then make a decision that sends us to either this node or the end for instance.

43:33 and then here is what a LangGraph agent looks like. So very simple graph here. We have the LLM and tools and you can loop as many times as you need to and then you finally decide to end. And the state could be something like message history. So I have a live programming demo for LangGraph as well. Let's make sure to be efficient with it but let's go back. Okay. Okay so defining a graph so here we go so structured output is possible in LangGraph too. So here I'm doing so that you don't have to re-internalize the program I'm going to be writing the same program again in LangGraph and Temporal when we get to Temporal but so we have event details where we are going to extract event details from an email and then so this is the same thing. We have the event name, date, start time, duration and then you can lock in your LLM to have a structured output. So we say LangChain LLM which I defined at the top so you can look at how you initialize everything later but LangChain LLM.with_structured_output and then you give it this structure here.

44:44 And so I'll run this and now we have our LLM that has a structured input and output. And then we'll say the state is going to be an email and details. So, remember the state is shared amongst the whole graph and so I'll run this and now I want to define a node. So, the node will take the state as input and update the state as output. So, I'm going to say I need an extract node which will take the state and it will add this details thing. So, I have email which will already be in the state. So, I'll use that and then I will write details and the LangChain engine will look at this and say okay, the only field that you touched was details. So, I will like update the details field.

45:27 So, this is what a node will look like. And then I'm going to build the graph. So, I'll build I'll make a state graph builder. I'll add an extract node which just is this function and then I need to add a start and an end and you can see you're adding basically the edges between these and then you can compile the graph. So, I've compiled it and now I I think it'll make a lot of sense if I just show you what the graph looks like.

45:48 So, we can visualize it. It's a very simple graph. We start we go to this extract node that operates on the state and it goes to the end node and there's nothing conditional here. We always follow the same path. So, I can invoke the graph on an email and say we have a meeting scheduled for this Friday and I'll run it and now you can see that the output is the state. So, the state is the email and now the details which all the extracted details go here.

46:17 Okay, so I know I went a little quicker here. Any questions about LangGraph and I will do my best to answer them but I know DSPy a lot better than I know LangGraph. Yes. >> How would you make like an edge conditional? >> Good question which is exactly what we'll talk about today. No, no, no, a good perfect question. I like the transition. Any other questions before I do that? Yes. >> What's the like unique feature of LangGraph or putting it in this format?

46:39 >> Yeah, what's the unique utility of LangGraph? Why why would we put it in this format? It's about like the orchestration of the graph. So, like you can do things like oh, retry this node and stuff like that. It's easier for observability and it's like somewhat robust and so like I think it's like state machines can be a nice way to think of a problem. It's also good for distributed computing too, like you can have each node operate on a different machine and stuff like that. so there's there's like some perks like that essentially. Like yeah.

47:07 >> So does LangGraph produce this graph for you or does the user need to put in that? >> so the question is does LangGraph produce this graph for you or does the user need to put it in? I guess like the way I would think about it is that you define the graph in LangGraph and then like so you have to do everything I did here you have to do when you're using LangGraph. And then it depends on do you I guess in to ask you a question, do you consider the user to be the person writing code with LangGraph or like the person using your agent system?

47:37 >> Latter. >> The latter, the person using your system? then no, they don't have to touch this graph or anything like that, you know, you just hand off your output and it'll run through the graph and stuff like that. Yeah. okay. So conditional edges. so now I'm going to define this calendar state, which is going to be the emoji and the status. so we're building a new graph here. This has a new state to it. and I'm going to create nodes. so pick emoji is going to be a node on this graph and add to calendar is going to be a node on this graph. and this will just be an LLM call to reply with the emoji and then this is going to be a call to have our tiny calendar add an event. okay, so hopefully you remember what all the nodes do and how that works. And then to create a conditional edge, you actually define a function and the function is going to or a method, I guess we're working in Python so it's a method. and we're going to say a route that takes a state as input and the output is going to be the name of the next the next node that you should go to. so if the state contains details, if it has an event name, date, and start time, then we are going to route to the pick emoji because remember that logic that we wrote in Python earlier where we tried to extract the event. If we didn't get an event, we We no event. If we do, we pick an emoji and we add it to the calendar. So, we're writing that same program here.

49:02 and so, we say pick emoji if we have one, otherwise route straight to the end. And so, this is now a conditional edge, and we can build the graph where we say we're going to add an extract node, a pick emoji node, and add to calendar node. And our edges are going to be We're going to start, go straight to extract. We'll add a conditional edge that goes from extract to the route, which is this function that we defined. And now we will go from pick emoji to add to calendar and from add to calendar to end. And I think the easiest thing to do is just look at what this looks like. this is the state machine that we've built. So, we always extract. If we have an event, we pick emoji, add it to calendar, otherwise we go straight to the end.

49:43 does this make sense? Cool. so, now I'm going to invoke this, and I'm going to say the dentist appointment confirmed for this time, and then this was added to the calendar. And if you remember, our calendar should be over here. now we've added a dentist appointment onto here. Yeah. >> What is the primary or difference between LangGraph and DSPy? Is it better for context management because it uses graphs? >> yeah, the question is what's the primary difference between LangGraph and DSPy? I think DSPy is more built around this like write your control flow in Python sort of like optimize your prompts. The These are like the use cases that people come to DSPy for. LangGraph is more about orchestrating, building like these robust graphs that can that follow some certain control flow. I think the interesting thing is you could actually compose them. Like, you could actually have DSPy calls be a node in your LangGraph graph. so, the use case is sort of like They they fit similar problems, but you can have different reasons that you want to use each one. And it's somewhat of also a matter of preference. Yeah.

50:57 Cool. And then last question? >> Does LangGraph have optimizers? >> Does LangGraph have optimizers? To my knowledge, it does not. but I don't want to say that with like with extreme confidence. Yeah, yeah. cool. and then here I have an email about open Reddit crowd open router credits have arrived. This one should not add to the graph. so you'll see this one did return just the event name, but it didn't actually add an event to the calendar.

51:25 Cool. And then how do how do you do agents in LangGraph? so similar to DS Pi, we can make an agent in just two lines here. we say from LangChain.agents. I won't get into it, but LangChain was like a LangGraph and LangChain are like connected. and you can say create agent. and then you say what the LLM is going to be and you list your tools. very, very similar. And I believe under the hood it's doing something like React.

51:54 and so the image here is like we have start, it goes to the model, loops over the tools conditionally, and gets to the end. And now if I say, "Please move my coffee with John to be 30 minutes later." just so you remember, the coffee is very shortly after class, so I need a little bit of a breather. so we're going to run this, and ideally it will go, and here's the list of all the actions it took in the messages. and it ultimately said I moved the coffee with John, and let's see that it actually did it. Now it's 3:30 to 4:00.

52:23 so that is LangGraph. Any final questions? Otherwise, I'll move back to the lecture slides. Yes. >> in a very unbiased way, how do you get >> No, yeah, yeah. >> All right. So if you want let's say you're doing a massive data processing project, and you need a consistent output format from like the evidence, which tool would you choose? >> yeah, the question is like if you had a massive project or like massive lots of data that you want to do data processing, which one would you use?

52:55 consistent format I actually don't think that the strength of either lies in a structured format. I think they both do that a pretty good job of that. So I would say the thing that you maybe lean to LangGraph for is like you want to execute like a massive thing distributed parallel like run like this huge graph orchestration. So, maybe like that would be a reason to use LangGraph. You can and again though you could use like DSPy within LangGraph and like optimize your prompts and stuff and then have the LangGraph do the like massive orchestration of that. And so, that would be like one reason to use that.

53:29 Good question. You had a question, too? >> just small first one. I assume you can use LangGraph for this. >> yeah, yeah. I I don't off the top of my head know but how to but yeah, there's a way to do that in LangGraph. >> So, you could assign the nodes to different different Can you assign different models? >> to different nodes? yes, yeah. So, the question was can you assign different models to different nodes? Yes.

53:53 What you do within a module is within a node is like kind of just free-form code. So, you can have multiple LLMs and stuff like that. Good question. Okay, sweet. I think this is a question and then I'm going to move on. >> When you're defining the graph for the agent in LangChain, can you modify it to be more than just like the model tool or is that the only thing that it does? >> yeah, the question is like can you modify the agent in LangChain to be more than the model tool loop?

54:19 Yes, I think so. They have some package called like deep agents or something like that. And I think that's what fits the use case you're describing. I I don't know the details on that but I I believe it is possible. Yeah. Cool. Let's go back to the slides cuz I spent I I think it was valuable cuz I feel hopefully we all learned a lot about how to write these things but I do have a lot more slides. So, let's go to the slides.

54:43 Okay, so LangGraph summary. LangGraph operates on the shared state over a graph. Nodes will read and write to the state. Edges define the control flow and some edges can be conditional. And then we have built-in tooling to create agents. So, the recap is that workflows will compose model calls, tools, program logic into the larger AI systems. The frameworks expose different abstractions. So, DSPy has signatures and modules, LangGraph has the states, nodes, edges, but you can use them to solve the same problems or you could compose them and and use the strengths of both of them. so we design the control flow and choose where the model makes decisions within this and improvement is going to require data and evaluation. So DSPi makes this explicit through the optimizers, but even if you were optimizing the system yourself by hand and redesigning the system, you still want that data and evaluations to do that.

55:36 Now we're going to talk about open-source agent harnesses. This is actually a relatively quick section and if if it's taking too long, I'm I may skip some of these, but basically now you are here in the framework landscape. so these are like pre-built agents essentially. And so what is an agent harness? this this comes from lecture one. so DSPi sorry, I don't know what I I think I'm thinking of DSPi D introduced what an agent like looks like whereas planning and reasoning, memory, tools, all these things. everything except for your kind of your external environment is what you would consider the agent harness. So it's like things that are built around the LLM to make the agent work. and and what differen- differentiates these different agent harnesses? well, there's a lot of design choices. the tools that you provide to the model, the memory management, context management, how the tools are executed, how those the tool executions go back into the context after the tool returns. different harnesses make different tradeoffs.

56:38 and a good harness must balance the success on the task at hand, the cost, the latency, what level of user control there is, and this all depends on the use case. Like Claude Code for instance, it matters of quite a bit how the user interacts with it. but some agent harnesses that we'll discuss are really meant to be just like solving tasks and stuff like that. So it actually doesn't matter how much the user is involved.

57:01 So the first one I'm going to talk about is Mini Sweet Agent. this one is very simple. It doesn't have any tools other than Bash. the history to the agent, so the context is built literally by just stacking everything that is seen, so you have a tool call, you add it to the context. you read a file, you add it to the context. So, there isn't like any super complicated context management. It's just basically what you are familiar with, like if you chat with ChatGPT, how it always adds to the conversation, that's how it handles context in MiniSue agent. it executes actions with a sub process.run, which means that instead of having a stateful shell, which is like the agent writes something to the terminal, it all of the state is preserved the next time it writes to the terminal. in in MiniSue agent is actually every single time it's starting up like a new terminal. which means you don't have to worry about like all these complicated things for evaluating agents with with Docker environments and stuff like that. So, that's kind of this is meant to be like a minimalist agent.

58:00 We have OpenHands. this is one, where it stores every step as like this event and then rebuilds the context from these events. so it has like this whole, look up of events to to put these together and so it's a instead of that linear history, this is like a more complicated way of managing the context. In this case, it keeps the persistent terminal, which is the contrast of MiniSue agent, and it has, like, different ways of stopping itself if it's failing. So, I put a few of these details in. I mean, I I I think you get the idea, like Pi agent is another simple one, only has four tool calls, no permissions system. Sto- this one stores the context history as a tree instead of a list, very simple. and then you probably heard of Open Claw, which, is like, very like very mainstream. people beyond like the community of AI engineers, is like something that like people have set up where like this is an agent that operates on your machine. It wakes itself up every 30 minutes, that's an interesting design decision. it runs as like one long process, where it's like consistently coming back to the main, agent that's running. builds its own identity from these like markdown files, stores memories every night. So, they they tried to make like something kind of unique here. but these are like different design decisions. So, my point in showing you all of these was just to say like these are design decisions that go into an agent harness.

59:25 Like, there is no one way to build an agent harness, and it's very interesting. You can put a lot of trade-offs into into what goes into it. And it also does make a a big difference. so, I think I showed this figure before on lecture two, but basically this was showing that Pi, Codex, Claude code, you run the models on it, and the success rate on Terminal Bench, which is this agent terminal benchmark evaluation, changes based on which harness you use and which model. So, these are things like check and optimize for.

59:57 And then another like interesting type of harness that this is a This one's from a research paper is meta harness. And the idea here is like the LLM can write its own code to to write its own harness, and you can then basically optimize for downstream tasks. So, you have like I want an agent that's really good at writing code for science. and so, you set up some evals, some science evals, and you say, "Write your own harness." And it builds its own tools. It's like, "Oh, I need I need tools that have to do with chemistry. I need to build these things and stuff." And they show that on Terminal Bench 2, which is the same benchmark I was showing on the other side, it can like evolve to make itself better at this task. Although, I would say like comparisons between like a meta harness and like Claude code and stuff is a little unfair because the way they do this is like they look at tasks from Terminal Bench and optimize it to be good at Terminal Bench, whereas Claude code is supposed to be good at everything. So, like take some of that with a grain of salt, but it's just like an interesting research idea I wanted to expose you all to.

60:55 So, to bring it together, there's no one way to build an agent. in this course, you'll be experimenting with science the design decisions, and that's what homework one is all about. So, you are building an agent harness for homework one, and you will have to think about these things about how do you manage context, what tools to expose, although we tell you some required tools. how do I handle memory? All of these things will go into the agent harness. and so hopefully you can build the next popular agent harness with if if it's something you desire to do.

61:21 any questions about that? Cool. okay. Yeah, one question. >> Are there like existing services that actually use this like let's say our own cloud subscription credits instead of doing individual API calls? >> oh, your question is there some that use your subscription instead of using API calls? >> And have the subscription instead >> Yeah, yeah. I think this sort of works I I think there there are people that have set up stuff to do this. It's maybe against the terms of service of some of these providers, so you have to be careful to to read into that. but I mean a very simple answer to your question is Claude code. I mean it works with your subscription instead, but open harnesses maybe maybe not in a way that abides by the terms of service, I guess. Yeah.

62:17 cool. I'm going to talk quickly about orchestration telemetry. so this is here on the framework landscape. and so why is orchestration and telemetry needed? this is a plot that shows you LLM release date on the X axis and task duration on the Y axis, and you can see that LLMs are solving longer and longer tasks. Now, one thing that is important and I if you look closely at this, you'll notice human time on the Y axis, not model time. so but you can imagine it these are things that take a longer time to solve, too. but so we have models trying to solve longer and longer tasks. And so agents are running for long periods of time, and so the infra supporting them must be reliable, or else you're just certainly going to fail the task if like the the machine running the agent has some issue.

63:06 so it's useful to have logs and observa- observability for debugging these long-running processes. And this is also like lessons that are taught from software engineering that transfers over to agent engineering. So, I'm going to talk about Temporal. and so, Temporal has your code on one one side of this, and the server will be on the other. And we have these activities, which are functional units that can fail. and so, you build workflows by linking these activities together.

63:38 and then you have workers. So, every activity can have its own worker. every workflow can have its own worker. And these are actually executing the code. And then on the server, you have a queue that says, "Okay, next I'm going to dispatch to this worker and this worker." based on like everything that you've linked together. And it stores all the execution history and state. So, I think based on how much time we have, I don't actually I think we have time to run the Temporal code. but maybe what I'll just quickly do is a quick demo. I think if you want to learn how to use Temporal, it's a very, very popular tool. I've spoken to many agent engineers that use this. So, I do recommend you take a look at this.

64:19 But I'm just going to run it and show you what the output is. So, essentially, I'll run these three things. And I want to pull up the Temporal server, so I can show you the observ- observability aspect. So, suffice it to say, I rewrote the same program in Temporal. so, even if you're looking at it on your own time, we did the same thing where we add events to the calendar. and I'm going to go I think up at the top here, I have a local host server that was set up for Temporal.

64:49 so, let me open this up and make sure I have the correct port. Okay, should be localhost 8233. So, now I will go back here and open this up. And so, when I go here, you'll see the interface for Temporal. And you can see all the things that we ran. and some of the things were like adding things to the calendar under a flaky worker. So, you can imagine, let's say my pick emoji function fails every once in a while, I can look at this retry workflow and you can see I started by extracting the event and then I had to retry selecting the emoji three times before I added to the calendar, which took two seconds and then it finally did it. And so you can do things like this to look at it and try to debug, look through the event history and see how the timestamps of everything and how things executed. So this helps you observe what happened and also it has like these robust mechanisms built in to retry when when parts of the system fail.

65:54 So I encourage you if you have some of your own time to to clone the repo and try this out or at least just look at the code. But you can see this is another example I guess of what a timeline will look like. So how like how it extracted the event and different different things to look at here. But I think I want to make sure we finish talking about all the things I have prepared for today. So I won't go through the code for Temporal at the moment.

66:21 So the last topic here is Okay, I wanted to talk about the different observability frameworks. So essentially we have other things like ML flow. You can see these are all doing similar things like it's showing you the inputs and outputs. You can look in and look at what actually executed. We have LangSmith. This one is related to the LangGraph ecosystem. So it's for observing Lang LangGraph programs especially. But you can see it ran you can see how many tokens it used, the latency. You can look at evaluation scores. So these are all about like debugging your system, understanding your system better. LangFuse is another one that you can use to measure like look over time like how reliable was your system, how many how expensive were things. So these are just This is the point of the observability frameworks. It's to be able to look and peer behind what how what is happening under the hood and and under your system.

67:22 So the recap is orchestration keeps agents reliable and observability enables debugging and monitoring. we pull best practices and tools from software engineering and these carry over into agent engineering. And then setting up the evals, monitoring, durable execution, that's the things that separate building agents from engineering agents. So last topic I want to talk about is just best practices and I felt this was important so I I did want to leave some time to talk about it, but are there any questions? I know, sorry that we had to skip the code demo for Temporal. Hopefully hopefully it's interpretable at home if you want to.

68:02 Cool. All right. so best practices first of all, I would say keep it simple. so you might not always need to build an agent to solve your task. in you may want to actually build a workflow instead or a compound system. There's a lot more reliability in and ability to debug a system when you have like a workflow. and so if you have a very niche thing like for all my emails I want them added to my calendar. That is a case where I feel a workflow is a good fit, you know?

68:34 whereas like if you have some more open-ended things like oh I want an assistant to manage my calendar for me, you know? That's a little harder to define maybe then you do need to build an agentic system. when you do build an agent, you should think carefully about what tools you should include in the agent. and if you have a lot of data, you actually might find yourself back all the way on the the left. It It does depend on the task like how many steps there are, but sometimes you might find yourself wanting to have something where I just have one input, one output and it's a in the model that's been specifically trained for this task.

69:10 I I think like an example could be something around like information extraction or something like that. Instead of building a workflow that like has multiple chain steps to extract a document, if you wanted to fine-tune a model to be really good at extracting information from a document, like that might be a case where you find yourself all the way on the left where you actually want to train a model if you have the data for it.

69:29 Another tip is to make failures observable and recoverable. And this is where the observability frameworks come in. So you should always be tracing the model calls, tool calls, state changes, errors, logging these somewhere. you should be setting timeouts, having retry limits, execution budgets, log the outputs, diagnose failures, and then just overall monitor the state of your system. An example here is in Claude code for instance. This is what I mean by one thing is like you can raise the error to the agent to try again. So you may be familiar I I'm not sure you can read this but basically like the model executed a tool call, it failed, and then the Claude code harness passed that back to the model and said like look, this tool call failed or like this code failed, like try again. And so this is another way you can make things recoverable in agents. So this is just a good practice.

70:22 An example of this is Shepherd. So this is again like meta harness, I'm showing a research example, although this this lecture is very like practical industry kind of focus, but here's another research example. So Shepherd defines these meta agents which will do which will do the observability layer for you. So it's actually an agent that does the observability and you can have your agent that does a tool call, it may have a bug, the meta agent can intercept this and say okay, try again. Fork this into another branch and then suddenly things work. So you have like a meta agent that that oversees what's going on and does this observability layer.

71:00 And so this is these are just different coding benchmarks again and you can see that they show that as compared to without having this like meta agent observing what the what the task agent is doing then you get lower performance. So you want to have some sort of like observability layer and that's that's the idea with this. The next tip is to run evaluations. So I think the best way to make design decisions about your agent is to ground them in evals.

71:32 and evaluations enable optimization. So, earlier I showed you we can define data for a task and we run an optimizer to make our system better at that task. This is all possible because we built an evaluation. You know, we had this defined that exact match score for our email data. And now we can actually optimize on top of it. So, building a proper evaluation is one of the most important steps in building an AI system and with your project proposals, I really really encourage you to deeply think about what the evaluation's going to look like for your project because this is like a somewhat underrated thing.

72:06 To build a good system, you absolutely need a good evaluation. Otherwise, you're just kind of shooting in the dark. Like I think this is this would be a good way to build this. So, if you have a good evaluation, you can test your hypothesis and see is it actually helping. So, spend a lot of time on your evals. Probably one of the first things you should do in your project too is really build a good eval.

72:25 And so, definitely think deeply about this. I'd also say choose your metrics wisely. Obvious things would be things like accuracy, but depending on your application, the cost, latency, all of these might matter too, especially if it's user-facing for latency. And then make sure that when you're testing, you test representative cases, diverse cases, edge cases, different failure scenarios, and put your system into these really challenging scenarios. So, don't just put it on the happy path because you might make design decisions incorrectly if you think it's always going to work a certain way.

72:58 one thing I'll mention is like it can be tempting to look at these like headline benchmarks that all the companies run their agents on and say that's what I need to optimize for. I would actually argue you should focus on your custom domain. So, whatever you're trying to build in particular like for your project or something, if you're trying to build an agent that's really good at helping you schedule your your your calendar. I'm on this because that's what we just talked about. Then you should build a calendar eval. Don't like try to show that your agent is really good at SweepBench because that's not what matters. Like that's not what you're using your agent for. Now, if it I I have this caveat, if you're building a general coding agent harness, yes, these are the right benchmarks because that's what the purpose of these benchmarks is.

73:40 but, if you have a specific domain that you're optimizing for, you know, that's the type of eval that you should be running. last sort of point here is on challenges and limitations. so, abstractions are lifting you further from the underlying system details. I showed you this with PyTorch earlier. somebody who just learned PyTorch can write, you know, a neural network without having to ever learn backpropagation. But, it's important sometimes to know backpropagation. And so, similarly, you could use the frameworks I just showed you today and write code for an agent without having taken this class. but, hopefully, you should, you know, want to peer under the underlying system and understand how it works. and I think it's important because you can build better systems when you really understand what's going on. And that is the point of this class.

74:28 So, I guess you already understand this, but it is a it is an important limitation to bring up about frameworks. Also, when building a complex AI system, it can be challenging to identify where the failures come from. So, I showed you very simple systems today, but let's imagine we're stringing together a dozen or a hundred language model calls. it's important to look at the traces, and you can use observability tools to help with this. You can also use agents to help you do this. So, make sure you try to really understand where failures are coming from and diagnose them.

74:56 and then, building on top of frameworks adds dependencies and reliance on external libraries to be maintained. so, the three libraries I showed you today are very actively maintained, and they can be good picks, but maybe some some library comes along that does something in a really elegant way that you really like. Another consideration is just like, you know, if I'm building this system that I want to last for 5 years, 10 years, more, is this library going to always be around? and is this always going to be maintained? So, there's a limitation there. And I should I have this little meme from a blog post that's kind of talking about some of the limitations of frameworks where it's look, you know, we have a framework that that does this magic stuff with LLMs, and it's like, well, you should understand it. Like, make sure you know what's actually happening.

75:36 Show me the prompt is the the headline that the blog post had. So, to recap, you engineer AI agents by building, running, evaluating, improving. This is like a cycle here. You build the agent, you run the agent, run your evals, improve the agent, and and just iterate on that. don't just think that the first system you build is going to be the one that you should ship. start simple and justify any added complexity. you should design for failure and recovery. Use tools like Temporal, for instance, that will help you recover from failures.

76:09 and think about what you're going to do if you're if a system fails. and then use evidence and evaluation to assess quality, cost, and latency. So, my point here is like, set up your evals, run the evals, make your design decision based on the evals. and just overall fully understand how your system works, because if something breaks, you know, it might have been an underlying thing. the same motivation I had for giving the lecture on LLMs, we should understand how the system works. Same applies here.

76:36 so, in summary, today we talked about what does it mean to engineer agents. we talked about frameworks for workflows and programs. We spent a lot of time on DSPy, so hopefully I've converted you all to be interested. we talked about open-source agent harnesses. we briefly talked about orchestration and telemetry. So, if you're interested in more there, I do recommend you check out the repo and and read the code yourself. and then lastly, we went over the best practices and and challenges that go along with using frameworks. So, that's the end of my presentation today, and I'll I'll remain after for questions.