Observing & Evaluating Deep Agents Webinar with LangChain

0:00 Welcome everyone as we get started. It's been a while since we've done one of these webinars, so we're excited to do them again. my name is Harrison. I'm the co-founder CEO of Langchain. joined by Nick, who's been here for a number of years and currently on the applied AI team. and we're excited to talk a bunch about stuff around evaling and debugging these new types of longunning or or deep agents as we call them. and so maybe some agenda setting and then and then some context.

0:37 So we we have about an hour here. I'll kick it off by doing about 10 minutes of some thoughts on kind of like evaluating these these deeper types of agents. and and introducing some of the new things that we actually launched yesterday, which I think are really cool. And then I'll hand it off to Nick after that, and he has about 20 minutes of a practical more kind of like hands-on showing how we use some of these tools to actually test, debug, trace these these these deep agents. and then after that, we'll open it up for questions in in the chat. So I think there so there is a Q&A box on the in the in the panel below. So leave your questions there and we will we will get to them at the end. okay so I think we can probably get started. we'll be talking about debugging and observing deep agents. We use deep agents in the title. By deep agents, we really mean agents that run for an extended period of time and do more autonomous tasks.

1:43 And so this is a the term deep agent is a term we came up with to describe things like claude code and deep research and manis these general purpose agents that would operate for extended periods of time. And and and when we saw that they had a number of kind of like common characteristics like they would all use a file system and they would all use sub agents and things like that. And one of the we I think it's also just reasonable to call these things agents.

2:08 Like these are what we think agents are. I think the issue is you know we we've been using the term agents for a number of years and agents haven't really worked and then I would say about six months ago they started to work and and and and by work I mean things again like claude code deep research manis and so the I think it's so we use deep agents as a way to kind of like differentiate these newer types of agents that we see taking off. and so as these things have started to become more and more common, we we've started to think about how the tools that we build to work with them have evolved. And so now I'm going to start sharing some slides that walk through a lot of our thoughts here.

2:46 so awesome. so what what what is different about evaluating first? So we'll talk about evaluating and then we'll talk about kind of like observing and debugging. So what's different about evaluating deep agents compared to like simpler LLM apps whether it be kind of like a single LM call or a chain. So when you evaluate a simple LM apps the typical thing that we see is you build a data set and this is a collection of inputs and outputs.

3:14 you define an evaluator. This could be correctness. This could be LM as a judge. It could be something. And then you basically run it in a loop. you you you take your agent, you run it over the data set, you run you get your your outputs and then you run the evaluator over the outputs and you can just run it in a loop. Deep agents are are different from these types of applications in in a few ways. So so one, they often times have state. So you're not just judging the output of the LLM, you're you're judging the changes that it's made to the state, whether it's if you think about coding agents because that's the most kind of like typical example. those would be files in the codebase. How how are those changing? the the output's oftentimes complex. so even if it does respond with natural language are now really long natural language responses or if it makes changes to code, it can make a ton of changes to code and in a bunch of different files.

3:59 So the so the output is generally way more complex than just like a single response from an LM. And each run needs bespoke logic. And so what I mean by this is that when you're testing changes to code, for example, like each test case might be, you know, a different task you want to do. And how you evaluate that task is is very different depending on task. Like in in in terminal bench which is a data set for test encoding agents. They they run a bunch of tests that are specific to each data point. So you now have kind of like bespoke logic and and so the patterns that we see for evaluating these types of deep agents are are the following. We see this bespoke testing logic for each data point. Each test case has its own success criteria. This is different than, you know, if you're just calculating accuracy of a classifier that's using the same criteria for all data points. This is different. we often time there there's multiple different ways to actually run the agents themselves. you might want to you might want to run them just a single step for individual decisions. You might want to do like a full turn and to testing kind of like the end state of what happens. but a lot of these agents are also conversational and so you might want to test the back and forth. And so how how does that happen?

5:07 And so there's multiple different ways to invoke these agents. and then third, this is related to state like environment matters. So you need kind of like clean reproducible test environments. the the main thing that we've been pushing here recently to help evaluate these agents is a pi test ingest integration. so this automatically logs inputs and outputs and full traces for each session or for each test. And so this is great. So you can track results over time. You can link failed tests to the agent execution. So if a test fails, you know exactly what happens. it stores these as experiments in Langmith. So you can track regressions and and and and the reason that this piest just integration is nice is because when you write a piest case, you just write code. You write code to set up the agent. You write code to call the agent. You write code to test the agent's output. And as we mentioned, there's multiple ways to run the agent. So you might want different ways to run the agent for different test cases. And then there's multiple logic or or there's bespoke logic for for each test case. And so you might want bespoke code for that as well. So piest and just piest for Python just for JavaScript provide really nice ways in in a pretty familiar software engineering paradigm to to write kind of like evals for these things.

6:21 So moving on to debugging deep agents. what what's different about debugging deep agents versus some of the simpler LLM apps? Simpler LM apps probably have a shorter prompt. So maybe you know a paragraph or something like that max and they have shorter trajectories. So maybe it's just a single call to an LLM. Maybe it's like a a chain. So it does like one retrieval call and then one answer. But it's generally simpler. When you talk about deep agents, they've got longer prompts.

6:44 You look at cloud code, it's it's prompts about 2,000 lines long. They've got longer trajectories. They can call, you know, 10, 50, 100 tool calls in a row. and and they're oftentimes conversational. they often times have a back and forth whether it's to ask is whether it's to ask clarifying questions or or the human correcting the agent or anything like that. And so this causes issues now. You you can't quickly tell whether a trajectory was efficient or whether it worked. Like you can't look at like a hundred tool calls and know whether it was being the most efficient or not. And same for prompts.

7:14 Like if it messes up, you don't immediately know which part of the prompt to change. If the prompt is just three sentences, you can seal that all in one go and immediately have an idea for which you know sentence to change. if it's a massive prompt, you don't know that. And so we've evolved evolved Linksmith in two ways to help debugging these these agents. we launched both of these yesterday. They're both new. We'd love feedback on them and Nick will walk through them in more detail. One is Paulie, which is an inapp assistant for for agent engineering. And then the other is Langmith Fetch, which is a CLI.

7:43 So walking through them, Paulie is a AI assistant for a engineering in Langmith. It's chatbased, so you can interact with it. there's three places in the app that you can use it. We wanted to we wanted to only add it into places where we thought it could be useful. We tried to be pretty intentional about where we put it. One is in the trace view. So you can ask questions like hey did this call the right number of tool calls? Did did it look like it was being inefficient in any tool calls? The next is in the thread view. and so here you can say things like was the user disappointed in the conversation? Were they asking you know did they have to repeat the question multiple times? And then the third is in the playground. So here you can say things like hey the the output is actually X. I want it to be Y. go change the go change the the prompt and in in a way where where where it makes it why and so it will go and and figure out what parts of the prompts to change.

8:33 Langmith fetch is the other thing. So so Paulie lives in the app. Langmith fetch brings lang to you specifically. It brings it in in the form of a CLI. you can you can you can pip install lang fetch and then there's two core workflows. One is like an instant pull grab the latest trace in a project. So if you're if you're developing an agent locally and you're just running it and it just messed up, you can you can pull down the latest trace and work with it immediately. And then and the second is bulk export. So grab a bunch of traces and dump them locally. So maybe you've run the agent a bunch of times, maybe you want to look at production data. maybe you want to look at some tests you're running. Basically pull down the latest 10, 100, 20, however many traces and then and then put them in your file system. And that lets the agent that lets the coding agent that presumably you know everyone's using coding agents these days that's let the coding agent work go over them inspect them see what's going on and suggest changes to the agent which also lives in your codebase so it's bringing these things into your codebase so it can start to edit code based on that that's the highle things that I wanted to share again like we think we think evaluating and debugging agents are different for these longer type of deep deep agents we have introduced a number of things piest Paulie and then Langmith Fetch to help with this. to make it all more concrete, I'm now going to hand it off to Nick, who's going to show you how we actually use these things.

9:58 Cool. let's get into it. I'm going to go ahead and share my screen here. if you guys can see the code. today I'm just going to walk through a pretty simple deep agent. There's actually a quick start for this agent in our deep agents quick starts repository if you're curious to follow up later. This agent is just a personal assistant. but we're basically going to walk through the full implementation of this personal assistant, the prompts that we give it, the different tools it has access to, and one specific sub aent for meeting scheduling. Once we have a good idea of how our assistant works, we'll run it over an example, and then we'll pivot over into Langmith, take a look at the trace, and then use Poly, use Langmith fetch, and use another, pretty interesting tool to try and make our agent better.

10:43 Cool. So for those of you who know deep agents or or have some familiarity, it's a pretty simple agent harness. So a deep agent is just a React agent, but it comes opinionated with a set of tools that we've seen to be pretty helpful for making agents performant in a lot of different situations. And one of the most important parts for a developer to get right for a deep agent is the system prompt. like Harrison mentioned, system prompts for a single chat application might be be quite simple. but we see system prompts for deep agents being quite long and quite detailed. you can give heristics on how the agent is supposed to work. You can give concrete examples of what it's supposed to do in different scenarios. and all of this is pretty helpful to bake in. And so just taking a quick look at the system prompt for our assistant here. I'm giving it a bit of a persona. I'm telling it it's a top-notch assistant for me. I'm giving it a little bit of background about myself. I live in New York, pretty busy. and I'm also giving it some pretty concrete instructions on how to handle incoming emails. So, that's how our personal assistant is going to work.

11:49 It's going to trigger when I get a new email and it's going to determine what to do with it. And so, you can see here that I've given it some triage instructions. There are a class of emails that I don't think are typically worth responding to and that I don't need to look at. And then there are emails that I think are definitely worth responding to where I want this agent to go ahead and write a response for me.

12:10 This agent also has the ability to do other things as well. So in addition to writing emails, starting new threads, it can call a sub agent that's focused on scheduling meetings. So the sub aent has access to my Google calendar and can schedule meetings for me. Or if it doesn't think I should look at an email, it can go ahead and mark it as red. The second prompt here is just for that sub aent. And so we'll see this all come together in a second, but this sub agent has a different sort of purpose than the main overall agent. We really want it to just focus on scheduling meetings. And this is intentional, right? We don't want to give the main agent too much to worry about at once.

12:50 And so whenever someone wants to schedule a meeting, this main agent will kick it over to this meetinguler sub aent. And this sub agent has two specific tools to look at my calendar and then to schedule meetings on my calendar. I also gave it some guidelines to follow about different times to schedule meetings and and how long to schedule them for. And so line 79 to line 93 here, this is pretty much the bulk of of creating the actual deep agent. And intentionally it's pretty short. We have create deep agent which is a factory function from our deep agents package. We have a few Gmail tools which I've implemented in a separate file. but these just allow us to to talk to the Gmail API and actually take actions. What we do here is we create the deep agent. We give it that overarching system prompt with instructions for myself and also on how and when to call its tools. We give it the one particular sub agent for scheduling meetings. So this sub aent is really good at scheduling meetings and interacting with my calendar. And the agent also has a few other tools around writing emails, kicking off new email threads, and marking emails as read.

13:59 And so just to zoom out here, I think the point that I really want to to hammer home is that deep agents are really quick to set up in in this sort of fundamental basic form. but this is really just the start of the battle, right? Like we have the the question now becomes how do we make sure that the agent actually works the way we want it to? How do we know that it's calling the right tools in the right scenarios?

14:23 and how can we also improve it over time against specific inputs? and so like most developers would do, I think the the natural next thing is just to run our deep agent over an example and and see if it does what we want it to do. and so here I'm just going to kick off our deep agent against a sample email thread. this is from my good friend Oliver Queen. and Oliver Queen wants to talk to us about Deep Agents. He's super interested, wants to chat about it, and wants to schedule some time to chat at 8 a.m. next Monday. so we're just going to go ahead and run this agent.

15:03 and what I've done here is basically just packaged this email into a single message for the agent to take in. So, it's just a simple message that says, "A new email came in. Use your tools. Handle it to the best of your ability. And what I've done here is I've tracked the response. So this is an agent running under the hood. We're going to get a full response back about how it handled it. and I'm going to print this out just for comparison because I think a default for a lot of developers, myself included, is is just print debugging.

15:36 What I didn't mention earlier is I have set in my environment the Langmith API key and I've also turned on tracing to Langmouth. and so if I swap screens over here to Langmith, I can see this trace is ongoing here. So this trace that's currently running has to do with the question that I just asked, which was an email thread came in. Handle it to your best ability. And we can see the content of the email here.

16:01 and like Harrison mentioned, right, like this trace for the steep agent is pretty complex. It's pretty long. it would take me a decent amount of time to click through each of these individual parts and parse through manually what's going on. And so that's why I'll open Poly over here on the side to see if it can do the job for me. Poly has a few default prompts. So I'm just going to go ahead and ask it to summarize what the agent did in the trace.

16:25 At the same time, we can click through and try to get an idea of what went on. so the model we're using here is Claude Sonnet. we called this task tool. This task tool kicked off our sub agent to go ahead and look at my availability to see if I could schedule the meeting for Oliver. And even just in the time that it took me to look through this one step, Paulie was able to quickly pull all of the information from these traces and give me a quick summary of what what the agent actually did. so we can see that the agent checked calendar availability, delegated to the sub agent like we just saw. It then drafted and sent an email response and then it marked the email as read for me so I don't actually have to handle it myself.

17:09 It also gave me some nice metrics like the agent successfully completed in 46 seconds. And so this is pretty cool. in the UI instead of having to click through each of these different pieces myself, I can get a nice summary from Py. Poly is also just a generic chat, right? So, if I had follow-up questions what, you know, specific days did it check or things like that, I could also ask Polly and, get some answers back there.

17:37 And so, Harrison brought up this point earlier, but what we've done so far is constrain to this this UI experience, right? We're using Langmith to get a really good sense of exactly what our agent did. and we can use Poly to give us a nice summary or ask specific questions about specific parts. but a lot of the development that actually happens for our agent, right, happens in our code. You can see here for comparison, the print statement is pretty pretty hairy and I probably wouldn't want to debug this myself.

18:07 And so that begs the question, how can I get the information that I see in Langmith into my terminal so that it can be used by coding agents like cloud code or the deep agent CLI so that I can improve my agent directly while working with a coding assistant. so I'm going to go ahead and use lang fetch here. it's just a package that we can install. So if we try to install lang fetch we can see that I already installed it. And now lang fetch work which works to pull the latest trace from a particular project. so what I'm going to do is I'm actually going to come back into lang and I'm going to take a look at the tracing project that I'm currently logging to which is for this personal assistant.

18:52 I'm going to go ahead and copy this ID here and I'm going to set this as the config that I use for linksmith fetch. So I'm going to do UV run lang fetch config set project uid and I'm just going to set this project as the one that I want to get traces from. Now if I run uvun linksmith fetch traces, it's going to go ahead call the linksmith API under the hood and get the most recent trace for me and pretty print it in this nice format.

19:27 and like I mentioned, right, like we can see this already in a much much nicer view by just going into Langmith. but this is really helpful for a coding agent to digest. So, I'm actually going to go ahead here and kick up our coding agent, the deep agent CLI. For those of you who haven't worked with the deep agent CLI before, it's super similar to Cloud Code. It's a coding agent. It has access to all my files, and it can suggest and make updates to those files.

19:57 And so I've given the the deep agent CLI agent a few instructions already. but I can just ask it here. Can you go ahead and fetch the most recent trace from Langmith and summarize it for me? And so just like other coding agents, there are different modes that you can run the deep agent CLI in. Right now I have it in approval mode where I want to approve every action. but I could also run this in yolo mode and and let it cook for me.

20:30 So, I just approved the action here and so it's running lang fetch traces in the background and it's just given me the summary. It's told me the different tools that the agent used. it's told me the overall workflow and what it just handled. Now, let's see if we can actually improve the agents behavior in this case. So, I maybe like some of you don't really love waking up too early. so I'm going to tell the agent that, hey, I actually prefer to sleep in.

21:00 I don't want to take any meetings before 9:00 a.m. Can you update my agent to adhere to this? And can you also write a test to make sure my agent follows these instructions? So, let's go ahead and see what it does. We can see that it's taking a few actions here. So, it's reading in two of my files. It just read agent.py and also test assistant.py, where I've defined my tests.

21:33 And now, cool. So, it's suggesting an update to my prompt. And it's basically just writing a new line here that says very important. Nick wants to sleep in. Nick doesn't want to wake up. Cool. So, that has now been edited in my actual system prompt for my agent. now it is working some more. and in a moment we'll see what it comes up with. Cool. Just to expand this a little bit, we can see now that it's updating and adding an actual test. this is a piest test. And looking carefully, I'll I'll go ahead and accept this, but we'll we'll take a closer look at exactly what code is in there. if I open up my test assistant.py pi and scroll down to this new code. I can see that it's a test specifically designed to test declining early morning meetings. I can see that the example looks super similar to the one that I just inputed. and I have a specific success criteria for this test case. So, the response should politely decline any early meetings, in this case the 8 a.m.

22:41 meeting time, and respect my preference. Then there's a few different ways I'm actually going to test the agent. And I'll go ahead and and kick off the test here so it can run while we're walking through this. I run the agent here. I run it to its end state. So I I take the email, I format it into that same input message. And then I get my response. From that response, I'm going to extract the tool calls. And there's a few different things I'm going to assert here. One, I want to make sure that we actually wrote an email response to the user. That's very important. If we didn't do that, there's no point in in using an LM as a judge and and we fundamentally made a mistake. So this is a very deterministic check that we can take based on the end state of the agent.

23:25 Then I'm going to do something softer. I'm going to use this evaluate tool call method that I wrote. And what this does under the hood is it passes in the success criteria and it actually uses an LLM as a judge to evaluate the result. So we we use an LL LLM as a judge with structured output to see if we get to see if the final output from the agent fulfills the result criteria. And this is really powerful, right?

23:51 Because for my different test cases, I can write different specific success criteria and have the elements as a judge judge these different criteria for each test case. It's also neat, right? Because I can assert different tool calls with potentially different tool arguments for my different test cases, too. Awesome. So, we can see the agent says, "Great, the test passed. Now, let's fetch the trace to see exactly how the agent handled it." we don't need to do that, but if we go over to Langmith, we can see that we ran this test case and it got logged here, in my tracing project as well with this input.

24:26 And in this case, we can see that it wrote an email response. And actually, it says, "I'd love to chat about it. However, I don't take meetings before 9:00 a.m. So, I've updated the behavior of my agent. really using the AI tools to help me out. So to kind of recap what we just went through here, right? We started with a pretty simple we started with a pretty simple agent, just a personal assistant agent. we wrote it with deep agents in just a few lines of code and then we tested it out on a pretty simple example and its behavior was was pretty reasonable. but there was a specific customization that I wanted to add on top of it. to do that I used deep agent CLI. locally the coding agent. I gave this coding agent access to lang fetch so that it could find the latest trace and understand the information from it. and then I asked for a specific change and it made that change, wrote a test for that change, ran that test and logged that back to Langmith. And in the case that this hadn't passed, right, the agent would then be able to continue iterating and it could fetch the more most recent trace to try and figure out what went wrong according to the test and how it could then update the prompt or edit the tools to to make it better.

25:53 Cool. that was it for the the practical example. so I can stop sharing my screen and we can maybe take a few questions. I can help read some of them out. all right. Let's see if we can do this. I clicked answer live for this one. Do you see anything pop up when I click answer live? >> I don't. >> Okay, I'll click answer live and then read out loud. Can poly eventually be used in the terminal so I don't have to tab back and forth and within the browser debug and cursor? Much smoother flow coding agents. I think this is where linksmith fetch comes into play.

26:35 so we don't currently have plans to expose poly itself in via the API, but we have linksmith fetch for that. how will one evaluate the multi- aent without a curated QA pair? Nick, do you maybe want to talk about kind of like evaluation strategies in general and how you might think about evaluating without yeah, a ground truth? Yeah, I mean I think there are a lot of different types of ground truths, right? So it kind of depends on the type of agent you're building. If it's a more like open-ended agent, so like let's say research or coding, there might be specific criteria that you want to assert your your final answer again and that might not just be a ground truth.

27:16 so for instance, for a coding agent, right, you might want to actually run the file that the agent came up with to see if it works. For a research agent, you might want to make sure that sources are cited properly. or some other success criteria like one or two specific facts were mentioned. so I think it really depends. You don't need a full ground truth answer in a lot of cases. Sometimes you just want to make sure that one particular thing happened for your agent.

27:42 >> Cool. let's skip around to okay this one from Cameron. Are those deep agent trace spans automatic? if you set two environment variables, they start tracing to Langmith automatically. Yeah. So you don't have to do anything else to your code. and yeah, we we do have docs on tracing. If you if you look at lang tracing and then lang chain integrations, you can you can see it there. does poly run with its own API key or do we need to provide one in order for it to work? So this runs with your own API key. We did that for a reason so that you don't have to worry about any data sensitive things coming coming to coming to our models.

28:27 so yeah, you provide your own API key and then it runs. do you suggest using deep agent CLI for lang chain related things rather than clawed code? I don't know if you have a take on this, Nick. I'll give my quick take which is basically I mean deep agent CLI is right now there's nothing about it specifically for lang chain related things but we are thinking about how to make it easy to customize things like deep agent CLI so that you can have like a CLI that is really good for lang chain things and so I actually view deep agent CLI as more of like a scaffolding for other maybe like more specialized coding agents in the future. Nick I don't know if you have anything else to add. Yeah, I think that's spot on. I mean, one thing that I've struggled with a lot with cloud code is that it gets import paths wrong a lot of the time, especially for more or like less adopted libraries. and so I think one thing that we could do that's really interesting is is have specific profiles that you can pull with deep aent CLI and get a coding agent that's really good at one thing.

29:42 >> are there any specific features that are only supported for Python versus TypeScript? I mean off top of head, so lang fetch is right now only a Python client, but it works for traces whether they're coming from Python or TypeScript. deep agents is both Python and TypeScript, but we do have some differences between the two. Nick, I think you've worked more on the JS side of things. Do you know the differences between deep agents and Python and TypeScript? I think it should be pretty up todate in terms of the the core functionality. so if you see anything, please open an issue on the repo. and it'll get addressed.

30:21 >> Cool. I'm thinking I can just abandon cloud code and use deep agents to develop an agent instead. presume I need to set up with API keys. Yeah. So and one of the cool things that we do, I think we support right now OpenAI, Anthropic, and Gemini. So you can also use it with other models as well. how does Langmith differ from other agent observability and eval tools, i.e. langfuse? I think there's a few things that we've seen. I think generally we've pushed the boundary a lot in what you can do with agents. so I think we're generally we're very scalable because agents have a lot of data that comes in and so my we have we have a whole team kind of like dedicated to working on on scalability. And then I think also there's a difference between tracing like single LM call apps and like these really complex agents. And I think these complex agents didn't really exist until like 6 months or ago or nine months ago.

31:14 And so I think everyone including us prior to to that was basically really focused on simpler LLM apps. and now only now I think are we starting to to tackle these more deep agent style things. And I think I think we're doing a lot of work there that that other folks aren't. can we use our own models to run poly/deep agent CLI or do we have to use one of the frontier models? so for I'll answer for poly and then maybe you can answer for deep agent CLI Nick. But for poly one of the things we are adding that should be in prod today or tomorrow is the ability to switch between different models. Right now, we only support OpenAI and Enthropic, but it's an OpenAI like compliant endpoint.

32:04 So, it can be any other endpoint that's running whatever model as long as it's OpenAI compliant. So, we've seen a lot of the open source models be hosted on platforms that expose them as OpenAI endpoints. And so, you could plug those in. >> Yeah. Right now for deep agent CLI, I believe we run with the frontier model providers. So, I think you can use OpenAI models, anthropic models or or Gemini models. but configur configurability is something that we're looking to add in the CLI in general.

32:33 >> Do you plan to support skills middleware as part of core deep agents live itself and not part of deep agents CLI only? yes not sure what time frame but yes I think skills bythropic are very interesting and we want to support them. is linksmith open source or are there or is there any cost associated with it? Can I use it for my local development? Langmith is not open source. It is our commercial offering. there is a free tier, so you can absolutely sign up. And if you're just using it for debugging, you should be able to use just the free tier. pass a certain number of traces. that's where there is some cost associated with it. And there is an enterprise option to deploy it on prem. but that's that's generally only an enterprise option.

33:21 how about difference with arise to enterprise tools? I think similar to the answer to langfuse I think it's easy to monitor a single trace or a single trace of a simple app but I think it's harder to monitor lots of of traces whether that's millions or billions coming in and do it respon do it snappily and also do it for these more kind of like longer and more complex deep agents. >> there's a there's an interesting question here about the example. in the case of a deep agent, let's say the agent did a couple of tool calls that are useless or has wrong parameters but eventually passed. How will the deep agent based eval catch that?

34:00 basically the length of a trajectory. so this is really interesting, right? So sometimes right you as the developer of the app have a strong opinion of given some sort of inputs, what exactly should the agent do next? And so this goes back to what Harrison showed about running an agent for a single step. You can just run it for that one chat turn. You can see what tool calls it makes and you can assert that it essentially calls the right tool or with the correct arguments. There are other cases like in this email example where I kind of don't know how many dates the agent is going to check, right? I mostly care that at the end of the day it does schedule a meeting for me and it doesn't really matter to me like what particular day as long as I'm free. And so that's a different thing, right? Then I have to run the agent end to end. And what I can assert against the trajectory is just that at some point it called the schedule meeting tool. And I don't actually care about the arguments. So I just called it I just care that it called the tool.

34:55 Cool. awesome. maybe we can maybe we can take turns choosing. There's too many questions. So maybe we can take turns choosing questions that we think are interesting and answering them because otherwise I'm just going to be reading down a list. One that I I I like here. Are you planning to add poly at the project level to summarize the most common problems the agent has or check multiple traces and and and diff? So, we're thinking a lot about where to add poly. we want to be really intentional with where we add it. We don't, you know, we don't want to add AI just for the sake of adding AI. And so, the three places we add it like threads, traces, and playground. even there we started with like limited functionality. So like you know you can't you know add an evaluator in the playground because we really wanted to focus on like what we thought was like the most critical stuff there.

35:46 some of the other places we are and and the other thing I'd say is that we are thinking about agents in other parts of the app that may actually be like longer running more background things. So for example we have an insights agent. So in in the insights functionality we have is an agent. It is not part of Poly. Why isn't it part of Poly? Because it's just a pretty different UX. Like you insights will run for like 5 10 20 minutes in the background. It analyzes like it analyzes like all the traces. So that's kind of similar to what you're asking here to summarize the most common problems the agent has. This is I'd use the insights agent for this. And this just isn't really something that you'd probably want to like have in a chat. It's more of like we have a dedicated UI for it.

36:27 We we list the jobs that have run previously. and and and so that's kind of how so so the answer would be like we are thinking of putting in other places of the app. We don't just want to add it everywhere willy-nilly. We will also have these other types of agents that make more sense for what we consider like longer running types of things. Another example of this would actually be optimization. So we're thinking a bunch about how do you hill climb on examples and should that be part of poly in the playground or should that be like part of these like async background jobs? And we're actually we don't actually have a super strong opinion on that yet. We're thinking through that live. But like we're planning to add a lot more AI and and then Langmith fetch we will add more things in as well to pull down experiments and and maybe even pull down annotation cues and things like that.

37:16 >> Cool. I I see a few questions here that are pretty quick. So does Linkmith support human annotation? Yes. Linksmith has an annotation cues feature that's specifically dedicated to that. is deep agent CLI free to use? Yes, it's free to use. you just need to supply your own API key. but then you can use it for free and install it for free. another interesting question here, what sort of memory is served with the deep agent CLI? or is there a way to customize the memory when prompting? so yes, the deep agent that runs as a part of deep agent CLI pulls part of its system prompt from a file called agent.md which it writes locally on your machine. And so what you can do is you can open that file yourself and you can add instructions which is what I did earlier today by giving it some instructions on how to use linksmith fetch. what you can also do right is you can prompt the agent to remember something. So maybe I tell it hey I have soccer every Tuesday at 6. this is something that the the deep agent can then write to its own memory files because it's it has access to the file system. And so instead of you actually cracking open that file and writing it yourself, you can talk to the agent and the agent has the ability to write that as well.

38:32 one here which is maybe more open-ended as well. Can you use tests to check things like token or context use? which maybe I'll even generalize to like what types of things can you test in in in tests? I' I've seen someone I I I mean you can test a bunch of stuff. So like you can test latency, you can test token use, context use. I actually haven't seen I mean I think we've seen people test functions that like create context. I think we've seen people test like hallucination and groundedness in context. I I don't know if we've seen people test that like, oh, I run an agent. Does it ever pass like 20% context fullness or something like that?

39:16 anything else that you've seen people test, Nick? >> sorry, I missed I missed that part of the question. I was looking through the, the question list. >> Can you use tests to what what do people what what do people test? So, like I've seen people test like latency. obviously like accuracy, groundedness, you can test for things like, you know, prompt injection, like security related things. any other any other types of things that you've seen people test for?

39:45 >> Yeah, I mean I think content is a big one, right? So if you have like an open-ended chat application, you can have some guard rails that run as some sort of online evaluation to filter out questions that you deem don't fit content. That's that's another one I've seen. >> Cool. maybe one other quick one that I'll answer and then I'll let Nick go. if we use if we use fetch to trace, do we still need Langmith subscription? Yeah. So, so Langmith fetch fetches data from Langsmith. So, the idea is basically, yeah, you still you still send all the traces to Langmouth and so we'll need a subscription for that although mentioned we do have a free account and then you just use fetch to bring it to your coding agent.

40:24 >> a really good question I see from Manuel and and Bruno. what are your thoughts on the side of building the evals for complex queries carried out by a deep agent where there are multi-step tasks with many points of failure? Any advice here on good practices on building the test sets? for me this is this goes back to running agents for a single step at a time. so you can always model the inputs to an agent as some sort of list of messages that are going in. Right? So maybe instead of letting a rate an agent run for multiple steps at once and then evaluating the end result, you just run the agent on the first input. it produces the first output and you can have a set of assertions on that first output that have to pass before you give it the next input. And so you really break this multi-step problem down to a bunch of single steps and you can have assertions in between these. And so that way also if the agent does go wrong and it if it goes off the rails, you're not wasting a bunch of tokens or money with with the agent doing things that are are totally wrong, right? you catch it at the first point of failure and then you can work on optimizing the agent until it passes that and then you can see if it passes the second, the third and so on.

41:40 one other question that's pretty easy answer is there a way to host a deep agent so that my team can access it and ask questions through a shared interface. yes there's two ways actually. So like one we have lang deployments. Langmith deployments is where you can deploy langraph langchain or deep agents or or any agents actually and it's agents that you build in code and then we have we what we call langraph studio which is where you can interact with these things after you've deployed them. We also have a noode agent builder in Langmith that's actually built on top of deep agents. so we didn't do anything in no code for a while. but we just recently did. And what we did there was we, basically took deep agents and we put it so that you could build it in a UI because deep agents are just a prompt plus tools. And so we put this in in the UI and now you can build these things in a no code way. And then you can also interact with them and share them. And we have workspace agents. So you can share them with other people in your workspace. And so this is still in beta. And so we'd love any feedback here. but yes, that's that's how you can kind of like share and access these these deep agents.

42:44 >> one more open-ended question that that I think is pretty interesting. What feature within like Smith have you seen resonate the most with enterprises? I can speak more for myself. What what feature resonates the the most with myself? but I think it's tracing, right? I think tracing is still super undervalued. I think eval is a buzzword that everyone is throwing around, but I think at the end of the day, right, if something doesn't pass your evals, you need to understand why it didn't pass your evals. And so, you need to go look at the trace or you need to have poly go look at the trace or you need to have deep agent CLI with lang fetch go look at the trace and you have to understand very granularly where your agent went wrong. so I think like when I at least when I think eval I don't think like large benchmarks. I don't think like 100 tests, right, that that are running and you pass 50 of them. I think a small blur set where you want to have 100% accuracy or passing and and making sure your agent does accomplish its its core competencies.

43:43 and before Harrison jumps in, I see another one from Shrih Harsha. Any ideas to make a swift or Android based lang chain framework on deep agent so that we can run the agents on device with an ondevice LLM? I don't know about ondevice LM, but I talked to Harrison about this a few weeks ago, but I think mobile is a super interesting UX. I've kicked off cloud code from my phone and and had it do stuff for me while I was off hiking or whatnot. so I think this is something to explore and is is super interesting to me.

44:14 >> Yeah. And and maybe I can add one thing to Nick's answer around what have enterprises found most resonate most. What one surprising thing to me is the amount of people using Langmith for kind of like for compliance actually. so we we didn't build Langmith to be like a compliance tool but a lot of the functionality that it provides in terms of like showing everything that happens inside or testing and benchmarking like that's exactly what compliance teams are looking for. They want to know exactly what happens inside the app so that they can ensure that nothing nothing bad is happening and they want to know how it performs on different things. And so we've heard multiple times that that Langmith is a really useful tool for ensuring that agents are compliant with with any regulations or things there.

44:56 and and then maybe one quick one I can I'll answer are any plans for Langchain Academy courses related to this to show off best practices in Langchain's opinion? we pro it's still so we're doing a bunch of work on like evaling deep agents and so no I no plans until now but it's a good idea but we do have a lingchain academy course on lang in general so we have lang essentials which is which is like a one-hour course and then or we have lang quick start which is a one-hour course and then lingsmith essentials which is a like three or four hour course and I think Nick actually taught that that the longer one >> back in the day >> back in the day and so yeah, for for Langmith kind of like like there's so much in Langmith that we actually didn't show during this this webinar. and so for a full deep dive, I definitely encourage you to check out the Langmith courses on the the Langchain Academy.

45:50 I'll I'll I'll drop a link to the academy in the chat. If people didn't know about this, we have a bunch of educational resources on Langchain, Lingraph, Lang Smith. and we spend a ton of time there. >> two two kind of related questions. is there an online evaluator option? and if so, how do you deal with streaming and and latency? and there was another question here that I I think I lost. oh yeah, in in multi- aents and orchestration, is there an eval test to prune agents, removing them when not used or or not performing? I think both of these actually really lend themselves to to online evals. for those who who are less familiar with the term, so online eval when your app is in production already. people real users are using your app. and those traces are showing up in your tracing project. Every time a trace shows up, you can run an online eval to sort of figure out what was the intent of the user. Did a certain thing occur? Did it occur under a certain time, etc.

46:51 yeah, I think online evals is super interesting, especially for deep agents, right? You can track how long they're taking to respond, right? That's super important for UX. you can track how many tokens are being used. you can make sure your research agents or your your more open-ended agents aren't churning forever before converging. so yeah, I think there's a lot to be done with with online eval and and langu supports that quite well. any additional features for fetch poly being developed? the two that are top of mind, one we want to let fetch pull down results from experiments. So right now it can pull down traces and groups of traces. is we want to let it pull down groups from from experiments. and then for poly, the big thing that we're iterating on is actually having it do some of the prompt optimization. So right now it can edit the prompt, but we want to let it edit the prompt and then rerun the the the whole playground to evaluate over the data set and then get the results and then see how see if they changed and then change the prompt again and and then then iterate on that. And so that's probably those two are the two things on the immediate term road map there.

48:03 I see a question as well can can fetch be used for monitoring deep agents runs let's say with a custom tool. So that is one avenue right you could give the deep agent a custom tool just to fetch traces from langmmith. The reason that fetch was dei designed the way that it is is because deep agent CLI has access to a shell right so it can just run the lang fetch command instead of of going through a tool. if for some reason you wanted to design an agent that wasn't running in a shell environment and didn't have access to that how lang fetch works under the hood is it uses Langmith's APIs to actually go fetch runs and traces and so that is something that you could add as as tool functionality as well.

48:53 can fetch be used for non-deep agents like a little simpler langraph? right now a lot of this functionality is optimized for agents which have like a messages list basically because we've seen that that's you know that's a very canonical way to represent the trajectory of these agents. if it's a more custom agent, then if it's a more custom langraph thing without messages, then even though it might be like shorter or run faster, it's actually more complex than some of these simpler things. So like fetch and poly are still there still like like the core like deep agents is actually a pretty simple agent under the hood. It basically just runs in a loop and calls tools and accumulates a list of messages. And I think that's like a a dirty little I don't know if it's a dirty little secret, but it's like the the the most like impressive agents actually use like some of the simplest kind of like cognitive architectures out there.

49:48 and and so a lot of these tools are really optimized for when you have this like tool looping architecture which is which is simple and is easy to deal with. and at least right now that's the focus. I do think poly works on anything. So poly we've given more latitude with but lang fetch we've we've we've kept a little bit more focused. I see a few questions about multi- aent setups. so in the past we've had more packages with more specified handoffs right where you have one agent specifically handing off work to another agent and then waiting for that to return. now this is really modeled as tool calls and I think this is inspired a lot by how we saw Claude code and deep research from Claude that that great blog post a while ago in Manis how those tools work. the agent retains the right to spawn sub agents and hand off work to them but the work is returned in the form of a tool message result. and one of the best benefits of this is context isolation.

50:50 So you could take a task that the main agent could accomplish on its own, but it might be a fairly isolated or siloed task, for example, like researching a particular topic, and researching that all in the main context window is really going to bloat that context window so that when you have your final response, you're going to have a bunch of ugly web search results sprinkled throughout. and so by handing that off to a sub agent, right, it's functionally a sub agent with the exact same capabilities, but the context is siloed. So the sub agent can do all of the work and it can return one final nice report to the main agent without returning all of those additional tokens in context.

51:32 one question around prompt optimization, reprompt optimization, DSPI integration, co-functionality or jeepa. mini rant here. Jeepa is a really fancy term for a really simple concept of just letting the LLM suggest improvements and then and then using that prompt and testing how well that prompt does. And so, yes, we're doing variations of that. I think the idea is like look relatively simple and so we probably won't be integrating with with DSPI just for that. but it it's yeah it's essentially Japa under the hood.

52:10 >> I see one comment here. deep agents how like how do deep agents run commands and and how do they they fire things off? So one chief benefit of deep agents is that they can run things in parallel. And so this can include tools, but this can also include sub aents. so maybe the canonical example that I always bring up is if you've got a complex research task and you're asking you know, who's the goat? Is it LeBron or Jordan?

52:36 instead of doing these sequentially, right, the deep agent can launch two sub agents in parallel, one of which will research LeBron, one of which will research Jordan. and then it waits for both of those results to come back before coming up with its its final answer. but yeah, parallelization is is really powerful for agents. and it can speed up the user experience a lot.

53:11 there's a there's a lot of questions around deep agents and so maybe rather than try to answer all of them because we are short on time and we should do a deep agents webinar in the future but high level for deep agents we're very very bullish on deep agents if you want to give If you want agents that are more autonomous or you're working on workflows where you need more autonomous agents, then you should probably switch over to deep agents from whatever you're doing now.

53:40 there are definitely cases where you still need workflows. but but I would actually argue that most of the things where you might think you need a workflow, if you're thinking about it in the context of an agent, it's actually probably better to use something more more autonomous like like deep agents. we're investing in a lot in both Python and Typescript. we think it provides way more value than just abstractions. So like a big value ad of Lingchain and other agent frameworks out there are the abstractions. deep agents has more of like a batteries included approach. So it provides like a planning tool and sub agents and things like that. so there's a bunch of questions about deep agents. I'd encourage people to check out the Python and TypeScript repos and leave issues or questions there. We probably won't get to them all here.

54:23 I think we're probably good on time unless there's any final things that you wanted to answer or talk about, Nick. >> no, I think it's good for me. Thanks. Thanks everyone for coming. really appreciate it. >> Cool. Awesome. Yeah, thank you all for joining. there will be a recording of this that we'll post later and we'll do more of these. So, hopefully see you in the future. Goodbye.