Building & Observing a Deep Agent for Email Triage with LangSmith

0:00 In this video, I'm going to show you how I use Langmith's new agentic features to help me build agents. We're going to use Langmith to help us build a simple assistant that can triage and respond to incoming emails. So, every time I get a new email in my inbox, my agent is going to read it and figure out what to do with it. We'll use Langmith to help us make sure that the agent is doing the right things. Let's start by looking at this agent.py file. In this single file, we're going to define a simple deep agent.

0:29 First, and also most importantly, we have our system prompt. Most deep agents have pretty extensive system prompts. You move a lot of the complexity out of the agent architecture and into the actual prompt itself. This prompt has some background information about who I am, and it also has some criteria and rules for how to handle incoming emails. When an email comes in, the assistant can take a bunch of different actions with its tools. It can write an email response. It can kick off new email threads with new people, or it can call a sub agent, which we'll look at more in a second. It can also just mark the email as read if it doesn't think I actually need to look at it. Our sub agent here is specifically focused on interacting with my calendar. It has specific prompting around how to find meeting times where I'm available and also book the meetings for me. And so with these pieces, we can assemble our deep agent. We have that overall system prompt with some background of myself and the direction of the different things that I want the agent to do. We give the main agent some general tools to write emails, kick off new threads, or mark emails as red. These tools are all defined in a separate file, but they basically just connect to the Gmail API and let our agent actually take actions.

1:41 Then we have that sub agent called the meetinguler. And this has access to a few specific tools for calendar interactions. Finding available times on my calendar and then scheduling the meetings. And so with just these few lines of code, we've built a simple personal assistant. What I really want to emphasize here is that it's really quick, easy to put together a deep agent. But this is really just the first step. The question now becomes, how do we make sure the deep agent actually works? How do we actually make sure it's doing exactly what I want in different scenarios? And how can I improve this deep agent over time?

2:18 The natural next thing to do is just to run the agent. So, let's do it. We have an example email here coming from a friend of mine, Oliver Queen. Oliver is emailing me about wanting to talk about deep agents, and he suggested that we meet at 8 a.m. next Monday. We can take the agent we just created and invoke it on a single input message. This message just says, "Hey, an email came in. Handle it to the best of your ability."

2:43 The agent will take this input email and call whatever tools it needs to handle it. Now, we do have a print statement here that'll print out the final state output of the agent. Let's go ahead and wait for it to finish. We can see the printed output of the agent here, and it's super hard to read. It's really hard for me as a human to parse through this. As an alternative to this print debugging, I've actually set this agent up to trace to Langmouth. All I did was set my Langmouth API key as an environment variable and set Lang Smmith tracing to true. And so now when I ran this code, all of the agents decisions and outputs got logged to Langsmith.

3:24 Let's go take a look at the trace. Langmith is our observability and eval platform. One of the first things you can do with Linksmith is set up tracing. And clicking into our most recent trace here, we can see exactly the input that came in, that same email thread from Oliver. We can also now see all of the actions that the agent is taking on it. One important thing to note about deep agents is that they have some built-in tools, including the ability to call sub aents through this task tool. The sub aent can then call its own tools. So this becomes a multi-layered, pretty longunning process.

4:00 This view is already way better than the print text dump of my agents outputs, but as a human, it's still going to take me a decent amount of time to click through each of these steps and see exactly what's going on. And so that's where Poly comes in. Poly the parrot is a new tool in Langmith that can read all the different runs here and lets me chat against this trace. I'm just going to use one of the default prompts. I'll ask to summarize this trace and we can watch for a second as Paulie tackles this.

4:30 You can see that Paulie is reading exactly what went on, listing out the runs. and eventually, here we go. Polly has given us a nice summary. We can see that we received an email request from Oliver. We delegated this to the meetinguler sub agent. We checked calendar availability for me specifically on Monday. We saw it was free, relatively late, and so we sent an email response confirming the availability and accepting Monday as that time slot. Cool. But to throw a wrench in this, let's say I actually didn't want the agent to do this. Let's say truthfully that I don't love waking up early and I don't actually want to take any meetings before, let's say, 9:00 a.m. Eastern time. So, how do I actually go about fixing this? If I navigate back to VS Code, there's a tool that we have called Langmith Fetch. It's just a package that you can install. And when you run Lang Smmith fetch, you need to set the config to pull traces from a particular project. So for us, I'm actually going to go back to Langmith real quick. I'm going to grab the ID of our personal assistant tracing project, and I'm going to paste that when I set it in the config of Langmith Fetch.

5:42 Now, when I run Langmith Fetch, it's just going to pull the most recent trace. We can see in a moment the full message history, the email that came in, what our assistant thought to do, it called a tool, it did its analysis, confirmed it works, sent the response, and then marked the email as read. Now, you might be thinking, why is this even useful, right? We just saw a much more detailed version of this in the Langmith UI. The benefit here is that by exposing this in the terminal, we now allow our coding agents to actually use this information. The coding agents can call linksmith fetch themselves and see what happened in the most recent traces.

6:21 I'm going to show you how you can do this with our deep agent CLI. I'm going to run our deep agent CLI. This kicks off our coding agent. And if you haven't tried the CLI before, it's pretty similar to Cloud Code. The experience should be pretty familiar. I'm just going to ask this. Hey, can you tell me what happened in the most recent trace? Summarize it for me. The agent's thinking and I've actually given the agent some instructions already in a markdown file about how to approach this stuff and that it can use lang fetch. So we can see the agent now wants to run lang fetch and I'm going to let it do it.

6:58 Now lang fetch traces are executing in the background and in a moment we'll get a nice summary from the agent. It says here this short summary this is what triggered it. The meeting was accepted and confirmed and great. So now instead of manually going and changing the prompt myself, I can also just talk to the coding agent. So I'm going to say, "Hey, I like sleeping in. I don't actually want to take early meetings, and can you make sure that in the future the agent only accepts these meetings after 9:00 a.m. Also, while you're at it, write a test to make sure that it remembers this successfully."

7:33 And so we've given it a few tasks here, but because I just ran lang fetch, we have that information in the context window. And so the agent is going to work for a while. It's going to list out files. It's going to read some of the files I have locally. And then it's going to start making some edits. So here we go. The agent has come up with a recommendation. It wants to edit agent.py. Specifically, it wants to edit the prompt. I'm going to accept this. It looks good to me.

8:02 Now, it's going to read our test assistant file and add a test. It comes up with this nice test, very similar to the example that I just had. And so I'm going to improve this by adding this to test assistant.py. Now, as a sidebar real quick, I want to talk about why I've chosen to write these tests in piest. Deep agents can handle a variety of tasks and the success of handling these different tasks can be measured in a lot of different ways. Writing in piest or or viest for JavaScript gives us the maximal flexibility.

8:35 I can in these tests assert that a certain tool call was made in a certain scenario. I can also in the same test use an LM as judge to check that my final result or my final email in this case followed some specific criteria. I've really found that when writing tests for deep agents, it's nice to have that flexibility to assert whatever you're specifically looking for for a particular given input. And both viest and piest afford a lot of flexibility here.

9:04 So now let's run the test. And the beauty of this is that in running this test, we're also logging more information to Langmith. And so after we run this test, whether or not it passes, we can ask the agent to list out this trace with Langmith fetch. See what just happened, see if it was acceptable. And if it fails, we now have this programmatic agentic loop where the agent has a clear reward function. It can keep iterating on this prompt until this test passes.

9:33 And so just to rehash the different things that we covered in this video, deep agents super easy to get started with. It's really easy with just a prompt and a few tools to come up with a pretty powerful agent. But that is really just the first step. From there, it's really important to figure out if your agent is actually doing what you want it to. The best way to do that is by setting up tracing to Langmith so you have full visibility.

9:55 In Langmith, you can talk to Poly and ask Poly questions about what your agent actually did during execution. And this is just a quick way to speed things up. I've seen deep agents that can go for hundreds of turns, takes several minutes, and that can be a pain to walk through yourself. Then we have linksmith fetch, which works in the terminal to pull trace information in. This is really powerful when you make it accessible to coding agents.

10:18 And so these tools are all intended to make it really easy for us as developers to work with AI while building deep agents. You can try out Poly and Langmith Fetch today. Let me know how it works for you. Pix.