Middleware for Managed Deep Agents

LangChain · 4m · transcribed 5d ago

Section Insights

# 0:00

Introduction to Middleware for Managed Deep Agents

What is middleware and how does it enhance managed deep agents?

Middleware extends the agent life cycle with customized behavior, allowing for policy enforcement, fault tolerance, and management of interactions with tools and LLMs.

Middleware can manage policy enforcement and fault tolerance.
It helps in tracking, transforming prompts, and adding error handling.
Great documentation is available for getting started with middleware.

# 0:56

Implementing PII Middleware

How does PII middleware work in managing sensitive information?

PII middleware redacts sensitive information like emails before the agent processes requests, ensuring that such data is not exposed.

PII middleware can block specific types of sensitive information.
It modifies input data before it reaches the model.
Using pre-built middleware is recommended before creating custom solutions.

# 2:49

Creating Custom Middleware for Logging

How can custom middleware be created for logging tool calls?

Custom middleware can be defined to log tool calls by wrapping tool calls with a decorator, allowing for tracking of tool usage in the agent's life cycle.

Custom middleware can enhance functionality by logging or tracking performance.
Middleware can be placed at various points in the agent life cycle.
Documentation provides guidance on creating custom middleware.

# 3:45

Logging Tool Calls with Middleware

What is the benefit of logging tool calls in middleware?

Logging tool calls helps in tracking performance and understanding tool usage, which can be directed to various outputs for analysis.

Logging tool calls provides insights into agent performance.
Standard output can be used for logging, but other tracking methods are available.
Prebuilt middleware offers a solid foundation for adding custom functionality.

Transcript

0:00 I'm Nathan. I'm a PM here at LangChain, and today I'm gonna be talking about middleware for managed deep agents. Middleware is a way to extend your agent life cycle with customized behavior. It allows you to manage things like policy enforcement, fault tolerance, and more when your agent interacts with tools, interacts with the LLM, or hits limits like token usage. We have great documentation about how to get started with middleware, and our middleware SDKs all play great with managed deep agents.

0:27 So you can take principles you may already know or use principles that already exist out in the wild to build great middleware for your managed deep agents. Some potential use cases include tracking with logging or analytics, transforming prompts, adding retries or error handling logic, or adding rate limits and guardrails. We'll add a few examples to a managed deep agent I've been working on. Let's get started. We'll add a PII middleware to redact email. In this agent, I have a tool which allows lookup by customer email, and we're gonna break that with the middleware.

1:01 But to start, let's show the current behavior. If I ask what plan is ops on, then the agent should be able to use that lookup by email tool to fetch the customer data and answer the question specifically. So it found the email, it ran the tool call using that email, If I look at the LangGraph, you'll see it's got the middleware set up, and it passes things to the model. And in this case, there's no middleware yet, but we'll add some, and we'll see this graph expand.

1:31 So to add PII middleware, I'm just gonna add into this middleware attribute here. In this case, I'm setting up PII middleware with the attribute email to be blocked. I want to use a redaction strategy, and I do want to apply this to the input as it passes through to the LLM. If I look back here, I'll see the middleware added to the graph If I go back to the chat, and I try asking the same question, but this time using the middleware, we'll see what plan is the email on.

2:01 And when I send this request in, this pre-built middleware will automatically detect the email and redact it and prevent the agent from seeing it. The agent will attempt to do a lookup, but it actually won't work because it doesn't get the email. So if I look back here, you'll see what plan is redacted email on. So this isn't even stored, inside of LangSmith. And if you look at the graph, you can see that this is passing through.

2:25 It runs the PII middleware, and then it redacts it at that step in the process. So before it even hits the model, the input has been modified in place. If you're trying to block specific types of PII, you could use that PII middleware. Other tools in here are really handy for common cases, so definitely experiment with different options here depending on what you're trying to do, and it's always worth using the pre-built middleware before you go ahead and build your own.

2:51 But in some cases, you're going to wanna define custom middleware. In this example, we'll create some custom middleware that just does something basic, logging. So I'll create a new middleware directory in my project, and I'll add this audit file. And in this audit file, I'm just gonna bring in the example here. And this uses this wrap_tool_call decorator, which is gonna automatically wrap any tool call that's used. And there's a bunch of different hooks from the middleware package that you can use depending on where you want to place your middleware in the agent life cycle.

3:24 In this case, this is gonna come around every single tool use, because we wanna log what tool is being used, okay, so I've set up the middleware, and now I'm gonna add this to my agent definition So if we take a look at the LangGraph here, we can see we have this log tool calls middleware and this runs at the beginning. And if I interact with this agent, for example, sending the same question that we've been asking already, and I take a look at the agent logs when I run this, we'll see this is going to actually log the tool calls that are being used.

3:59 Here, it's just logging them into my standard out, but obviously you could send this to tracing, you could track performance or whatever you need to do in this case. So here's the tool calls that were run. So that's a quick introduction to middleware and using both the built-in middleware and the custom middleware functionality. We have a ton of great documentation on building custom middleware, so I definitely encourage checking that out to see more if you're trying to get more creative with how you build.

4:30 In general though, the prebuilt middleware is a great place to start, so definitely give it a look if you're curious about adding more custom functionality around your tool calls or controlling the set of information that's sent to the LLM

Summary

Nathan discusses the implementation of middleware for managed deep agents in LangChain, emphasizing its role in customizing agent behavior, enhancing policy enforcement, and ensuring fault tolerance. He demonstrates how middleware can be used to manage sensitive information and track tool usage through practical examples.

- Middleware extends the agent lifecycle with custom behaviors like policy enforcement and fault tolerance.
- Use cases for middleware include logging, prompt transformation, error handling, and implementing rate limits.
- A PII middleware example shows how to redact sensitive information, such as customer emails, before it reaches the LLM.
- Custom middleware can be created for specific needs, such as logging tool calls during agent interactions.
- The middleware SDKs integrate well with managed deep agents, allowing for both pre-built and custom solutions.
- Documentation is available for users to explore building custom middleware and leveraging existing options effectively.