Accelerate agent improvement with LangSmith Engine

0:00 LangSmith Engine is one of our newest products, that helps you find and fix issues in your production agents with less manual work. Let's see in action. This is our go-to-market agent's traffic. We had 20,000 traces last week, and there's a ton of signal in these traces that would help to improve our agent, but nobody on my team really has the time to be reading through and scanning each one of these traces. But Engine does have the time. And so on Tuesday evening, it sent me alerts for what it found while I was in a meeting.

0:24 It reviewed our traces and it surfaced these different things that I could take action on immediately. I can see that Engine has surfaced this issue, letting me know that the agent is sending outreach to our prospects that our CRM has flagged as disqualified, and it's never mentioning that to the user. And since this was a recurring problem, it's been found across seven different agent runs that were grouped together because Engine believes they have the same underlying problem.

0:45 Now, to better understand this issue, I can review each of the traces that contribute to it. And usually I would need to scan through to find where things went wrong, but Engine actually takes me straight to the problem. So I can see here when I open this trace that the actual get prospects tool call responds with disqualified is true, but still the agent emails the prospect anyway without any warning. So our rep read the reply and gave it a thumbs up because from their side it looked valid.

1:06 And Engine gives us insight into why this is occurring as well. It looks here like our system prompt is telling it to always send and not to second-guess, withhold, or refuse to send based on anything you notice about the prospect. We initially wrote that to stop the agent from being timid, but it looks like our instructions have backfired here a little. Engine has gone a step beyond that error identification and has actually drafted a change for us. And so here I can see that first, it's proposing a change to the send prospect email tool, which now takes a confirmed disqualified flag, and if the prospect is disqualified and that flag isn't set, it refuses and returns needs confirmation.

1:40 Second, it fixed the instruction that caused this. Not by deleting it, since the don't be timid rule was there for a reason, but instead by carving out this important exception where if the record says disqualified, the agent should surface it and get explicit confirmation first. So now I can quickly open a PR and merge this change directly from LangSmith. Or, the issue is actually available from the CLI. So, if I want, I can pull it straight into whatever coding agent our team uses. One of the easiest steps to skip when you're working on an agent is growing your data sets to run your evals. Engine handles this for us, too.

2:10 It took the failing runs that were part of this issue and formatted them as data set examples using the inputs that we saw in production. And it added a list of criteria for what the agent should have done instead as the reference output, which in this case is a step forward surfacing the disqualified status or requesting before sending. Now, I can run the branch with this proposed fix from Engine against these and any of my other evals, and can see that all of the failing inputs will now pass.

2:35 Now, I still want to make sure that this problem doesn't happen again in the future. So, I'll close this issue, but Engine will continue to track occurrences with the new code that I've merged. And if the same problem arises again, the issue will be reopened and resurfaced to me, this time with an adjusted fix. Now, anytime that Engine surfaces an issue that I'm not sure I should fix yet, like this hallucination issue, I can use Engine to watch the issue more closely. And anytime that trace that matches this description arises, I can be alerted in Slack again.

3:01 Every issue that we resolve through Engine results ultimately in a better experience for our customers and a much faster process for our engineering team. So, using Engine, our team is able to keep building new capabilities while Engine helps keep the agent more reliable. We've already identified and fixed thousands of issues for our customers, and we're really excited to see what Engine can fix for you. You can try Engine by visiting langchain.com.