Why We Built LangSmith for Improving Agent Quality

0:00 Over time, people are going to have to get much more methodical about testing. And that's where eval come in. That's where online eval come in. It's it's less of vibe testing two versions and putting more rigor behind this. >> People mainly know us for our open source, but ever since the start of the company, we've been building Lang Smith, a platform for agent engineering. And the things that we started off there were tracing and eval playground and observability and all these kind of like agent ops LLM ops things that that are really needed to bring out the reliability in in your agents as you build them. And we've we've added in more and more features and we're launching a bunch of new ones this week.

0:45 And so we're going to talk about that. my name is Harrison. I'm the co-founder and CEO of Langchain and have been working with Bagatur here to build a lot of these features. Do you guys want to do quick introductions of yourself? >> Sure. I'm Bator. I'm an engineer at Langchain working on Langsmith a lot on both our observability and eval features. >> And I'm Tenushri. I'm a product manager at Langchain and also focus on Langmith and been have been paying attention to a lot of our observability features lately.

1:14 >> Cool. So when we when we first launched Langmith, there was a few things in there that we kind of built up over time. I think tracing was one of the first things we added just like seeing the exact steps of what went into your agent and why. And this is kind of like an equivalent of like debugging. And then and then we built up kind of like offline evals where you'd create a data set and then define some metrics and run it against that. And that's like the equivalent of like unit tests and and kind of like software. we added in a prompt playground because we saw a lot of demand to bring kind of like product and other nontechnical people into the product and have them iterate on prompts and even kick off experiments.

1:48 And so we have a prompt playground and prompt hub now. And now we see a lot of folks going into production and so we have you know tracing very related to kind of like production observability. And so there's a very natural tie in there. But we started thinking more and more like, hey, we're we're getting these millions of traces a day. Like, how can we help people get more value over them and find more insights and and and and things like that? And so, there's two kind of like groups of features that we're launching this week that I'm pretty excited by. we're going to walk through them in order.

2:23 the first one is insights. And so, Bagator, you drove a lot of this. Do you want to introduce insights to us? Yeah, happy to. so like you mentioned, these days we see a lot of our customers are not just going to the sort of v 0ero to v1 creating their first agent and trying to get it out in front of their users. They've got agents in production. They've started to get millions of traces per day. And the questions they've started coming to us with is I've got so much rich information in Langmith. Can you tell me something interesting about how my users are interacting with my agent? how my agent is behaving, where my agent is and isn't doing well. And so the question we asked ourselves is how can we help our end users actually understand their agents and their their own users better when we have so much and such rich information. We're launching this new thing called insights and and we can dive into it in a bit. But the high level idea is that we are going to automatically find trends for you in your traces that either indicate to you these are the types of ways your end users are interacting with your agent.

3:31 These are the questions they're asking. This is the functionality that they're asking for or we're going to tell you these are the ways in which your agent is behaving. This is maybe where it's making mistakes. This is where it's hallucinating. Whatever it may be. and so that's sort of the core idea behind insights. you mentioned kind of like spotting trends in data. What types of trends what types of things do you think people will get out of insights?

3:53 >> Yeah. maybe maybe to go into a few concrete examples. we had one customer who built a sort of co-pilot into their product and their product managers came back to us and said, "We are sending you a bunch of traces of how people are interacting with our co-pilot that's embedded into our product. We want to know which parts of our product they're using the co-pilot for the most. And so that's one example of a type of insight that we should be able to generate for you.

4:20 >> Like product usage basically. >> Exactly. And like whatever wherever this agent is embedded, what are the actual features within that product that it is being used for? so that was one example. Another more classic example is is sort of like the engineers who are building the agents come to us and are like can you automatically identify the places where my agent is making a mistake where it is using the wrong tools where it is explicitly erroring whatever it may be and can you like categorize those types of failure modes for us and and give us examples of where those are happening that's another good example are these like customizable like if I want to if I want to explore something that's specific to my use case can I can I do that easily?

5:00 Yeah. So the idea is that there's a few sort of like built-in modes like this product usage and error analysis mode. but then we also expose a lot of the knobs of this algorithm to you as the Langmith user so that you can really guide our insight generation algorithm to to sort of find the patterns that you are most interested in. So there should be a lot of configurability in all that. >> Maybe not like in super detail but like how does this algorithm work under the hood?

5:25 >> Yeah. So, so this was actually inspired by a paper that came out I think earlier this year from anthropic where they had a sort of very similar question with their sort of generalist agent Quad. They wanted to understand over the millions and millions of conversations that people are having with quad what are the general topics of conversation and they came up with this algorithm that they call quo that pours over all of these conversations and ultimately generates this sort of hierarchy of categories so that you can sort of nicely like zoom in and zoom out into the granularity that you care about to understand the topics of conversations that people are having with quad. And we were really inspired by that idea. by this point we've sort of like completely changed all the internals of how it works to sort of generalize it to what lang traces can look like, which is a lot more broad than just what a a sort of like chatbot conversation can look like. but the like end result and and the highle idea are all very similar to that anthropic paper.

6:26 >> What was the hardest part of getting it to work? Well, I think the hardest part was and is that we are sort of dealing with arbitrary agent payload. What an agent looks like today is is sort of like very diverse and increasingly diversifying. It's not just chat bots anymore. There's sort of like background agents that are, you know, there's no like chat history for for us to really be looking at to understand exactly what the agent is doing. yeah, there there's like being able to very generically generate insights from arbitrary agent payloads is is something that is like we're still working on and and trying to improve.

7:06 >> And so I can try this out today. I can go into the product. I'm assuming I need to send a bunch of traces. The more the better for it to look at patterns for. How else do I get it to work? Well, like if like Yeah. Like what what tips or tricks would you have for people using this? Cuz this is this is an agent under the hood basically this algorithm. >> Yep. Exactly. And so like like with any LM based application, the key is giving it sort of good instructions and the context it needs. So there's a few places where you can tell us sort of exactly what your underlying agent does and exactly what types of insights you're most interested in in generating and sort of like greater clarity and specificity with which you can give that information, the better and more interesting insights we'll be able to generate.

7:47 >> What do I do with these insights once I have them? >> It's a great question. Insights is a very generic feature. It really depends on what you're doing. If you're the product manager trying to understand what like product areas people are engaging your agent on, then maybe you're taking that information and like updating your road map to rep prioritize and make sure that you are actually focusing on the features that people are most asking about. if you are sort of the AI engineer who is iterating on this agent and you see a failure mode that is happening with much greater frequency than any other failure mode like for example your retrieval step is never returning the right context then like now you know this is what I need to focus on to improve my agent.

8:28 if there's like a a type of question your agent keeps getting that you didn't even know was a potential question that they would have to engage with and you just haven't given it the tools that it needs to be able to answer that question, then you might be updating your agent architecture in that way. So, it it really varies, but hopefully it's always something fairly actionable. >> What do you see as the distinction between evals and insights? Because they sound overlapping, but what's the difference there? Yeah, I think insights is really about discovery and it's often about sort of like the known unknowns or the unknown unknowns. With eval sort of know what you're hoping to find and you can define it well up front and then you can run that over your traces. with insights. The idea is we just have a lot of this like trace data and we're trying to bubble up something interesting out of it and discover patterns out of it. And then oftentimes once you know of those patterns then you can go translate them into eval if if there's for example some like failure mode that you want to have a better monitor on going forward.

9:28 But this is sort of like a precursor to even getting to eval. And that's actually a good segue because a lot of what you worked on for this launch was around evals and specifically around this concept of like thread evals. So we we've talked a little bit about some of the evals that we've had in the product before like some of the the offline build up a data set type things. So so I guess first like what are threads and then second like what are thread evals and and why should people care?

9:54 >> Yeah, good question. So threads are are really entirely userdefined. depends on the product interactions that you have. so for example, if you have a chat GPT like style interface in your app, maybe each thread represents each each chat window, >> each conversation, >> each conversation. Yeah. Yeah. Or or maybe you have a more of a co-pilot like interaction in your application where maybe each thread is just a user session. what did a user ask across my app to this conversational agent? So really depends on how your your user interactions are and threads are totally customizable. it's really just unique UU IDs that you add to traces, >> but it's like a sequence of repeated traces that you can stack up on top of each other and have the same context conversation some somehow related.

10:46 >> Exactly. Yeah. Exactly. The key thing is you want to define a relationship between traces. And so where where threads and and and evalplay is sometimes actually the unit of threads is much better for certain types of interaction patterns that you're evaluating >> like what what are some examples? >> If you're evaluating over kind of single terms or single traces, you might just look at the context of a user message and an AI message and that's enough for for that style of evaluation. But if you want to evaluate end to end user interaction, so something like what was user sentiment across this entire thread, did the user express frustration? You really need to build up context and have an entire entire thread. Or if you want to evaluate things like tool calls, the trajectory of what tool calls are like within a conversation, you need to have kind of end toend view of that conversation and then and then dig into the tool calls are made. is agent getting stuck somewhere and and doing the same tool call over and over again.

11:44 So it's complimementaryary to kind of single turn evals but it's a different lens of looking at things. >> So so we've talked about evals for threads. One of the things I mentioned earlier was this concept of like offline evals which you know I I described as like building up a data set of examples and running evaluators over it. >> This is different than that like h how is it different than that? >> Yeah. So the the two ways to think about it is offline emails are over a set of known examples. So these are cases where you want your agent to to exhibit particular behavior. but when you ship something to production, the the things that your users are trying to do don't always match up with the things that you intended when you first built the product. And so that's where the online evals piece comes in is building up evaluators that measure your agents performance in real time as traces are coming in. and again, threads are are a really great unit to use for this because if I want to let's use that user sentiment example again. If I want to measure user sentiment after I just shipped a change to to my prompt and see if that improved the quality of user interactions, you could do that in online eval because they run on real time over your over your production data.

12:59 >> Cool. And so, so we have this new concept of threads and one of the things we can do is we can evaluate them. What else can we do with threads? >> I think there's some really interesting levels of visibility and then also actions across the product that we can enact on threads. So I think one metrics and analytics are a big piece here. So if you're bubbling up and looking at the level of threads, things you might care about is like what is the cost of each user interaction. or if you're running evaluations across threads, you want to see that data over time and plan it over time. So we'll be doing a lot of work to introduce new kind of agent thread level metrics into our dashboards. Second thing is we have you know this whole flow of you go from traces to a data set or you go from traces to maybe you havememes that are reviewing data and we want to enable that over threads as well. So going from a set of of threads that you have coming into production, maybe you want to spot check those when there is negative user feedback and have have someone look over them. So enabling more flows with our our automation rules feature over threads as well. Those are the next big things coming.

14:07 >> Cool. So So we've chatted a little bit about evals. We've chatted a little bit about all the stuff you can do over production data. Maybe to close things out with a with a with a spicier question. There's been some takes online recently that like evals are dead. All you need is kind of like AB testing and online testing. How do you guys think about that? >> It it's a provocative take that doesn't like it seems way too black and white to me. I think like the maybe the general sentiment is that in a lot of types of things that people are using agents for, you have no good like coverage of how your end users are going to interact with your application. And so offline evals are never going to fully give you like a 100% guarantee that your agent is going to behave the way you want them to. And therefore offline evals are useless. And that just seems like one it's super domain dependent like it depends on exactly what your agent is doing. I think in some cases agents are very narrow and you can't actually really define a lot of what is happening before you ever ship it into production.

15:07 But then two, I think it's also like just because you can't get complete coverage of every type of interaction doesn't mean it's not still worth enumerating the types of interactions that you know of and that you want to make sure your agent is sort of knocking out of the park. And so we think offline eval are always really useful to like be building up even if they're not for example you could ship without them potentially but then like build them up over time every time you see a bad interaction in production and and those can go back into an offline data set and then those offline evals serve as regression tests every single time you want to ship a new version of your application. So, it's like there there's nothing that binary about it. And it seems sort of obvious to me that like some form of offline emails is always going to be useful. And then yeah, the there's also like people say emails are dead. I think they're usually referring to offline emails. Online emails are also a thing. Online emails, I think, are especially useful because it you can always be extracting some additional signal from your online traffic. and so why wouldn't you want to be doing that? The other thing I'll mention is we find that teams that are building often start out building with use cases that have pretty high margin for error. Like the agent gets something wrong. maybe the human intervenes. or yeah, the human is is is very very in the loop.

16:24 But as you have more I know Harrison, you talk about ambient agents a lot or agents that are used for more critical use cases, the like rigor that goes into testing is going to need to increase over time. and we also have some teams that are building like just AI native applications where you have lots of AI interactions throughout the application, but the user may not even know it distinctly. So I think over time people are going to have to get much more methodical about testing and that's where eval come. That's where online eval come in. It's it's less of just vibe testing two versions and putting more rigor behind this is going to become more important.

17:01 >> Cool. So testing's good. Both types of testing are good. Un unspicy answer to a to a spicy question. well I think that's all we've got time for today. I think yeah we're we're really excited about this launch. More and more things going into production is great for the industry as a whole and there's you know there's a lot of unknowns when you launch an agent into production and we just want to decrease the the number of unknowns and hopefully insights and some of the threaded evals can help do that.

17:28 So, thank you both for for walking through a lot of these features with us. >> Totally. Thank you.