What Is LangSmith? Explained in 5 Minutes

0:00 Hi, I'm Katherine from Langg. Today I'll give a five-minute tour of Langmith, our commercial platform that helps developers improve LM application and agent performance at every step of the development cycle. Specifically helping with observability, evaluation, and prompt engineering. While Linksmith works well with Linkchain's open source frameworks, it is framework agnostic, designed to work with any agent framework or no framework at all. Let's dive in. One of the biggest challenges with agentic applications is understand how they reason and take action. With Langmith, tracing comes out of the box.

0:36 All it takes is setting a few environment variables or tracing decorators and Langmith will automatically log each step of your application's workflow. Linksmith gives you the tools to inspect overall performance as well as inspect individual traces to root cost effortlessly. The run tree view helps you debug a trace, showing a detailed log of its trajectory towards its final output. This can also be helpful to identify expensive steps or understand latency bottlenecks. The threads tab allows you to group traces into a multi-turn view, making it easier to understand conversation history and view how users engage with the agent. Each tracing project includes a turnkey monitoring tab that provides built-in charts on volume, success rates, and latency along with the flexibility to define your own evaluators for continuous monitoring of qualitative metrics such as response conciseness, helpfulness or hallucinations. To improve the accuracy of your elements of judge evaluators, Linksmith automatically collects examples where human corrected the judge. These examples are seated back to the judge with viewshot prompting helping align judge scores with human preference. One of the biggest challenges users face is trying to find insight from their production systems which involves a sea of rich text conversations and unstructured information. Linkswift provides alerting to proactively raise issues. You can set up alerts to pager duty or web hook to notify you if quality dips or latency spikes beyond a set threshold.

2:15 You can also set up automation rules that help you action on the data to improve your app. A common flow involves taking interesting traces in production, filtering from feedback stores from building online evaluators to build a test suite or to send for additional human review. We believe great evaluations begins with great observability. Let's now dive into a links with evaluation features. The annotation cube provides a dedicated space for auditing traces. This is especially useful for teams where subject matter expertise is needed.

2:49 Here, annotators can inspect traces, leave feedback, or add them as an example in a test suite. Linksmith data set and experimentation features makes it easy to systematically test your application's performance. By curating a representative data set, whether by manually creating examples, using AI to augment data, or pulling from production traces, you can continuously measure and track your agent performance as it improves. Once a data set is created, you can run experiments directly from the SDK or through the UI. This flexibility allowed both developers and non-technical stakeholders such as product managers or business users to participate in the testing process. After running an experiment, the results are centralized in the experiments tab, making it easy to analyze performance trends. For example, our team tested whether adding a reflection step to an application architecture would improve results. We ran two separate set of experiments to compare the impact which helped us assess the trade-off between improved quality and increased latency. A problem many teams consider when optimizing their applications. Experiments can provide clear visibility into the trade-offs you're faced when building and scaling your application. Linksmith also makes it easy to compare experiments side by side. You can select to experiments and quickly toggle through the examples to see which one regressed or improved, giving you clear insights into the impact of your changes. Beyond helping to identify areas to improve, Langswift makes it very easy to iterate, optimize, and collaborate on prompts. The PROM playground provides a scratch pad to test different models, adjust wording, and see how changes impact responses. A side-by-side comparison view helps you compare and analyze variations between prompts. Langmith also provides a prompt canvas tool where you can give feedback on the prompt and get AI optimized versions, making everyone on your team a profession prompter.

5:00 When you land on a better prompt, you can save it in a centralized prompt library so teammates can reuse and refine. Linksmith also tracks commit history, making it easy to track changes over time. No matter where you're in the agent development cycle, Linksmith can help from building, evaluating, and monitoring, all in a centralized platform. Thank you for joining this quick tour.