Tracing Claude Code to LangSmith

0:00 Hey there. Today you're going to learn how you can set up tracing from Claude Code to Langmouth by just setting a few environment variables. Let's dive right in. First, you're going to need to create a Langmith account and generate an API key. Once you have an account, navigate to your settings page and create a personal access token. This is the only key that we need to connect Claude Code to Langmith. Let's copy this and we'll use it in a second. Claude Code is one of the most impressive and useful AI coding tools to date. I use it every day in my work including on this open deep research agent which I have open here. What's nice is that cloud code can emit open telemetry standard events for monitoring and observability.

0:39 Langmith can then collect and display these events to give you a full detailed log on everything that happens during your cloud code sessions. To set up tracing to Langmith, all we need to do is set a few environment variables to configure the hotel export from cloud code. The first environment variable that we're going to set is cloud code enable telemetry equal to 1. This basically turns on telemetry for cloud code. Next, we'll specify hotel logs exporter equals OTLP. This specifies the output format to use open telemetry protocol. We'll also specify that the log should be sent with HTTP transport and JSON encoding. This is the format that Linksmith ingestion was built to accept. Now, this is the key piece. If you're using Langsmith Cloud like I just showed, this is the endpoint that you'll specify for logs from Cloud Code. If you're using a self-hosted instance of Langmith, your URL will look a little bit different. You can refer to the docs in the description to go get your URL.

1:37 Now, we're going to take that API key that we generated earlier and set that in our headers. This is going to allow us to authenticate and connect to Langmith. We'll also specify a tracing project. This will specify where our cloud code traces show up in Langmith so we can find them easily. Finally, we want to enable logging of user prompts and inputs. So, we'll set this variable to true. That's it. All we needed to do was set these six environment variables.

2:04 Claude Code is now emitting events that are captured and displayed in Linksmith. Now, let's see it in action. Let's go ahead and start a Claude Code session. I'm going to ask Claude Code to broadly describe to me what Open Deep Research does and how it works. We can see that Claude Code is reading quite a few files. This will take a while, so I'm going to skip ahead to when the answer has been generated. We can see that this response is pretty helpful. Let's see what we traced in Langmith.

2:34 We can see a new trace here in Langmith named Claude Code. If I click into it, I can see each of the individual things that Claude Code did. I've logged the question that was asked initially. We can also see the model names, token usage, and latency from the model requests that claude code makes. Cloud code also sends up costs associated with each request. And we can also see all of the operations that claude code undertakes, like reading files.

3:01 Now, let's ask a follow-up. We're going to ask Claude Code to write a cloud.md file for this repo. We can see that in this case it comes up with a to-do list and asks us to execute a few commands. We'll play along with it here and go with it. Now let's check back in on our trace in Langmith. In that same trace, we can see that we now have more runs which correspond to the different actions that Claude Code took. One thing to note here is that while we can see the user prompts that we input, we don't have access to the actual system prompts and messages that Claude Code sends to the anthropic models. We also can't see the raw model outputs that come back. This information isn't exported by Enthropics hotel logging. However, we do get token count and cost measurements. We can actually see that after the second question, the total token count and cost for this trace has gone up. This is because each cla code trace is tied to a session. In other words, everything that I do in this session of cloud code is going to get logged to this trace. The waterfall view is particularly interesting. We can already see groups of runs based on timestamp for our first user prompt asking about a repo and then our second user prompt asking cloud code to write a file.

4:16 Tracing cloud code to linksmith can also be really useful for organizations trying to monitor general usage. API usage for cloud code can easily match or exceed that of a production application. Langmith has these pre-built dashboards to help us see the total number of traces over time in any tracing project, as well as stay on top of any patterns in token usage or costs. To recap, you can set up tracing from Cloud Code to Linksmith just by setting a few environment variables. I've linked a doc in the description with written instructions and those exact variables spelled out. Thanks for watching.