How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph

0:00 Customer support use cases are very broad. It can be like issues like lost and found or issues like damage claims or issues like taxes. We have like 500 million types of interactions every day. And we have almost like 200 to 300,000 user queries per day, which is a lot. So we work for riders, drivers and applicants. One of the metrics we go for is what's like the resolution rate. So how much can we solve our customer issues without escalating them to human agents?

0:35 Our platform covers about 30 to 40% of use cases before we adopted LangSmith. Our agents were pretty basic. They were very deterministic, maybe like a one single short LLM call with a huge prompt and a bunch of internal APIs it can access. So there was no like sort of sophistication in multi-turn conversations. There were no evaluations, proper evaluations. The observability was a big issue because we kind of launched a few agents in production and sometimes it did not behave as we expected it to be.

1:11 But then we didn't know what to do about it. We didn't know which component inside that pipeline broke the response, whether it was a tool call or like a prompt or maybe like the LLM itself hallucinated in between providing a response. It was difficult to debug issues. It was getting difficult to scale. So we wanted our engineers to focus more on more complex use cases or maybe like foundational improvements and not spend all of their time building agents.

1:45 We tried to bridge this gap by creating a self-serve platform for our non-engineering people like PMs and ops, where they can use their own domain knowledge. and natural language prompting skills to build agents and ship them to production without making a lot of code changes. So it's merely like a config change, and they write some prompts on LangSmith Prompt Hub. We as engineers can review them, but they are like very easily deployable to production. So what we did was we have like a meta agent and all the sub agents are kind of registered as nodes in that meta agent.

2:25 And we do that dynamically when we start the service. So every new agent now it's just like a config which we have in our runtime repository and prompts which we have in the LangSmith Prompt Hub. So we just pull that and like the platform composes the rest of the things. So it's become pretty easy. And because of LangSmith tracing, the debugging and evaluations, testing, everything has become like very smooth and very easy. It was a huge upgrade to ship to production.

2:56 But then with this agent builder platform that we have created, that has reduced from six months to a few weeks, one to two weeks. And that's pretty exciting because now, like we have free time to focus on more complex and foundational improvements while PMs and other ops people can directly ship agents. So the iteration loop has become faster. Shipping faster has enabled us to create more agents and like cover more issues in production. And that's definitely resulted in an increase in the resolution rate by 16%.

3:31 So yeah, since we have started using LangSmith and LangGraph. (upbeat music)