Vizient’s Healthcare AI Platform: Scaling LLM Queries with LangSmith and LangGraph

0:04 Hi, I'm Blake Rhodess, director of AI strategy and technology at Vizian. Vizian is a healthcare performance improvement company. Our clients make up 97% of academic medical centers in the US, more than 69% of the acute care hospitals, and over 35% of the amulatory market. We are building a generative AI platform designed to help healthcare providers access and analyze their data more efficiently. Our goal is to unify silo data sets and enable users to extract key insights from patient outcomes to clinical benchmarking in a way that makes datadriven decision-making more intuitive and accessible for providers of all sizes, even those with limited resources.

0:39 Before integrating Langraph and Langmith, we encountered several significant challenges. One of our biggest issues was scaling our LLM queries using Azure Open AAI. We faced token permanent limitations which throttled responses and impacted performance. Additionally, we had limited visibility into system performance, making it difficult to track token usage, prompt efficiency, and reliability. Continuous testing wasn't feasible, which meant we were often addressing performance issues reactively rather than proactively. Furthermore, our multi-agent architecture introduced additional complexity, requiring better orchestration to ensure agents worked together efficiently without generating inconsistent responses. The lack of Langmith's observability tools early on also resulted in technical debt, making it harder to iterate and improve our workflows over time. Integrating Langraph and Langmith was transformative in addressing these issues. With Langmith, we gained the ability to accurately estimate token usage. This allowed us to properly provision capacity in Azure OpenAI. The tracing and observability tools provided real-time insights into system performance, making it much easier to diagnose and fix issues, especially during high stakes moments like live demos. Langraph provided the structure and orchestration we needed in our multi- aent workflows. Perhaps our biggest aha moment was realizing how much easier development would have been if we had used Langmith from the beginning. Early graph visibility would have saved us from creating and accumulating technical debt. Since implementing Langraph in Langmith, we've observed several improvements. First, we successfully resolved LLM rate limiting issues by optimizing token usage and thorough put allocation, ensuring a consistent user experience. Our development and debugging processes have also become significantly faster thanks to Langmith tracing tools which allow us to quickly pinpoint and resolve issues.

2:23 Our shift from manual evaluation to automated continuous testing has dramatically improved system quality and reliability. Additionally, we are now able to rapidly turn beta user feedback into actionable improvements, enhancing the platform's effectiveness. Start with a slim proof of concept, then model one high impact user flow and langraph. Wire that into Langmith on day one and treat every run as a data point you'll want to use later. Define a handful of golden query response pairs up front. Tag them in Langmith and let them drive your acceptance test while your graph grows.

2:55 Finally, I recommend budgeting a short weekly review of Langsmith's run history. 5 minutes of inspection usually saves us hours of detective work down the road.