Section Insights
Introduction to Morningstar's AI Initiatives
What is Morningstar's focus in AI products?
Morningstar is expanding its externally facing AI products and applications, including the live MCP server and AI assistants in its software.
- Morningstar is a financial services firm specializing in investment research.
- The AI accelerator team is dedicated to enhancing AI applications.
- Key products include the Morningstar MCP server and AI assistants in licensed software.
AI Assistants in Morningstar Products
What capabilities do AI assistants provide in Morningstar's software?
AI assistants in Morningstar Direct and Advisory Suite can answer queries about ETF holdings, visualize data, and fetch analyst research.
- AI assistants enhance user experience by providing quick access to information.
- Users can easily visualize data and get the latest research on funds.
- Agentic capabilities allow for interactive queries.
Challenges in Observability and Debugging
What challenges did the team face in debugging AI agents?
The team struggled with limited visibility and cumbersome debugging processes, often feeling 'blind' to the agent's performance.
- Initial debugging was complicated and relied on limited data.
- Lack of a common language for observability hindered effective troubleshooting.
- The team needed better insights into prompts, models, and outputs.
Improvements with LangSmith
How has LangSmith improved the debugging process?
LangSmith provided a common language for observability, reducing anecdotal debugging and saving significant time with its infrastructure.
- LangSmith enables better observability and tracing of AI agent performance.
- The team can now rely on structured data rather than anecdotal evidence.
- Time savings from using LangSmith's tools are significant.
Team Collaboration and Metrics Enhancement
What benefits has the team experienced from using LangSmith?
The prompt playground in LangSmith has fostered team collaboration and improved metrics such as time to resolution for tracing.
- Collaboration on prompts has become a team effort rather than an individual task.
- Metrics related to debugging and resolution times have significantly improved.
- The team is committed to continuing with LangSmith due to its positive impact.
Transcript
0:00 Morningstar is a financial services firm known for investment research, ratings, and data. I'm currently on the AI accelerator team and we're the team focused on expanding Morningstar's externally facing AI products and applications. Three in particular that I'd like to talk about are one, our Morningstar MCP server, which is live and connectable today. We also have AI assistants directly embedded into our licensed software products. So Morningstar Direct and Morningstar Direct Advisory Suite, we have AI assistants directly in these products with agentic capabilities.
0:30 So you can surface things like, oh, what are the holdings of a given ETF or index fund? You can visualize data that Morningstar has. You can fetch analyst research to get the latest and greatest on a particular fund or what's going on in the market. Visibility at the beginning was quite cumbersome and debugging was extensive. Some complicated Splunk queries, which only gave us a limited picture into what was going on within our agent. Effectively, some of it you could classify as flying blind.
0:58 Some of the things we needed to see were what prompt was used, what model was called, what tool was selected, what the output of the tool was. So without LangSmith, we didn't have a common language for observability and tracing. And so debugging becomes anecdotal. Someone saying, you know, I saw a bad trace once. We now have features that we didn't have before. With LangSmith online evaluators, that's infrastructure. We don't have to build ourselves and that's significant time saved.
1:23 LangSmith's prompt playground has allowed us to easily share prompts across team members. It's no longer an individual exercise, it's a team effort. Some of the things we're looking at are time to resolution, how quickly can I find a trace? We're only a few months in, but all of our metrics have been significantly improved with LangSmith and I can't go back now. (upbeat music)
Summary
- Morningstar MCP server is live and accessible for external use.
- AI assistants are integrated into Morningstar Direct and Advisory Suite, enabling users to access ETF holdings, visualize data, and fetch analyst research.
- Initial visibility and debugging were challenging, relying on complex queries and anecdotal evidence.
- LangSmith has improved observability and tracing, providing a common language for debugging.
- Features like online evaluators and prompt playground have streamlined team collaboration and reduced development time.
- Metrics such as time to resolution have significantly improved since implementing LangSmith.
- The team emphasizes the importance of collaboration in prompt sharing and debugging efforts.
Questions Answered
What is Morningstar's focus in AI products?
Morningstar is expanding its externally facing AI products and applications, including the live MCP server and AI assistants in its software.
What capabilities do AI assistants provide in Morningstar's software?
AI assistants in Morningstar Direct and Advisory Suite can answer queries about ETF holdings, visualize data, and fetch analyst research.
What challenges did the team face in debugging AI agents?
The team struggled with limited visibility and cumbersome debugging processes, often feeling 'blind' to the agent's performance.
How has LangSmith improved the debugging process?
LangSmith provided a common language for observability, reducing anecdotal debugging and saving significant time with its infrastructure.
What benefits has the team experienced from using LangSmith?
The prompt playground in LangSmith has fostered team collaboration and improved metrics such as time to resolution for tracing.