Run Root Cause Analysis in Claude Code with AI

0:00 Everyone, welcome to a Wednesday lesson around root cause analysis in cloud code. Thank you for joining us. Today's agenda is going to be roughly 40 minutes or so of me talking. So, combination of slides and then also demoing some stuff, some workflows in cloud code that that hopefully will will be helpful on this on this on this very specific topic around root cause analysis. Now, before we dive in, so just a quick intro for us here. My name is Hai.

0:38 I am part of the AI Analyst Lab. We do a lot of um courses, free workshops, and and also email series, free email courses on our website if you want to take a look. We've been doing this for a couple of months now or a few months now. We open sourced a AI Analyst repo. Probably many of you have heard it or have used it. If not, we'll get into a little bit of a preview on that one.

1:08 And I'm currently the head of data at Ontra at a legal tech company called Ontra. Previously at different consumer tech companies like LinkedIn, Pinterest, Meta, Nextdoor. That's where I met Shawn and Shravya who's part of this this crew. So, yeah, nice to meet everyone for those we haven't met and nice to see folks that that that's been with us before. Shawn, over to you. Hey everyone, I'm Shawn. Yes, as Hai mentioned, I I lead the AI Analyst Lab with him and Shravya.

1:40 I'm a principal data scientist. Same legal tech company Hai is at. Been in data for a little over a decade. Most of it in in product, but across B2B as well as uh kind of [snorts] uh you direct consumer products. Um yeah, [clears throat] most recently I'd say last 2 years been focused on AI evaluation and uh genetic analytics. Part of that was like kind of more classic product data science.

2:12 And then our um our third member of our team, Shravya, I think she's going to be joining us towards the end of the session. She's currently on a very relaxing uh uh retreat somewhere in North Carolina, I think. So, she's probably at the spa right now or or getting a sound bath or something like that. Um whatever they do at those things. But uh uh yeah, glad to glad to be here. Thanks for spending the the hour with us.

2:43 Cool. All right. So, um we'll get into this uh right now. Uh just before we get into the actual content, does anyone if you were to describe what root cause analysis is, uh how would you describe it? Put it in chat. What's one thing that jumped to your mind when you hear this these combinations of words? The five whys. Oh, that's such a Oh, two people said it. Yeah, so good.

3:13 Decomposition analysis, yep. Volume versus rate effect. Not my problem. Root cause principles. Principles, structured. Okay. Cool. Yeah, that's good to good to hear. Good to see uh where what what people how people would describe this. Um so, Where's my button? So, root cause analysis uh typically is trying to answer like an observation.

3:44 So, uh you have a some sort of metric, right? So, in this example that we have here, um probably folks in this room might have gotten very similar questions or slash uh have some have very similar curiosity when when when certain anomalies happen in your day-to-day metric. So, let's say, you know, this is not uncommon to hear, "Hey, revenue's down 12%. What what happens?" Uh that's generally almost like a trigger phrase or some variants of this would be uh would be would trigger a root cause analysis to diagnose and decompose down to something that is actually useful and actionable. And so, um you know, right here in the second line here, you have until end of day go. Uh typically that's, you know, like if this if this question comes from executive or like a VP of of some sort, uh generally it's pretty urgent. Um you know, coincident or uh surprisingly or unsurprisingly, when revenue goes up 12%, it's less urgent.

4:48 Uh but uh you know, uh it's a way to it's a way to pinpoint as precisely as possible the story behind why a number moves and what the biggest contribution to that is. So, we'll get into that in a little bit in terms of how to actually do it. Uh okay, cool. So, when folks get that question, the uh there's, you know, like in terms of common practice, uh I'm I mean practically, not theoretically or not best practices, um three things, just being a little bit uh uh you know, sarcastic here. Number one is probably panic. It's uh "Hey, down 12%." Uh sounds like sounds like pretty urgent. Uh but like where do we start? There's so many ways to slice and dice, no clue how to how to act. Um you know, that's pretty common, especially if, for example, a metric is new, brand new, you've never seen it move before.

5:47 Probably for something like revenue, it is well understood. Um, but you know, there could be billion other metrics that might still be important for the business. And uh, if you haven't seen it before, generally it's not as uh, straightforward per- perhaps. So, you know, like there could be merit to panicking. Um, the second one here is uh, uh, just, you know, pick a dimension that you have a dashboard for and then see, you know, if something moves and just be like, that's that's that's it. That's the cause. And then just tell a story off of the only dimension that you have access to, for example. Uh, that's pretty common as well.

6:23 Um, and then the third one is uh, go super deep into certain, you know, like paths. So, for example, it's like, oh wow, like this looks like it's uh, you know, went up a lot. So, let's look in look in there and then, you know, the more that you drill deep into, the less you lose the specificity of hey, exactly how much of contribution when I go this deep, is it actually affecting the top line and how much is it actually explaining it? So, this is kind of like what it's uh, referring to, drilling to nowhere. Uh, you know, like if if it's unstructured, if it's not very specific, uh, you get to a lot of motions to try to get to a path that may not actually lead you to the right to the right direction.

7:07 So, um, you know, like I've I've seen and Sean's seen this in the past. Um, I've been guilty of it myself when I first uh, you know, like uh, either you get on boarded to a new area or uh, have done this kind of like without uh, a lot of reps in the past. Um, it is a lot of uh, you know, like hey, see what we can find and uh, hope for the best. Does this resonate with folks?

7:33 Hopefully. I saw Sean's comment throwing spaghetti on the wall. Yeah, that's a good >> Yeah, yeah, yeah, yeah, I mean, the drilling nowhere thing is pretty dangerous, too, because it's also just like you can also see all these totally unrelated like spurious correlations that actually have no nothing to do like it might mislead you. Like red herring is basically, right? That can just totally mislead you around the cosmos or something else is going on. >> [snorts] >> And it's not coming from Um, well, I'm sure you'll get into like the right way to do it, so I won't spoil it.

8:05 Yeah. Okay, cool. Sorry, my computer applications are going crazy here. Uh, okay, cool. So, we're going to go through a pretty simple framework to understand or to ground things in as specific of a way as possible. Um, and this framework actually lives in our open source repo that Shawn could share the link to. This is the AI analyst system. It is open source on GitHub. Anybody can clone it.

8:37 Everybody can use it. Everyone can build upon top of it. Uh, it's got many different skills agents. Effectively, this is a agentic agentic system that mimics what a senior product data scientist would behave like in the analytics workflow. So, think of it as the AI analyst system plus clock code, you have a very, very capable intern or a data scientist partner to help do analysis and stuff. And obviously, within that, there's a bunch of skills agents. And one of one of those is root cause investigation, which bakes in the the framework that we're going to walk through.

9:19 And it will know what to do on its own as as kind of like a AI system. All right, cool. So, what do we actually do when we are encountering a uh, and a request or encountering the need to understand what is causing the thing that we're looking at. So, we call it, I guess, the level zero to end decomposition, and meaning there could be multiple different multiple different dimensions, multiple different channels, multiple different dimensions that you can slice and dice by. Um, but there is a way to think about how to do how to exhaust your possibilities in each one of these. So, the the sort of like the idea here is, uh, we have a level zero, you go from you go actually state the metric, and then, uh, uh, making sure that you're clear what is needed to what is being looked at.

10:21 Level one is pick a dimension, so something that you have a hunch on, and then level two is, uh, uh, within the, uh, or I guess, back to level one is find the segments that explain the move the movement, and then, uh, level two is within that segment, pick another dimension, break that again, and then you keep going until you kind of like figure out something that, uh, holistically explain the majority, if if if if not, uh, if not the entire kind of like, uh, uh, uh, movement in the metric that you observe.

10:57 So, start at the whole, break by one dimension, and then drill into a segment, and then keep repeating, um, and just being very systematic about it, uh, so that's, um, so that it's not like a, um, wild turkey chase. Let's see. Okay, cool. So, level zero. So, you know, obviously, in order to actually diagnose something, you have to define it. So, define the metric precisely. So, the kind of like the problem statement we saw, revenue was down 12% uh, what happened? That was not exactly super precise, right? Like revenue over which period you know, how do you count revenue so on and so forth we can imagine you know, like if people say something like hey engagement is down go figure it out like down in which in in which like what what does what does that mean?

11:53 What what is engagement? How do you measure it? What time window? What is the grain? So on and so forth. So first most important thing is to be precise around what that metric is and where the observation comes from. The second one is the quantifying the the change. So is it absolute? Is it relative? The comparison window. So down relative to what? Um that kind of stuff. So is it down 12% for the revenue example that we gave is it prior to your four-week average? Is it week-over-week? Is it year-over-year?

12:27 What is it? So be very clear on that. And then setting the bar. So you know, like maybe something is not worth investigating for example. So you know, it may be down 12% versus like a week ago that had an unexplained increase of say whatever like 12% and you're just back down to baseline. So there then there should be no panic on that. So just making sure that stuff becomes precise and it's not just a hey it's down 12% let me start firing up all my dashboards Let's let me start firing up all my reports and stuff like that to to go off on on the on the on the adventure.

13:09 Does that make sense? Okay, cool. Now once you are able to pick the like you know, hey perhaps you're you're being um you're you're being extremely precise. You know exactly what you're comparing against, you know the exact metric, uh and uh uh and and and stuff like that. Um now, you go into kind of like the um uh the uh uh the next up here, which is level one. So, picking a dimension that explains the most variance. So, the the uh the you know the easiest way would be reach for a dashboard, for example, to look at that's uh you know like things that you have uh that's already charted. Um like that's that's the easiest part. That's typically pretty pretty simple to do. And then, you know like you eyeball it and uh and uh uh and and and stuff like that. The idea here is try to be as exhaustive as possible. So, like if you know that, you know like the segments that you care about for your business, for example, is the demographics, the uh the regions, the um the uh the the acquisition channel, uh the uh you know whatever else um like that pertains to your business that you know are pretty important and that you you're uh uh that that typically is um is something that you know your business is sensitive to, just make sure that you check all those dimensions. Um pick understand kind of like across these most common dimensions, which one explains the biggest the biggest contribution to the actual change. So, let's say like if your revenue is down 12% um and that's amounts to, for example, like $5 million, and then you look at these different dimensions and you saw maybe like region is uh uh uh uh you know like is is uh APAC is down, let's say like 4 and 1/2 million. That's your that's your direction. Like that's where you should go first even if let's say like another dimension says, "Hey, there's this I don't know like TikTok acquisition channel is down, you know, like 80%." Maybe that's still worth investigating, but it doesn't explain kind of like your your your the the biggest the biggest proportion of your pie if you will. So, just making sure you look across and then kind of like figure out the the the the one that explains the most of the change that you're observing because most likely that's going to keep that's going to point you to the right the right story or at least the the right direction where you can keep slicing.

16:07 Cool. So, which one has the most uneven impact? Okay, and then you just keep going with this. So, within that segment or within that dimension, for example, then what is the next segment that's that that's that's helpful to to keep going. So, almost every piece of kind of like data like you know, region, it might you might be able to break it down by devices, for example, or you might be able to break it down by marketing mix or marketing channels so on so so on and so forth.

16:41 So, here is where if you could keep slicing and if the contribution continues to remain kind of like a big basket of of the proportion of the of the total change, then you can keep looking at what is the most granular way and that typically is where your root cause lies. So, the specific example here is revenue is down in EMEA. That is typically too broad and like you know, like that's that's a good observation that points you to a direction.

17:16 Very unlikely that something is in Amia that is like that is only affecting it. Maybe it's seasonality. Maybe that's true. Uh or you know, some external events and stuff. Internally, it is it it it there might be more of a story to it than just, you know, simply this big piece of of the pie. And then, you know, like when you start kind of like drilling it in in deeper, for example, um in the example that we are that we're going to show in the cloud code demo, we're going to use a fictional data set that that we have in the in the AI analyst repo, where just think of it as we have a fictional company e-commerce company called Novamart.

18:02 Think of Amazon. So, exactly how how how it works. And perhaps, you know, like there's membership people. Um like there's a cut around whether a uh whether a shopper is a member or not of Novamart. And so, in this example here, it this is getting closer. Maybe it's the the members that sign the the paid members in Germany or UK on iOS is contributing to most of that decline.

18:34 Now, that's getting closer and you know exactly the isolated area within the region that's explaining kind of like the top line. Now, you can keep going as well where, you know, hey, why is iOS down? Uh it could be because it's a particular channel that they're landing on that is that is that is different. And so, you know, like as you keep going, you're going to start seeing the the specific cause of something if it indeed is an anomaly.

19:07 Hopefully, hopefully that helps. Oops. Okay, cool. So, the whole idea really is like, "Hey, if you have a dimension, um you go look at what's the biggest contribution, and then you slice it again by the other dimension types that you have or segment types, and then you keep going where, uh you know, like the more that you do it and the more that you maintain um your perspective around, "Hey, how much movement this actually explains the top line?" Then, hopefully, at the end of of that exercise or by the by the, you know, like nth nth cut, you're able to see like, you know, "Hey, this is probably the anomaly that I'm that that I'm seeing that explains the whole thing." And then you can start correlating against other other things.

20:01 Here's a reusable prompt for, um you know, what we just talked about. Uh where, you know, you can run it in any of the AI chatbots or any of the uh your favorite ChatGPT or Claude or whatever that would pretty much mimic that um that's uh that that uh that whole process. Uh and you just have to give it the the context, for example. Okay, cool. Uh so, 20 minutes in. Maybe just a to to pop in here for a second since like so, we've been talking a lot about like just the uh you know, the methodology and the framework of just root cause analysis in general. I'm sure a lot of people here have done root cause analyses. Um this [clears throat] obviously, this lightning lesson and a lot of stuff we do at the lab is more around like, "How do we use agentic systems to do this?" And I mean, I think you know, Hyde just shared that prompt and he's going to go and and demo some stuff in Claude code in a bit. But, I think one of the the things I just want to call out here why it's really helpful to have agentic systems um versus just us do it, is that all this stuff can be done at a level of robustness and depth that we just do not have time to do. So, there's if we think about like the steps we just went through, right? It's like, okay, is this is this real like line on the metrics and stuff? Is this a real problem based on the baseline? Then we form all these hypotheses around all of these dimensions we want to drill into. We don't just want to like throw spaghetti at the wall and look at everything. We want to have like strong hypotheses around them.

21:40 Aजेंटic systems like this is going to be like your co-pilot to think of like, hey, are these really like solid hypotheses around like dimensions we should check? And then they're going to be able to go down and drill down and that into that one to end space that I mentioned at like in parallel at a speed we just wouldn't be able to do. So, like in my job at any company, like I I spent I remember a couple times when I was at Nextdoor that I spent two months on some like code red, code yellow root cause analysis trying to understand why our sessions were down. And for me, just as like a human, like I'm like going through all these different dimensions even in a structured way, and I would have to keep circling back to new ones as more data came in.

22:29 Um, it's hard for me even when documenting it to kind of like, uh, keep track of all the threads and what links together. And so when you have a system like this where it can literally have sub-agents do all in parallel and then write all its finding to like the same knowledge base, you're just able to manage and store like that that context of the problem so much easier and expedite the, uh, you know, the speed of the the analysis of the root cause analysis, but also the depth as well. So, that's why how found it like really powerful um here and it's and it's nothing you know, it's not you can add some like more fancy um like robust causal inference methods to this to make sure like, "Hey, this is like you know, the truly isolated driver."

23:18 But even just at the like most naive level of segmentation um and attribution uh that which is like it's not going to screw that up uh it's extremely powerful. So, just kind of want to highlight that since that's more what this this session's going to be about and and obviously what we focus on in like our boot camps and our full week course. >> [snorts] >> But yeah, back to back to you, I. Cool. Yeah, great points. Um yeah, uh I remember it used to take uh yeah, hours to just cut cut one document and go on and like just keep going. So, um with the help of Claude, for example, and the Agentic system, you'll we'll we'll we'll see it we'll see it run.

24:01 Uh okay, cool. So, back to this uh scenario. So, again, the whole e-commerce fictional company that we talked about, Novamart, that's what we're going to run some some uh some prompts against. Um so, the setup here is we know that in June 2024, we know we have a a ticket spike. So, tickets would be um you know, call center supports around people calling in to uh to to get uh customer supports. And so, uh tickets were way up in June 2024 uh versus May baseline and um and then, you know, we'll we'll just use Claude code um with natural language prompts with the AI analyst system and uh you'll see that we don't write any code, um but it's going to do all the the drilling, the dimensions, and then we'll uh we'll see we'll see what that looks like.

24:53 So, let me share my other screen here. Okay. Can people see this? Do you see a black screen here? Yeah, and maybe just to get a feel for the room, too. In the chat, if you wouldn't mind dropping if you've never used Cloud Code before drop a zero in the chat. If you've used it, but you haven't used it for analytics, drop a one. And then if you've used Cloud Code for analytics tasks, drop a two.

25:22 >> [snorts] >> Thanks. Okay, cool. Yeah, a lot of people have have either not used it or have used it, but not for a Yeah, Robert Robert, you're a two, man. You're a two now. You You're in it. >> [laughter] >> Robert, you're a two. >> 1.5. Nice. I will be in a few weeks. Yeah. >> [laughter] >> There you go. You will be after the boot camp. You can You can officially >> You'll be You'll be three. Yeah.

25:47 Cool. Um so, yeah, so what I'm what what you guys are looking at here, for those who are not familiar, this is I mean, this is I think think of this as the AI analyst um repo that we cloned to our to my to my uh to my to my computer. So, basically the link that Shawn shared with you all, um you can use this I'm in VS Code. So, you can clone that entire repo down to your local machine, and then I can work with it via Cloud Code, which is this uh this terminal thing here. Now, it looks a little bit intimidating, especially for folks who haven't used terminals in the past or are not, you know, like maybe software engineers or technical folks, um but it really is just an interface where you just chat with it once you have it set up and installed.

26:35 So, what I'm going to do is um I'm going to start pasting some prompts in, where I'm going to say, "Hey, can you describe what the Novamart data is about? What's the business model? Etc." So, just to warm it up. Like, just for us all to understand what what this is that we're that we're looking at. And this is probably a good practice, you know, anytime that we look at a data set that we're not familiar with. What we typically do is like explore it a little bit, right? Like, you know, what are we looking at and stuff like that.

27:11 And with this company, Novamark, we have a lot of rich information about it in in in the form of the whole context of what this business is and how they make money, what kind of data we have, that kind of stuff. So, here's what it responded with. Um, what it is is a simulated e-commerce business, as we talked about, not a real company, models the full year of operations from 2024 to 2025 for online retailer, so on and so forth. It's giving me the, you know, has 50,000 people, sent 47,000 orders, 6.5% 6.5 million events that it's tracking, 500 products. Business model is a direct-to-consumer online marketplace with a transaction with transactional you buy some stuff with a membership subscription. So, kind of like a $99 Amazon thing, Amazon Prime.

28:09 And it's it's got we've got data around acquisition, engagement, monetization, retention, so on and so forth. So, lots of these information that it just gives me. And I'm just asking it in in in in English. So, I'm going to do a follow-up prompt here cuz we already know the story a little bit. So, how is ticket volume over time look like? I'm grammatical mistake, but that's fine. Not English major. Can you make a chart?

28:40 So, in here, I'm actually asking it to explore the data a a bit. So, um the setup, we know that as part of this as part of these this uh data set, we have we also track ticket volumes for call centers. And so, you know, I'm interested in what that looks like over time just to confirm again back to our back to our um uh framework just to confirm that what we're what we said in the slide is actually true. And so, what's better way to to to see that than to uh to actually look at it charting out.

29:16 So, what it's doing is it's basically looking at the the instruction files. Think of the repo like the the different folders and files here as instructions for Claude to behave like a senior data scientist. So, what it's doing is uh hey, I've got a bunch of instructions around here. So, I'm just going to go look at the right ones to make a chart, to uh explore data, figure out where the data is, that kind of stuff. And then it's self-healing. So, anytime that it runs into errors and stuff, it'll figure it out on its own. I almost never read too much into what these are cuz I don't understand most of it. Um but it will fix itself and it will figure out how to how to actually um you know, keep going until it can't figure it out, then it will tell me like, "Hey, here's here's something that I need from you."

30:06 So, now it's done. Uh it's done doing the charting. Uh and uh it also summarizes sort of like the uh the uh the the findings here. So, here's the chart. So, I'm just going to look at this um and this spike here is something that I'm interested in sort of like taking a look. Uh and uh and this is and and it bakes in sort of like the uh storytelling with data best practices for charting in the in some of the skills and agents.

30:42 Uh and so that's why this looks very different than when it's charted here versus if you maybe just prompt Claude to be like, "Hey, can you graph can you make me a chart?" For example. Hey, a couple questions. Hi. Yes. Just to uh just to pause for a second. Um Question, do we need to call out Question around this being synthetic data and can you use this on real data in a real company? Yeah. Yeah, definitely. So, you can connect it to your data warehouse, your data sources, you can give it CSVs. There's native connectors for I think Snowflake and DuckDBs and all the other and a few and a few ones.

31:31 And certainly for the stuff that's not on here, for example, like Data Bricks, I don't think it's part of it. You can ask Claude you can work with Claude to get that set up, for example. But yes, this is portable to you know, real data everything. And then questions around Can you use this Is it just Claude code or can you use GitHub Copilot? Can you use it with Codex? Um I just say with GitHub Copilot, we have not developed anything, but we have multiple students from our boot camp and also our five-week course who have extended this um to leverage GitHub Copilot. I know a lot of companies, especially if you're working in like banking or healthcare, you're limited on what you can use, but and and you probably you probably do use Copilot at least. Um so, we haven't developed anything, but we could probably connect you with some of our students who have.

32:26 Um and we're talking about building at least some kind of frameworks around that, but we haven't yet. And then hi, I know you've messed around a little bit with Codex on it. Yeah. Yeah, and I think Shravya is going to develop the Codex >> [snorts] >> repo for it. Yeah, so I raised my hand because I wanted to give a quick introduction. Hi, I'm Shravi Madipalli. Sorry for giving an introduction so late. I joined it right now. I had an appointment. So I I am one of the AI Analyst Lab founders with Shawn and hi.

32:55 So great question about the Codex. I'm currently building that right now and we plan to share it in the advanced boot camp. And great question about the Copilot, too. That's something that I want to dabble in as well. Have AI Analysts everywhere, guys. We are in cloud code now. We want to get to Codex. We want to get to Copilot and also open source. I know there's a bunch of people that have legal constraints in their companies. So we want to see how can we build systems that could be maybe sitting locally in your service. So yeah, stay tuned.

33:26 I I I did tell the I did tell everyone that you were at the spa, Shravi. And I'll say you're glowing right now. You're glowing. You look very happy and refreshed. Like you're not coming out of like meetings on like your like fifth coffee like me. Like you look you look you look like you had a had a nice relaxing day. So thanks for thanks for jumping in. >> guys. It's amazing. Totally recommend this. Along with a bunch of our courses, I recommend this resort as well.

33:52 Nice. Wow. We'll we'll put it on a slide. What whatever the resort Shravi is. Yeah. And yeah. Yeah, running locally would be awesome. Yes, we are looking at that right now. Right at this moment. So okay, so the next prompt here I'm just I'm going to give it is can you do a root cause analysis on a payment ticket spike in June? And then it's going to run off and do the uh uh do some stuff here.

34:24 And so uh So what it's doing is first it's going to verify that whatever spike I'm giving, it can actually quantify and characterize it, and then it will drill down, and then it will have to sort the different dimensions that we talked about at the earlier introduction about the data set prompts, and then it will sort of reason its way to figure out the exact you know level zero to end decomposition method. It will go in one, take a look, and then if it's a dead end, zoom out, keep going, and all of that driven by the hypothesis.

35:05 So, confirmed, let's see. June payment jumped 41% of all tickets. July fell back down. Crucially, attempts grew smoothly. So, it's now reasoning on its own around hey, this is the observation. This is my next step. This is a way for for us to follow along in terms of what Claude is thinking and what it's taking in terms of the the the next steps and the different things that it has in its repertoire.

35:36 So, again, hitting some errors, and that's cool. It knows what to do on its own. And so, it fixes it. So, seems like it's already found some root cause, and then so it's now building some charts around it, and then writing it up, and very soon we will be able to take a look at what it actually comes up with. All right, here we go. So, it is now listing some of the findings that it's got. The root cause defective iOS app release version 2.3.0. So, this is pretty deep. Like, this is actually a few layers deep in terms of the the the different levels that we talked about.

36:19 Like, how many how many times do you slice and dice? It effectively has done quite a bit of dimensions and segmentations in the different tables that it's got access to in order to arrive at hey, it looks like this is the isolated incident incidents here. It says why I'm confident. So it's got skills to triangulate its own findings and then figure out that it is not, you know, like just making things up and and and stuff like that. It's got the different ways for it to check itself.

36:51 And so, you know, hey, it's like hey, I'm I'm here's why I'm confident. Now, assuming this is all cool, like we still have to check the working stuff. The here's the so what. So it doesn't just kind of like stop at hey, here's the, you know, I answered your question, that's it. But it keeps going in terms of hey, what does this mean in in the context of the knowledge that I have about the business that you have of the data that I've already looked at, what usually would mean that we can we can use this information for basically. So, um So I think the kind of like the I'll have it write it out so that we can see it even better than on on just the screen here.

37:36 So, I'm just going to do can you write this report along with the charts to a report in Google Doc. Actually, it outputs a chart, so I can look at this. Uh so, it basically isolated to hey, a version of the of the iOS release shipped on this on this date and then the payment defect was detected off of the the the tickets and then something was fixed and then it was fixed a few days later and so it dropped down back to the baseline. So that's kind of like the, you know, like one graph that tells them all kind of thing for its conclusion. So I'm just going to give it can you write this report along with the charts to a I mean yeah, uh spitballing not nothing nothing very formal English here.

38:29 Um so it's going to it's going to produce a Google Doc and and then we'll see kind of like what that what that looks like once this is done. Hopefully you guys are following along in terms of just you know, I know this is pretty quick and you might have a billion different questions around hey, how does this work? Like that kind of stuff. If you clone the repo yourself like right now, you would be able to do the exact same thing that we've that we've done.

39:04 Export skills now it's looking at MCPs. So my cloud code is connected to Google Docs as well. So that's why I'm able to have it do the um uh do the export and it knows exactly what to do. Now it's probably going to take a little while but I already have the thing produced. So I'm going to show you that as we let it run so that we don't just sit here and wait wait for it to do wait for it to to load.

39:43 Uh let me see. I'm going to share the screen here. Can you guys see this? Google Doc? Yep. Cool. So this is the kind of like once it well once it it's done running its thing, we should be able to see something very similar which is hey, here's the root cause analysis around the June 20 2024 payment spike. Here's the conclusion. Here's the impact of it. Here's the evidence chart the same spike that we just saw in the in in flat code itself and then it goes through the different evidences of why this is an issue, why this is why this is real based off of the data and why this is confined to iOS only. It's giving me all these detailed information around what it's done. So, the different I don't know severity issues, uh order linkage flips, so many different slices and dices that it's able to do and that it's able to summarize. And so, um gave me some judgments around why this matters, uh recommended next steps, confidence. So, at the very top there's the uh how confident it is in this report. It says pretty high and here's the itself grading itself in terms of based on the stuff that it's done, the findings that it's it's looked at, um why it's why it rates itself uh pretty confident in this uh in this analysis.

41:14 So, um you know, and sources, queries, everything is logged, uh that kind of stuff. So, that's basically the demo. I know it's pretty quick, but just wanted to just show people where as Shawn said a little earlier, if the if these were done by hand, it would have taken probably quite long. Versus if we systematically just prompt, for example, um flat code that uh we're able to get to the same, you know, like probably a much more robust um and exhaustive understanding of uh of what the actual root cause is.

41:53 Yeah, I mean, it's not even just like the the long the length thing. It's just like we wouldn't do some of this stuff. Like there's just like stuff that I would I just wouldn't do. I'd be like I have I can't stare at this screen anymore. Like I got to get outside and touch grass. Like I'm losing my mind doing this root cause thing. Like what what am I doing with my life? And now I can have a agent do more of that. Um there was a question here high around um if the event is driven by multiple causes, will the agent only surface the one with the biggest contribution?

42:29 Um it will surface everything. So, things that would contribute to like if there are multiple causes, it doesn't just go to the the one and only or the biggest or anything. It would flag that um you know, hey, I'm seeing two anomalies here. Both probably need to be looked at. Um so, it really it it's not hardcoded in a way that's like, hey, just find me the deepest, but really reasons itself through the uh all of the driving factors that could be possible.

43:01 And then there's a question also around um how do we ensure our repo holds up across different models, Sonnet versus Opus, as well as model updates. So, model updates, we test out. So, right now, like we're starting to test it out on Codex with Codex. We're starting to test it out with um some open-source LLMs. We haven't tested out with any of the Gemini stuff yet. Um with [clears throat] all of the Anthropic models, we basically test out every model update. Like even in our last five-week course, uh Opus 4.7 came out a week before our course started, and we actually tested some stuff out. And we we re- we changed some things in the repo itself. We have like a plus version of this repo that our students get access to that we updated for 4.7 because it was going off the rails a little bit.

43:47 And we also re-recorded some content. So, we actually don't only update the repos, we update our course curriculum that we run every month for the new model updates, and we'll continue expanding that to other frontier models and open-source models as well, and maybe even something around like GitHub Copilot. In terms of Sonnet versus Opus, this does not work well with Sonnet. There's probably some tasks that do work well with Sonnet, but uh the we've been kind of working at this for I don't know, like maybe like 8 months at this point. We were like trying it and then like it didn't really really click until February with Opus 4.6 where we were pretty confident.

44:23 We're like, "Okay, we should like really something open source cuz it works quite well." But prior to that like 4.5 didn't work great with Opus. Sonnet doesn't really work that well for a lot of the stuff. I mean like this works well with somethings, but it's like very confidently wrong. So, that's why I don't like using it with Sonnet. Um >> Yeah. Regarding Sonnet, what I wanted to share is if you're sure like it's purely execution, you there's no like analytic analytical thinking or reasoning, uh and it's like executing the exact uh you know, rules that you've put in in skills, you could try using Sonnet. It's not like it's not going to basically think by itself. [clears throat] But for things like this, the root cause analysis where it has to think through and you know, question it and and also like can kind of come up with some validation loops, you uh you'd rather do Opus.

45:12 Yeah, anything with reasoning. Um Yeah. Yeah. We should probably do something. I mean, we just there's so much everything's moving so fast. I'm sure everyone here always feels like they're constantly behind. I definitely do. And so like there's such a big backlog of things I want to do in terms of like I'd love to be able to go in there and just uh like have for every single like most granular task have it assigned to like the the the cheapest model that does it um to the highest quality, but you know, obviously haven't had time to do that.

45:42 But if anyone else wants to and share it with us, then that'd be awesome contribution. Yeah, that'd be so cool. All right. So, just to wrap things up, um you've seen a bit of the AI analyst system and we have an upcoming boot camp just this weekend. So, in 3 days, we're going to have a 2-day boot camp that goes through building your own AI analyst. So, many of you may have questions around, "Hey, how is this built? How can I build it myself? How do I use it in my company?

46:15 How do I do X, Y, and Z? How do I build build on top of it?" That's exactly what we're going to go over. Like, you can use the the the repo yourself, like completely open source, free. You can just use it, figure it out on your own. We also have a bunch of free content as well around it. But, if you want to build with us to learn the how do you design it? How do you actually do skills and agents here? How do you connect to the different tools that it could have connect to different um uh context in your company and stuff like that. Um 2-day boot camp, 4 hours each day. So, Saturday morning uh Pacific time, Sunday morning Pacific time, 4 hours each. You're going to be in a room with many different other folks who are doing this too to um to understand how to actually build agentic systems um and the patterns associated with it. Uh for attending this lightning lesson, we have a 20% off coupon code or discount code, RCA20. And uh this expires tomorrow end of day. And uh you know, we'll paste in the link here as well to the actual boot camp that you can sign up. And you know, ping us with any questions that you may have or or anything. Uh we would love to have you there.

47:29 Yeah, and then just a couple other things. There was a question around uh unable to attend the weekend boot camps. >> [clears throat] >> Can I access the content outside the boot camp? So, um couple options here. And yeah, totally understand people can't go on weekend boot camps, especially people with like kids. Um I don't know how Siraj teaches these boot camps. Uh she's she's nuts. But, uh like I have no life. Like, this is just It's I do. So, this is fine. Like, my dog, it's fine.

47:58 Uh but uh the we do have some people who could not make the boot camp, but everything is recorded during the boot camp. We release the recording on Saturday and Sunday about 2 to 3 hours after the session ends. Um once the Zoom like processes everything. Oh, as well as all the kind of material and uh resources covered. So, we had a few people in our boot camp uh last month just go through that async recordings, and it went really well for them. Um something we've talked about doing is like doing another office hours during the weekday um following the boot camp for people who couldn't make it live.

48:39 >> to Slack, right? Sean, you get Yeah. Yeah, yeah, yeah. We basically have a Slack channel for every course, and so uh even though you're going to do it async, you're going to be added to the Slack channel. You look at all questions people had, all discussions, all outputs, and learn from where people, like, you know, had questions and outputs from, and you could ask your questions there, too. And we'll reply, but the most important thing about these boot camps and that we are actually really excited about is the community aspect. We have people from multiple levels, from VPs of data to, like, you know, act staff ICs, and they learn so much from each other by sharing with each other. So, uh those Slack communities and all are going to be really helpful, too. Even though you're async, you could just, you know, be have a conversation there.

49:26 Yeah. And then um Yeah, at some point we'll do it on the weekdays. We just haven't haven't done that yet, but we will definitely consider doing that um in the future as well. And the only other thing I wanted to add because I know like half the room when we asked like the 0 1 2, have you never used Cloud Code before, have you used it, have you used it with analytics, there was a lot of um from So, for people who um So, you can join the boot camp and never use Cloud Code before. Like, that's that's totally good. But, if you're like not ready to do like a full-on boot camp and you just kind of want to like get started, get it installed cuz you've never touched it before, or you're not familiar with the terminal, we do have like a a pretty reduced price, like 3-hour intro workshop. We just ran it for the first time this past weekend.

50:19 Actually, there's some people in here like like David and Robert did it um last weekend with us. It's 50 bucks. Um we're going to do that again. We're going to keep doing that monthly, I think. So, we'll do that in a few weeks. I think on June 10th, so Wednesday morning. Um so, what it's like 7:00 to 10:00 a.m. PT. Um Yeah, thanks Robert for the for the shout-out. So, yeah, it's like a very mini boot camp. It's um we basically just like walk you through live how do you install a Cloud Code, how do you clone a repo, how do you download the data, how do you run your first um analysis. So, it's pretty introductory, but if you're kind of in the in the haven't used this before and you just want to get set up, then that's a really nice first step.

51:08 Yeah, Rich. Yeah, that'd be a good one, I think, probably for you. Since it's coming up. Yeah, we'll paste the link here as well. Yeah, what we'll be covering in the boot camp. Uh we'll also have everything that Sean shared in the like what we did in the intro boot camp so that you get everything set up, and we'll also share with you how we build that repo. Uh because when we ran our first boot camp, that was the most hottest questions, like can you help us understand what went into the code and how you built it and you know, all of that. Because I I understand that most of you wants to take this and you know, share it with your team or start, you know, building and like, you know, 10xing your work output. And knowing the brain behind how we got there would probably benefit you, and if you share with your teams, benefit them. So, that's the intent of the boot camp that's coming this weekend.

52:03 It's actually pretty exciting even for us because this is our first time sharing with with the world. We We actually have this information not shared with anyone yet. Yeah. Yeah, so we'll be like walk through like, "Okay, what's the different components of a genetic system in the context of the analytics? What is the like design approach that you take to uh build each of those components?" Then like purposefully try to break it and then identify why it broke and then apply fixes to it. And then we'll also do some spend some time on like connecting to different MCPs like Google and Ocean and stuff like that. Um Yeah, it should be it should be pretty And then of course we'll run we'll run analysis and and walk through the three parts of stuff, but I think the best value is from uh people not just like taking someone else's tool and using it, but knowing how to like build it theirself as we kind of like enter this world where like building and and execution of code becomes really cheap.

53:09 Derrick has a very good question. Do you want to take that, Sean? Oh, yeah, definitely not just for technical folks. There's no coding. It's all just natural language. Um we are going to have different kind of breakout rooms in there as well. So, if someone is like more technical and is more familiar with stuff, then they could be in that breakout room. If you're like totally uh a a non-technical kind of person or less technical or intermediate, there'll be a a breakout room for that um to make it approachable. And then if you if you're if you're if you're already building a genetic systems and stuff and like this is too kind of like intro vanilla for you, there's an advanced boot camp that it teaches you too. Um Yeah. Um I mean, you guys saw that there was nothing coding related in the demo and a lot of the pattern is is going to be like that where once you get it set up, once you get an understanding of it, you're just chatting with it.

54:08 Hey, I do want to answer Robert's question around um I think his question's more around like Oh, framework. >> Maybe the maybe the five-week course kind of thing. Yeah, great question, Robert. So, boot camps tailored towards more of building your own systems, understanding the AI components of uh I would I would call it kind of like at the forefront of doing AI analytics. Uh we have another that we didn't really talk about here. We have another course around um a five-week course around AI analytics. It's called AI analytics for builders. Uh I think Shravya pasted the link here where we really help people to become analytically independent, meaning even if you're maybe if you're a data professional or if you're not a data professional, we give you all the foundational knowledge, the frameworks, the best practices for how to think about different aspects of the analytical workflow, just like a root cause analysis being one of those lessons, for example, out of I don't know, like 80 or 90, um that we would equip everybody who takes the course with the best practices that accumulated over our collective history of being in the field of what good looks like and what solid means in terms of analytics and uh and then go through them systematically while delegating the actual execution to AI, just like cloud code that you see. So, that is more of a five-week course where we distill as much as possible for all the different framework and and and and things such that everybody can absorb and learn those and then become really really self-sufficient in their analytics journey. So, next time when you ask question or when you think of you know, like a data question or something, you already know kind of like what how do you how do you best do it to make the most informed decisions.

56:13 Yeah, the way I think about it there's always been boot camps around how do you do data science or analytics. [clears throat] This is your kind of end-to-end data science analytical framework boot camp but instead of the execution layer being taught in like Python or R SQL being taught through cloud code. Great. Thank you. I can stay over a bit if people got more questions. I know we've been answering questions about boot camp but also questions about causal analysis or the repo or can be anything around genetic analytics.

56:50 And if people don't have questions, that's fine, too. But happy to stay over a little bit. Yeah. Let's stay over a little if folks are interested. They're speechless. All right. Okay. No questions is fine, too. But again, hit us up on uh on uh LinkedIn or um feel free if you're in the Slack community, you want to join the Slack community, um hit us up there.

57:21 Um intro to optimization stuff. We're going to kind of go over that in our advanced uh this is this is referring to Rich's um question chat. Um, we're going to kind of go over that a little bit in the advanced stuff like some of the kind of like uh, code optimization loop stuff where we where you build like some kind of evaluation scores and then can't have like the system itself kind of like optimize um, what's going on but it'll probably be like a couple hours of that course not a full course itself but which I think it could be. Maybe sometime in the future though. I think it's a good idea.

58:07 >> [snorts] >> I've been doing a lot of that at my work but um we haven't been teaching much of it. I do feel like with a with just the agents and how how good everything's getting like optimization and simulation it's going to have like a crazy um moment in terms of like analytics. Um Totally.

58:38 Okay, cool. Great. Um, join our Slack community as well. That's uh, where you can find us. Uh, you can also find us on LinkedIn. Uh, you can DM us on Slack and uh, yeah. Otherwise Yeah, let me drop the I'll drop the invite link to Slack right now. Okay, cool. We'll give it we'll give it 15 seconds. Yeah, yeah. If anyone If anyone wants to join Slack who's not in there yet. >> [clears throat] >> Cool.

59:07 All right. Thank you everybody for spending your time with us and hopefully we'll see you again in a future session or in the upcoming boot camp. Yeah, we have another free session this Friday on insights to action in product code so should be a really good one that highest leading. Um, so we'll see See You'll see me again on Friday. You sign up. All right. Later, everyone. Thank you. Thanks.