How to Design Analysis in Claude Code

0:00 something that cost teams more wasted time than a bad SQL or a broken dashboard. This is basically designing a wrong analysis. Uh you probably know the feeling, right? You spend a week building charts and cohorts. You present probably to your product leader or a VP and then all they say is interesting but that's not what I'm looking for. So today um I'm going to show you how a sevenline framework prevents that and an AI system that makes the whole process automatic. Okay, let's get started.

0:35 Before we get started, we start with a typical quiz. Can you basically give your answers at a quick poll? Hi, I'm Sean. We look at uh please let me know once it's done what is the most popular choice. So this is your question. If your VP says conversion dropped last week and you have 1 hour almost like an SLA that you reply in an hour because it's your VP, what would you do? Do you do A, B, C or D? One thing I want to let you all know, no judging here at all. I think all options are pretty common like we all actually go through all these options, but I want to make sure what would you do first?

1:15 Can you please start giving your answers? Are we? Yeah. Looks like we're getting some messages. >> Got a lot of C's. Got some A's. >> Yeah. >> Okay. >> Got a B. >> Okay, that's awesome. Okay, so is the popular ones between A, B, C. >> I think C is the most popular, but we got we got a lot of A's also. >> Awesome. >> Two ones at A and D or no A then D. Ah you know what uh so jumping so here's the thing right um I think we have some good set of people if you're starting with C but there's a lot of people that I've worked with in my career that uh we start with A and B I myself have done it I myself have started with A and B right but you know I I understand that that's the instinct to just jump in and understand what it is but we probably need clarifying questions to understand what does VP even mean when he means by you know conversion rate drop. Does he mean week over week, year over year?

2:26 What does it what what conversion is he talking about? Is he talking about like sign up to activation conversion rate? Is he talking about like people free user to upgrading conversion rate? So knowing these basic questions and also hey why are you even asking this? Do you have a vortex that you have going for to report on these things or is it just something that you're curious about? All of these things would help you like understand how do you want to go about answering that question. So most of the time especially when I started out so much of time was spent in just assuming answers for these questions and going ahead and you know doing these analysis but you would benefit a lot and a lot of cost uh like you know time is reduced if you ask those clarifying questions. One thing I want to say before we go into the next slides is that this um design analysis uh uh like workshop today would also help you for interviews. If you're you know applying for interviews, if you've done data interviews and data science stuff, you definitely see yourself actually having at least couple of interviews. For example, in companies like Meta and any tech in tech company in Bay Area for sure has a how they have a question like literally this dropped last week. What do you do? It's literally this question and they are looking for you to come up with a framework and a set of things. So uh by product along with learning how to use cloud code to learn it in your work today, you would also probably learn how to do it the best way in an interview.

4:06 Cool. So let's go to the next slide. I want to say something. What will you have at the end of probably optimistic by by minute 32 you'll basically see a structured analysis plan document the kind uh basically that you would share with your pod the pod in the sense the team that you're part of your PM your tech lead before you spend a week running queries. Basic I've seen a lot of junior DS if they have a question they jump right into analysis and start putting things in a dock on the analysis but agreeing with your stakeholders on what you're going to look at for the analysis is the first key step and there are there is a framework on how we want to do it and this is what we're going to cover today. Okay.

4:54 And you would basically see uh the final output as at the end of the session. Okay. Before we do the big demo, I want to show you something quick. Uh, this has been a style that I've been doing in my last three workshops. I want to show you something super quick and in the cloud code and we can jump in into the theory behind it and then wait, where's my claw code? Okay, cool. So, let's get started. Uh, this is like a quick demo. I want to Oops.

5:32 I was using Whisper Flow to get started and I I clicked on it quite early. So, let me just do this. My VP of product just sent me this conversion rate dropped last week. Can you look into it? So, I use Whisper Flow to get me this and Claude code. As you see, we have a bunch of skills and agents that we've built that we're going to cover in the boot camp starting tomorrow. Uh this is the initial set. This is not using the the fancy skills and agents. This is something a question framework and stuff helps us get here, right? So what does it do? It asks which conversion rate recommendation like you know what do you do and where it breaks and then what does drop compare to what prior week trial trailing week same week last year like so many things could be looked at and the hypothesis tree ranked by likelihood drop is one funnel stage not drop is in one funnel is not at all the category is product first check conversion by stage week over week and then you have things like traffic mix shifted device browser broke like all these wonderful reasons that are the hypothesis for something like this. Go get it started, right? And then it says investigation priority has checked these three things first and then draft reply to your VP. Isn't it pretty cool? It gives you a slack message as well. So thanks looking into this today. Quick questions. Which conversion rate are you seeing a drop? Any known changes last week? So it also helps you with that as well. So this I would say is the most minimum uh cloud code uh framework that it uses based on all the data that we shared with it. Um and I also have a fancy demo later after we cover the theory behind what we what we're going for today. Okay, cool. So uh what the system produced basically in the one sentence. It's not even a detail prompt, right? And I didn't give in even any instructions because this is uh all of that knowledge it gathered from the cloud MD files. It gathered from the uh existing skills agents and the context we built in our repo. And what did it was the output?

7:52 The system clarified what conversion means because there are four different conversion rates in most products, right? And it clarified what drop means and dropped compared to worked. uh it generated seven possible causes and ranked them causes the hypothesis right and it told me where to get started. Now those likelihood rankings are based on general patterns but in your company you basically give it enough context that it picks up the context that you're working with in your company and it'll help you with like what are the hypothesis for the context that you gave. This is something that it picked uh today in demo that we are showing like generic context and generic hypothesis. Okay. So the framework gives you basically the structure in which that you want to plan your analysis and the type of questions you need to ask the person uh who basically you know asked you to go do the analysis.

8:47 I'm going to give you a free prompt today uh basically that you could use in any of the web uh tools in chat claim because uh not every one of you would have access to the complic some of the agents and skills that we're building that would be uh we will be sharing them in the boot camp uh we have the boot camp tomorrow but we'll also uh share with I'll also share with you this free prompt along with this you also have access to the free Oh, give me a minute.

9:18 Uh, you also have access to the free air analyst repo that we have uh that has uh a skill for analysis design as well. The complicated analysis design and tools and skills for it that would be part of the boot camp. Okay. So, uh what I showed you is like the system working, right? One sentence in and structured plan out the free prompt I just show you gets you the most value. and also the free AI analyst repo that we have in GitHub that could also give you most of it. But uh if you want to run this investigation uh in like a repeatable fashion and you want to build those skills and agents yourself, we are running a analyst boot camp. Uh tomorrow is when we are starting. So today is the last day that you get into this cohort guys. So here we have like a claw code um a code for you to get at least 20% off. We also have a fiveweek oh it says as sync but it is also a sync course sorry for that but we also have a fiveweek course where we go through the entire detail of how what we built in these agents uh how do you even come up with these questions what are the metric trees how do we do the root root cause decomposition like as a data scientist in tech what are some uh workflows you go through we have a course coming for that called AI analytics for builders Okay, now let's go to the next step which is the sevenst step framework that I was talking about that's uh built in into our uh claw code skills and agents.

10:55 So before you run any analysis, you should be able to fill in seven lines, right? Um I call this analysis design brief. Uh it's not a checklist. It is like a thinking tool. Each line forces a decision that most people like maybe at least the most people that don't have context about how things work in data science they skip. So question forces you to pin down the metric, pin down the platform, pin down the time frame. So and this comes from clarifying what the initial question came out to be. Right?

11:30 And another very very important thing a lot of people especially younger uh junior folks miss is the decision part. So especially if VPs and you know uh pro like the head of uh something with big titles come up they do not ask hey what if I give you this wonderful analysis that you're looking for uh answering this question what would you do with it if they're I'll you'll be surprised with some of the answers some of them would just say hey yeah I just thought you know this could just help us um get more clarity about who what a user wants that to me is not a great decision. Imagine if they'd say we want to run this experiment and we want to prioritize this framework and assign engineers to go after this feature. That's the type of decision you'd need. Uh anything that says something like oh I was just curious about this is not a great decision that you want to spend cycles answering those questions with that analysis. Right? So ensuring that you have the decision ready before you go start exploring the analysis is very important so that you don't end up uh spending a lot of time on it.

12:41 And another like a bunch of things like you would need are the hypothesis. The hypothesis basically makes you commit to what to expect. So you can uh you know actually be wrong. Maybe there's eyebots as people say, "Hey, there is something happening with our new feature that just cost this." So that you could do the analysis and prove it wrong or prove it right. Um, and comparison is the next one. Comparison is where people default to compared to last month without even thinking about seasonality, natural experiments, what's happening out in the wild, some that's not related to your company, but you know something that's happening like outside like in the ecosystem. And then the next one is segments. Sigments is where Simpsons paradox and other things hide. You might see something at a higher level but when you start seeing segments then you see total sometimes you would see entirely contradicting things. New users are absolutely doing great but it's uh something like u like existing users or older users account age with more than two years are where uh we are struggling uh for this metric.

13:53 You might come up with something like that if you start looking at segments. And another very important thing is confounds. uh it is like uh confounds is the one everyone to be honest kind of skip as well. So three things that changed at once and which one caused it? There could be something like uh happening in the ecosystem happening like there's an experiment, there's a marketing campaign, how do you define which is the cause behind it? Having them listed out is absolutely going to help you get started. And then criteria uh is what separates investigation from rationalization. Right? We'll go through basically each of these uh this with like real examples and also possibly do this with cloud code in like one prompt.

14:42 Okay. Bad versus good. Before I go, hi Sean. Anything in chat that I want to know about or pause because I've been talking too much. You can go ahead. >> I can go ahead. Okay. Awesome. Cool. Uh question. So the left version looks reasonable, right? I mean, uh you've got a metric, you've got a direction, you're even breaking it down by channel, device, but which conversion? Browse to cart, cart to check out, check out to purchase. Uh like, you know, like last week compared to what? The week before, same week last year, a trailing average.

15:22 uh what what do we mean by you know all of these things knowing these could mean so many things right so the right version pins down the exact metric the exact platform the exact comparison um and you can design an investigation for that and the decision on the left if you see that it's like we need to understand what's happening with conversion so the team has visibility that's a pretty you know um I would say something that accounts to like a not a great decision that you would spend a lot of time working on. So the right version names like two specific actions like reward the check out flow or brief marketing and have a deadline you know so so that the changes um that you propose as part of the analysis the recommendations actually have uh like an outcome and um it also helps you understand how deep you want to go in the analysis or how fast you want to turn around the analysis. So the decision is the one that's another one that's pretty important.

16:28 Okay. And let's go through these quickly uh as well. The hypothesis, comparison, and segments. Um the left version has hypothesis which is I mean better than nothing, right? But pulling the data and confirming is confirmation bias baked in. So the right version is falsifiable. It predicts a specific magnitude uh specific category, specific customer type. you can actually be wrong because you're going to test in the analysis to go after uh you know verifying the hypothesis and there's a chance a a good chance that your data returns that hey this hypothesis was not true and then comparison week or week and month over month sound like solid uh analytics but last month might have had a promotion the week before might have had a holiday so the right version finds a natural experiment uh also along wrong with all of these things like a mobile removed the widget first or desktop still has it but mobile removed it basically uh same customers but different timing so all of these uh are something that you want to look at in detail when you're talking about comparison and the similar thing for segments as well trying to move on fast so that we have time uh for the demo okay cool so the next one is the compounds and the criteria so we'll control for uh for the seasonality and check if anything else has changed. That's sounds reasonable, right? That's one on the left. So, it's not necessarily bad, but we can definitely, you know, enhance it so that it actually becomes useful. So, something on the right that you see is that loyalty program restructured same week and marketing also ran a 15% off new customer promo. And guess what? You might also find something with the data pipelines too like data engineering changed the mobile tracking pixel midmon. So in reality uh I'm sure all of you deal with this as well. I deal with this a bunch in my uh like in my experience as well is that you have multiple confounds uh happening at the same time and that's what makes this entire you know designing the analysis plan and also doing the analysis quite tricky. So having that written down helps you u like go through the data and actually see attributions of these multiple styles uh to ensure that you covered for them in your analysis.

19:00 Now what about the criteria? So on the left something that looks reasonable but not too great is that if the data supports the widget theory we'll recommend restoring it. I'm not sure if that's like you know the the detail that we'd want. So the criteria on the right that we'd want to be slightly more detailed is that uh accept if consumables dropped greater than 3 percentage points on mobile but not desktop because the loyalty change doesn't explain the timing right and reject if the drop is uniform across all categories and platforms then that would mean that it's not a mobile versus desktop thing. So having the criteria locked down is also pretty important.

19:43 Okay. So uh one thing uh I think I've said it multiple times already but to make sure that I I actually nail it down is every failed analysis violates at least one one line you know so the scoping black hole someone says look into conversion and you spend three days on a wrong metric the fishing expedition right no hypothesis means that you'll find a pattern in anything I remember even at our previous place Sean used to be very good at forcing stakeholders to like guys give me a bunch of hypothesis.

20:18 You want me to go after this question. I have this hypothesis but what about uh like if if you have this question ensure that you know the product enough that you give me these hypothesis. I remember this tough conversation from last uh experience with Sean and others. But yeah um make sure that not just you you ensure that you work with your stakeholders and your teams to get uh these hypothesis written down and also keep your you know stakeholders accountable so that they have a list of hypothesis that we they want us to look into as well and yeah um another thing that you'll see is that uh like Simpsons paradox is something that people miss as well if we the overall number lies and the segments tell the truth. So ensuring that you have a bunch of segments already look into and especially not just any segment I know geo comes up very easily and you know things like u device comes up very easily but I'll tell you something in your company in your context there are bunch of segments that are not as generic um in my company specifically if people were like a paid person before or not is a segment that you wouldn't probably see as like generic segment that helps for every type of product but it is absolutely something that shows uh always having Simpsons parad effect in my uh like you know context. So what are the segments that in your context in your company that you would see a difference and you ensure that you look through them uh as part of this analysis right and then the attribution error is the other thing three things changed at once the confounds basically and you picked the one that confirmed your theory but guess what there could be so many other things that you missed out on if you did not write them at the get-go in your design analysis done and then 2 minutes of design prevents two weeks of wasted work. um uh in in the sense 2 minutes is I'm not talking about the entire design plan coming up with it but actually the framework check did I get the uh question clarified did I get the hypothesis all the hypothesis listed out you know that little uh ensuring that you check those boxes um would help you definitely like you know get to a place where you don't end up two weeks from now as like oh no I need a V2 of this my V1 was entirely like not useful. So ensuring that you go through these would definitely stop you from having like a failed analysis.

23:03 Okay. So now a little sneak peek into the system that we do today. Uh we basically do uh kickoff analysis design scale. This is I heard myself for some reason. Uh this basically kicks off all these three uh like you know this entire system and in you you might have seen this in your place as well. You basically have a question and you come up with a V1 analysis and then you share the V1 analysis. You have this meeting for it.

23:36 You share it with your stakeholders and then you get a bunch of questions or stakeholder feedback and then you take all those feedback and you take all those questions and you run it again come up with a new analysis plan that incorporates the feedback and also your van analysis and then the V2 analysis pops out. This is the system that we do today because this is the regular cycle. That's something that I've seen at my place and I'm sure most of you see this as well. question, V_sub_1 feedback, V2 design, and then V2 output. So, I basically have this entire loop. I'm not sure if you'll have time to go through the entire thing, but I can definitely share with you uh the first one, uh the design for the V1 and the V1 analysis plan by itself. Um and the findings and what do uh V2 came up with. The last two things I probably not have time to demo with you but I can share with you the output of uh what's something I have already created that I could share with you.

24:40 Okay. So one thing that I want to uh again talk about is in the boot camp you don't just watch the system like how we're doing today. You basically will build it. So the skills the agents the document pipeline all of it becomes part of your personal AI analyst repo. You could already have this uh because we do have a free AI analyst repo, but this boot camp would cover a bunch of new agents and a bunch of new skills along with helping you create them yourself so that you wouldn't need others to train you or like you don't have to watch this, but you could actually create one.

25:18 And maybe you do this in your company wherever you are, but you train other people on how to do this. So yeah, uh today is the last day to get into the boot camp. Tomorrow uh this weekend is when we are going to have the boot camp. Okay. So today's analysis we basically I think uh jumping into the demo. Uh let's I'll try to do the demo in the next 10 15 minutes so that I have the last 10 15 minutes for the questions. Let's see how we'll do. Uh so what is the uh demo that I'm going for today? We'll be doing cart loop online marketplace for homeg goods.

25:57 This is something that you probably might have seen it in your companies if you're a data person. If you are a PM, you probably do this to your data people as well or maybe your own analysis that you want to kick off. Basically, there is something with repeat purchase rate. Let's say a PM has a hunch, right? I think our repeat purchase rate dropped because we removed the post purchase recommendations widget last month. Customer support tickets mentioning I used to see suggestions after buying have increased. This is absolutely a hunch currently. It doesn't have any data that's backing up. it doesn't have any uh metrics that say because uh you know they probably uh maybe don't have an experiment as well that looked at this removal and looked at the metrics change. Um ideally when these changes happen you want to put that behind an experiment but there are a bunch of reasons why uh in companies this doesn't happen today every change is not behind an experiment. So the data team gets asked these questions. Um and let's get into understanding how do we do this. Okay. Uh what is this demo tool going to be? Uh we are going to basically take the hunch and basically go through the PM's hypothesis. How do the agents sharpen these hypothesis?

27:24 Scan through the whole thing and do the terminal. And then we'll have this for you. The V1 analysis plan, the V1 findings combined with feedback and then V2 analysis plan. So the terminal is basically agents thinking and documents is basically what you'd share with your team. So let me jump in into the cloud code. Anything quickly Sean or hi before I jump in?

27:56 >> Greg says uh I've been the source of these hunches many times. transparency. >> No, no. Uh, you know, no judgment here, Greg. >> Well, you know, when I I I kind of posted on this on LinkedIn, I think like the whole analysis design thing is like a translation kind of process when an analyst gets hunches or questions from or comments from stakeholders. But I also think like it doesn't have to be the the analyst doing that. can be like anyone, right? Like if like stakeholders, non-analysts have have their hunches like they can go through this analys analysis design process to kind of like get it a little more into the language of the analyst. Although I don't know if I was a if I also had a stakeholder that was like, hey, do this do this very specific type of analysis like Claude Code says you should do it. Then I don't know how I'd react to that either.

29:01 I like I mean I just like the translation piece here on either end. Okay. So what I'm doing I'm calling it analysis design skill here which in turn will call a bunch of skills and agents to get this started. So all I'm doing is whatever I just told you I am uh copy pasting here I'm giving it the data as well. Let's see what will come up. Okay. So, analysis design pipeline. Uh if you see what it's coming up with, give me a minute. I didn't give it. Uh so, if you see this, it basically is talking about uh analysis design pipeline. I'll investigate this in three stages. Hypothesis sharpener because there's already a hunch, right? How do we make sure this hunch is actually a testable claim? So that's the first one.

29:58 And then the second one is confound scanner. Find everything that could make this wrong. Um and the third thing is the investigation plan. Prioritize what to check first. So these are the three stages that is going to run this. Okay. So now stage one hypothesis shortener. It basically is looking at the stated cause, stated effect, implied metric, implied time frame because it was a pretty detailed u like you know hunch with details and also the assumptions.

30:26 The assumptions was widget was the primary driver the drop is uniform like no other changes occur simultaneously. Those are the assumptions obviously because if that's something that you have a hunch and then what is missing it basically talked about everything that's missing here. So it has a testable hypothesis and also a secondary hypothesis. It also has the metric definition. It has what is like you know uh gives you notes like the nuance as well like repeat purchase here means any subsequent purchase not same category purchase like uh and also talks about like this is going to be different if it's going to be that. And then it talks about comparison groups, natural experiment like what this versus this and widget users versus non-widget users, before versus after the same population and stuff. And then it also has accept or reject criteria. Accept if widget users show greater than 2x or you know things like that. Reject if you know so and so. So these are like a bunch of things like this is the analysis design brief that we have and investigation priority.

31:35 uh what are the multiple steps that we have and it prioritized it based on why we want to prioritize it right so and then it has a stage one summary so what it says is testable hypothesis best comparison and what is the key insight now it starts stage two so it gave you the debrief of what it's going through fully and then goes through the stage one gives you debrief of what's your hypothesis sharpener and then goes to stage all of these demos. So this is a demo, right? All I'm trying to show you is what is the capability of claw code here. How do you do this at your work is something that we could discuss in the boot camp and also something that you basically give it the context give it the memory and skills that your company and your business would you know have the nuances of your area which it would take into consideration while coming up with this different stages and you know the confounds and other things. Okay. Now what is the stage two? It started the stage two and there's a confound scanner in the stage two. It's basically going through uh the claim removing the post purchase recommendation widget led to a 3.77 ppb decline and uh it comes up with what are the other concurrent changes it found. So how does it uh in in an ideal world in your company how could you do this? What I do is I have MCPS attached uh to my experimentation platform. I haven't seen it is attached to my uh Slack uh most important Slack channels where we keep sharing our experimentation updates. It basically my system my claw code has skills and agents that grasp these context from these multiple places that you gave uh early on. And this is to demo you that we kind of already this already picks it up from the existing data set. But that's how you would do in your like workplace.

33:39 Okay. So it goes to the confirmed scanner. It found hey product has these changes, marketing has these changes. Technically these things happen and there are also data quality threads the severity of them. And what are the sele selection biases like a a bunch of things a bunch of alternators and their plausibilities and then it gives you a an entire confound scan report right uh and then this is stage uh this is the stage two summary as well and now at the end we get stage three. So this is the final analysis brief uh it basically goes through the question removing uh the widget and then the decision the hypothesis. So this is like the seven uh uh like know frame workflow that we had and this is what it gives. Let me give plot code.

34:31 So okay cool. So uh review the plan or should I proceed with V and execution on the data? I will ask for it something. Let's see if this going to do this live well or not. If not I'm going to share with you the V doc. Okay. Can you talk to this? Can you move the design plan into a Google doc that I could share uh with my stakeholders? Use the Google Doc creator and export skills.

35:02 Let's see if this comes up. Uh so until then I let me read. So it's basically what it's doing currently um is it's checking a bunch of skills and agents that I have to ensure that my Google auth is correct and you know it it has the Google export and the review skill set. Okay, while this happens uh anything that's happening in the chat that we want to chat about this might take a few minutes.

35:36 Sean or hi? Yeah, Michael has a question and this is probably prevalent I think uh from a couple folks as well. How much do you need to explain your company product to the tool or does Claude figure it out somehow? >> That's a great question. Um like if you I don't know if you all tried cloud code in the before version. Uh when compared to that it needs a lot less information for sure. To me, in my experience, if uh if you have Oh, looks like my my token expired. Sorry.

36:16 Mhm. Okay. Send it to Let me stop sharing my screen so that I give it my O and then I can get back. So, I was kind of winging it to try this online uh like live, but I mean this is not uh this is something that you'll definitely run it and you hi and Sean, do you want to answer that while I do the O setup for Google? >> Yeah, I think so. I think it's pretty good at reasoning and it's pretty like intuitive in terms of like even if you didn't provide that much detail, but obviously it's going to be way better if you do, but it doesn't necessarily have to be like you sitting down and explaining everything like I will literally so I have it hooked up hooked up to like notion MCP and Google MCP and I also have repos with a bunch of like my past analyses and Um, I sometimes go as broad as I'm just like go search not notion in Google or and I have it connect to my Slack too and my Slack for like this topic around this product and past past things I've done in this area or if I want it to be a lot more focused I'll just copy and paste a bunch of notion links or Google doc links like hey read through all this relevant information documentation.

37:39 Um, also something extremely I mean like cloud code has OM, right? So it's like really good at reading through code bases. So if you can just give it the repo to your product's codebase, um, it can make connect a lot of the dots in terms of like how things flow through as well. Um, so yeah, MCPs, other sources you can connect to make this thing like way more powerful. Um, so I don't know.

38:10 I'd say the more information you give it the better, but like even like no information is going to get pretty good. >> Yeah. If you if there there's a there there's also a really fun exercise to do is that if you have it just crawl your company's website, it could it will be better than if it didn't have that context. So any incremental information you provide it gets that much better. >> And then Greg had a a question for you.

38:41 Shia, could you remind which um information sources cloud code accesses here? Like what is it interrogate to check for other concurrent changes to the target metric? >> So you're talking about this demo per se or generally at work like you know where it could get this information from? And you either one either the specific case here or but I think Shane and uh also answered it generally and the the rule of thumb is the more information the better. Um >> yeah yeah yeah exactly uh it so it cla is so darn confident it if you don't give it information it's going to take information from what it knows obviously from the web about your specific area or context and the more information and the nuance you give it your work the better it gets and the first time you work with it the feedback you give it and the next time and the next time and by the time you work on a week or two weeks with it it's super smart already and it does work like way faster than you would um if you had like a bunch of these uh you know skills in place already on how to talk to it, how to create these things from the get- go.

39:53 So guess what you see this uh Google doc created and formatted be analysis plan is it in this deck. Okay, let me oh I'm sharing my cloud code. So let me share with you what it produced. Okay. Uh this is what it produced guys. It basically gave the V1 analysis plan repeat purchase the context, the questions, the approach, the known risks, the investigation priority, what would change your conclusion and the deliverable.

40:26 This is something that you saw that purely like was generated by cloud code. And uh once we have this it basically would we we could use this and start the finding. So uh ideally if I had time because we only have 15 minutes I could share this analysis plan feeding that to claude code and claude code generating the analysis itself as well and then we could come up with hey the this is the feedback on that analysis. Can you come up with a v2 design tool? So, I have all of that for you that I kind of already did and I'd like to share that. Give me a minute.

41:11 Okay. Uh so what it did uh is I fed it uh I basically did this on the background. I got uh the V1 analysis plan and I got it to execute on the analysis plan because it had the data set handy and then it I gave it feedback on it and then it gave me the V2 analysis plan. So that's what you see. This is the second one. I couldn't do that in today's demo because it's too long but that's what it says.

41:43 It's analysis plan V2 revised after V1 findings and stakeholder review background. In V1 we investigated so and so. It basically gives you all the context to and it gives what we even establish that there is a real material drop and you know all of that but it also talks about what we even could not resolve because we we generally don't answer all the questions on the earth about uh like you know something in the first iteration. So you go in the second iteration and then we talked about what was not handled and then it talks about the stakeholder feedback incorporated.

42:16 It takes the stakeholder this was the concern and this is how the V2 addresses it. So it talks about the revised questions it talks about approach and you know and a bunch of things. Yeah I mean each uh company does it differently. I just shared one approach. Uh the other approach is a set of questions a set of segments and all of those written down as well. So yeah we have 14 more minutes but I want to share something pretty quick with you all.

42:46 This generally I always like it uh at the end of uh any session we could ask Lord hey can you share the architecture of how you solved the above. So uh it basically gives you everything that it did. This every time you work on a complicated problem, you want to understand like how it went about and did it. We'll see basically the entire loop. You could ask it to be even more detailed than what it is what it's giving it right now. So this is what it is. SL analysis design. Now repeat purchase rate dropped architecture preview shows the user three-stage plan data profiling uh come out coming up with this and then the stage one came in and then the stage two and then the stage three and then the o pre-flight we had an issue with o ideally if you add o it looks like it's been a week more than a week I did my o so it triggered my o but if not you wouldn't have this stage let's say you have this and then google doc creator we also have agents and skills for google creator export. Uh and yeah, this is how the entire thing it talks about how many skills were used, how many agents were invoked and what tools were used and what MCP was used.

44:17 Okay, this is where my demo ends and we have 12 minutes. Ahmed, I have a question. Do we cover any data visualization dashboarding in the boot camp or sixe async courses? >> Absolutely. uh that's literally uh one of the very important uh aspects uh of uh the courses. So we basically cover even more than that Ahmed uh we cover visualization dboard coding and the most important thing storytelling and also what you want to visualize in dashboard basically question framing as well because all of those are very important into understanding that. So yes, we do cover that in the uh six key course and it's not async. It's totally sync like it is basically live boot camp. So live course that Hi Sean and uh I we we run.

45:10 Yeah. Uh I'm sorry for that typo in the slide but yes we do cover that. >> Did I miss any questions from anybody? Anyone else have any questions? Feel free to come on. come off mute as well. >> Yeah. >> Raj, do you want to speak to your question? >> Ah says take my money. Okay.

45:42 >> Yeah, absolutely. Hi, thanks for the nice presentations. So, I just want to highlight one of the problem statement I am getting on a regular basis like I have the semantic layer, I have the visualization part ready for the AI part. uh what I'm struggling is how to connect the particular semantic layer to the visualization part like I have the data in my database like uh data warehouse like snowflake and I want to visualize uh using GPT or cloud anything anything that is working perfectly fine.

46:14 So how to connect that particular piece that particular snowflake data to cloud and that that should be refreshed on a daily basis. So that is some that that is something I am facing a major challenge in between like so are we covering that particular piece uh in incoming upcoming segment like how do you are you planning to do that >> I think we'll go more into like the full like how do you implement this into like a production environment more in our fiveweek course. So we'll in our fiveweek course we actually will work with cloud code. We'll also work with some other third-party tools uh like enterprise tools that do some of those for you. Um this weekend what we'll talk more about is like how we like build a genetic system from scratch and I think we could work with you in it actually to be like hey let's try and build an agent and scale that refreshes some like HTML dashboard on a regular basis. We go into MCPs a bit as well too. So, I know that there's like depends on like what your company has, right? Like we have like um Sigma at our company and they have an MCP where you can create dashboards directly from cloud code. Um so like we could potentially play with something like that probably more in the fiveweek course, but I don't know maybe we could try and mess around with it tomorrow or and Sunday. We're basically going to have like how tomorrow's structured, we'll have some like lecture time, example demo time, and then we'll have big blocks both days for about 90 minutes where we're going to do breakout groups um from like beginner intermediate to advanced >> and everyone will be kind of like building their own thing with one of the instructors involved. So, if that's like a project you wanted to work on during one of those sessions, I could definitely like try and play around with that with you. Um, but yeah, obviously production pipeline's more involved. So, probably like more the fiveweek course.

48:23 We do run a a kind of two for one thing with the fiveweek course. So, if anyone takes the boot camp and wants to do the full fiveweek course afterwards, we just deduct whatever you paid for the boot camp from the fiveweek course. So, you basically get two for one um without having to commit to the fiveweek course up front. Did that answer your question? Yeah, that answers my questions. So, uh there are like a particular two problems I'm facing like one which I've mentioned another one is the deployment part development the particular dashboard into some somewhere so that uh everyone can I can I can share with any any stakeholders on a regular basis or they they can refer that particular link so that they can get to know about the numbers without depending on me or the my team. So those are the two problems we have.

49:17 >> So >> one way one way I've done it right now is so um so like if you were to create like customized dashboard right you have to host that somewhere internally at your company which is totally feasible. We're not necessarily going to go into that because every company is to is completely different with like how they want to host things internally. So there's there's like the MCP versus like with like uh like your whatever your BI tool is route if it offers it. Another thing I actually do is I have a skill where uh you run this skill and it refreshes like a Google doc or a notion page for like a very particular stakeholder that has like a full-on report of everything I know they want.

49:59 Um that's like a kind of nice like hacky middle ground way to do it as well. And that's definitely something we'll cover uh this weekend. >> Yeah. And about the BI tools, we as a team trying to eliminate the BI tools because it's a pain. It's a pain to maintain and build >> using BI tools in the era of like AI. So that's why that's why I'm trying to uh save some time and eliminate the BI tools >> but happy to know more about that. But I think the major problem I'm facing is about connecting the data warehouse to the AI part where I'm so yeah happy to know about that particular piece.

50:39 >> Yeah, I I am I'm on your side about eliminating BI tools. I know not everyone is but uh >> I never want to maintain a dashboard again. So I like >> I like this idea of eliminating them and even getting stakeholders uh self-s serve empowered enough where they can be >> building their own dashboards themselves because if you think about it like >> they know what they want. They know that the question what they want but they need like the trust in the data. So if you we can create a system where they're like hey I want to add a filter to this.

51:11 I want to add another dimension but I don't have to go to an analyst. I can just do it myself. That'd be pretty cool. There's obviously trust stuff with the data there, but that's kind of where I see the future going. >> Yeah, >> thanks. >> And and the line of sight is pretty clear if you have one of these like I mean even if it's not one of these uh kind of like uh bigname data warehouses like you probably have some you know snowflake data bras or or something like that there there's a very clear path to to doing that.

51:43 >> Yeah. I mean like data bricks, snowflake, I'm sure all of the bigname ones have MCP set up. Uh but for other things uh that probably your company doesn't have an MCP or don't have one. You know where be it what something you could do is show them what good looks like. Hey, if I had access to this, you put a CSV there and show them guys this is what I want and I'm going to help you just uh you know get so much more money for free if you had this u you know like these type of systems set up for us because in some companies I know security is a big thing uh people are worried uh about that so we don't uh there are bunch of ways to go around it uh like to ask these tough questions and you know management when you show what is possible and what is stopping you from getting there uh they would we could people will find some ways to get you there.

52:45 >> Yeah, absolutely. I agree with that point. Yeah. >> What other questions? Anything else? >> Hello. I have one uh thank you first for all of this presentation. I think now now is the time where all the data should be self service right because actually I'm pointing to get that in my company but what happen if I apply this for example h with the afterlife where where can we uh set up some kind of feedback loops or or framework of h improving that analysis ICS for example uh we have something that we want to decrease which is the retention in my company and we are preparing with the with code like uh something to prevent that. Okay. So uh in in your business area or in in your companies, how do you manage to uh possibly propose the plan but actually uh get some feedback in in feedback loops to re to make this work uh something functional, right? because you are providing the insight you are telling where we have to go and how that happened with that insight after uh are we going to cover that maybe in that session or um so I think you're so just to to come up to kind of summarize so you're asking like after we get to like the insight from the analysis how do you then like implement that afterwards and then get like a feedback loop going so like >> consistently is being executed.

54:43 >> Exactly. Exactly. It's how how do you manage that cycle? Because this is potentially is very help for for make the insight and validate hypothesis. >> Yeah. >> But in in your framework how do you manage to okay how can we continue improving that or seeing if the the people are applying the action plans that maybe we can propose right? Yeah, we do. So, so in our fiveweek course, the last week we spend on basically like insight to action. It's more about like how do you like it's more like an organizational change. How do you like share out your information? But I think so there's that piece of it where it's just like presenting and getting buy in etc. Uh but I think there's also like a question there around like okay I armed with AI like I have these insights can I do it myself or can since so I've been doing some of that at my job which is this is kind of what I think is the future of like the data role where it's like you start going more end to end where instead of just us doing analysis we start going upfunnel and building data foundations but also start being like all right we have some insights It's like can I now whether I work in go go to market or in product can I actually go into some code and leverage cloud code to like try and test and proof of concept some of these things out. So I've done a bit of that at my day job. Um, and like I've found it pretty compelling to be like I've take I've got the insight and now I can like kind of have like a sandbox of like the production code where I'm going to try to apply it myself and then I work with like software engineers to be like hey like I'm actually seeing like uh increases in retention or decreases or in the predicted retention or whatever based on what came out here. can like I kind of pass this off to you to like implement that or like work on it some more. There's also this whole thing the past like couple weeks um auto research like from Carpathy where it's like an actual like autonomous loop where the code is like if you have some type of metric to optimize the code like the agents change the code itself then see if that like brings your metric up or down does an analysis comes up with new hypotheses tests it again I think there's like a whole field of software engineering that that goes that that direction. We're not going to cover that stuff in this boot camp. Um, and we're probably just going to cover like more of like the share out stuff in the fiveweek. But, um, yeah, I'm playing with an idea about doing like a twoeek course on that called it's called automate evals. Um, and maybe we'll create something else too. But I think definitely like coming out of this, you should feel empowered to be like, "Hey, can I can I go the next step forward?" Like the insight doesn't have to stop at me passing it off to a stakeholder.

58:00 >> Exactly. Perfect. Shane, thank you. Thank you very much. >> No problem. >> Cool. >> I think you had a question. >> Yeah. Um, do we want to answer this one last question here? >> Yeah. >> Okay, cool. Uh so uh actually Maxine if you're still on on the line do you mind sort of like reading out your question to the group. >> Hi thank you so much Pi for giving me the opportunity. I'm not quite sure if my background is quite noisy here. Uh so I'm currently looking for an entrylevel position as a new grad and in some to some degree that I find it even more competitive than like a junior or mid-level physicians but I'm also in the position that I don't feel comfortable comfortable validating all the output for codes generated by large models on my own which is different from this linear data science position. So I'm not quite sure at this stage of my life if you would suggest someone like me to first focus on getting myself um in the door first by drilling those interview style questions like time simple practice instead of focus too much on um integrating AI workflow with my um project.

59:33 >> Got it. Yeah, I think that makes sense. Sean, do you want to take it or I can take it either way? >> Uh, why don't you take it? I can I can add some color to whatever. >> Okay. Yeah, I I'll take a crack at it. Um, and so um you know, like my thought is uh the sort of like AI is going to help almost like accelerate your learning if you will. Like if you um if you actually look at the open- source repo like it's got a lot of best practices for you know like the many different components of being a data scientist effectively in the analytic space and um you know like being able to leverage that from a for example like um you know like hey I uh I know what good looks like for you know like framing questions. I know what good looks like for designing and analysis. I know what good looks like for validating and stuff like that and I know how to use it with or I know how to uh I know how how to uh do it with the help of AI in this new day and age is actually pretty powerful to like you know like um uh be like your learning companion um along the way. So if I were understanding your question correctly, um the advice I would give is to sort of like use it to supercharge you. I mean like there's the component of like you know hey we want like you know should we understand the fundamentals and like you know do do it really really well uh first uh before kind of like you know the math before the calculator right like uh that that analogy. Um, I actually think doing both at the same time would actually give you an unfair advantage um over everyone else who's kind of like, you know, just I don't know, like grinding the traditional way.

61:22 I don't know if Sean, you agree with that. >> Yeah, I think so. I think I think it's like uh it's like a it's like a challenging time on either side of the spectrum where it's like if you're coming in totally new. Now there's this like very new AI thing that is able to do a lot of the stuff that like was more kind of entry level analysis work before. on the other side of the spectrum like a a totally new way of working is is being created and so people who have been doing this work for a decade or two or three decades are having a hard time adjusting to basically unlearning things that used to be very good habits that now become bad habits. Um, so I I think the only thing I would add to it, I agree with everything Heis and the only thing I would add to is I think like the job of like most people are changing is changing. No one really knows what it's going to be, but like at least right now like my bet is and where I'm investing is that the job's changing from like me doing analysis to me building aic systems that do analysis for me. Um, right now I have to validate a lot of that. Um, who knows what happens later on as these get better and better.

62:43 You're going to need less validation and less validation. Just like with like, you know, GPT3.5 a few years ago was like, oh, it hallucinates all the time. And now it's like, yeah, it still hallucinates, but I mean, like, you're not checking for hallucinations as much as you used to. And that that's probably going to happen like industry ride with other validation stuff, too. So either way, like I think my just TLDDR there is like investing in learning how to like onboard and nurture and build out agentic systems I think is going to be kind of like a new valuable skill rather than like just doing kind of like the traditional way of learning analysis. I don't know. Anything else high on that?

63:27 >> No, you nailed it. I mean definitely a tough time. Um and uh it is our belief that if anybody not not just experienced or not just um uh entry level who master and are comfortable using for example right now cloud code uh is probably going to have uh again back to the same spiel unfair advantage over others. Okay cool. Uh thank you guys. Unless there's any question, uh you can catch us in the Slack community. Um and uh yeah, hopefully I I think we'll see a lot of you uh in the boot camp tomorrow. Uh if you want to join, please feel free to join us as well. Thank you. Thanks everyone.

64:14 >> Thank you very much. Good stuff. Thank you. Bye.