Reframe Vague Questions in Claude Code with AI

0:00 um through this in more depth. This um today's session is around um basically cutting wasted analysis time. So um you know a lot of uh I mean any any job really but I think especially in the data field it's so easy to go down different rabbit holes. It really depends a lot how your data team's set up. you know, it's not it's not um as easy if you're like embedded in a team in a product that you're working with.

0:31 And so you have a lot of context around the problem you're solving day-to-day like if you're going to the the daily standups, the weekly uh planning meetings with like your, you know, engineer and PM and designer, everyone. But a lot of data teams aren't set up that way. A lot of folks are, you know, understaffed in the data or so. So maybe you're covering multiple teams. Maybe you're more of in kind of like a serviceoriented centralized model and that makes understanding the context of the problem really challenging and um questions can come in that actually seem like pretty concrete and pretty specific around how you're going to answer them and really easy to spend you know not days but I mean like weeks um doing an analysis um even communicating with your stakeholder that whole time but it's like you're just kind talking past each other or like speaking different languages, but you feel like you're actually saying the same thing.

1:28 And then you come through with the analysis and you get some coordinating, but this isn't really what I'm I'm looking for or like thank you, like I'll take a look at this, but then nothing ever happens. A lot of that can be solved um upstream by just, you know, reframing questions, answering the right asking right questions, getting the the right context. Um, and something I've been finding lately is, you know, there's a lot of talk and and I'm not I'm not just finding it around like this kind of reframing questions things, but but anything around like rigor and experimentation um around um uh like checking myself on if I'm looking at spirious kind of random relationships. there's all this talk around like you know LLM is like being you know hallucinating and um kind of being biased but as humans like we're very biased as well. So I actually do find there's a lot of opportunity when we build gentic systems oursel to kind of just like put checks and balances on our own bias and our own thinking so we can have guard rails and make sure um we start our analysis from the very beginning in the right direction. So um Strav is going to talk about that a bit today. Can you all see my screen?

2:43 >> Yeah, we can see I can see your screen. Can you all if you maybe drop a yes or no in the uh chat, but I can see it. >> Okay, >> nice. We got some thumbs up. >> Yeah, I'm I'm on this new computer and I have I'm currently doing Zoom on web browser. I realized I didn't have the app until like last minute. So, sorry folks, if there's some technical difficulties, but thank you so much, Sean. Uh we could do a I could do a quick intro about myself. Um Sean did go over a bunch of it but um Straa Mipali I uh am the co-founder of this along with Sean and hi uh we lead the AI analyst lab here I have experience of around 14 15 years now in data science starting with Microsoft and most recently at superhuman uh which is Grammarly uh so uh have been into data science and silicon valley for a long long time right now. Okay. So, uh like Sean said, uh we are basically going to look into the most I would say the the this the first question, the first thing that we want to do when data or analysis comes into place, right? So, we run uh like a bunch of courses. We run cloud code analytics boot camp course. We run uh fiveweek uh frameworks of data science course called AI analytics for builders.

4:05 And the first thing that we teached in that that we taught in that course, we had one cohort out and the first thing we taught in that course is about framing your questions right because if that's not correct guys, if you're going after the wrong problem, the entire work that you're going to uh that you spent your bandwidth on is not going to be help like you know it's just utter waste. So and we knew about the importance of this but to be honest until our students actually picked that's the biggest thing that they learned we were uh you know like super surprised. Uh so uh this was something that we did as part of our uh course cohort discussions where we asked people what was the single most valuable thing you learned. the most fa the most popular uh answers was uh that you know push back on the question and actually reframing the question because you see especially it happened with me as well when I started my work in data um we are used to getting questions from so many different stakeholders for so many different reasons and do this is I would say uh more relationship more about data uh like it's it's a combination of so many skill sets not purely data skill set and but that totally affects the impact of you as an individual that you could bring to the you know to the to the company wherever you are so that's the reason we believe this is one of the most important skill set this is beyond technical this is I would say in the frameworks and thinking in analytical thought process and also your personality type like we are so used to saying yes for everything now uh this uh I I hopefully reframes your thinking like it did to most of our students. Um and like trust me they all had uh like AI analytics like a repo of 60 skills and agents like they had all of that with them that was also very important that uh that people really like that but this which is like the most basic thing came out as like the top one that people write.

6:21 Okay. Uh so where are we? uh let's talk about um like making this whole thing concrete right so let's say we have same AI same data but we have two tabs right so we have a vague question and a sharp question and what is the difference between both of them right so if you look at a vague question uh analyze user engagement you know these are some things that you probably keep seeing a bunch of people like uh your stakeholders come and ask you hey what's the user engagement you know um and this is basically could be so many things when we talk about user engagement right so it it gives everything but also nothing it's basically is like three paragraphs of generic observations because uh like DA is relatively stable but some segments are worth monitoring nothing you can act on so this is like what a vague question can you know encompass it could mean so many things versus what a sharp question looks like which segment drove the engagement drop last month so we can decide where to focus. So you can actually see there are particular frame like you know segments of this question that you can divide and understand what each of them mean. For example, there's a name segment, right?

7:42 Because it's talking about a specific area, a specific user group where you you might have seen the drop. And then it is also talking talking about it being a quantified driver like understanding the what behind it. And it also has like a clear recommendation. For example, what would we do if we know the answer for it? Which is the most important thing that I would say especially people who are you know just starting out or people who are used to the service based of data types of you know data uh service databased orgs. We are not used to this setup at all where we are uh where we basically take a question in do all of the you know the complicated backing ourselves and finally end up at a place that we could have gotten to far more uh uh like you know in a far more shorter time with far more clarity but you know we basically didn't start there and hence it took so much so so so long than when compared to like without it and Guess what? This is something that's not just at work. This is something that you'll see in multiple places. I myself have like you know we just until now spoke about how whatever we're talking about question framing like changes how you work right for example can you look at why signups are down and then you spend a day on the charts and then hey this is not what I meant I meant something else you know this is something that we see on a day-to-day work but guess what where this is more important this is actually uh one of our students mentioned this too on how this question framing helped them like craft track uh you know get better at interviewing which is if you go to interviews like if you look at what meta is like data analytics questions are they're pretty they start very vague as well like how would you measure success of this feature it doesn't uh basically cover any of the specifics and it's upon you as an you know interviewee how you're taking this question and how you're segmenting it down into multiple segments and how you're actually tying a metric to and how we are actually ensuring that you ask the interviewer the right questions about hey what are we trying to understand with answering this question how are we going to execute on this right so these things are something that interviewers look for as well not uh so this skill set that you're going to learn today um it's it's just going to go way beyond your work and another very interesting piece uh that totally you know came together in today's world is AI. So you might have seen this a lot.

10:26 This is what uh you could call like context engineering. You could call so many things where if you give AI something vague, it has a lot of scope to cook things up, right? You could call it uh I mean if you don't give it enough information, it's going to basically assume a bunch of information, right? It's kind of like, you know, an intern you work with that doesn't have all the information. So ensuring that you're specific and you probably have maybe already a system like AI analyst uh system that we have like a repo full of uh you know skills and agents that we built even our free repo has them guys.

11:07 Sean can like share the free repo where we actually build the system where if you ask a question to claw code um it actually tells you hey this is too vague you know can you can you get more specific so you could build an AI system that does that for you so that you don't have to be the person that's you know telling it every time to be specific if not you better be specific because the more vagger vague our questions are the harder it would be for the you know for claw code to actually understand and help you solve the problem in the right way and this is where we could we basically talk about the exact specifics of what I've been talking about right so this is basically a three check framework you could call it like you know a 30-cond gut check right so every analytical question should pass all these three questions what are those first one is decision type. I would say this is the most important one which basically talks about what decision does this question inform. If you can't name a decision, you stop right there. This is probably the hardest as well. Imagine if you have a VP of product coming to you and ask a question and you are like hey uh this is pretty uh like you know thanks for this question. Could you please tell what will my answer for this question change what you and your team's working on or how do you want to take this answer and implement something because at the end of the day if you're working on something that doesn't prove to be impactful it's not going to help uh for anything. It's not going to help for your work that you do at your place.

12:54 It's not going to help you get promoted. It's not going to help you you know if you want to interview what did you work and what's the impact. you're not going to have a full closed loop, right? And the second most important thing in this framework is being data grounded. Can I actually answer this with data that exists? I know this sounds pretty basic, but I'll be honest, there's so many questions that come to us for data teams where the data is not entirely fully present or half present or like not fully baked. And does it make sense to answer this question with the amount of data that we have or do we just wait go implement get the infrastructure talk to engineers get the tracking implemented and then come back to it maybe next quarter instead of giving halfbaked answers that are not going to help you make that decision that you probably you know pushed your stakeholder to give you right like what's the decision they're going to come up with the decision but guess what if you don't have data or if you have halfbaked data it doesn't help and the third most important thing in this framework is being specific enough how will I know or how will you know basically when you're done is there a metric a comparison a time frame you know to make sure that you have some sort of a bound scope creep is the most important things or the most uh you know thing that weighs anyone down so ensuring that all of this is under the scope is also very important Yeah. So before we spend an hour analyzing, spending 30 seconds here um to understand does my question actually satisfy these three rules. Is a decision tied? Is the data grounded? And is it specific enough?

14:40 Now let's go to the next one. Okay. So let's actually watch one get sharp, right? So let's start wig. Why are signups done? No decision down from what? No, we don't have for whom. We basically have it really vague around saying no like you know no decision like we can't basically make out anything from this question, right? So how can we make it slightly more specific? Why did signups drop 12% last month? So it's better it's better than the last time uh the the first question, right? We have a number now, but what would you actually do with the answer? So, the decision is still unclear. Right now, we can make it even more sharper. Did the pricing page redesign cause the 12% signup drop in the 2 weeks post launch? And the decision comes, should we roll it back?

15:39 Right? So, this basically has the decision which is rolling back if it's not great. Yes, we got that frame that that part of thing figured out. Now, what about data? The data uh is a data grounded? It is because we are looking for signups plus and we also have a launch date and we have all the specific things. So, that's data groundedness. And what about the specificity? We have things specifically saying that attribution and we also have recommendation. So this is basically I would say the most minimal thing where we start um to ensure that we go from a vague question to actually a sharper question.

16:23 Awesome. So do you so that's this is something that you would probably have something at work but what about like I have an example for interviews as well right? So this let's say an interviewer asks this. How would you measure success of the new search feature? This is a pretty common question guys. Like if you go to any of data scientist uh or like product analytics or maybe even data product analyst questions, you would find a metric or a measuring success type of question, right? And most candidates uh especially like from my experience the more junior ones even more they immediately start listing the metrics saying hey Dow retention engagement I'll probably look into this that and that right so yeah that maybe feels productive but it's actually the trap the candidates who get offers and actually offers for higher you know level roles they actually pause and ask clarifying questions first success for whom?

17:27 Right? Because we are talking about success for this new switch for but who is it we're talking about and ensuring that is it power users or everyone. So we need to have a tier one set of questions and then a tier two set of questions under it. Uh so question like success for whom is a tier one question and a tier two question under that is what segment of users is it power users or for everyone. And then >> [clears throat] >> uh another set of question that you could ask is measured against what is the baseline. Right? So this basically also this framework around ensuring that the specificity and ensuring that you have data groundedness and ensuring that there's a decision tied to it. It's also going to super like you know it's going to be super helpful for you for interviews as well. So the question that could be pretty uh you know targeted that you could start work on the interview is once you get um the clarifying questions answered and get to a question something like what it says at the bottom for search targeting power users what's the primary success metric the guardrail and what would be good enough to keep investing um like you know look like at the end of 30 days right awesome So a sharp question usually carries a hunch. It has a hypothesis.

18:47 The most important thing that we need as data people. Uh irrespective of you know it doesn't you don't have to be a data person. Irrespective of uh your role if you work with data you need to come with a hypothesis because all we're trying to do is either validate or invalidate the hypothesis. And we have multiple ways to do that right. We have Amy testing we have causal inference. We have correlation analysis we could you know validate to some extent but what we need is ensuring that we need to have that hypothesis and the move that separates rigorous analysts from everyone else is that uh you basically have a rejection condition as well. What would confirm the hypothesis and what would reject the hypothesis? Right? So before you touch the data you need to basically say it out loud. What result would prove you wrong? Here's the midmarket churn. uh you know is driven by failed on boarding not pricing. What would confirm it? You could say something like hey ch clustered in accounts that never finish setup. You know pricing absent from exit interviews that these are some things that would confirm it but what would reject it? CH spread evenly across onboarding states. Pricing dominates exit interviews. So this is something that would reject this hypothesis. So it is very important uh to ensure that you have a hypothesis and you have pre-written rules on what confirms it and what rejects it. Right [snorts] now uh another very important thing uh is not every question deserves the same effort. Uh so score each one on two axis. You have impact and you have physibility. Right? So top left is basically uh something that you do first. It basically is a question that clearly drives the decision and the data already exists. That's where you start every single time. But what about the top right? The second part of the metrics. It's like basically you plan for it. Uh high impact but you're missing the data or you're missing instrumentation. So you probably need to you know um stake it out for the next quarter. Okay. And what [clears throat] is the like you know leftmost one? This is something that you would use it for buffers. You basically do if you have time. This is basically something that that are quick wins and there are dashboards. You basically uh ensure that when your bigger work is blocked, you have something to fill your bandwidth with. Right? And the most important one of all is the skip one. The bottom uh you know rightmost corner in the bottom, right? This is the most important thing because a lot of uh asks uh so many data scientists and analysts I worked with that needs to be skipped actually falls under one of the other categories that you don't want it to happen right so it's basically hard and uninteresting this is the graveyard of half you know finished side quests right ensuring that you say no early and ensuring that you pinpoint to why you're saying no right that it's low impact, it's low feasible, you don't have data grounded, there's no specific decision you could make. And these are some things that you need to ensure that you write down and talk with the stakeholder and share with the stakeholder and give them a reason for why you are skipping it. But if they don't want you to skip it, they better come with those answers related to the type of decision they're going to make.

22:19 If you give the answer and they let's say it's a product manager, they ensure that the data groundedness happens. They work with engineers to have the tracking ready and you know get everything done. Without it all of those go to skip requests. Okay. So where are we at time? So we probably I have a few notes as well. I'll share this uh with you but I'd probably go to the demo instead of going through this because I have a a good something good to share in the demo. any any questions any discussion so far before I go to the demo?

22:58 >> Um I think a couple questions. Um one's around do do you define the metric yourself or do you rely on the on the stakeholder? And then there's another one that um maybe Hi can jump in on because you can answer in the chat but what about in the case of ETA type analysis? Um >> okay I mean the metric question guys it's um defining metrics and coming up with uh we actually have Kai actually shared how do you define metric we we have like an entire uh segment in our course that talks about it but it's the most important part as well because all these steps that come at the start of your work if they're defined wrong then your entire work goes you know haywire. to ensure that you come up with a framework of what a metric means and what uh what is the type of metric you want to like you know how do you want to define it and you work with the product partners or the decision makers whoever it is into understand the goal of the metric why you want the metric and what is the decisions that it's going to help change in the product once you get those decisions from the product partner I think it's the data person that comes up and accepts and you know verifies and uh like validates the metric but you want really tight partnership with your decision makers and product partners to ensure that uh they are on board and they know what they're going against because you don't want to be vague. you need to be very sure about the specifics uh and as a data person you need to come in with the you know the caveats and frameworks around it because you don't want the metric to be too specific in a certain area. You want also it doesn't have to be too vague so high up that it doesn't help make any decisions right so as a data person you bring that perspective and as a product or finance or whoever is the partner that you're working with they need to bring their perspective of the goals that they are trying to achieve with that metric. So that would be my answer for the metric one. But yeah, and hi.

25:04 >> Yeah, we have a we have a session next week actually that's free. It's called developing northstar metrics with AI. Um so if you want to if anyone wants to join that um I just dropped a link in the chat and uh if you can't make it but if you sign up we'll get you the recording that for free as well. Um there's actually another question in here. Um this is a little off off topic, but not super off topic. um are you guys already building agentic systems for self-s serve analytics at your companies? What is the expectation now from analytics folks? So I think there's two directions for this. There's the uh agentic analytics where the data team is leveraging is building their own agentic systems to um either automate out their work in such a way that is more robust or um faster. So increasing breath and depth of the analysis but the agentic system the analyses the analyses are still done by the the data team but it's just through you know the agentic systems they build more than just like direct SQL python R or whatever um that's something that we've um been doing for several months that all of our students have been doing for um several months and we're seeing this as kind of like the new I would say expectation for data professionals. is moving from doing analysis yourself to building agentic systems that um do a lot of that work for you where it makes sense and then validating um that analysis.

26:33 Uh the other side of the coin is like building self-s serve uh analytic systems for you know the rest of the company other orgs where they're actually like a you know like a someone from the marketing department has a question they're able to type it in in some sort of centralized agentic tool and have the analysis themselves. There's a ton of enterprise solutions working on this, right? You know, companies that have like hundreds of millions of dollars in in funding trying to solve um this problem. This is a much more complicated problem because they cannot validate the answer themselves.

27:08 In the case of, you know, a data professional building out a system and running the analysis, they can look at the numbers, they can look at the code um and they can say like, "Yeah, this is right or wrong." they can review it just like a software engineer would review the code that um like a co a genetic coding tool would output. Um so we focus on the the former and less on the latter although we are kind of in talks with a few partnerships with some of those enterprise solutions and we're happy to talk more about that offline or or one of our courses.

27:41 >> Awesome. Uh >> maybe I'll let you there's some other questions here but I'll let you maybe like if you want to keep going and we can get to these. >> Yeah, we can get at the end. Sure. Awesome. Okay. So I'm going to give you a bunch of uh like because you signed up guys because you signed up for this we I'm going to uh give you a bunch of uh you know free PDF resources as well uh that contains all this information that I shared and a lot more uh for example like how do you come up with the right question give you a bunch of examples of uh some questions that you could ask in your u like you know interviews if there's a question that comes up what is the type of example that you need to you know ask and stuff. So, for example, um yeah, so uh definitely uh check them out. I'm going to send them over an email. Okay, what are we doing right now? I'm going to have two demos.

28:32 Hopefully, time permits. So, the first one is going to be in Cloud Web UI. I'm sure um if you join our a bunch of other courses, we like the intro to cloud code, we you could do all of this in cloud code. My second demo would look at that. But since uh if you don't have cloud code or if you have not used it with our rael, this is something what you could do with claude web UI. Okay, I'm sure this is something that you could do even with chatbt as well. Okay, so this is a prompt I gave it here. And what does it say? It basically tells that you are a question quality coach.

29:06 Your job is not to answer questions or do analysis. Your job is to help me turn a vague question into a sharp answerable and uh you know something but fast but coaching me through the few targeted questions right I gave it three modes I gave it a work mode basically if this is a question related to your work and I gave it an interview mode let's say if this is a question that you asked in an interview and you're prepping for interviews right this is something that the interview mode would help you and there's a rank mode let's say you have a bunch of options and there's uh like a vague question what how do you rank all different questions and then it basically has step one running the three check framework that I just shared with you and step uh two is something that talks about how does this framework work for interviews and then how does the rank mode works right and then uh it talks about synthesizing it in step four so begin and look at this begin by asking uh for my way question request or problem and whether it's a work situation or interview situation and a problem you should rank framings for. So this is all the entire input. I'm giving it to Claude web UI and let me ask it a question and see what it gives as an answer. Okay, >> can you drop the prompt in the in the chat too shabby people are asking?

30:24 >> Okay, let me just do that so that >> and someone asked if they can transform this into a skill. >> Oh yeah, that's a wonderful question. This is something that uh you know uh we have a free repo. I'm sure Sean might have shared it. We'll share it again in the email, but you could look through our free repo and look through like the skills and agents that we already built, right? Um, but if that's something that you need help with, we actually have the intro to cloud core analytics workshop where you could, you know, uh, work with us. But yes, these are some things that you could definitely build skills and agents for. Cool. So, I gave you that and I'm giving you my simpler question as well.

31:07 Okay. Um, yeah. So, what's my question now that I gave it the entire prompt? I'm asking it. It's a work situation. My VP just slacked me. Hey, can you look into our chore numbers? They seem high and leadership is asking about it. Seems familiar, at least familiar for me. So many of these type of questions that keep coming, right? So, what does it say? It says running the three check framework on looking into our churn numbers. Guess what does it say?

31:34 Decision title. No, that there's no decision in this question. So, it crosses it out. Is it data grounded? Kind of. Churn is measurable but high versus what? So, it's not also fully data grounded. So, that's kind of a maybe. It's not a fully now, but it's almost a maybe. And what about specific enough? No metric, no segment, no time frame. So, and also there is no definition of what done means because you could just keep answering this question for weeks and weeks and still not get to the answer that the requesttor is asking for. Right? So, this fails two out of the three. Let's and it talks about let's close the biggest gap first. So, it is asking if you got a clear answer on churn, what would leadership actually do differently? And it also asks for examples fund a retention initiative.

32:28 Does it change pricing? Reallocate to the growth team. Just report a number upward. So what is this decision feeding? Right? So it's asking me the question and let's give it an answer. So um I just gave it. You could guys this is something I'm doing. Uh you could have something entirely different that you could work with. Okay. So what I'm giving it is figuring out whether it's one segment we need to fix or it's a broad problem basically where do I point the retention team. So yeah there is a decision now you could try uh the same thing with another answer and see what you get and you know you could share that with us but I gave it this um to demo something. Okay, so it says good this is a real decision where to point the detention team that makes this a resource allocation question right not a reporting exercise which is pretty good now it is talking about the next gap being data so the first gap the decision tide is done now we are looking at data which of these exist cleanly plant tier acquisition channel 10ear cohort oh oh my god we could have so many things right so let me give it a few things let's say that that I kind of have. I'm saying I've got plan type, region, tenure, monthly revenue, and last login per account.

33:51 Now, it says it's a solid one, right? So, it's basically asking um how do you know you're done? Right? Uh the the the final one. And let's say I'm going to give it uh I basically uh end this when I get a ranked view of which segment is churning the worst. So, I know where to focus. So that's where I'm going to end it. Awesome. So heat is now giving me the sharpened versions of the same question.

34:21 Look at this. Because you could have multiple styles, right? And it's giving me all of those styles. It's giving me a concentrated view and it's recommending that. It's basically giving me which segments, the plan, tenure, region, and stuff that I could look at. It is giving me a rate versus volume view. It is giving me a trend view. And these are a bunch of options. So, it's asking if like, you know, what is something that uh you want out of these things, right?

34:49 I mean, I could keep going until I get, but I want to get to the clawed code demo as well. Um, or maybe let's do one more thing. So, you could go on with this until you get to a specific question, but I want to try one more thing for the interview question and then go to claw code. Okay. Okay. Let me just say okay let's move on to interview question. Okay. And this one's an interview question. How would you measure the impact of a new recommendation algorithm?

35:26 Right. So it is basically doing the same thing. It goes to the interview mode. It is not in the uh you know it is not in your um the the work mode. And look at this. It is asking about clarifying metric, understanding context, planning the cuts, confirming the decision. Trust me guys, I've done like at least hundreds of interviews and you would expect people to do this, but no, people actually don't do this at all, especially confirming the decision.

35:55 So, you want to ensure that you do this twominute decomposition of of this question live with the interviewer and keep asking them for do they agree? Do they want you to go down this direction or not? And hopefully this prompt helps you to do some mock interviews yourself. Okay. So, let me move on to cloud code and I'd like to share you a claw code demo. Okay. Let's try to do it in the next 5 minutes so that we'll have around 15 minutes to for answering any questions that you have. Okay. So, where am I?

36:34 [clears throat] What are you looking at? You're looking at cloud code. Um, and this is visual code, VS Code. Uh, we actually teach you uh in one of our courses how to, you know, install all of this yourself. Um, and you know have uh how do you have the repo? So the repo that I have on my left is the is our entire repo that we built all of this. For example, if you look at the repo uh for the boot camp, this is the repo that you give for boot camp. These are all the skills that we have analysis design, you know, design spec and you know all of that stuff. But I'll get into the detail later. Let's jump into the claw core prompt. Okay.

37:18 So, uh what am I asking here? I'm asking it you are the question framing layer of an AI analyst system. Like I told you, we have so many of these skills the analysis design, you know, ARV analysis like you know so many of these uh and I'm asking it instead uh if I have a vague request like look into our churn numbers they seem high and leadership is asking what is the process that the claude code repo that we built is going to go through right and this is an asky diagram of everything that cla code runs. for you.

38:00 Let's say you have this question. You have this repo at your hand, right? Look at this gate zero. This is like the first gate that it looks at three checks. Decision tied fail. Nope. It's not great. Like we said, is it data grounded? Fail. The churn numbers, which table, which churn, no metric is named. And what about the specific enough? Again, a failure, right? So, it basically is giving me a score of 0 by3. And it basically is telling me that you have to I'm rejecting it and we are going to you know you need sharpening it right and in case I ask for the entire ask diagram for you to understand what all things the system is capable of doing so that you would understand to get to a specific question let's say we get to a better place and what is this what does the skill pipeline look like right so uh this is invoked to uh and there's another thing all of these get invoked to sharpen the question too Right? So question framing shuffling. So it basically helps you to get to the right question and what does it ask you?

39:02 It basically asks you hey what is your goal? Is your goal retention? Is your goal revenue or is it just a broad narrative right? And what is the decision you want to do? What changes if churn is high? And what does high mean? Is it relative to the target trend or a cohort? And who decides and by when? This skill is the most important skill the question framing skill that goes through the threespec framework that we have and ensuring that the question ladder gets done right and once that's done it basically goes to the metric spec this is basically the data groundedness right so what is the churn type you know what are the multiple steps here within the what's the denominator who is at risk is it like a voluntary one is an involuntary one basically all multiple segments that you get to within like understanding what the churn metric actually means and what are the multiple segments you can divide this churn metric by right and then there is data audit. You want to make sure that that particular data is actually available to query right and you want to make sure that you know the data is clean that everything is reliable and you can actually go and query it because specificity is not the only thing you need. You actually need data that you can rely on to actually go and execute, right? And then another important thing is actually now that you know all of these things, how do you want to prioritize your question? What are the other things that are at stake?

40:31 What is the amount of dollars that's at stake? What is the uh you know uh like you know what is the stop rule? When do you think you need you'll stop this? And is it worth the return on investment? Like the amount of return that you'd get on this investment is it worth your time or not? Right. And these are basically the forks that go that we go through like the metric baseline window segmentation, the ranking logic, the threshold, the unit, the output decision. All of these is baked in into the skills and agents in the system.

41:04 Most of this I would say is already part of the free reper. But our AI analyst plus repo that we share as part of our courses has has even more detailed uh description into like what uh like one if you ask it a vague question how can the repo help you get to a better place. Yeah. So this is how it looks like and uh one final thing I want to share with you is um I want to ask something. Oh where are we? Maybe I can skip it so that we'll have 15 minutes. Is that uh what do you want to do, Sean?

41:41 >> I go ahead and get go ahead and ask your question and then we can we can see if people have questions. If people have questions, feel free to drop them in the chat now and we'll get to them after. >> Yeah. Straw goes through this. So it basically went through uh the question right and it it basically gave the exact sharp end question that um I know we didn't give all these answer but because for the demo's purpose I basically took something based on few assumption and basically has a sharpen question for us right has monthly voluntary logo churn risen above 3% target in the last two quarters and if so which customer segment is it by plant year and tenure um and is that driving the increase and so leadership can decide whether to fund a retention intervention this quarter or not. So this is the sharp end question that solves all these forks like you see here. The metric, the baseline, the window, the segmentation, the ranking logic, the threshold, the unit which is customers like you know accounts, seats and you know whatnot and the output or decision right and it also ensures that all the checks are done the decision type the data groundedness and specific enough and the most important thing the rejection condition. If you remember we talked about being having a hypothesis and ensuring that when do we reject the hypothesis should also be you know uh like documented and tracked and thought of and if voluntary logo churn is less than 3%. The target across both quarters and no single segment that's when you say that the report is not elevated and recommend no spend and we stop. So we have a rejection condition here which is very important right and then we have the data scope note what is the source what it excludes what is the grain and what is do we do if if you know if you're blocked yeah so it goes through all of this and it gives you this and this guys is just just the first part of the system if uh maybe I have something here to share with you oh Um so if you want to understand the entire system uh of what actually happens now that you have a question that we basically share in our courses I can briefly share with you while we we could probably keep taking questions while I run this maybe Sean. Yeah, I think something to emphasize about this is like um like reframing questions like it sounds easy. It's just hard.

44:19 There's two kind of there there's obvious things when like a vague question comes in, you're like, "Oh, it's super vague." And then there's like questions that come in where like you kind of don't realize it's vague, but it but it actually is because it doesn't have everything here. Like it doesn't have, you know, I I've started plenty of analysis in the past and be like, "Oh, I have my metric. I have like the kind of decision that's going to made from be made from it. But then like there's all these other components later on like oh like actually what window are we talking about? Oh what are the filters that need to be applied be applied? What's like the threshold we care about? And I go back to my product manager or designer engineer and everyone's like oh yeah I don't know what about what do you think?

44:59 What about this? And like this just like gets everything done upfront for you. And it's not that it's like, you know, like there's no math here. There's no like crazy like um sophisticated like thinking. It's just like all these things that we don't have time to do or kind of like forget to do. And now it's like every single analysis you do, every single little question, you can run it through this system. It's going to take a few minutes and your like question is gonna become so much more powerful and you're going to get a like um like uh I think Shabby was showing the slides earlier like a decision around like hey do we kick the analysis off do we need to like plan more um do we just like put this in a in like a backlog to do later or do we like politely decline the analysis um so it kind of guarantees up front that you know you go from just like these are fake numbers But like imagine like 80% of the analyses you do um don't actually have action come from it. By adding these gates at the very beginning that becomes like drastically smaller and you get a much larger percent of your analyses and the time that you spend actually being um actioned on in a company. And that means that means that like yeah the company's doing better. That means you're invited to more more important rooms around like how do we make decisions around the company. that means you get the promotion, right? So, it's a it's a really like not crazy challenging skill to learn, but it's so important and so many people miss it. And it's not just junior people. I know senior managers of data science who suck at this who are just like, "Oh, yeah. Oh, we should track." Wow. And it's just like, "No, dude."

46:45 like people who like who focus their whole thing around like reporting a metric and not actually relating the metric back to like the actual reason we're here to build some sort of user value of a product. Um I know CEOs who suck at this. I know like um leaders in product who suck at this. So it's not just like a junior thing also. Um and now you can have systems where you just like build it in to like automate out out that like those errors.

47:10 Um yeah so Shravia you're sharing here like the full system here and you can see like the sharpening the question is just like the very first part of this kind of end to end um system that we built and that our like our our students and clients have like built on top of like way more than us. And then you start with input and then there's a stage one you basically scope and plan and there's a data audit that happens and then you actual now that now execution starts you basically measure and what exactly happens there and you diagnize right if let's say you find something in the measurement how do you diagnize that and then you stress test it and then you close the loop there's so many other things as well uh I tried to you know get it to so this is what hap so it basically if you run like run analysis pipeline. These are a bunch of things that get that you know called the agents and skills that get called as well. Um which question framing hypothesis data explorer uh you know source tie out uh the descriptive analytics the root cause investigation.

48:20 So you you'll see all of them here on the side. These are my skills. Uh these are the repo skills that we have here. The deck critic if you look at it the data quality check comparing data sets closing the loop. So we have basic you could invoke all of these skills by themselves if you want to stay in control and if you want to ensure that what is the output of this particular stage of my data analysis which I would recommend [clears throat] 80% of the time but let's say you've done this loop multiple times and you know that the type of question you have and the output you need then in those cases I just run the end to end pipeline that basically goes through all of these skills one after the other and gives me an entire pipeline And since I've done this so many times and validated every step already, I'm confident that my final end to-end pipeline is a good one. So just wanted to share with you the capability of the entire system here. And it to be honest doesn't cover other areas as well like experimentation and tracking.

49:17 That's going to be a different set of system and skills that get generated. But yeah, I'll stop here. >> Yeah, we can pop out a few questions. So if you got questions, drop them in the chat. Um we can raise your hand and ask them as well. Um two questions that are basically the same question here. Um so Anukica asked like what is the difference um between our three courses? Um yeah they said uh they're working as a product analytics in their current or their goal is to fully integrate Agentic AI workflows and tools into uh their work and bonus helps in interviews. Um, and then another person asked, um, hey, the repoo looks really great. What would we learn from the paid sessions?

50:01 Um, I can answer these. So, we have three, we have, uh, four courses actually technically. So, we have and they're all coming up really soon um, in the next few weeks. A bunch are kicking off. They're recently ranked uh, Maven's top 100 courses also. And so Maven is running a a sitewide sale on their their top courses where it's 25% off I think throughout through the end of uh this week. So that stops on Sunday. That's better than any deal we give. We usually give like 20 or 10% off. But um just something to flag. So there's four courses. There's one um I'll say what they all are and I'll go through them.

50:38 Intro to Cloud Code Workshop. Cloud Code Analyics workshop. There's Cloud Code analyics boot camp. Advanced AI Analytics Boot Camp. And then AI analysts for builders. So, our intro to cloud code analytics workshop. This is super introductory. Um, if you have used cloud code before, uh, you probably don't need to take this if you've never, this is for people who really have never used cloud code before. Um, it's like um, our like introductory course. It's like it's 50 bucks. Um, and we spend three hours and we just walk through step by step how to install cloud code in terminal inst get VS code going. You clone the repo and then you run your first uh analysis end to end on some synthetic data using the repo. So um this is open for anyone. there's no prerequisites like you can never have coded before in your life but we do find that um like that setup stage is challenging for for some people and once you get past that first step you're kind of off to the races like you're all like curious smart driven people so we're this is like just our goal like hey let's get people setting up set up and running their first announce to end so that's coming up next Wednesday on June 10th 7:00 a.m. to 10:00 a.m. Pacific time. Um, and then we'll record it.

51:53 Also, if you sign up, we'll send out the recording. We have a bunch of people who just followed along with recording. And there's like also a step-by-step like guide you can walk through as well. So, that's the that's the workshop. Super introductory though. Um, but great for a first step into cloud code. Then we run a cloud code analytics boot camp. Uh, not this weekend, but next weekend we'll have that. Um so it's Saturday and Sunday and then we have a bit of async material following through the week that you can um as you start uh working with cloud code analytics at work and then we have office hours that's optional on the following Friday after you've ran through uh the agentic system at work for a week. Uh this is about building agentic analytic systems. So it's about you know learning the foundations of agentic systems how they differ from chat. um you start by building out your own skills and agents. Um then you run some analysis with those. Then you um implement our like full AI analyst plus repo which is kind of our private repo we're constantly developing on. You um add you port over skills to that from your from where you get your own. You build on top of that. You connect to data warehouses. Um, and then we do some uh kind of like 101 previews into stuff around like uh context management and uh validation and open source but like those are like more like just like uh introductory um parts of it. Our advanced boot camp then goes all in on that. So our advanced boot camp is focused on how do you build evaluation systems and validation systems to know that the agentic analytics output is correct? How do you manage context across your team um and organize it even like for yourself? How do you how does this perform against other models? So this cloud code analics boot camp is just for cloud code advanced AI analytics. Uh we go into codeex. We look at open source models as well. Uh and then so that's another two-day uh boot camp. We might extend it. We're experimenting a little bit with the boot camps right now. So, we have a two-day with some async and then office hours for the cloud code analytics boot camp in a weekend from in a week from now.

54:02 And then July, we're actually going to move this and try it on weekdays. So, we'll do like just two hours a day, Monday to Friday instead of four to five hours a day, Saturday and Sunday. So, that's an option for you. We we reiterate on our our courses every time we have a cohort to try and make them better. So, some of the feedback we got back. And then our last one is AI analytics for builders. That's a fiveweek course. Um, and this is more about like end-to-end analytical thinking. It's less about building the agentic system and more about like how do I use uh how do I use aentric systems to do the actual analysis work. And so you could think of this as kind of like your classic data science analytics course except the execution layer is no longer in SQL R and Python directly. It's in uh it's in cloud code or codeex. So uh we teach we start the first um week is all around kind of like framing questions.

54:59 Um the second week is all about like setting up your tools and your system. Third week is about um correlative analysis. So think like trend analysis, segmentation, root cause analysis. Fourth week is around causals. So think experiment design and analysis and causal inference. And then fifth week is around um kind of like uh getting insight to impact. So it's about uh presentation, managing stakeholder, that kind of whole kind of thing. Um yes, those are our four courses and like I mentioned all of those are 25% off right now on Maven until Sunday I think is when they when they uh end that. But so if you just go to the course, you don't even the promo code is top 100, I think, or Maven 100. but it'll be auto auto applied since it's a sitewide sale at the moment. Um, and then if you miss it this week and you want to go on later on, usually we offer some, you know, if you need a promo code, DM me. Usually we give like 10 to 20% off folks who come to the lightning lessons.

56:02 All right, that was pretty salesy question answer. So, what what are some other uh questions? Oh, Maxine was asking uh my post about moving from tech into acting. No, that's just a joke. I'm not going into acting. That would be pretty good. There's plenty of people who who get into Hollywood later in life, so maybe I'll do that in my 70s. Um, no, I am leaving my full-time job, though. I'm currently a principal data scientist for a legal AI firm or a legal AI tech company. And tomorrow's my last day as we go full into agentic analytics research education and consulting. So, you know, leading uh co-founding this this lab with Shravian High. No acting yet, but who knows?

56:50 Um what other questions we got? 4 point uh she's have you guys noticed the difference between 4.7 and 4.8 for an analytics? Man, I feel like it kind of topped out at 4.6. six. And I've read a few people who leverage it for more like product managing stuff and less for coding stuff. And their opinion, I don't know if you listen to like Claire Vo's podcast, but I think she said something like like as the 4.7 and 4.8 models got better at coding, it got worse at strategic thinking. And I kind of think the same thing for analytics. I often just try and set my model back to 4.6 six because it does a great job with analytics and it's faster and cheaper.

57:37 Um, but [clears throat] I don't know 4.7 4.8 I don't think it like deteriorates the analysis but I'm not getting much of an added benefit. So whenever a new model comes out we test them. Um, and then also OpenAI they just released a data analysis um flow this past week too. I was actually talking to some folks on the codeex team there and so we'll be investigating that quite a bit. Um we don't want to just we've kind of like primarily been focused on cloud code the past few months but now that we have some more time we'll be investing on that and then also open source LLM's um high has been investigating that lately.

58:20 So that'll be in our advanced boot camp but we'll probably do some sort of free session on it. Haven't tried Codex mobile Valeria. Um, however enable self-service. Oh, yeah. I'll have to check this out. Thanks for sharing, Modita. >> Yeah, that's cool. >> The self-service stuff's so interesting. I mean, it's just like there's there's not a great solution out there for even any of the enterprise stuff around like how do you validate that the numbers correct? like Snowflake um cortex is is amazing. It's it's writes a great SQL and stuff, but there's like there's not really a great system of like saying like, "Hey, is this number correct or not?" Like when I talk to my anal engineers, I'm just like, "Hey, how do you know this is like correct?" And their answer is usually like, "Oh, I think it should be if we build this table out." And it's like, "Man, I've built like really good semantic views where I I'm sure no one can screw up the number." And then I give it to like someone in the seauite and they've figured out some creative way to screw up the number. I'm like, why did you ask it that? Um, so like we focus a lot in our advanced course on like validation.

59:34 It's kind of why we're leading more into like empowering people who like are checking the numbers themselves. Otherwise, you end up in this weird situation where like um like you're just in this like reviewer mode where like say like one of your stakeholders runs an analysis and then sends it to you and then you just have to like go through the whole trace of the conversation they had with the agent and check the number anyways. It would have been just faster if you ran it yourself. [snorts] [clears throat] Um, we've talked about open claw a bit, but we haven't done anything anything there.

60:09 Any other questions? I know we're over I have like five 10 more minutes. If we have other questions though, >> tomorrow we're gonna do we have some a bunch of stuff coming up in the next two weeks that's free um that I can just drop links in for right now. So tomorrow we have one on agent experiment design and analysis. Um this is free. We're actually working on a separate repo called agent XP. It's like pretty like early stage right now, but it'll be an open source thing kind of like the AI analyst, but it's like purely focused on experimentation. And the goal of it is to remove the bias that the humans have on experimentation because it's like with experiments, I don't know how many of you have ran experiments. I ran a lot and it's kind of like you're grading your own homework. Uh like the people like me who's building the building the product, building the feature is the same people who's creating the experiment and then reading the results.

61:19 So it's like you really want it to be good. You're like so incentivized to like find some part of this experiment that was a success. And so uh this this repo it's not necessarily around like the instrumentation of the experiment itself. It's more around like how do we set up and automate guard rails so that like we're not biasing the results and we can um make sure that's like oh yeah this is this is truly a success or not.

61:46 Um kind of like how we did the the question framing here. Um so we'll have that and then Sravia is leading another really cool one next week called publish analysis everywhere with cloud code. Um this is about using um MCPS to get your analysis output in like several different um platforms. This is going to be really cool. I think that's one of like the really powerful >> parts of this system is like the MCP um work. And then next and then I talked about this other one already uh which kind of goes hand inhand with this question framing one is develop uh northstar metrics with AI. And so this will be well today was around like question framing. This will involve some of that but it will be around how do you like create really rock solid metrics um that actually reflect like the user value what you're trying to build. So those are all free sessions um check those out and uh what was the reason behind keeping this open source? Yeah. I mean, my philosophy is now it's like um like there's a couple reasons like one like I think like building is so cheap now that it's kind of a little pointless to like try and like build product and put it behind a pay wall for some things. It's like you guys, anyone here can go build the AI analyst like you know like we teach a boot camp on it where it's like we believe in like this our a few days people should have like the skills it takes to build it theirelves. So it's like um that's one reason. Uh, and then the other reason I think is just like since like everything is like changing and developing so fast, like I just think the best way to learn is for is having kind of some sort of open source community where we're all kind of like testing stuff. A lot of the really uh like I don't know valuable parts of the of the boot camps and the fiveweek course for me is all the live time with all of everyone who joins because people take what they learn and then immediately use it at work that week and find out way more better ways to implement it than I would and then we take that and we share out across the across everyone. So it's like it's we're we're entering this field in energetic analytics where it's not like hey like there's decades and decades of of stats best practices that you can just kind of read books on or like people just have like the expertise and know what it is.

64:24 It's like I don't know if this is the best way to do it. I don't know what the best way to do it is going to be in a month or two from now. So I think it's really important for people to kind of like be like learning in public. Like I think we can go like fast alone but like way further together, right? So that's like the reason behind open source. Open source is also a nice hedge, right?

64:48 If I screw things up, it's like, hey man, it's free. Like you didn't pay money for that. >> Um what other questions? I got a couple minutes, but we can also close out >> and I'll send you uh a bunch of PDFs, guys. I'll be sending you three PDFs with some prompts and uh you know, and later as soon as we have the recording the recording as well. >> Cool. Yeah. So, check out that Maven 100 deal. I think it runs till June 7, whenever Sunday is, they give that 25% off. It's not just our course. It's like all the top courses on Maven. So, um definitely would recommend checking out Maven um in the next 3 days because there's other stuff on there too if you're not looking for analytics um that you can get a good really good deal on right now.

65:39 All right, thanks everyone. Uh hopefully we'll see some of you tomorrow at the experiment at the edge experiment design and analysis one >> and get your questions there as well. We'll mostly have like last 10 15 minutes like we did today. Okay. >> Yep. >> Thank you so much attending. Bye guys.