Frame Questions That Drive Decisions with AI

0:00 lightning lesson around uh framing questions that drive decisions with AI. Uh my name is Hi. Uh I'm here with my colleague Sean Butler and Shravia Medapipali. And uh today we're going to talk a little bit about uh how to ask really good questions so that uh you can leverage AI, you can leverage data to validate and or um you know drive decisions that really matter for you. So, um, a little bit of intro and then we'll get right into the content. Um, again, my name is Hi. I'm the head of data at a legal tech company called Entra. I previously worked at LinkedIn, Pinterest, Next Door. Uh, that's where I met Sean and uh, and Shravia. Uh, Shravia or Sean. Do you guys want to give a little bit of intro about you guys?

0:51 >> Go for it, Shravia. >> Okay, I can go for it. I'm Shravia Maripali. Hello everyone. Um I'm uh origially from India, South India. I speak Telugu. I see Ekil asking this question. That's my native language. I've been here in the US for a a long time. I uh have a I did my masters here and I worked at Microsoft. I worked at uh Next Door. That's where I met Sean and Hi. And we kicked off our data neighbor podcast from there. And now I lead growth data science at uh Grammarly, which is now called Superhum.

1:25 So yeah, that's about me. >> Hey everyone, I'm Sean. Um, I'm a principal data scientist at the same company. Hi. Works at Entra legal tech company. Been in tech and data science for about a little over 10 years. And yeah, teach a course on AI evals. That's kind of my like primary focus area. Um, and then I also teach this course with high stravi around AI analysts for builders. So excited to meet you all here. Um yeah, if you want to connect, I'm going to drop a link to our Slack actually in the chat as well. So we keep the conversation going after the chat here and if as we kind of like go through feel free to drop your like LinkedIn in the chat as well and we'll get in touch with you.

2:15 >> Cool. Awesome. Um yeah so um we as Sean mentioned and we'll get into a little bit later uh in the in the uh presentation um we run a couple courses one is on uh helping or I guess it's called AI analytics for builders and uh the whole idea is uh sort of like if we zoom out from this lesson we wanted to help people to become analytically independent uh being able to ask really great questions get really great insights with the help of AI and uh delegate the execution to AI instead of you having to run all the uh all the queries and an analysis. And then there's also um an AI evals for product development that Sean runs. And then we also have really great uh free lightning lessons coming up almost one a week for the next uh many many weeks. Uh so uh check them out on data neighbor.com.

3:11 Cool. All right. So let's see. So, let's get into why we're here today. And, uh, by the way, uh, we're probably going to run this for 45 minutes and then, uh, can stay over for questions. So, um, if you have any any questions you have, um, feel free to drop it in chat at any time and, uh, uh, Sean, uh, feel free to interrupt if, uh, if there's something that we should cover. All right, cool. So, let's get into why we're all here today. So today uh you know I think the the premise is pretty simple. Um it's so the idea is some people can extract gold from their questions while others get garbage. So probably I'm hope I'm I'm guessing people here have seen have seen that some people ask really great questions that always seems to be uh consequential. Some people always get into the running around the circle kind of kind of questions. So what exactly is the difference between these type of questions and uh uh and how do we actually get better at it? So the idea is uh your question quality really determines your output quality.

4:21 This is especially true in the age of AI because the more generic things that you ask of your LLMs or your chatbot, the less precise of an answer or probably an answer that does not align with what you're actually thinking about comes out. So we wanted to uh given our uh collective experience in the tech industry of working with stakeholders to really ask questions, really ask uh thoughtful sort of like um uh uh things. uh we wanted to just uh kind of like bring this lesson together to to make sure um you know like we understand what are the elements of those um and uh you know like one thing that we can sort of sort of assume is that given the same set of data um uh uh and we're going to run a little bit of a kind of like a poll here uh two different questions which one do you think is a better question that actually would lead to better answers, better outcome. Drop in the chat, one or two.

5:32 Come on. I'm expecting some ones. No ones. Oh, VJ wants one. VJ says one. Nice. Cool. Uh I think we have almost consensus here. Um so, uh two is certainly better. It's definitely longer, a lot more context. And in this lesson, we're just going to go over sort of like breaking it down into what what makes it better for uh you know, like G versus number one, but I'm glad that uh everyone almost everyone chooses picks number two.

6:09 All right, cool. So, uh what you're going to get out of today uh by the end of this session, you will know how to frame questions that drive decisions. So, we're going to go through a simple three item question quality checklist. So, that's a very um uh simple framework to evaluate any analytical questions you may come across. Number two, we'll show you how to use AI as a coaching partner. So, moving beyond just answering m answering your questions, but using it to sort of like give you feedback to uh help you refine questions. And then number three is uh you'll take away the prompt that we actually would use here uh afterwards and uh that'll be in your f in a follow-up email so that you can uh uh whenever you want to ask when whenever you you you want to refine a question uh feel free to sort of like play around with it.

6:58 All right, cool. So question quality checklist. This is the the the most important sort of framework that we uh uh that we should all take away. So uh it's very simple. It's three steps. So almost like three elements that really make a question that is analytically sound and that's answering it would actually give you something give you a direction or give you an answer that you can actually act on immediately. And those components are uh number one decision tide. So when a question is asked does this inform a decision like what decision is it? So almost like think of it upfront uh an anti-attern to what this is and we'll go through some examples and also evolutions of how to think about you know like from something that is not to something that is uh in all three elements of this. Um you can imagine something like hey it would be interesting to know and then you know like some some question uh is probably not a decision tied uh doesn't have the decision decision tied component of it because you know like what are we going to do about it or what are we going to do with that information um so that's what uh the number one check is the number two is data grounded so can I actually answer this with data So sometimes there could be a lot of questions where hey it's just impossible to find the data or the data is just uh like in like uh does not exist and so even if the question is wellformed it's impossible to answer. So, you know, I think I hope this is uh this is also obvious. And then number three is uh is the question specific enough? Um meaning like uh you know like it's bounded somehow. So it's not like a boil the ocean, look for everything under the sun kind of thing, but it's very narrowly targeted at something that would be sort of like, hey, if I can get to this answer, I'm good to go. And so we'll break down what that looks like. um in the next few slides with examples.

9:12 All right, cool. Um I don't know how to animate. So that's why uh uh uh I I would have I would have done this with uh with sort of like animations of one two three in sequence but I don't know how to do that. So the answer is already here. But let's walk let's let's walk through it. Uh so example progression of a question um from vague to sharp uh in this example is the very first version of this is why are users turnurning um and uh what's what's the uh I guess what's the you know is that a good question is that not not a good question I guess with chat it's a little bit it's a little bit hard but it's not a good question um because number one the decision is clear. So what are we going to do with this? Like is this just interesting? Did an executive ask me why users are turning or what am I trying to do here? So the decision is unclear. Uh Justin, great great response too vague.

10:14 So like you know like which users what time frame? There's a lot of stuff that's missing here. Nothing is specific. Nothing is grounded. Uh so if we progress down to number two, refine this a little bit, we can ask what's our turn rate by cohort. That's a valid question, right? The the question itself is getting a little better. The the sort of like the elements are still missing some of them. Um so the data does exist, meaning like the measurement is much clearer like we're looking for turn rates uh by cohort. But what do we do with this information? Like is this a good to know? Are we trying to figure out what to invest in or are we just trying to be like uh you know like uh I don't know like insight of the month and good to you know like uh some something to uh something to impress others. So um you know like the sort of like the decision is uh is is what's really missing here.

11:14 And then number three, like another refinement, another attempt at refining it could be which user behaviors in the 30 days cor in the first 30 days correlate with churn. So we can trigger interventions before they leave. Now it's a very long question, but it does hit all the components where the decision is clear like we're actually trying to intervene. uh like something triggers us to think that churn is something that we should be investigating and that we should want to actually take action to intervene but we need a direction for how to intervene and so uh like decision is clear the data we have again like how to measure turn first 30 days that kind of stuff and then it's very specific it's like hey this cohort is what I'm looking for and first 30 days is what I'm trying to correlate that against do you guys see kind of like the the evolution here from super vague to something much more refined.

12:15 And you can imagine like answer number three is a lot less work but also a lot more targeted than number one. Right. >> Cool. All right. >> I like the comment in the chat too around like what does churn even mean? Like churn can mean I've had I've been in in conversations where there's two exacts talking about churn and one's like did the customer leave and the other one's like does the customer not use the product and they'll pass quarter or are they using the product and not using paid features like churn also in itself is nondescript.

12:56 >> That's a that's a really good point. So you know if your organization or if you uh already have sort of like something well defined then that's certainly much better in a you guys are in a much better place than if the company doesn't even know what churn even mean or haven't agreed to it. So yeah it's a great that's a great point. Uh the previous example assumes churn is universally aligned in your company. Uh but if not, we have a lesson next week to actually chat about how to uh how to how to make that happen.

13:31 All right, cool. So, let's uh let's move on. All right, cool. Example two, progression two, uh from left to right. So, first iteration of a of a question. How are users behaving? Is that a good question? Did that give away? All right. I see some Oh, well, no. I see not not sorry. I see head shaking. So, that's good. Um, cool. So, yeah, way too big, right? Not a great question. Terrible. Um, what what what does behaving even mean? How do you even measure it? What time frame? What are we trying to do? That kind of stuff, right? Uh, number two, this is probably better, right? What features do power users use most?

14:21 like at least the behavior part is a little bit more guided now like uh what features are they using. So uh that's you know like that's that's better than what how are they behaving behaving in what way. So this is at least you know within the confine of a product what are actually what are people actually doing there um and specifically for power users. Again, the decision here is missing like what are we doing with this information. So that's uh you know like there's many flavors of of answering this question or crafting an answer to it. But if the decision is unclear, the answer is also extremely open-ended and and also the process of executing that for that answer is going to be is going to be uh not a fun not a fun thing to do.

15:06 All right, the final iteration of this guy. Which three features correlate with 30-day retention above 50% for users acquired in the last quarter? And should we surface them in onboarding? You guys see like I mean again like more words much more ver verbals. I'm not I'm not advocating for more words or more verbals but I'm advocating for more targeted way of framing your question. And it doesn't have to be spelled out specifically, but you should always keep in mind that these are the elements even if I don't say it out loud.

15:41 That's where I'm going with it to get the best sort of like output either from uh a data team if you work with them or uh from AI uh like just like everybody has access to. So you know like uh I I I hope this is this is clear. Number three is way better. three features very much uh like decision the decision is is is laid out here what you're going to do with it. The data is again like we can we can debate what how retention is defined assuming it's already defined then you know like the data hopefully is already there and uh and very specific and you know also with a threshold that you that that that we know defines what good looks like. So, um you know the uh again like the more that we can get closer to number three or to the right, the better I can guarantee you your your answers is going to be and the less back and forth you're going to have to uh work with whether it be AI or uh uh or or humans to get that answer.

16:47 Cool. Let us move on. All right, example three. I put this in last minute. So, uh let's see let's see how this goes. Cool. So, assume we have a marketing uh performance question um around uh return on investment. So, left to right again, how's our marketing performing? If you've worked with marketing teams, you've probably got something very similar in the past. I see Justin nodding vehemently, so I'm guessing you work with marketing quite a bit.

17:19 Uh yeah. So um very you know like very classic question that's like hey how how's our ex performing? That's that's sort of like uh a lot of the the questions that either the high-ups or yourself or your stakeholders um would would ask because that's the easiest to to to to answer. You don't it doesn't require any brain power, doesn't require any thinking. You just blur it out and like let the data guide you, right? like that's the that's that's sort of like the the thing that most people like to do. Um but again if we run it through the checklist uh or the framework decision is unclear performance against what what would you change if you know this information like always be asking yourself that. All right moving on to the middle which this is a another refined question from the very beginning. So what which channels have the best ROI?

18:12 Now this is getting better getting getting much better actually but again decision unclear best compared to what threshold and what do you do with this list? Um again the the thing that's missing in a lot of questions that we've come across um in our experience has always been what decision are you trying to drive? And so the more that you can think ahead of time why you're asking this and what are you going to do with the information, the better the outcome or the output you're going to get um to to actually get at the the the the real question behind the question uh that you're actually looking to looking to ask. All right. And the third refined final state is which channels exceeded our 1.5x return on ad spend target last quarter and should we shift budget from underperformers to scale them. So return on ad spend is just uh uh uh every dollar that I put in uh in advertisement how how many dollars do I get back? So like I get back dollar a$150 every dollar that I put in. So uh this question gets to uh understanding um uh understanding sort of like uh hey should we like should we be allocating re rebalancing our budgets for for marketing purposes.

19:33 You guys see how this evolution is makes things much more clear and doing this exercise grounds you into sort of like a decision forcing answer at the end of the day. Like you cannot not get a answer that doesn't drive decisions if you think of the decision up front. Cool. All right. Um let's get to a common push back. Uh I'm I'm I'm guessing some of you might have encountered this. I certainly have many many times. Um uh in fact, I encountered it last week. So uh uh what about if I'm just exploring or what about just letting the data uh guide us or I'm just looking at the data? Uh nothing like nothing truly I'm not deciding on on anything. Uh but I'm just looking I'm just curious uh what you know like that that seems like a pretty common thing, right? like like why can't I just be exploring? Um the rebuttal to that why that is ineffective is that even exploration needs a direction. Very in very few circumstances would you be like I'm okay with the entire map canvas being the sort of like uh all possibilities of uh of of exploration.

20:54 Like that's that's just not not effective, right? Like if if you're if you're setting sail to some destination and you're like, I don't know where it is. I have no hypothesis. I'm just going to go, you know, like whichever way. That's probably not not not not uh not not not the way that you would uh spend your resources and stuff. So reframing the word exploration as a decision is actually itself getting you a lot closer to an effective sort of like a effective question in itself. So some some examples that I listed here uh instead of saying I'm just exploring user behavior or I'm exploring X uh instead you can you can just reframe that into what patterns would change our next sprint priorities like you're you're you're trying to look for something that does something for you. So say that something out loud or that ladder something. I'm already mixing a bunch of somethings but like uh you get what I'm saying. Uh so uh you see how like one is extremely vague, the other one is forcing some guard rails on or I guess at least forcing some direction to the very vague thing and uh answering that itself forces you into uh looking at the right things. The second one is uh I'm just looking at the data.

22:18 Um so you know common push back again uh another way to reframe it is uh what would I need to see to recommend we invest in X or what do I need to see to uh to do to do Y uh uh and like you know like that that that sort of framing gets you to be sort of like refining your questions to be like hey I'm I'm actually interested in this direction or or I'm actually looking at uh uh wanted to to see this and that. Um, and then the final one that's uh that's listed out here, I don't have a decision yet, so I'm just looking. Uh, the the the rebuttal to that is what's worth investigating further is your decision.

23:00 And do not overlook that. Like uh if if if you have some some hunch of like I don't have a decision yet, but I'm interested in this area. interested in that area itself is a decision because you have some sort of knowledge subconsciously consciously that there might be a smoking gun in this area that I really wanted to dive in. So list that out and that becomes sort of your uh your your decision. So undirected exploration is like a fishing expedition. You're just trying to you're just trying to to see where the fishes are. Uh but directed exploration is discovery with purpose. So that's the way to to to really think about it. So next time if you're like, hey, I'm just exploring, reframe it. If you hear your stakeholders or your people that you work with telling you to, you know, like, hey, go dig up this data. I'm I'm really curious. Uh, force them to like, hey, let's let's let's rework that a little bit up front so that we can get a lot better answers down the line.

24:02 Cool. All right. So, let's see. Oh, okay. Cool. Uh yes. Um uh we brought up this uh or previously uh uh one of you brought up this point that how do you even define a metric like what what does churn mean? Uh uh you know like how how how do we get to a place where we can actually ask some of these questions or if the actual measurement is not clear then how do we even ask the questions?

24:29 Um we you know next week around this time or I guess next week Wednesday exactly a week from now um uh I'm also running another lightning lesson around uh the title for it is design metrics that don't lie. Uh it's certainly a clickbay one but the idea is what it says. It's uh how do you actually come up with metrics that drive decisions? So not the qual not the question uh because we cover it today but the actual measurements how do you actually come up with nonvanity metrics that actually gets you to measuring the stuff that really matter. We see this struggle a lot uh in many many companies and uh it's not a straightforward one and so we'll provide very similarly like some framework uh and some uh uh some not quite checklist but like uh like ways of thinking about it so that uh so that we can all so that you can all like uh take away and uh uh and go through and go through them uh uh in you know like as you as you embark on any exercises like that. So you can uh feel free to scan this uh QR code that gets to the registration page or go to bit.ly/metricxedesign uh sign up there and uh you'll get the recording even if you can't attend. Um but we'll we'll follow up with uh with links to these as well. Oh, and Sean put the links. QR doesn't work. Sorry, Bob. My QR codes still apparently is not great, but we'll we'll we'll we'll follow up.

26:02 Cool. All right. Uh let's see. I think we can skip this practice moment. Yeah. Originally I was going to have uh everyone do a practice, but let me let me skip this. Okay. So, uh key insights humans or AI will execute whatever questions you give it. The checklist uh that we just went through ensures you ensures that you're asking the right questions. So uh you you you've probably heard of some var variations of this quality input leads to quality output. Uh in other words, garbage in garbage out. So uh spend time upfront to make sure whatever the input is is not garbage. Then you can you can basically ensure that uh whatever output it is is also not garbage.

26:51 All right, cool. So um let's see. So yeah, so we'll get into this segment of uh how to actually use AI as your coaching partner uh to sort of like go through the things that we uh that we that we went through. Um so I I'll I'll do like a I'll do like a quick demo here. Um but assume the question is like uh either you asked or you've gotten this question or you've heard of someone brought up a very similar question. Hey, can you look into our support tickets?

27:23 They've spiked recently and the support team is overwhelmed. Not sure what's driving it. So, well, I guess uh drop in chat yes or no. Is this a good question? Feel free to say yes. It's cool. I think it could be a good question. Victoria says no. If Dit says yes. Sean says I hate support tickets. Okay. No, Natalia. Cool. Okay. Well, we'll ask AI if this is a good question or not.

27:59 Uh, so let's see. All right. So, I'm going to switch window. So, this is a prompt that you all are going to get uh and I'm just going to go through go through it live with you all. So, this is chat GBT. Hopefully, everyone has uh has has used it uh in the past and know what it is. So, let me see. You guys can see my screen, right? This chat GBT canvas. Cool. All right. So, I'm just going to copy and paste my question in.

28:27 And don't worry, you don't need to read the whole thing here. And I'm going to give it a question. All right. So, this is a question quality quote blah blah blah. What question do I want to give it? So, I am copy and pasting the exact same thing here. Uh, all right. Cool. So I pasted in the exact question that uh that was on the slide and uh uh and then this is what it's telling me.

28:58 So uh thanks for a raw question. This is exactly a kind of starting point that benefits from sharpening. So basically this is not a good question. So let's work on it uh in a variable light way. So um uh you will see that it's going to walk through kind of like one by one um on sort of like the the things that we talked about in the checklist. So, what decision would you make differently? What would you do if you know this this this this uh this information? Uh and I saw Patrick says we want to focus on fixing them. Um uh fix fix stuff. Let's call it uh well, I guess fixed stuff is not great either. So, okay. So, bug fixes.

29:42 Bug fixes for high priority stuff. Right. Getting sharper. So I'm getting the validation from AI. Uh and then the next one is data grounded. Actually, what data do I have to be able to do to to to look at that? Um so I'm going to say take a volume by day, right? And then so now I get to checklists. Um and is it specific enough? So how would you know when you're fully answered this? And it's giving me some examples.

30:14 So I can say um you know like normally uh I know anything above uh 10% week over week spike means bad news. So let's use that as a threshold. See what I do what I did there. Cool. So now I passed all the checklists and uh and and and things like that and it helps me refine them into uh into uh like a like a like a question that uh uh that we saw uh previously. So it gave me some some examples of it and uh uh and then these are uh these are certainly much better ones that uh that we can ask. So, this is almost like a kind of like a like a async coach that you ask AI to uh to to help. Uh it could either be a question that you already have or it could be I have this vague idea. I wanted to formulate it into a good question kind of uh kind of use.

31:21 So, um either way works. And so, we're going to switch back to here. So, this is this was the prompt. And so I'll be giving everybody this guy. Uh okay, cool. So, uh this is what we did. We did a demo walkthrough of uh a conversation with an AI coach. Um so exactly what we what we saw. If you found an answer, what would you do? Uh what data do you have access to and how will you know when you're done? So all the three elements that we talked about.

31:55 Cool. And then uh the refined questions. These were some of the some of the candidates not quite exactly um uh from the outputs of of uh of of chat GBT but like the flavor is the same. So for example like which ticket categories drove the biggest increase in volume past two weeks should we prioritize a bug fix blah blah uh or are certain user segments driving disproportionate ticket volume and should we prioritize fixes for those segments. Uh both of these are really really great questions, extremely targeted and we know exactly what we're going to do when we get the answer to it. So that's what that's where we want to get you all to.

32:36 All right, cool. Um so uh I could have very well given everybody here just like a did this question pass this or you know like just give the answer directly of uh you know like uh every time when you feed it something it'll give you the answer. Uh but you know like our philosophy is that coaching is better than grading. So, uh, if you notice what just happened, we you didn't just get a yes, no is good, is bad, but you had a conversation with that AI thing, uh, through kind of like the this this, uh, this this prompt, and it helped you discover the gaps in your thinking. uh and by the end you understand why a question was weak not just that it is weak because uh let's let's be honest it's uh uh you know like especially in the world of AI where think where getting an answer is so easy the process of the fundamentals like acquiring the fundamentals knowing how to actually do it yourself first before delegating is extremely important and we really want to help everybody to sort of like develop that muscle And so the checklist is a skill. The prompt helps you internalize it faster.

33:48 That's that's that's all the philosophy that that we have with uh with this. All right. Cool. Um wow, that was a very quick almost 40 minutes. Um check your email after this session around uh a PDF guide on question quality. A it will have the uh coaching prompt that we just went through and then we'll also send over this deck for your reference. Um or if you missed any of the uh the slides or want to want to uh refer them in the future uh definitely, you know, feel free to feel free to leverage them and you you you'll get them in your inbox today.

34:27 Um cool. And uh we also have a free AI builders community. This is a slack channel uh that we recently uh stood up to uh to just kind of gather people who are interested in the whole uh you know like AI uh building community. Um you know regardless of uh what your uh sort of like what your focus is. So we have uh uh members uh uh this is all three uh we have folks from all over the world um who are uh PMs who are um uh engineers uh who's trying to build with AI who is trying to learn more about AI uh and so definitely join us um we'll we'll share a lot of uh uh free resources and uh uh we would love to have you join us. So the link to that is bit.lyic /iconnect.

35:20 Cool. And uh as I mentioned at the very beginning um we also run this course on Maven that is called AI analytics for builders. The idea that we all that Sean Travia and myself feel extremely passionate about is to help people become analytically independent. Meaning you ask really great questions. you have a very analytical mindset. We teach you all the framework that we've learned over the years in our own profession in order to be able to sort of like uh uh kind of like um you know ask great questions uh make really informed decisions with data but you you can delegate the execution of getting those answers of getting to uh you know like like quality results through the use of AI instead of you know like learning all the technical skills and stuff like that. So um the thing that we sort of like uh uh put an analogy on is uh in today's world there is you've probably heard of vibe coding right like everyone can anyone can uh vibe code stuff even if you're not a software engineer to build um applications or write software code uh you can use AI to help you become a product manager um through you know like uh uh uh uh doing you know like the product requirement uh documents and things like that or you can use AI to help you be a designer even if you're you have no no idea how to design stuff.

36:53 uh what we're seeing missing in the market is really around the data piece like how do you become analytically sufficient self-sufficient such that uh you don't need to rely on a data analyst or data scientist or if you're a data professional you can be better at being more effective in how you do sort of like data work uh and offload a lot of that to to uh to AI so you can focus on the most important things so we run a fiveweek course that starts starts in April. Um it's going to have pre comprehensive training around all the things uh uh all the things that uh that that that we learn around analytical analytic workflows and mindsets and how to think about it and uh and then we also teach how to how to do those execution with AI. Uh you'll have a capstone project by the end of the by the end of the course. Uh we're running a limited time discount um to everyone who's uh who's uh who's joining this uh lightning lesson. 25% off uh valid through February 14th. Promo code AIBuilder LL at checkout uh if you scan the um the the the the thing here. And I forgot the forgot to put in the link, but we'll we'll have all that information uh uh in the follow-up email.

38:13 >> One thing I'd like to add is we collectively, hi um Sean and I have a 40 years of experience. Like if you look through multiple companies, this is our first attempt to try to get all the knowledge that we gathered over the years into a single course. So uh we are basically trying to get everything to you guys in a way of h like with AI now like uh try to use our experience and how how how do we get to use AI and get to these? So yeah, just wanted to iterate on that.

38:48 >> Yep. um uh AI is not quite replacing us quite yet and we know where and how uh where it's going uh through the podcast that we do on the side as well. So um certainly we'll share uh a lot of those uh to you all. Um so final thought and we'll get into Q&A is uh the question isn't whether you use AI for analysis or AI for anything really everyone is going to. The question is are you going to be the question? Are you going to be the person asking the right questions or the person generating garbage? Don't be the latter because it's very easy to fall into that. Uh and uh uh you know like just as a mantra in the AI world, we wanted to emphasize this.

39:33 Uh cool. Okay, that's the that's the end of my repaired uh uh programming. So I'm going to go back to here and just leave it up. And uh if you have any if you have any questions, floor is yours. And uh I'll I'll I'll stay on for however long folks want to hang out. >> Yeah, I think Sean and I can also answer any questions. Uh so one thing I'll probably like to add is that through all this experience in data and data science what I've observed is critical thinking and curiosity and going in the right direction is literally what like leads to impact. It saves you a ton of time. Imagine I can I can see myself like you know 10 years ago when I started off I went in directions that were took me a long time but leaded to me in places that didn't lead to any impact but today if the same question comes to me how do I handle it is like so much so different and so much better and that is purely the experience teaching me which angle do I pursue how do I get my curiosity to you know tackle me and what questions to ask the question that you get and things like that. So that's the reason I think today's score today's session by Hyde really resonates um and it's something that we literally use it in our day-to-day jobs as well >> and we're running this uh as a first cohort. So we're new to sort of like uh uh teaching in this way. So you're you you you can if you decide to join us, you're going to be helping us uh uh shape uh where where where this goes.

41:26 >> Has anyone spent a bunch of time on an analysis or or something because just because like I don't know a question was phrased in a way that they interpret it. I've definitely I feel like early in my career I've spent like probably months sometimes on analysis or building a dashboard or something or some readout because VP of product had some question that came to me in a Slack or in the hallway and then it was like months later like I actually didn't really care about that.

41:57 I was like telling everyone else like oh I'm doing this thing for the the VP of so and so. Um, wonder if anyone else has experienced that. I just find like asking good questions is also just extremely important for your career. Like it could be like a career maker if you can tie if you can get your workflow and your mindset where like as much of your work turns into something happening at the company, a decision, an action, something changing because you were there like because you were basically the key. A lot of that just comes from making sure the question is framed from the very beginning.

42:43 BJ's asking, can you show me examples of weak versus strong AI questions for my use case and help me write mine, rewrite mine? Uh, yeah. I mean, we'll send you the prompt and uh certainly you can uh you can iterate with it and it'll tell you why. Um, you know, some questions are better than others. >> Yeah, we could talk we could talk about that in that Slack community too, VJ. Um, if we want to go a little deeper, I know we don't have like a whole bunch of time here, but definitely down to talk through your use case.

43:17 I mean to add Vijay basically the more context you gather from the people who is asking the question the more context you give to AI the better the direction of you know you you solving it comes from and uh exactly the framework that hi shared as well is going to like literally help you in in like getting there faster. >> I think to your point as well having patient thoughtful feedback is it's important but it's difficult sometimes in that I'm definitely an example where was asked by a higher up to do some analysis that they thought was important that was important to them they wanted this this dashboard this kind of metric and then we spent some time taking it from Excel prototype to the Tableau dashboard but um actually with the frontline leadership below the executive um it wasn't very meaningful. Didn't have the right immigration context included really made make sense of some of the higher level subs and so you know could have saved a lot of time getting that beforehand before you know the effort to put up a high level sub rate that wasn't actually useful.

44:36 >> Yeah, it's a great that's a great point. Um, uh, one secret trick that's, uh, I guess not so secret because we talked about it quite a bit is, uh, uh, you know, like if if if you're the one that ask that asks questions, uh, always ask yourself what decision I'm trying to make with this information. And if you're the one that's being passed the question to go solve, ask the person that gave you that question, what decision are you trying to drive with this question? Uh I can tell you in at least in my experience uh probably 60% of the time they would be like actually that's not a really good question. So let me just uh you know it's cool. You don't need to work on it. Uh uh and so you know like just asking one follow-up question could uh potentially save 60% of the time. So uh that's a very high ROI thing to do.

45:29 >> If if this is a predictor model that you're trying to look for the time saved. I think the first feature that you'd get is did you ask that second question or did you just take up and just go work on it you know uh I would say as a new manager when I was starting out as a leader the first thing is like oh my god this CPO asked this thing let's just you know get my team to work on it right versus uh irrespective of who the person is hey uh why are you asking this because you could get it from here already and would just you know trying to be curious And then they're like, "Oh, you're right." And then stop. Like the entire uh whirlpool that starts when a sea sweetite person asks it just gets died down right after.

46:16 >> Yeah, totally. Uh I see a question from Sean. Hey, how's it going Sean? Long time no you'll see. >> Hey Sean, nice. >> That's cool. Uh the question is are there differences in structure and types of questions you would ask to a human versus AI? Um I mean structurally no. Uh I guess practically yes because humans have emotions and you need to meet them where they're at. Uh I would say AI is certainly a lot more uh well I mean this is recorded like in in the future state where AI understands everything like you can sort of like you know you can stretch it to its limits right uh and it it's not going to be like hey I'm I'm I'm hurt by whatever you're asking me so uh human has the emotional aspect so you need to meet them where they're at you need to make sure sort of like uh you take empathy and all that kind of stuff in mind uh I think AI is much more free flowing and uh uh the sort of like common denominator of both is the more to Shia's point the more context you can kind of build them up uh whether human or AI the better that they're that they're going to be at in kind of like giving you the answer and then you know the context could come from the question the questions themselves as well that's why that's almost like um uh sort of like the uh the P 0 there >> to double click on what Hi shared about to humans you should meet them where they're at. It's actually the most important thing and it's actually a well-developed skill as well. Knowing where the person's coming from. In fact, you could the the way you converse with them and the way you understand the question and talk with them might just make your relationship with that person in a way that you're not like against them asking the question or against working on something. You want to definitely tell that or you know show that attitude when you ask a question back uh because that would help them basically before another question comes they know how to use the self-s served tools that are already available right and it's not that data doesn't want to support it but it's more that um we are here to support but your question could be something that's already uh you know being able can be answered with existing stuff or might not be even important enough and the other question behind the question is more important.

48:49 >> Yeah. I feel like sometimes you have to like like you said like explain. It depends on the persona you're working with. Sometimes you have to explain why you're asking a question because some people can be taken back when you question their question. It's like I'm not trying to question you, dude. like uh I actually want to really help you and so if you can work through this with me then like we'll make sure that whatever you need gets done and you look really good and everyone wins. Um so like always trying to like reframe that way. You know how to do it with AI. I mean maybe we should be doing that with AI. I don't know.

49:31 Cool. Um, cool. And, uh, we don't have to hang out here all day. Uh, we'll again, we'll send out the deck, we'll send out all their information. Uh, probably send out the recording tomorrow, uh, as well, so that you can always watch back or, uh, uh, you know, like if there's any segment that you have more questions about, feel free to reach us on uh, in in this uh, uh, in this Slack community. uh we're we're very responsive and uh you know we hope to be able to sort of like connect every like-minded person in uh in today's world uh so that uh uh so that we can you know like we can share information, share knowledge and help everyone succeed.

50:17 >> Awesome. Thank you so much >> everybody. Thanks for joining. See you all next week. Uh design metrics if you want to join us. See you.