Size Product Opportunities in Claude Code with AI — Data Neighbor Live

0:00 All right. So, yes, welcome everyone. Welcome to this uh lightning lesson around opportunity sizing in cloud code. Uh my name is Hi. Um I work at a legal tech company called Entra and um I was previously um worked at data science teams at uh LinkedIn, Nextoor, uh Pinterest, Meta, um all the big consumer tech companies. Um and uh next door was actually where I met uh uh my co- co-hosts uh today uh Shravia and uh and Sean. Um Shravia, do you want to introduce yourself a little bit?

0:40 >> Sure. Hello everyone. I'm Shravia Maripali. I lead the growth data science team at Superhum. It was previously called Grammarly. Um I have around 14 15 years of experience now. I majorly worked at Microsoft for 6 years. Uh moved to the Bay Area. I was in Seattle. Redmond moved to the bay, worked at eBay and Next Door and like Hi said uh Sean Hai and I met at Next Door and we want to do something in the world of data and here we are. We started with the podcast guys. You you have to check out the podcast uh from the past. We also keep sharing some of these sessions later as well there. So if you want to check out any old sessions definitely check that out. And AI analystlab.ai is where you would find everything that's related to current workshops, future workshops, courses and whatnot. Yeah.

1:27 >> Yep. Cool. Awesome. Um Sean is on the road today, so it's just going to be me and Shavia talking. So if you have any questions or if uh in between you have thoughts or comments, uh we'll drop links as well in the chat. Um feel free to just uh uh you know like put put things uh put things in the chat. Cool. Um so as Shavia said uh we are from AI analyst lab. Um so this is a uh almost like an open source community where maybe many of you have seen sort of like uh we open sourced uh a repo called AI analyst um and that's fully open source fully free that uh effectively turns claude and claude code into a senior product data scientist um who is very well-versed in how to do sort of like analytics uh in uh uh in the in in Silicon Valley uh let's call it uh but it extends to many different uh areas.

2:24 So today we're excited to talk about uh one component uh in the AI analyst uh repo to sort of like um hit on this topic on opportunity sizing. Uh in parallel we also run courses. So we have one that's coming up actually next Monday on AI analytics for builders. This is a fiveweek course where we teach people um on the kind of like the analytical thinking and the foundations of how to become analytically very sufficient in your day-to-day job. Um so that's think of it as almost like a um with a with execution delegated to AI.

3:03 So think of it almost as like a um like a condensed version of uh all our years of experience working in this field between myself, Travia and Sean. So, uh, combined to be probably over 50 years of, uh, of of experience here. Uh, yeah, that that sounds kind of crazy, but that's that's that's what we distilled into. Um, and then we also run a boot camp, like a weekend boot camp on, um, building AI analysts yourself. So, um, we ran our first cohort just a few, um, just a few, uh, no, a couple weeks ago or a week ago, something like that, very, very soon. And, uh, or very recently. And uh um I think maybe some of you have taken it um and uh it's uh it was it was really fun. So the idea was really to get people to actually building and then seeing for themselves kind of like what they're capable of in terms of building an agentic system [clears throat] in the context of kind of like the uh uh the AI analyst. Um so if you guys are interested in that uh we'll drop links throughout uh and uh let's get started.

4:12 Cool. Okay. So, um has uh before I get into this, um does everyone know what opportunity sizing even means? Drop in the chat. Just a quick yes. No. >> Yeah, just a yes. No. >> Okay. Says I do a lot of time. Nice. That's >> awesome. Time to Sam. Yeah. No. >> No. Okay. No. >> Nice. >> Cool. Okay.

4:43 >> That's a good mix. Some of you. Yes. No. >> Love it, Greg. Love it. Um, cool. Yeah. So, um, yeah. So, just to define it a little bit, um, opportunity sizing really is like, hey, if we were to work on X, how big of an addressable market are we serving? um meaning like how do we translate that into something that's uh uh that's of a common language usually down to the impact basically it's like you know hey if I were to do this what is the impact quantified um so I think the kind of like in uh on on on this slide uh there's a I think Greg kind of kind of hit on it um most of your opportunity sizing is a almost like a madeup number um at least for people that I worked with I mean like you Different people are different in terms of how they how they uh how rigorous they are in terms of the the the sizing opportunity. Um but many of them is actually a you know let's call it like a madeup number or like a pie in the sky sort of number uh on a slide deck. Um so on the left um many of the sizings could look like uh something like you know pick an optimistic assumption and then multiply it by a very big total addressable market and then uh put in a slide and then hope nobody asks kind of like where the numbers come from like nobody asks you to decompose like you know hey what's your assumption and whatever it is. So typically you'll see that in the form of uh hey this is a multi-million dollar opportunity or this is a $10 million opportunity something like that right on the right which is where we want to get to in terms of being rigorous and being thoughtful and thorough in how we're thinking about these things is that um it should always be a formula that separates kind of like what's assumed and then what is the actual math that gets you to like a believable number and And then all you're doing is checking the inputs like figuring out which inputs are grounded, which ones are assumed that then you know you can always debate and like have judgments around what it should be um and things like that. So um kind of like on the right side is what we're trying to get to um in this in this lightning lesson and that's part of some of the skills and agents that we encoded into the AI analyst repo.

7:19 Okay, cool. So, when we boil it down to like, you know, hey, what does that actually even mean? It's pretty simple, right? Like it's, you know, think of it as like one formula, just three inputs and that's it. You land at the impact at the very end of this formula. So, the first one is um stated very simply. It's like how many users or transactions are actually in scope that we're affecting. So if you were to make a decision on doing something, you probably would know like what is the um the subsets or what is the audience that you're hitting that would benefit from the change that you're thinking about making. So that's one just like the baseline number for what that what what that number is. In general, a disk you can calculate, right? Like you can pull from a user base, you can count how many uh how many actions or whatever it would be uh it would be um it would be affected. And then number two is the rate of improvement like what do you believe could be possible to uh to actually um drive the change in that number that you're looking to move. So um you know this could be anywhere from like zero to 100% or you know even even even higher like for example like hey I want my I want to double my conversion rate or something um on on some sort of uh on some sort of product flows that's totally fine uh and you know mostly you just have to figure out kind of like you know either based on experience based on industry benchmarks based on what's believable um what that improvement improvement rate could be from a uh from a sort of like a you know like not a madeup sort of way. So you can always ground it um as well in in uh in prior context. But that's the second component here. And then number three is the value per unit. Uh we call it the value per unit. So if you could translate what you're doing into dollars, what would that be in terms of every single unit that you are uh able to sort of like lift in the improvement? Um, and so, you know, pretty pretty uh pretty straightforward. If you go if you just multiply through this whole formula, then effectively you have your impact dollar amount. And then what you're doing effectively is like looking at these three components and then figure out how to get data um to kind of like plug into this thing and perhaps even construct a range of possibilities. um you know given uh given the movements of each like given the range of each one of these.

10:01 So impact formula users affected times improvement rates times value per unit. Uh that's uh hopefully that is uh that that is that is pretty straightforward to folks. Okay, cool. Um, so let's walk through some of like the uh good versus bad examples of each of these and then uh and then we'll get into some uh some other examples. Uh and then at the end or towards the end we'll get into kind of like the AI component uh claw code.

10:35 So the first one is users affected. The bad example which actually I I hear a lot myself perhaps some of you might have heard it in your uh in your day-to-day as well. Uh it could sound something like hey all our users would benefit or something like you know hey a big portion of our population would would be affected. Um I hear that all the time especially coming from you know like for example some of uh perhaps marketing teams or um you know like teams that aren't uh used to sort of like uh thinking in terms of numbers.

11:07 What a good example could be is like hey we have 12,400 monthly mobile users who started the checkout flow but don't complete. like that's really concrete that is grounded in data. So like literally the difference is extremely vague versus extremely specific. Um same thing for the second row here if you look at it improvement rates I the bad [snorts] example here is uh we think conversion will double like I mean you know totally fine like we can we can think what we think. So um you know that's never wrong. The better believable part of it is something like hey mobile checkout conversion rate is 33%. Desktop is 52%. So do you think it's pos it's plausible to close half of that gap which would get us to 42%. like that is a lot more um concrete in terms of what the possibil the range of possibility is and like if you pick a number between this that would be way more believable than let's say like hey conversion rate will just double so it's like 100% or whatever that's just not how it works and then the third component here is the value per unit. So um bad example could be hey each conversion is worth a lot.

12:23 So the question is what does a lot mean? uh and you know again like in terms of thinking in the range of possible numbers a lot does not give you anything specific. So um so always think like you know like on the right which is a good example is that hey today our average border value is 67% or $67 at 75% margin. Uh so the uh per conversion we it's worth $50 something like that. So now if you just kind of like multiply these three together and then each one of these you have the confidence just like what we just went through then probably it's going to be a lot more convincing whenever you present your case for why we should or should not invest in in uh in in certain improvement sets or feature sets. Does that make sense to people?

13:21 >> Cool. I saw in the chat someone says uh would accept oh would not accept. Okay. I heart attack for a little bit. I thought someone says they their CFO would have would accept worth a lot. >> Also everyone >> uh please uh you know uh give your comments or any questions in the chat as well. I can I can keep answering or I can like whenever high pauses we can take any of the like important questions. We do have Q&A at the end though but just saying Cool. Um, yeah. Awesome. So, um, you know, coming back to another question here. Um, or I guess another scenario here. Uh, let's call it like, so imagine you have like a e-commerce company. Uh, and we're going to get into an example company like that in our cloud code section session as well. Um, in that think like Amazon and then you have a feature for save for later. So, it's something like, you know, hey, I I don't want to buy now, but I don't want to pick again uh in the future for the stuff that I'm interested in. So, like, you know, I have a safer feature later.

14:29 Uh and it's a $5 million opportunity. And then the pitch here is uh lots of users browse but don't buy. If we add the save for later, more of them will come back and purchase. And this could be huge. Sounds pretty reasonable on the surface right now because you've seen sort of like you know hey how can we be concrete in the previous couple slides uh that is actually that that actually adds to the credibility of like you know hey if we can improve it in a numbers grounded way then this would become even more uh effective in terms of a pitch. So how would you do that? Um you know just some just some just some uh some some examples here. Uh again if you look at those three components uh what is the users affected on this uh in in this sentence it is extremely vague. The users affected here is lots of users. So um you know like not that's not very helpful uh and probably people aren't going to be patient about uh you know listening to pitches like this. Um the fix is be concrete. So 28,000 monthly users who viewed three plus products but don't add to cart. Like that is extremely specific, very concrete in terms of the addressable market like right off the bat. Second one on the improvement rates. Um again like very vague here, more of them like we can get more of them to come back. Uh well what happens if you attach a number to it like a believable number? So it could be 8 to 12% recovery rates which comes from an industry benchmark. So you're not making that number up. It is grounded in something some sources that would be a lot more effective than the uh than the uh than just like you know hey more of them or like it'll be it'll be great.

16:21 And then the value per unit the vague part is could be huge. Uh yeah anything could be huge or it could be small could be anything. The way that you add credibility to it is uh again attach a number to it that is grounded in some sort of experience or data that you already have. So $45 average first purchase from recovered users. Um that that itself would just complete this whole this whole uh whole flow and make this pitch a lot more concrete for uh for whoever the recipient of the pitch is.

16:54 Hopefully that makes sense. Okay. Uh maybe drop in chat. Um A or B, which one do you think is better? Well, I mean, well, yeah, I I'll let you judge. A or B? Oh, well, I gave it away. Who's going to say B? Or who's going to say A?

17:25 B is better. B is better. B is better. So, this is a trick question, guys. It's not B is better because B doesn't have a pitch. I'm just kidding. It's >> Yeah, I forgot to forgot to actually make a pitch out of it. Um, but the idea is there like I I think you guys got it. It's uh um you know, instead of like no no trail like instead of a pitch that's non-traceable, like think of it that way. um have something very specific that every single assumption could be discussed or could be debated um by different people. And then as long as you present your first version of how you arrive at that um at the beginning, that's going to be almost like setting the anchor for what the recipient would ultimately probably um you know like latch on to if that makes sense from a psychological perspective. So, uh, you know, like the more that you can ground your things or ground your ground your opportunity size in as many verifiable sources as you can, then the better.

18:35 Okay, cool. Uh, okay. So, a quick plug. Um what you just saw is one lesson out of I believe we have a hundred lessons in AI analytics for builders. This is our fiveweek full course around um kind of like developing the fundamentals and the foundation of what good analytical thinking and judgment um should be and uh while delegating sort of like the execution of analysis of the analytical workflow to uh to AI. So um this is yeah one one out of a 100 I think you know plus or minus 10 let's let's call it that but we have a lot of framework we have a lot of practices and we have a lot of uh a bunch of stuff uh in it that starts next week so next Monday is our day one of uh of of of this course you guys if you are interested in signing up 30% off for attending the lightning lesson um and what you're going to learn in that course is framing questions, using cloud code as your AI analyst, defining metrics, debugging funnels and root causes and then uh almost like um you know like design experiments, size opportunities, storytelling, how to convince people based on the outputs, how do you drive decisions, um things like that. So lots of framework um again encoded based off of our 50 years of experience. Uh we're giving our HOA but that's cool. And one thing I want to add to that is you'll get a repo. We have a free repo right now that you could use.

20:16 Like you all should definitely check that out. I'll share the free repo link as well, but you will get an advanced version of that repo guys. And it is pretty mind-blowing the amount of things that the repo could help you literally do today. So to understand that, you should check out our free repo once. I'll share the link right now. But if you're part of this course or the any other course, we'll basically continuing to add what we are learning into that repo. So currently it has up to 50 skills and you know 50 agents. They take any workflow and make that AI workflow.

20:50 So you'd get that repo, you could take that repo, plug into your work and start using it. So that's something that you'd get as part of this course too. >> Yep. And uh yeah, and we open sourced um sort of like the AI analyst repo as well. That's uh that's free um just not as powerful as the kind of like the one that we share in the course, but you can also feel free to check that out as well. It's uh totally accessible.

21:19 Okay, cool. Uh so let's see. So you learned the formula. Uh here's a prompt that can help you check um kind of like you know what's missing whenever you need to or whenever you get kind of like a proposal or a pitch from someone else like what what should uh what should be included. Um so I can let me just uh I'll share this with all of you like I'll send it out uh later uh after the course along with the recordings. Um but like you know before we get into cloud code this is kind of like a very simple prompt that uh all of you and we'll we'll actually look at it together um to kind of like uh to uh to see how that works. So let's see do you guys see my claude web?

22:05 >> Yes we can see it. >> Okay cool. All right. So I am going to paste in uh so okay before before I get there let's let's analyze what this prompt well let's look at what this prompt does. So this is like the finding claude to be like hey you're a senior product analyst reviewing opportunity sizing um and the opportunity is uh uh and we'll paste something in here like whatever pitch you get that's sort of like the uh opportunity and then evaluate it against each of the components that we just talked about right and then for each component give me a rating of is it clear is it vague or is it missing and rewrite the opportunity with specific numbers and why you would flag it the way that you did. So, roughly that's kind of like the setup for what this prompt does. So, you can fire up your favorite chatbot, anything like um you can fire this in co clot, co-work, chat, GBT, Gemini, anything. So, I just have Claude open. Um so, I pasted in this guy. Same thing. Um formatting is a little weird, but that's okay. So you remember there's a role, there's an opportunity, and then uh this this whole thing. So let's just see what happens.

23:28 Um let's see. So a question about why claude and not chatbt. Uh you can paste in any chatbot you want. Um like literally anything. Grock, Gemini, anything. So this would just work. Uh okay, cool. So let's see. So it basically evaluated this opportunity pitch that we gave it. We should improve our mobile checkout experience if it's clunky and we're losing sales. Like I didn't give it anything else. Just just uh just this line. So it says okay in the component of users affected that's super vague. So we're losing sales means nothing about scope. Who are these users and stuff like that. So it gives you the critique around why it's why it's vague and then how you can fix it. So concrete estimate to plug in for midsize company e-commerce sites assume 500k. Um so a lot of these are assumptions but like it gives you an idea how to sort of like plug in the numbers yourself and why it's doing the judgment that that it's doing. Um number two improvement rate is missing. So clunky is a vi clunky is a vibe not a benchmark according to claude. So there's no definition for what clunky means. Um but here's how you can anchor it. For example, uh internal um you can have a conversion rate on desktop versus mobile experiments. If you had run any prior AB test around checkouts, uh industry benchmark, these are all these all look very reasonable and is exactly kind of like what uh most people are probably going to uh anchor themselves on. And then again, number three here, value per unit, that is also vague. Losing sales implies revenue, but doesn't quantify it. No average order value, no margin, no LTV, long-term value consideration, stuff like that. Um, and then it says, "Okay, for a really good um uh pitch.

25:24 Uh, here's here's a full That's a very long one. Uh, here's a full sort of rewrite um that gets extremely specific about each of these components. So, feel free to play around with it. I'll send it I'll send it all send it out to all of you guys. Um, and uh, you know, plug it into any chatbot and the next time when you hear something about opportunity sizing, especially Greg, uh, you can plug it in here and it'll tell you, it'll tell them what's uh, what they need to come back with.

25:55 >> Cool. >> Hi, a couple of questions. Do we take them quickly now? So, the first one is why is the role important in prompt? Naveen asked this and Otto gave a very good response, but maybe a quick something quick. >> Oh, okay. Um, yeah. So LMS generally are sort of like a world knowledge. Think of it that way. And uh if you ground it at the beginning of what their role is, it almost makes them very specialized right from the from the beginning. So it goes from hey, I know everything. I'm very generalist to like, okay, let me look straight into just this slice of uh of of role that you're assigning me as. Um so, you know, definitely try it out and see what the differences are. um that's generally been sort of like the uh a prompting technique from the very beginning of LLM.

26:43 >> The question I have was the same question could be coming from a finance person could be using it. It could be a product manager, a senior product manager. Everyone does opportunity sizing. So if you say that only the product analyst will give you this answer, a finance person answer will be different. That doesn't make any sense. >> Yeah. I mean I I I could see that. I could see um their answer should be the same. Actually, that's a really good idea. Like, if we try it out on having it take on a different persona, it'll be interesting to see what uh what that what that looks like.

27:19 >> So, >> can we do it now? Can I do it now? Maybe change. >> Uh I do want to show you guys quad code. So, this one is more like a almost like a starter uh to so that everyone has access to the uh the chatbot. So Naven, you'll get access to all these uh you know uh prompts. You could definitely try it out. Like hi said uh you will see a different persona. So as soon as let's say you're a junior analyst with senior analyst, you'll have slightly more nuance and flavor to it. I would probably think as an LLM that's what that would do as well. So if it's a deterministic answer, then probably the exact number would slightly be the same, but the flavor in which they share it is probably going to be different. And the the reason why we give personas is for LLMs to make it to make the scope uh down as easy as possible. Uh so that it thinks in that direction of a product analyst, right? But product analyst does have product and analyst in it. So it it could uh pick up things like a product manager too. But yeah, uh you can ask these questions later as well. Uh there's a slack community. I I'll share the slack community nav. So you could join there and ask these questions and we'll try to answer them too. Okay.

28:30 >> Okay. Sounds good because I'm a director level and I want to understand is there a difference now coming around here if I'm going to ask my team to use the LLMs people could be asking different questions with different levels and at the end you have to have the same answer you can't have an answer coming from an analyst one number a product manager >> absolutely you hit on a very important topic that's literally what hi and I we do as part of our work too so the validating systems and how do we ensure that the answers are deterministic so that's a big topic that we could discuss maybe in the Q&A or it could be in a in the Slack channel. Navin, I can share the link. Okay, >> sounds good.

29:07 >> Thank you. Yeah. >> All right, cool. Uh, thank you for that question. So, let's uh let's get into cloud code. Uh, let me find where my thing is. Share um this. All right, cool. Can you guys see my screen here? Am I >> Yes, I do. >> You see a black? Yeah. Okay, cool. So, I am in um for those who are familiar with um IDs and uh uh text editors and cloud code in terminal. Um this is my setup.

29:44 Uh this is VS Code. Um so, think of VS Code. For those who are not aware, VS Code is one of the IDEs out there. So, on the left I have file explorers u folder file folder explorers. Um on the bottom I have my terminal and then on the top I can preview any of the files and see what actually is inside. So this is my setup for quad code. We go into it in the courses and boot camps and things like that. Um but if you you use quad code yourself, find whatever setup that makes that uh that's best suited for your use case and for how you feel comfortable doing it. This is the way that we all are pretty comfortable with.

30:23 Um, so that's sort of like the the intro to what we're doing here. Um, and because our people did people follow the news yesterday, Opus Claude Opus 4.7 dropped, right? Um, because of that, uh, I'm actually going to do something very different here, which I'm going to have Opus 4.6 six and Opus 4.7 side by side so we all we can all take a look at kind of like what uh what the differences would be given the same sort of uh setup.

30:56 Um let's see so and for those who are not aware um when Opus 4.6 6 came out that really crossed the chasm around the analytical capability of AI/Claude that makes a lot of the AI analyst repo you know skills agents and its ability to become like a senior product data scientist possible. So um you know if you sort of extrapolate all that then uh 4.7 is only going to become more capable with the harness of of AI analyst. So on the repo here we have AI analyst plus.

31:33 This is the souped-up version of the AI analyst uh that you that you see in the open source uh repo. Um so this one has gotten a lot more you know like more agents. So there's a lot of these uh for example like um deck builders, Google slide reviewers and things like that. Uh it's got more a lot more skills. Um and for those who are not aware skills and agents, agents think of it as like you know it can go and do stuff. Uh skills would almost be like the recipe for how things should be done. Um for example, so the combination of these things make sort of like a a complete harness for this repo to act like a senior data scientist. Uh okay, enough of that. I'm going to fire up side by side. So, I'm going to split my terminal. Uh, you guys see a left and a right side, right?

32:27 >> Okay. >> I'm going to fire up Claude. Never done this before, so we'll learn together what what it's going to look like. Oh, no. What? >> That can't be the case. Yeah. Okay. Yeah, it's the same computer. So, >> cool. So I have side by side right now. As you can see I'm in Opus uh 4.7 here. So I'm going to make the right side.

32:59 Um let's say model Oh, I can't go back to 4.6. Uh how do I switch to Opus 4.6? So I don't know how to switch back to Opus 4.6 since it's not available. So I'm asking Claude to do it for me. Um so run model and pick opus 4.6 or type model opus 4.6 directly. So I'm just going to do that. Um well >> interesting. I was trying to do this. So this this is something that we uh basically uh get for if we try to do things live and >> Oh no. Okay.

33:46 Huh? Okay. Well, I guess we don't have 4.6. We only have Opus 4.7 now. >> They absolutely replaced it. We had it in the morning. >> I guess uh quad and fropping moved too fast. Okay. >> Anyway. Okay. Well, I guess not then. Uh let's just look at 4.7. Uh let's see. I'm gonna exit this one. Go terminal. Okay. So over here um just a little bit of a setup. Uh we have in the AI analyst plus repo we have a um set of or I guess a bunch of data or um like a data set um that is based off of a fictional company called Novamart. Um this is actually what we teach in the course and the boot camp as well. Um think of it as the thing that I just talked about probably 10 10 20 minutes ago. Um, this is a fictional e-commerce company, think Amazon. So, there's, you know, like you buy stuff, you check out and things like that. So, it's got access to this uh uh this data set here.

34:54 So, I am going to uh let's see walk you guys through step by step on what we're doing here. So, what can you tell me about the noar mart data set? So, we're going to have um we're going to have Claude code help us sort of like do lead up lead us up to the opportunity sizing uh side of the house, but we're asking it to kind of like give us an overview of what this is. So, it's done thinking Novamar e-commerce is an active data set simulates an e-commerce company. It's got this many users, this many orders, this many uh events um that is tracked uh over in 2024. And so it's got 12 tables and stuff like that. So notable patterns, it's got a checkout funnel. So if you add stuff to your cart and then you start to check out, you pay and you uh you make the purchase and stuff like that. Uh that's sort of like the uh some of the some of the data that's available here. Now, let's see.

36:06 So um I'm so I'm going to give it my next question. I'm interested in uh let's see checkout conversion by device. Can you do an analysis on that? So uh right here uh literally I'm asking it to just give me a quick analysis around uh what are you seeing on a conversion rate basis by device. So like things like um mobile desktop and stuff like that. um and uh and it will go do it based off of some of the skills and agents um that we have encoded and we'll take a look at exactly what it what it's doing in terms of uh the skills and agents that it invoked.

37:04 Okay, so it's done says strong signal gap is concentrated in the start to payment steps. Uh it's going to check for a Simpsons paradox uh and see if uh if uh if there's any anomalies in there. So it says web beats mobile in every single channel. So you can always follow along as you um as you kind of like let the let clock code uh do its thing. And then it runs into an error. Um but it's cool. It's uh self-healing. So it understands how to fix itself. it learns, it knows how to uh get around and or um resolve kind of like errors that it hits.

37:46 Okay, cool. So, it's done doing its thing. So, it says web converts at 40% from start to complete. iOS is 29%, Android is 28%, so mobile user is 30% less likely to complete than a web user. And then it gives me kind of like the breakdown of this uh in terms of where kind of like where uh where the drop offs are. Um so it's lists out the entire funnel. So from start to payment attempt, from payment to payment complete, and then sort of like if you multiply them together, what is the uh conversion rates? Uh so it gives me sort of like um uh you know like a narrative on you know what is uh what it's looking at. So it doesn't just dump me the numbers. It does some sort of validation checks around um uh whether this triangulates with other data points um in the data set itself and then it gives me the sources of where it's doing where it's getting stuff and it's uh uh it's giving me sort of like hey what should we be doing next um to either make the analysis more robust or explore more more uh more angles. So the next thing I'm going to do is you know like this looks fine. Um, what skills did you use to do this analysis?

39:12 So, I want to I'm curious like what exactly did you do to uh what what did you use in this repo that helped you kind of like do all of these uh instead of you know like it could be multiple many different ways of uh accomplishing the same goal. So it's uh it says um uh didn't invoke any of the skill but I apply principles from se several skills. Um so it's got a question router, it's got a visualization pattern skill, we have a triangulation skills um and uh and and a whole bunch of other uh stuff.

39:48 it didn't invoke any of the sort of like analysis design because it's a pretty simple uh question uh question framing guard rails and things like that. So um so you can always ask claw to be like you know hey what what did you do what uh what what helped you get to kind of like some of the uh some of the the the the final outputs um what was helpful. Okay, cool. So, I'm going to show you this one here, right here, which is opportunity sizing. So, on the left here on my agents, I've got the where is it?

40:28 Um, I have this agent called opportunity sizer. So this one basically if I click into it it's like uh hey if I invoke this skill it's going to quantify the business value of an opportunity um or or do a sensitivity analysis and identify which assumptions matter the most and then it's got like instructions for how Claude should be doing kind of like an opportunity sizing exercise. And so we're going to we're going to use this to see what that looks like. for this for this question. So um you know we looked at conversion rate by device and right now I'm interested in hey if we fix or if we improve upon the mobile checkout conversion rate uh what would that look like in terms of you know like bottom line and impact. So um right here at the beginning I am doing the uh you know like I'm invoking I'm calling the agent and then I give I'm giving it sort of like the parameter like hey here's the question that I'm interested in uh getting at. Can you can you state your assumptions and what's believable?

41:41 So we'll let it we'll let it do its thing and let's see uh do we have any notable questions here? So u one question from Greg was uh did you specify the conversion metric somewhere uh the final checkout for uh from when item placed in the basket or you know when user first hits the site etc. >> Yeah. So uh when you first load the data set to the repo or to the AI analyst um it will flag sort of like uh hey based on what I understand um these are the metrics that matter um based on the column names based on kind of like the the the context of the data set itself and then for anything that you're like hey no that's not how you would do it then you give it more context. So in this case when it was first connected to the data set itself um it figured out uh correctly.

42:41 Okay. So let's see. Okay. So it's it's it's going to do some data pulling for the uh for the sizing model. Um you know like we talked about the three components in the impact model. Um what it's doing behind the scene is exactly trying to do that. like hey can I get concrete in some of these components in that in that model such that when I give you the sensitivity analysis or the opportunity size I can explain it myself as in like claude not me uh let's see cool um so I This uh opus 4.7 is uh is smarter but it also thinks harder so it takes a little longer than 4.6.

43:38 Has anyone tried Opus 4.7 yet? >> There's a chat going on with uh I think Anit uh has been talking about he's seen a difference in the uh the information retrieval across like browsers. So, I was just telling him that we'll have we uh do some we need to do some uh deep research work over the weekend to get our course up uh and working for 4.7. So, we'll know soon. Uh but yeah.

44:09 >> Oh, yeah. um forgot to mention that um for the fiveweek course um that we're about to start next week, we're actually going to re-record our week three to week five content because now Opus 4.7 came out. So we actually just finished recording and so we have to like redo that. Um so any any piece of analysis or demos or like exercises that we have students do in those uh in those weeks um when if you're interested in joining us when you see it it's going to all reflect 4.7. It's a crap ton of work that's for sure. Um but I think sort of like uh you know like we can't even get back to Opus 4.6 as you saw. Um, so it would be disingenuous to be like, you know, hey, here's the here's the thing.

44:57 And then like, oh, oh, 4.7, we can't really pinpoint the uh uh the behavior. Um, okay. So, let's see. So, it's done its opportunity sizing. So, basically, um, actually there's a report, so we can actually look at it. Um the the main idea is that um uh it it's got like a base case that it identify uh 15% relative lift on mobile conversion and it says um let's see base case bottom line and like if you calculate those numbers what it flows through it also gives you a range of uh hey if the pessimistic um like assumption is 5% then this means this much revenue, this this much gross profit based on the data set. If it's base, then it's this. Uh if it's optimistic, uh it's this. So, and then what it could look like in terms of what effort, what level of effort we're talking about, uh in terms of uh what's believable here. So, it's also got some assumptions around um the cost of doing this uh and so on and so forth. So it's also outputed a um uh a report for us to read. So we can actually check that out.

46:15 Uh sizing conversion. So it's this one. Uh it's a markdown file. So it's very friendly to to uh to AI. But then for humans I can just do a preview and it will look much better much easier for us to to look at. So annual impact. So again like that's the bottom line that it really calculates kind of like the one pitch thing. Um uh let's see so topline revenue impact of almost 300k uh 13 gross profit. Uh if we assume some sort of engineering cost confidence is medium uh it's data backed from the starts. So the very beginning how many people it touches and then the average order value and then the margins are all data backed uh the lift magnitude is not as confident in. So uh gives you the full kind of like uh math around uh what these are in terms of the in terms of um in terms of actual concrete number based off of the analysis. So it it's not like kind of like making things up. It's actually going into the data doing the analysis like we saw and then uh putting it up here and where the lift assumptions come from. Uh so it would tell you kind of like hey here's how I thought about it.

47:40 Here's where the assumption would be. Uh things like that. And then it'll also give sensitivity analysis. Um and uh and then like you know it's uh right now it doesn't have a lot of context around the business the data sets or anything like that. So um this is almost like a best guess based off of what it's uh reading from the data. U break even analysis it's got scenario analysis uh base case believable what's believable what versus not. So as a human with judgment you're supposed to be the one that really be critical about some of these assumptions and uh and what it's uh what it's outputting. So, um it gives you a base almost like a basket of things for you to kind of like think about. Um but it really stops at that. Like you shouldn't just take it and be like, "Oh yeah, that's that's all cool." Like I'm just going to oneshot it and and use that as the uh end all be all. So final prompt for this exercise.

48:39 I know we're running out of time, which is show me a sensitivity table of conversion rate lift versus average order value. So if we kind of like vary the assumptions um in uh in conversion rate lift like what we can achieve um but also like average order value because an incremental order may not be worth as much um over time if we sort of like intervene like this.

49:10 Uh let's see. Cool. So, it would have kind of like a uh it would have like a kind of like a 2 by 2 here. Um, actually, can you visualize this for me? I'm a visual person, so I can't really read these numbers myself too much. Uh, so I'm just going to ask it to to do it for me. Um, and the only thing I would say is 4.7 is pretty slow.

49:48 [laughter] >> Probably everyone's hitting it right now. I've seen that happen. Uh, yesterday Claude was uh also down, right? Yesterday morning. >> Yeah, yesterday was down. >> Yesterday, day before, I think. Yeah. >> Uh, okay. Cool. Um, we can take questions. I need to leave at 1. So I do need to to have a hard stop so we can answer questions as as this thing is uh training itself. Anything that >> So one thing that Jackie Becker wanted to uh ask sorry Jackie if I'm pronouncing it your name wrong. Uh she had a question I think they had a question around are you also connecting snowflake or looker to help with giving it more context or any other recommendations. I told her that the this was around when you were giving clawed web UI. I said it's mostly web UI but we when we do cloud code then we do connect is what I gave. So she asked like a deeper they asked like a deeper question like unders so this example didn't have the connection set set up. I would love to understand what BI have you seen best results for pulling opportunity sizes together. Uh yeah and they wanted to catch up on the recording later.

50:58 >> Cool. Um yeah know that's a great question and in the boot camp or and in the course we also touch upon this pretty in depth. So context is context domain knowledge uh is everything right like claw doesn't out of the box understand any of your company stuff or your what the strategy is what the uh you know like what the uh what the road map should be or what previous you know analyses or numbers were were like. So one of the most important thing is to be able to connect it to context. So you know like and uh by context we go beyond just the warehouse like we don't we don't just think you know data warehouse think notion pages Google docs um uh emails slacks uh everything everything with an MCP is know is is uh is context that you can actually bring in and um it's actually pretty straightforward to bring that in um as long as your organization approves for example like you AI security legal use cases and things like that. Hopefully that hopefully that makes sense. But like the more that you pull in from as many as diverse sources even if they're not like numbers or like data related uh sources, it's going to be gamecher for your analyst to actually understand um and give you much better answers.

52:25 one uh question from uh Ashwini before we go to Otto. Uh is this course relevant for a product manager who is not a data science major? >> Yes. Yes. Uh actually we built it for non or I mean like uh we built it with an audience for like the title for our course is called for builders. So anybody who builds would be relevant. Um the idea is like we want to empower people whether you're data scientist or product manager or designer to be analytically independent. So meaning like you have the foundations, you have the judgment from an analytics perspective to make decisions and then we also teach you to kind of like leave delegate the execution to AI so that you can ask better questions, have AI do the thing and then you make better decisions off of it. Does that make sense?

53:18 >> Yeah. One thing I'd like to add to what Hi said is uh anyone who works with data is going to benefit from it. People who work with data would probably use these skills that they generated. So because all of this is going to be around AI and cloud code. So data scientists could come and understand how they could use cloud code to to replace their workflows and get an understanding of these frameworks and get the repo that they could take and use in their work and non-data scientists like anyone uh who is not a tech data scientist any data analyst non techch or product managers they could understand what does a tech data scientist workflows look like and how do you use AI to do those workflows yourself uh and understand more so yeah Otto Yeah. Hey there. Thanks for all of the amazing content. I'm going to try to keep it short. I got a kind of a two-parter. I noticed you have a ton of agents and markdown files. When it comes to actually developing those, is the approach more so hybrid of like AI produced with uh human input? And then the second piece is you know with this example that was being shown is it fair to say that this is very bespoke to the actual data set of like Novamart itself uh which would be more realistic for like a business setting your AI is continuously learning on your data producing the outputs. Um so hybrid inputs on markdown files and is it more so bespoke to the actual data set where you can't really funnel it to other data sets. Yeah, let me answer the second one and Shabby feel free to jump in if you have thoughts around that. Um so your AI analyst is going to be bespoke to your thing like if you're you know like think of it as you clone it once and then you start giving a context about your company and then you develop it like that almost like grow your data analyst uh alongside you with the context of your company and stuff and then I will have no clue what how you've grown your your thing but we teach you how to grow it like basically like how to how to grow it how to set it up how to uh you add more skills, add more agents that would be relevant for your business. Uh, and so on and so forth. And potentially this could go beyond just like, you know, right now it's like product data scientists, but you can imagine the framework is basically transferable to financial analyst, marketing analyst, uh, whatever analyst out in the world as long as you have the expertise to be able to develop alongside with it and learn or help it learn. So, and relatedly to your first question, when we developed this, it is a um kind of like a uh hey, we know what the best practices are for analytics workflows because we've been doing it for so long.

55:55 Um claude itself off the bat also knows some, you know, like how to do data analysis, for example. It's just not in the same way that we, you know, like that someone has, you know, decades of experience would do. So, we basically give it all the best practices. we check kind of like you know where it doesn't make sense uh where needs to be enhanced and stuff like that to make the version that we have today.

56:21 >> Awesome. >> But you can absolutely you can absolutely build a very similar system with your own expertise. So it's it's it's all possible. So I think that's kind of like the the main uh the main thing. >> Yeah. I just imagine with new questions, you know, you have the opportunity sizing, you could advance that to like Monte Carlo simulators and just keep going and going and going. Um, and it's all built around the the the root of the data set. So, awesome. Thank you so much.

56:50 >> I think we could take one last question. The uh Ashwini is asking live inerson lessons for for the course like how how does how how does that >> Yeah. Yeah. Great question. So, um this is a hybrid. Um so, the we have the lessons recorded so they're async. so you can watch them at your own pace. Um, and then every week we have a kickoff at the beginning of the week and then two office hours dotted throughout for people to come in, ask questions or share learnings or um or you know, just pow out or like just talk to us. Like we're we're pretty friendly too. So, you know, you we can we can we can definitely talk. Uh, and so um that's kind of like the setup. Um, but the lessons themselves are pre-recorded and you can always kind of like go over go through them at your own uh you know like however many times you want. Um, and so that's that's the hybrid component of uh of the course.

57:46 And then what is the location? Location is online. So it would be virtual. Cool. Um, I need to drop because I have a meeting with our CM chief marketing officer. So uh Stravia no I I need to drop off too. >> Okay never mind >> meetings as well. So we try to sneak these workshops in guys to give you as much value as we can while we obviously have our day job. So really appreciate you all joining. Uh we'll share all these resources with you all. Yeah.

58:20 >> Thank you everybody and uh yeah we'll we'll send a follow-up email with the recording and uh and the resources and stuff. Thanks all. >> Bye guys. Thank you very much.