Design True North Metrics That Drive Real Decisions with AI — Data Neighbor Live

0:00 Uh welcome again everyone. Thank you for taking the time this morning or this afternoon, this evening, wherever you are. Uh welcome to this blank lightning lesson around designing metrics that don't lie. Um uh honestly, this is a little bit of a clickbaity title, but what we're going to talk about is exactly what it's saying. uh we're going to go into a framework around defining something called a true north metric and then also the inputs that you can you can decompose into to help move that metric. So uh before we begin my name is hi uh I'm the head of data at a legal tech company called Entra and uh I previously worked at companies like LinkedIn, Pinterest, Next Door, Meta. Uh I'm here joined by my uh uh we call them co-hosts because we also co-host the podcast. Uh Sean Butler and Travia Mapali. Um would you guys like to introduce yourself a little bit? Hey everyone, I'm Sean, principal data scientist uh the same company I works at. Been in data science for about 10 years.

1:04 Hello everyone. I'm Stravia. Uh I'm a senior manager in data science at uh superhuman previously called Grammarly. I worked uh hi Sean and I knew each other from next door and I started my data science career at Microsoft. I was there for six years and yeah been here for a while like more than a decade now for sure. Yeah. >> Cool. Awesome. And um yeah so uh we also run a couple courses and we'll we'll have more information later uh in the slides. um one on AI analytics for builders uh effectively helping everybody regardless of your profession to be analytically independent and also leverage AI to actually execute on analysis on any analytics task to help you make better decisions. So that's a that's a course that we run um and separately Sean uh himself is also running an AI evals course for product development. So um you know in the age of AI development uh the way to evaluate if your product is working or not is very different. So that's uh that's that's sort of like a dedicated deep dive uh around that. Um we have a lot of free lessons just like this one coming up as well. Uh and all the course information you can find on data neighbor.com.

2:20 Uh all right cool. So let's get get into it. So what are we talking about today? Um we are talking about obviously metrics. Uh who here have seen this problem here? You've got a lot of metrics at your company and you don't know what to do with them or you don't know what they are. >> Yes. Type one. >> How many metrics do you look at per day in the chat or you're supposed to look at per day?

2:48 You ever ask someone to like give you their metric and they give you like seven in return? Yeah. No. Which like which one? I guess only Sergio faces this problem. Cool. And that's cool too. Um this is a very common problem. Um honestly where uh we have um uh dashboards uh especially if you work in for example in tech um uh there is a lot of dashboards around there is uh even in Google Analytics uh has you know is a dashboard in itself and if you go to the engagement tab there's a bunch there's a billion metrics right Sarah uh if you if you remember those uh different tabs where at least for me when I looked at it I'm like what am I looking at here what what what do they mean kind of stuff. So uh very common problem that uh a lot of teams measure either the wrong things or are measuring a lot of things.

3:44 Um and uh uh and so you know like some would go up, some would go down but really none of them focus on hey this is what you should be looking at so that you can measure whether our product our services is actually delivering value for our customers or our users whatever your business may be. Um, this is very common as I said, uh, but very fixable. So, we're gonna we're going to talk through how to how to think about it.

4:15 All right. Actually, yes. Okay. Cool. So, um I'm just going to highlight a very simple example here where um metrics can also mislead uh which is actually worse than if you have a lot of metrics and you don't know what to do with it. Um so uh the scenario here is we measure something called daily active users. Um are people familiar what that what that is? Have you heard of it before? So roughly something about how much how how many times or you know like if a person has shown up to your to your product or business. Um and uh we count that as kind of like a DAU. So it's a very very um uh very common metric for a lot of companies like uh uh social media in the social media space um and just generally. So let's say um you know like the scenario here is uh uh we saw daily active users went up 30% and everybody celebrates right. So so it's like oh wow great up and to the right and then down the line we're like oh wait hang on turns out a lot of it is bot traffic. So like these are not humans. Um I guess in today's world it's a little less uh less uh less like an anomaly but um previously when only humans browse browse uh browse the internet uh like bot traffic is real and that is not something that we care about. And so um you know like probably the team could be making decisions off of the DAU trend for the last uh however many months and then uh let's say they saw okay yeah we will launch this feature and then uh it drove a lot more visitations uh so we keep building and doubling down on it and then 3 months later they're like oh wow this is actually not true uh not real traffic so they wasted three months building on a false signal. So that is actually very common uh if the metric defined up front that guides the road map or the plan is not actually pointing towards the right thing. So better metric judgment would have saved them.

6:22 Um again this is a very sort of like a very simple highle example uh but this happens all the time and uh more more times than uh companies and teams are willing to admit. Cool. Okay. So today's framework. So what's our solution? Um solution is to equip you with a framework that you can take home to uh to follow every single time when you have to define metrics. So uh this is what's called the true north plus inputs framework. This is used by a lot of companies um in the Silicon Valley. So Airbnb, Slack, Netflix um and uh many many others have been modeled off of the exact same framework. So sometimes you might have come across um a term called Northstar metric. It's synonymous with true north metric. So um either way it's fine. Um basically step one is to define what's called true north metric. So one metric that captures the core customer value of the products or services that you're offering. So uh like this is basically think hard to the end state of what you're building. What would it mean when you have delivered value by the end of it and is there some sort of way to to measure that that is sort of like the true north. So this this actually takes a lot of time a lot of debate and uh you know sometimes it's obvious sometimes it it's really not. So uh focusing on this is going to lead down to sort of like number two and number three here. Number two is once you have a true north metric defined then you you can decompose it into what's called input metrics. So actionable levers that teams can actually uh pull to influence the ultimate true north metric. So we'll get into some examples um uh uh just in in in just a little bit. And then how do you actually then define those input metrics? You can actually do this decomposition which is number three. Um it's called the BDEF decomposition.

8:37 We'll uh we'll we'll we'll explain what that what that stands for. uh but almost all inputs you can imagine would be very similar have similar flavors that falls into one of these four buckets. So that's going to be a checklist that you can uh that that we'll go over in just a little bit. So the key insights from this from this uh from this slide from this entire framework honestly is that the true north is the outcome that you measure and inputs are the levers that you can pull to actually influence that outcome. Does that make sense to folks?

9:15 Yes. No. One, two. One for yes. Two for no. Three, four. I did not pay attention. And that's okay. >> Cool. Okay. >> Lots of ones. >> How north would be the true north metric? Um, as north as possible. So, meaning as close to value as possible. That's the way to think about it. And if it's not possible to measure that, go one step south. If you can't measure that, go one step south. Kind of like doing it, think of it like that. I'm just painting it like uh conceptually here.

9:52 >> Yeah. The way I kind of think about it um is kind like I think about like what's the the person as a human trying to do in their actual life irrespective of the product. Like imagine your product doesn't exist. They're trying to get something and they're trying to like get entertained maybe if it's social media they're trying to get some workflow some job done some task like define that and then like how said is like can you translate that to a behavior that's represented like some like signal for that in your product and then a lot of things you can't and that's when you keep go and then if you can't you can build it you can try and build that signal into your product you can work with your engineering teams try and build build that way to capture that otherwise keep going South.

10:40 Yep. Good question, though. Uh, all right. Um, true north is a metric that captures core personal value specifically multiple products that a company might have. Uh, each product to have its own northstar metric. Then you could have like a company Northstar metric, too. But sometimes it's harder with the company stuff because that's a lot more lagging. So, it's a trade-off of like you want you want a Northstar metric to be something you can like kind of get a vibe on in like weeks, not like months or quarters. So, when you start going to like a holistic kind of company value, um sometimes you don't get that signal till later on.

11:19 >> One other thing I'd like to add along with what Hi and Sean just added is uh generally core customer value will also be linked to what business uh gets value out of. For example, Meta's daily active users means more activity that says users are, you know, benefiting from the product, which is amazing, which is the cro customer value here. But it also implies more ad revenue from these people coming in and more revenue and, you know, meta's uh like actual business value going up as well. So mostly the uh there's a big correlation between true northstar metric where and the uh like you know business uh like what the business gets out of getting the notar metric up as well.

12:04 >> Cool. Um let's uh let's move to the next one here. So okay, bad versus good true north metrics. And uh here are just some examples of it. Um, again I'm using an AI tool if you guys are aware of it. Gamma I don't know how to do animation or else it would have been a lot more suspense. Uh, you can imagine but like now it's like all wide open. Um, so uh on the left we have the bad ones and on the right we have the good ones. The bad ones could actually be a lot of these things that most people think are good metrics. So, for example, daily active users, revenue, and signups.

12:48 Like, these sound pretty reasonable, right? Like most people report uh on one flavor or another of something like that, but they're actually not good true north. And we'll get into why that is the case. Uh the really good good true north metrics are things around um for example, for Airbnb, it's uh how many nights are booked. Uh, for Slack it's how many messages were sent and then for Netflix it's how how many hours were watched probably like I'm going to spell out the exact difference between these two but like do you see any difference between these two columns here just like you know just feel just by feel like they're different right one is a lot more specific >> the other one is much less uh yeah business versus behavior. That's right. One is customer oriented, the other is company oriented. Yep.

13:49 Uh cool. Uh share example of B2B. Yeah, we'll get into that. Cool. Uh so let's see. So you know why are the good ones good ones? Um they typically share some of these traits here. Uh so let's do a counter example which which which we've been talking about for a little while here. Uh daily active users tells you daily active users is not a good one. Why is that? Because DA use tells you people showed up. So like they they they visited something like they could visited a website. They might have visited um like an application or whatever it is doesn't say anything about whether they actually got value or not from that product and services. So, it's just, hey, they showed up and we count them. So, there really is nothing that says uh like unless your unless your core business is for people to just show up like just show up regardless. Don't do anything.

14:50 That's fine. It's it actually doesn't tell you anything uh in in terms of value. That's probably the the shallowest thing that you can measure. Now, why is nights booked a good one? If you think a little bit more about it, it actually means a traveler found a place to stay and the host got paid. Like you have two constituents as a as a as a company for Airbnb to to to sort of like provide the services for and this one metric captures the value from both sides.

15:22 Not all metrics is can be sort of like uh uh can can be like that. But this is almost like a perfect example of uh of uh of just something that that touches upon uh kind of like both sides of the equation if it's a micro marketplace sort of product. And then just generally good true north metrics are leading indicators of revenue. Typically could be revenue could be uh other sort of like company metric that uh um uh that uh uh that someone on the chat observed uh before.

15:55 And it's also a direct signal of customer value. So they got value out of it. And then if you do this really really well and you move this metric up and to the right or you know like down and to the right if that's your if if if that's what the metric form factor is then we have really good faith that revenue and business outcomes will materialize. Does that make sense for people?

16:26 Hey, hi. I had a question in the chat from Nissan around examples of true north for large B2B enterprise. I don't know if you got this one already. >> Uh large enter large uh B2B enterprise >> rather than consumer tech. So, so I mean we work for a B2B and so like I can just give you example from our our work. So I just typed this out to someone else. So like our product is a legal tech product. It has it has many products within it. One of the products I work on is called market builder. Uh it basically helps lawyers gives them suggestions and automates a bunch of the edits to do during the contract negotiation process. So edits on a document is what you can think of it as.

17:08 And we have Northstar metrics around basically the time we save them from the start and finish of like a contract comes in and then a contract goes out. But we also have a bunch of guardrails on like the level of quality of those. So it's like a metric that kind of like combines both of those together and that gets a signal of like hey if we're we're maintaining quality of edits and people are like taking these and actually like a lawyer reviewing them and not changing much more after it and it's reducing their time like those are pretty good signals that that B2B product is working. So, I think B2B SAS is actually pretty good because it can it relates a lot more to like people's actual like productivity and workflow. Sometimes the consumer tech stuff's hard because it's like engagement entertainment base. Like I don't know, social media northstars are hard because it's like this person's uh spending a lot of time on TikTok.

18:07 Like that's they must be having a good time. But it's like I don't know. That's that's really healthy for them. But uh I don't know. Does that answer your question? Yeah. And we'll have an example actually for we all to kind of work through uh in just a little bit. It's a B2B. Uh cool. It's like uh and then Catana it's it's like so it's I think product adoption is a good signal like someone's using the product adopting it like we can infer they're getting value out of it but if you can that's like a little south of the northstar metric like if you can find something where it's like hey what are they doing in the real world and are we impacting that like for my example with the lawyers of like can we get the time stamp of when we know they like received this document from a from one of their customers, from one of their clients, and then they sent it back, that's like more of a signal like they're they're adopting it, and then that adoption is actually causing them to have like a uh more efficient or higher quality or a lower cost uh workflow.

19:22 >> Yeah. And uh once we get to the input metrics side uh you're you're going to see the difference between between these two and you'll you'll understand uh for example like adoption metrics where does it actually fall? Good question though. All right cool. So let's uh let's move right along. Uh so um this whole true north metric framework uh like you know it's it's an iteration. So it's not like a uh uh hey once we once we define something we have to stick with it forever. That's just not how it works.

19:54 Um metrics is always iterative. So the business changes uh the information about the business could change and uh we would know learn we would probably understand more of customer behavior um uh as as we sort of like you know operate a business for longer or operate a product for longer. So, uh, even Netflix, I mean, says got it wrong first, but it really is like an iteration. Um, they started with something like subscribers added, uh, and then later shifted to hours watched as we saw in the example slides. Why was subscribers added not useful? Because it didn't really tell them what to do. Um, what about the subscribers? Did you add, you know, quote unquote kind of like, uh, very very lowly engaged subscribers?

20:41 Do they actually watch anything? Are people actually uh you know getting getting value which probably is entertained um by your shows and stuff like that? Like it's just not clear. It's what we call a vanity metric. Uh measure something for the sake of measuring stuff. Uh still still useful in in in the grand scheme of things but like not for sort of like understanding value that you're delivering as a product or services. So once shifted to hours watched reflects values delivered.

21:11 So um you know like again like did are are the people entertained uh and when it moves they know that uh their content strategy is working or not working and they have a way to understand if that's uh if that's because of uh of the things that they they they put out. Uh okay cool. So the way to evaluate your um true north uh is uh it's called seven test checklist and we'll send all these out afterwards. Uh so um don't uh don't worry about uh taking notes and stuff. I know seven is a lot but generally you have two criterias. One is practical, one is strategic. So on the strategic side we've talked quite a bit about uh some of these uh uh some of these angles reflects customer value. So not just activity like not just did they do something uh unless that doing something actually leads to value or is actually the value in itself uh it leads to hopefully business outcome I wouldn't say like revenue but like most likely a business is exists to you know like to make revenue to make money so eventually leads to that business outcome uh it is hard to gain so you know like you can imagine if I define my true morph be how many clicks on a button in uh on on a website uh and that's my true north then that's very easy to game right like I can make it neon I can make it flash I can make it bigger like then people would click more um so uh something that is hard to game uh uh one number you know like this is this is kind of you know more creative than uh than it is uh uh technical uh a number that the whole company or the organization or the team can rally around so like people you know like people feel like uh they're invested in uh practical criteria is that it's actionable so you can actually it can actually inform what you what you can do uh uh uh later down the line uh it is understandable so time and again um uh I've come across metrics that are defined that are very convoluted so by the end of explaining what it means uh you know like most people have already checked out so that's also not good uh measurable so you can actually measure it so if the data doesn't exist or if it's just no there's no way to put it in a numbers format. That's probably not practical.

23:36 Does that make sense here? Cool. Right. Moving right along. Okay. So, we have true northmetric and everyone here is an expert at defining that now uh or at least knowing the uh the the high level framework of it. Uh let's talk inputs. So um what's input metrics? Uh so teams don't really I mean teams do own true north metrics but then you would have sort of like actual sub teams that own the different inputs. So the levers for moving that true north metric. a very simple example here. Uh and sorry used another consumer example here but uh we can get into we'll get into that the B2B um example in in an exercise in just a little bit. So imagine you are um actually I don't know if Shravia that's what your company does. uh let's call it like Shavei's company has some has a has a product called uh Grammarly, right? Like helping people to write better, be grammatically correct, use AI to refine writing. Uh uh I'm just going to make it up. Uh perhaps her company wants to get as many users to try out the product as possible so that the uh uh there's a growth team that Shravia is on that basically is responsible for a true north metric called retained signups. So people who don't just sign up, give you an address, email address, and you know, never use the product, but like they actually stuck around for some time after they've signed up for uh the you know, like the the the Grammarly uh uh widget. Uh and so that that's a reasonable true north.

25:21 Now, if you imagine sort of like if we just express that in a very simple equation, what what does what does that actually mean, right? like you can actually decompose it into the different uh percentages and rates and stuff to arrive at the exact number here which is how many tra how much traffic can you attract to for example Grammarly uh and then what's the conversion rate because you have to like go through uh a signup flow to to do that right so you have to first land on Grammarly's website somehow uh and then uh what is the conversion rate on signups from those traffic and then of the ones that signed up what is their retention rate, however we define that. Um, first seven days, first 30 days, doesn't matter.

26:05 This is basically what it is to that that that helps you get to a retained signups sort of um uh sort of final outcome here. Now, if I were tasked, if I were the head of growth, the growth team, and I'm like, okay, move retain signups, I would have no clue how to do that. Like that's like what what levers can you actually pull? But when you are able to decompose it a little bit more into you know one team responsible maybe the growth marketing team responsible for driving traffic to the website and then you have another team that's responsible for making the signup flow really really efficient or really really pretty then then you're you're optimizing for the middle and then you have maybe an onboarding team that makes sure for every single new person they have a really good experience when they onboard to this product. then they in impact the retention rates. You see how these different teams can work together to actually just move their respective inputs and then the true north would actually move along the way.

27:10 That's kind of like roughly the structure that we're talking about here where the true north is the outcome which sometimes is a lagging indicator. The inputs are the levers that you can actually dial up or down by the effort that you put in. And then each team knows exactly what they're influencing. So like the the growth marketing team knows that they need to drive traffic. The onboarding team knows that when somebody signs up, I need to give them a really great experience and that's all they focus on. So when all these things sort of like you know or even not not not all of them like even one team is killing it and the other teams do nothing, your true north still moves.

27:53 The thing with and the thing with input metrics why we don't just have input metrics why we need because like right like you could think like oh traffic sign up conversion retention rate those are my my stories um input metrics these ones are pretty straightforward but like so you're like building a product sometimes they can be like highly gameable so I mean even something like yeah traffic I don't know if you're just like getting traffic like by like bots like hi said or by like paid traffic or by like email blast. So it's like people are coming in once but then they're not retaining then it's working against all of those other input metrics and so your northstar metric won't actually go up.

28:36 So the Northstar metric is like a really nice way to validate of like hey is this actually a strong input metric or is the input metric too gameable? Yep. Good point. Cool. Um, again, not I mean like most metrics don't decompose so nicely. Uh, like like I I'll be honest here. It's very rare that you get something that decomposes into like an exact equation where you just dial up dial down and like you know exactly what the the true north is going to do. A lot of it is uh sometimes you have to have faith, sometimes you have to have proxies and that's all okay. The idea really is like we understand intentional from a like intentionally that uh that the the the metrics that we choose we have faith that it actually could move true north. It passes the smell test. It passes the uh sort of like the the the gut check.

29:35 >> Hey, I had a couple of good questions in the chat. Um one maybe get through this this one first and then >> from Nitton again. So all these metrics have like cost to maintain and also cost of resources to like argue about them and have meetings around. Do you have any rule of thumb for like the the max number of metrics a team should be? >> I guess it probably depends on the the size and stuff like that, but it's pretty it's a pretty good valid question.

30:01 >> Yeah, it is. Uh no exact rule of thumb. Uh but you can imagine a hundred is too too many. One is probably not enough. So, um I'm not going to I'm not going to say like it's somewhere in the middle. It is somewhere in the middle. Uh the the point that you've that you've raised is actually valid one. How much attention can you hold while you can still not go crazy with these numbers? >> What the kids do? 67.

30:28 >> Is that thing is that appropriate? Um that's kind of an appropriate uh use of this. >> Six or seven. Six or seven. That that seems that seems fine. One thing, go ahead. >> I was just say like I think something that that I do like kind of like what I said before like take your input metrics, see which ones have the highest impact on your northstar metric and also have like um like you get the highest leverage to move. So basically most ROI and then you can kind of like make a list and and some of that you can use data to do. Sometimes you can talk to experts, sometimes it's just intuition. Um, and then you can like prioritize them and then out of maybe instead of capping the metrics, cap your meetings or whatever where it's like, hey, we're going to go through this stuff like we're going to get through these really important ones first or we're going to have our team assigned to moving these really important ones first. Um, kind of it's more of like a resource problem and just like prioritization at that level.

31:28 Um, another thing that I'd like to add to Sean is that once you get into war rooms of why your Northstar metric dropped, you'll know what are the main reasons why your Northstar metric dropped. You go to the metric tree and see what what effects your notar metric moves. The more and more we get into those modes to understand why your noticed moved, you would understand what are the top metrics that influence it. One other thing I wanted to uh add uh on the questions I think JY I don't see their name uh their question was about having weekly like seven day like how do we have it time bounded the metric 7 days 21 days like you know and they were also asking questions about Q1 like in different quarters like how do you do that so I had a short answer there um not sure if it was uh helpful but wanted to give like a live understanding of what I meant uh based on the type of product. If you are a daily product, let's say, you expect a user to come to see and you know look at the feed on a daily basis like um Instagram for example, I would say that's a daily metric. If you have a weekly product where you see like you know work week almost like you have users you think like a project management platform or you know something like that that would be a 7 days and if you think it's a monthly like tax tooling or something I think Pinterest you used to do m um monthlies as well uh where you expect like it's event based or you know something that's related where people come in once a month and that's good enough then that's uh a monthly product for you uh And there's also like a question about AB tests. Like it's a very good question.

33:06 It has tons of things uh in line. So we'll probably do a bigger deep dive in our course later. But uh to answer in short, AB tests are time bounded by the test itself. So you would have metrics for the test. Uh basically based on how long the test is, you have metrics to measure the metrics within that AB test. Yeah. >> Cool. And then for uh like a dental practice, it might be a six-month thing because you know like that's uh or quarterly probably. Um so that's very different. All depends on the context of your of your business and product that you offer.

33:45 Okay, cool. So moving right along. Love all the questions by the way. That's uh you guys are awesome. All right, cool. Uh did we go through this slide? No. Um, okay. So, how do you actually uh choose inputs? Like how do you how where do how do you find your input metrics? Um, a this this is sort of like the what what what I was alluding to uh a little earlier. The BDEF framework um probably a better way better than this, but uh generally you can uh you can you can imagine most of the input metrics that you can think of would probably feed into one of these categories. So the first one is breath. So how many people like how broad are we talking about? Um and again using Airbnb as an example here. It could be active listings or it could not be number of guests. So like how broad generally are we measuring something. So that could be a flavor of an input. Um depth is the second one which is how much per user or per customer kind of kind of metric. So in the air in the case of Airbnb it's average booking value or how many average number of nights per booking like things like that. Um so it gets into like uh uh just like you know how deep you go in terms of uh uh providing the uh uh the uh the the the inputs and then the third is an efficiency metric.

35:14 So um you know like how I guess how smoothly or how much time is saved things like that. um or you know like what what is conversion rate? So this would be you know Airbnb's case when you search for for for a for a place to stay what how how likely are you to actually book it once it's uh once it's surfaced to you. So efficiency is u is on that. Uh and then number four is just frequency. So, uh how many times um has this been done? Uh and uh uh and and just how broad um uh is the sort of like repeat behavior is. Uh so these are sort of like these almost accomp uh these almost encompass kind of like all all the uh all the inputs that you can that you can imagine. And so it's it's one it's one way to sort of like you know uh uh put put it in a uh a a framework around uh uh okay my input belongs belongs to uh to this bucket here.

36:21 Okay cool uh let's see okay so we actually get to this uh B2B exercise here. So diagnose this metric. So a product manager at a team collaboration tool. So think Asana, Monday.com or Trello. Uh so hopefully you're all at least familiar with some of these uh some of these products or maybe like does anyone have no idea what these are? Type of two if that's if that's the case.

36:54 Okay. So everyone knows. Great. So product management uh sorry project management uh uh software or tool uh and uh this uh product manager uh comes and says hey our metric is uh monthly active users and we want to grow it. What is wrong with this uh with this metric? Real quick, type in chat. Frequency is off. Vanity.

37:26 Nice. No business value. Nit showing up is not useful for a collaboration tool. doesn't reflect value. Yep. Cool. Great. Awesome. So, yes, exactly. Uh, so problem with this, why it fails? Everybody got it. Doesn't capture value. People can show up and do nothing. And as a collaboration tool, what does that actually mean? Like, it means nothing.

37:59 They didn't get any value out of it. Uh, it's vanity. So, uh, yeah, like I think Sarah said it, uh, oh no, it's your guess. Uh, no one knows what to do with this metric. It just it it just is um you can't really move the needle on MAU here. Uh so you know like not not in face value at least. So yes, you all got it. This is a bad metric. All right. So what's an improved version of it? Um so an example improved version of uh this uh uh collaboration tool could be that the true north could be team projects completed per week right like without even going into the detail of what that even means like how do you measure team projects what is a completion and you know like per week is it seven days is is it is it like the Monday to Saturday or whatever it is uh you can feel that this is different right Because this this gets more specific and uh existed. If if your if your idea is to be a tool, you actually help people complete something, right?

39:09 Um and then the inputs to this metric could be any of these uh following and or and or could be more. So the breath on the breath side could be active teams with a shared project on the dub side could be tasks completed per project. So if you imagine collaboration tools, you probably have a lot of check boxes on uh things assigned out to different people and uh you know like checking checking them done is actually a good thing. Uh could be efficiency. So measuring the time from project creation to first task completion that itself is pretty valid like how many people are actually taking the project seriously or taking the tool seriously using it to track. Uh could be frequency. So weekly active collaborators on uh uh on the project itself like any of these could be really really good inputs to this uh much more refined true north uh versus sort of like just monthly active users.

40:03 Cool. And uh you know you can imagine uh if these were sort of like defined like teams are going to be able to know um or people teams or uh are going to be able to know like you know what what their role is in uh in in moving this True North. Okay. Oh, okay. Quick. Uh best true north for a food delivery app. Think Door Dash or Grab or Us something like that. One, two, three, four.

40:39 Give everyone 15 seconds here. Think what's the value? A lot of ones, twos. Yeah, this is a hard one. Three. Average order value. Two.

41:11 What does it mean by a success and failure in one? Great question. I don't know. Uh so I mean we can we can define it however we want. So assume it's a order actually delivered to the client or the customer and then how many orders were ordered or or placed. Okay, cool. Great. Uh so a lot of mostly centered on one and twos. Drum roll. Let me find a button here. Okay, answer is number two. Orders delivered per month. Uh why is that?

41:46 So if you remember on true north metric a few slides ago what we're after is driving the outcome or deliver the customer value that we want. So for a food delivery app you can imagine uh if we actually deliver the order then the sort of like the the transaction ends right like the value is delivered um like you know people are placing order and they get it and they can eat. So, uh why is number one not a um uh not a good uh or I guess uh number one is not actually not a good true north depending on your team uh but it can be a very good input. So delivery success rate is a ratio. So um you can double your order and the rate stays flat and uh it would look like nothing changes unless your team or your organization's true north is to actually drive efficiency on you know like making sure that the the the the orders are not dropped in any way or for shape or form. Uh YS3 not a not the best um uh metric here. Average order value. I think some of some of you picked that. Uh it's depth. Uh so going back to the BDEF uh framework uh thing, this is one of the more like you know how how many um you know like yeah like like something per something sort of measurements. And then for weekly active customers uh it's uh again it's not value delivered like uh customers can show up but if you know we fail they can place an order but if we fail to deliver to them they didn't actually get the value. They could be placing a lot of orders. Uh but it would actually lead to really pissed off customers.

43:31 Uh cool. Yes, one is an input. All right. Cool. This is this is really a hard one. Uh I made sure to make this not as straightforward. The other ones are all like ma and stuff. >> The and like when you work with your team to do this like um there's kind of like brainstorms and stuff you can do, but it is hard to land on these. Someone asked earlier in the chat, how often do you revisit it? And I feel like when a team is first wrapping their heads around like what's our northstar? It's like I feel like the first month to 3 months it's like you're every other week like oh like actually we're going to do a different metric and like you know you're reporting these to people high up and so it's kind of awkward like every two weeks like oh we changed our metric again but I think that's just part of the process like take that first quarter with like the understanding of like it's not going to be perfect. We're gonna try some things out and like learn what makes sense and then like you'll revisit more and more.

44:29 >> Yep. >> Yeah. It's really hard with with new products that don't have historical data. A lot of that's talking to um hopefully you have like some like codevs or beta users or a small sample like a lot of that's talking to users or prospective users. >> Yeah. And if you have no baseline it's it's cool. >> Yeah. >> Trust your gut. It's fine. um go from there. Got to start somewhere and iterate. Probably the iteration window is going to be shorter. Um but like you have to start somewhere.

44:59 >> I shouldn't say this as a data scientist because it like is not good for getting data scientist jobs but it's like if you have a brand new product like probably don't need data scientists also. There's no data like you probably don't don't need them yet. >> That is true. And I saw a question around what about total order value delivered? Uh that's a really good one. Um it's actually in the in in a very similar vein as uh something like total signups. So the number can only go up into the right. So um like you know like only one direction sort of metrics um is typically not suitable for a true north because everything you do it's only going to go up or at worse stay flat.

45:39 So it doesn't actually give you a measurement of the state of uh of value that you're providing whether it's going up over time or going down over time. Good question though. Okay, cool. Let's move right along. Uh, okay. Transition to AI demo. So, at the end of this uh as a followup to this session, um I will be sending you all this deck and also a couple prompts for uh any of your favorite chat bots, Claude, ChatGBT, Gemini, Grock, anything uh that you can actually practice with u with uh based on what we talked about today.

46:15 All right, cool. Let's uh to the next one. Uh yes, so we have a couple free workshops coming up. Um these are the most immediate two. Uh that's uh that's that's happening. So if you're interested, uh certainly, you know, like scan the code or uh uh or uh uh go into the go to the links. Uh we have one tomorrow that Sean is hosting to on designing experiments for AI features. And then uh next week we're really excited about this one which is analyze data with cloud code opus 4.6. uh we have a lot of really exciting um and I would say almost pioneering uh materials around actually how to effectively use AI to do all the analysis work uh as as much as possible meaning like uh we and I I'll actually show some of that uh later uh on you know like just delegating the execution work to AI and have faith that it actually knows what it's Uh so we're super excited about uh these uh these sessions and uh certainly sign up. Um they're free and uh they're happening in uh in the next uh week or two. One other plug. So like how high show you lots of like links and stuff.

47:34 Um if you just go to AIA analyst lab.ai I dropped the link in chat. Pretty much everything's on there. It's like that's like the directory like arc. All our courses are on there. All our free workshops are on there. We like free email courses. We have a podcast. Um we have blogs. We're going to post like analyses up there too. So if you just want one link, AI analystlab.ai is the link. >> There you go. Um coded up two days ago and uh this deck did not incorporate that yet.

48:06 >> And a bunch of things to uh rewind uh regarding all our you know like everything everywhere you could touch us on. We are on LinkedIn all of us Ravia. Hi and Sean. We post a bunch about all our research in the cloud gold and you know AI analyst area. We have a bunch of free workshops coming up. We have two courses. The first one is about the AI analytics for builders. Uh and you could find about that as well in on Maven. Uh you could also find about that in like AI analyst lab.AI that Sean shared. And we also have another course coming up uh which is going into the depths of how do you like you know incorporate uh claude code 4.6 into your daily data science and analytics or even as a PM or as like anyone working with data how do you incorporate uh these workflows the AI workflows into your day-to-day. So if you have any questions about those courses and you know want to understand more if that's a right fit you could set up some time with me uh and I could just help you understand like if uh are those courses a good fit for you or not and you know we can go from there. Yeah.

49:16 >> Cool. Nice. All right. Cool. Uh we're not going to have time to do the actual demo, but we'll I'll I'll send you all the all the uh all the prompts and stuff. Um but um uh we have a couple prompts where it helps you. It kind of you know acts as a coach to um to help you refine and iterate iterate on metrics and uh it will ask you very similar questions that we went through on the framework and so uh you can always run through your specific business your specific service specific product uh and what you're thinking about uh against it.

49:50 So what we're doing is uh so this is like the the demo output if we were to do it um in chat GBT. I actually have it here but uh we just don't have time for it. Uh it'll show you sort of like hey is this failing at this step? Is this failing at here? Is this uh you know actionable and things like that and it'll have help you give uh it'll give you suggestions for how to how to really think about it um beyond you know what you proposed.

50:17 Cool. Um I do want to leave some question or some time for Q&A. So uh I'm going to end in the next couple slides. Uh so remember your true north metric is the outcome that you're measuring and all the inputs that we went through are the levers that you can pull to actually measure it or to actually influence it. So this chain has to be very clear to you in terms of the cause and the effect. So learn the framework and then let AI help you move faster. That's the tagline.

50:49 Hey, and I know we're over we're only got five minutes. I can stay over like 30 minutes. Um if if you got bounce, no worries. But I can also stay over for folks. >> Yeah. So nice. So Sean will stay for Q&A and uh certainly feel free to connect with any of us. Um and uh we can we can chat more. Uh we have a free Slack community uh on um just AI builders generally who are interested in uh analytics AI um and uh just sharing knowledge sharing discussions all free feel free to join us uh we have we share a lot of free resources and knowledge uh on that as well bitlyic connect the slack uh slack community okay and uh Shravia have gone through this uh quite a bit um So we have this AI analytics build for builders course uh where we go in a lot more depth around some of the you know this is one of probably the the the 60 70 and 80 90 lessons that we have planned. Uh so um we're going to help our goal is to help people to become analytically independent. What that means is knowing how to ask the right questions, think like a seasoned data scientist or data data analyst would do. Um, and then be able to delegate execution of those work to AI and actually get very very good results. That's our promise and that's our goal. Fiveweek training starting in April for this group and the lightning lesson attendees. We have a 25% off uh promo code for you. uh you can scan the QR code or go to bit.lyanalytics builders and uh uh and that'll be uh that'll that'll get you all started and feel free to connect with again with uh with any of us.

52:38 Okay, cool. So that's the that's my last slide. Uh we're going to go into Q&A, but before that I want to show you a few of the things that we've been that we've been doing. Um and uh and then >> hey before you get into that high or you can open that up. There's another thing that we just created this week that's brand new. It's a boot camp. It's a weekend boot camp. It doesn't go into the whole like analytical thinking framework like this one does. If you are like familiar with kind of like analytical framework and analytical thinking already, but you just want to like get set up in cloud code for data analysis, like effectively building like your own AI analyst in cloud code, we're going to do a two-day workshop. It's like a weekend workshop.

53:21 It's going to be four hours each day. Um, we this is new because basically we we tested this out. We weren't really able to do this very well like a month ago, but with Opus 4.6 coming out, it's like really remarkable what it can do now. So, that's why we're kicking off this new boot camp. Um, if you take the boot camp and then you want to take the full course later, we'll just subtract whatever you paid on the boot camp out of the full course tuition. Um, but yeah, I don't know. That's an interesting way to get your feet wet if you just want to do like the technical aspect of it. And by technical, it's like set up cloud code and speak English to it. You don't even have to type. It's not like you're you're coding. So, it's pretty interesting. Happy to chat more with that. If you go on LinkedIn and like look at some of the stuff me and Hy and Stra have been posting about this week, been sharing a lot of the learnings on that. Yeah, totally. And that's a good segue into something that I wanted to show you guys. These are from the AI analyst that we built and that we will help you figure out how to actually do it yourself. Um, this is a full deck, the presentation included, like the actual deck and the imagery and the charts and stuff uh included. This is like the very early iteration of of of this uh of this AI analyst. uh it would basically incorporate all the um all the best practices and like figure out all the logic all the reasoning uh things like that. So you know this one is done on my own um uh product uh sorry my own uh uh real estate uh uh business and so this is this this data has never been seen. And then here's one that Sean did tested out on sort of like public data from Hawaii tourism and then it gave us a uh basically what's wrong with tourism in Hawaii almost one shot. Uh it does its own checks, understands what to what uh what insights to pull from and then what you can do about it. We actually tagged a Hawaiian tourism board to share this with them on LinkedIn as well. And then finally, this is the latest iteration of our AI analyst. Uh just hot off the shelf this morning. Uh this is an interactive HTML presentation. So the other ones were PDFs. This one you can actually navigate um on campsite data. Again, public data, pretty pretty niche. Um but you can imagine this deck helps you understand if you were to book a campsite because I had to do it. How do I do it strategically? And uh what's the timing and where should I go? So uh and I can't really demo it here, but like this button when I click it, it also has presenter notes like exactly what I should be saying at each slide. Uh that's that's that's included in in in this uh in this output from our AI analyst. We did not write any code. We did not we did not write any code. We did not write any um uh any of these uh slides.

56:25 We didn't do any of the formattings. We didn't do any of the recommendations or takeaways at the end. Um, all of this is AI. So, we really think it's a gamecher with Opus 4.6 from Claude and we want to make it accessible to everybody. One thing, one thing like just to emphasize like like we're not selling a product or anything like we're just going to give you the repo like that we have, but like you can build this yourself. That's what the boot camp's about. It's about building an AI analyst in cloud code that is extremely specific to your use case and your data and what you want to accomplish. And like I'm going to post about this later, but it's kind of insane. You can actually seed this in such a way where you have like a right now we have like a 60,000 line code repo that does all this. What I tested last night was effectively creating like a single markdown file.

57:25 that's like a genome or a seed and give it to cloud code where it just and feed it like what I want to do and give it my data and it's like go build the data team for me and over like several hours it just spun up a bunch of agents and started expanding itself to like rebuild from scratch its own but now specific to a different use case uh I'm working with. So that's kind of what we're going to teach here. Um, I mean if you we're doing a free workshop on this next Friday and I'll just give you the link to the repo then too for the free one and you can fork it. But uh that's what the boot camp's about. Not trying to like I don't know. I'm not I'm not trying to go like build Tableau or something but I'm hoping that we're hoping to like help people enable themselves to build things like that themselves.

58:14 >> Yep. Totally. And I haven't written code in a long time. Um, so I my my my day job is to manage. Uh, that's that's basically all I do. Um, but I feel like I'm now able to replicate something that I was able to do and do it faster, better, and uh uh and and more efficiently. So, >> and another another example that's not here is um so 13 years ago I did Microsoft internship and I I got a case study that was sent to me to you know and it took 3 days back then that got me the actual on-site interview into Microsoft I could replicate that in 45 minutes with cloud code and that too with the previous version with the newer version Sean I need to try the one that you changed probably it's even uh you know it's going to take even way less time but it's incredible how much we could get done with Opus 4.6 and we're glad that we are able to test this ourselves and we could teach you guys too.

59:13 >> Cool. Awesome. Um, so we didn't get into dedicated Q&A time. Um, but Sean's going to be here to take questions from any of you who wants to stay over. Uh, I need to hop unfortunately for uh, a meeting. But, uh, thank you all for attending this. This has been great. And uh thank you for all the engagement. I look forward to speaking with you hopefully soon. >> Same here. >> Sorry.

59:46 Any questions? All right. Let me let me read your thing here. Maxine course is mainly teaching sen workflows. How do you recommend students hunting level positions? This on their resume or their interviews. Yeah. So I think for Okay. So I think for your question, Maxim, I think the fiveweek course gets into a lot of this stuff. So, the fiveweek course, uh, just to kind of break it down for you, it's like the first week is, uh, more like analytical thinking, kind of like the thing that Hi went through here where he's teaching you how to frame questions and build questions and build metrics and a lot of other stuff.

60:37 Like these are just like core skills around analytical thinking. The second week is getting set up with cloud code and um potentially some other um some other analytical analyt AI analytics platforms like hex or something. Weeks three and four are then getting into like drivers analysis and causal inference. Like we're going to teach those actual um concepts, but then we'll show you how to implement them in like with the help of AI, but we're going to teach the concepts themselves in that one. And then week five is around like okay, how do I like storytell and provide a narrative and bring this to like um like you know my manager or my exec and like get done basically.

61:21 So that's still like that course isn't like hand it all off to AI. It's more like how do I do the endto-end data science workflow but hand off a lot of some of the technical components to make it faster. Um the boot camp in itself is a lot more of that like okay let's work with you to like build our own like AI analyst or data team within cloud code. We're not going to go into the kind of like analytical frameworks there, but we're going to go a lot deeper into cloud code itself. I don't know if that answers your question.

62:05 Cool. Uh, what's the repo you were talking about? Um, we're going to share it in our free workshop. Let me see if I can pull up the link to the free workshop. I need to clean it up a little bit before I before I share it out. Um but uh yeah, it's just the repo for our that we have like built like the AI analyst in. So it has basically like 20 like different agents like like agents that are like UX experts versus like storytelling experts for like vers like um driver analysis versus like causal inference experts. And then it has like a bunch of cloud code skills that are already created. And then it has a bunch of like helper functions in Python for like making all of like the beautiful charts that Hi showed as well as like how to like knit together into a PDF. Um so we're going to share that repo. Um if people want to just like fork it and use it. But I think what's more interesting in the boot camp is like building it from scratch yourself because then it's like hyper tailored to your use case um and like the brand of your company and what you're trying to work on. But let me try and pull up the free lightning lesson for you and I'll drop the link.

63:28 We'll share the repo once we and kind of walk through it in this and do a big demo. And that lightning lesson is going to be like a twohour thing next Friday. And we we didn't you saw we didn't really like reserve as much time for Q&A at the end here. We try to get to them during it. But that one we'll really try and do one hour walk through, one hour full Q&A. Okay, that should be the link to the free workshop.

64:03 No problem. Any other questions? Let me Oh, the other thing, you know, I read a blog post on this this weekend of what we were talking about. So, you can also check this out. Uh the title is somewhat clickbaity. Uh replacing your data team, but you got to do what you got to do for SEO. Oh, nice. She started already. Okay, cool.

64:38 Other questions? No worries. If there's not other questions, I'm going to hang around for I can hang around for 22 minutes, but okay, here we go. Zero to one stage. When's the optimal time to define your Northstar metric? I think so. I mean, in the zero one stage, you're kind of probably already defining Northstar metric without actually defining the metric itself because you're probably talking a lot about like why are we even building this thing? Like what's the problem we're trying to solve for a user? Um, so I think you're probably already having a lot of that conversation in terms of like what value you're bringing to the user. Uh, now you might not have data to actually like create a formal metric that you're going to put in a dashboard or a SQL query or a Google sheet or whatever, but you can still like have that kind of user value statement and then be interviewing users or even it's like with the team like having like sessions where you're working through your product and seeing if it's actually hitting that. Um, I think in terms of like so so you already have that. I think it's really good to think about that early on, you're probably already doing that. And then um eventually when you start collecting data or when you're like ready to collect data, like if you like have the product out in front of someone, have your team think as much as possible like what do we need to create within the product where we can like get some reflection of that user value. Like you you might have to build something. you might have to like make a workflow or a time stamp or something that doesn't exist yet that can actually kind of like give you signal around that and then instrument that data like as early as possible. So then once you have it like collecting and flowing through you can naturally translate that into a metric. Um did that kind of answer your question?

66:44 Uh >> yes absolutely thank you very much John. Sweet. What else? What other questions? What do you think about aha habit moments? Are they input metrics for Northstar metric? Should we validate input metrics that they actually impact Northstar or aha habit moments? Do you mean like um the thing that's like I don't know uh Facebook had like their like once someone has like seven friends or something there's like some like trigger. Yeah. Yeah. Okay. I think that's like goals. I think that's like you have your northstar metric or maybe it's your metric. Input metric. I think it's probably like a goal of like that you're trying to reach rather than I think it does relate to Northstar metric. Could relate to input metric too. So actually what I've done in my work with input metrics is I have looked at historical data and looked at the relationship between uh one of my input metrics like around like say like the quality of our AI outputs and then one of our Northstar metrics like the time saved um from someone using this feature and I plot those together after I do like some kind of like regression analysis to like basically uh control for everything else going on.

68:16 So I know the real relationship between those two and I see like okay what's the point at which like increasing quality which is my input metric suddenly gets a spike like uh like uh more increment for every incremental increase in quality I get even more and even more incremental increases in my like northstar metric that's like I think a reference point around like both someone's having the aha moment Right. Then um and then you can follow that all that kind of relationship all the way back even to see like some sort of like uh deceleration when people no longer get like incremental gain if you put more into the input metric and that's like that might be a signal to like oh let's look at a different input metric to move the north star. But yeah I think so I guess to go back to your question what do you think about the aha habit moments? I think they may be the key milestones along an input metric that have significant increases to the northstar metric. So that's like what teams might goal around like, okay, this quarter we want to get this input metric to X because we know that means we'll hit that aha moment or that means we'll hit some deceleration of like the ahas or whatever and we should focus on something else. Does that make sense, Sergio?

69:45 Cool. What else? got 17 minutes. I'll let it be awkwardly silent in here for two minutes as people are as if people think of stuff. No one can drop. What else is going on? Is anyone here in California? So, I live in Tahoe and uh it was straight up dirt on the ground two days ago and now I have four feet of snow that I'm looking out my window. Sorry, I'm just trying to kill a silence. But man, that's crazy storm.

70:32 Where did I learn to apply these analysts frameworks? Oh, before your first data related role. Oh, yeah. I did not Yeah, I didn't learn these in in school at all. I learned through a lot of failure early on in probably my the first five years of my career where I was like doing analysis and like analysis that like VPs or directors or someone asked for and I would work on some for like two months and deliver to them and they're just like, "Oh, that's interesting." And then nothing would happen. And it's just like I think I got sick and tired of it because it was like okay uh my stuff's not getting implemented. But then I al that also impacts stuff like your promotions and raises and all that stuff because it's like even if you're working hard, even if you're smart, like if you're not having impact, which I think a lot of these analytical frameworks basically like force you to do analyses that you know action will come out of and impact will come out of then like it's really hard to progress in your career. So I don't know like I think part of it was me getting like sick of it and like switching to a different mindset. I think it's really helpful to have mentors. So like I worked I've worked with Hi who earlier you obviously who gave this presentation for about six years now. She's like really good mentor to me. I had some other managers and mentors and also like colleagues. I think a lot of it's just like learning on the job and with other people. But if if like your whole goal is just like I got to set myself up for making it actionable, like what are the frameworks that can get me to do that?

72:08 Um that's some of the best way of learning. But it's it's really I think it's pretty hard to learn beforehand. But if you know this stuff, a lot of it's going to show up really well in your interviews. Like all these frameworks, it's just like ways of thinking would resonate very well in the in the interviews. Um, yeah, you know, Sarraia just did something the other day that actually, so she's doing a session next week or the week after that is like leveraging AI to help with product sense interviews or something. All right, let me find the link for this one because this one might be good for you. This is another free one.

72:54 Uh crack product sense in analy interviews with AI. Okay. Interview might listen. All right. This is one. So Savi's been working this. So Sravi is a senior manager in data science. Like she's interviewed hundreds and hundreds of people um over her career. and she was just like playing around with Claude the other day to like help create like um like a interview kind of like simulation or interview like training partner. I think it actually I don't know. I mean I I think it's kind of a a nice way to like you can like you can do like simulate mode and see how it would work through a problem or you can like test it against yourself. This is just like a little fun side thing she was doing. But um maybe maybe go to that.

73:46 She's going to have some she'd be able to answer some good questions for you too. Saying hi from Berkeley. Nice. How do you see the future for data analysts? What should we do? Learn should we switch to data engineering or product management? Pl Dude, man, the plumber thing. Like I'm seriously thinking about it. My my brother was looking into it a few months ago. like like being a plumber is actually like you have to do this whole like apprenticehips thing is actually kind of hard to get into. Um I don't know though seriously like things that we do with our hands like could be very important.

74:20 Um how do I do actually envision it though? Uh I mean I think product management is a pretty good route. I think we'll see a lot of people kind of trans taking on a lot of the skills of the product manager. I think I thinking like a product manager does you a lot of benefit if you're a data analyst or data scientist working in product like you basically want to like for me it's like I want to basically do a product manager's job but I want to be slightly more technical and I don't want to deal with as many meetings or like talking to as many people like I don't know that's what that's what I like about being a product data scientist anyways I think AI I think product managers will actually come out pretty good on top there like especially technical PMS. I think a lot of product data scientists will kind of like they'll kind of mesh together into like technical PMs. The other thing I was talking to high about as we're building this data analyst thing with cloud code is I think it's going to be like people data scientists and analysts who build systems rather than fully doing the analysis themselves. So if you can build like systems of agents and that can like solve the right questions for you, I think that kind of evolves into the job like managing data agents. But it's pretty hard to tell if you asked me like kind of like four three months ago even like even like a few weeks ago before this Opus 4.6 thing I was I would have been like oh there's a lot of stuff AI can't do yet. But um I don't know if that answers your question. I'm still trying to figure that out too to be honest.

76:00 I think a lot of people are actually Yeah. Nobody knows. But I mean I don't know like could be a good route. I was like firefighter right like in Tahoe. Like fires aren't going out of style. Like more fires every year. So, they're going to need more people in that industry. Um, what else? What other questions? Got 10 minutes left.

76:38 Awkward silence. Trying to think of anything to talk about. You guys watch that new Game of Thrones show that's coming out on HBO? Been watching that? Pretty good. Last episode was pretty crazy.

77:09 Uh, I chose anti-gravity because it was free. But, um, I mean, honestly, so yeah, I guess here was my workflow. I was using cursor uh at first and with cloud code also. And then I kind of liked having them both because uh I could talk to cursor any model and cursor and it's not eating up tokens like cloud code does. And then I was like, I'll switch over to anti-gravity because it's it's free. Um, now I have to pay for the cursor subscription.

77:48 Um, and then over time though, I ended up just like not really caring that cloud code was eating up my tokens and don't even really use the agent and gravity anyways. So, I don't know if you necessarily do that. You could just probably do VS Code, but I don't know. Any gravity is pretty good. They're probably just going to make it better, too. But any any uh any of these products could work like Windserve or uh cursor or anti-gravity or whatever other ones there are other questions.

78:32 Oh man. And yeah, the open claw thing is pretty interesting. I was just starting to mess around with setting it up there night, but I actually I don't want to put it on my own computer. So, I'm going to buy like a MacBook Mini or something to put it on. Oh, yeah. Docker container is good idea. Um, I haven't set it up yet. I've watched some YouTube videos on it, but it does I don't know. It does get me thinking. So like like one of my limitations right now is like I'll be like trying to work with these agents and it's like okay I got to go do something right now that I can't really be in front of a computer anymore with it would be nice to like have them keep talking to them and having the AI analyst like keep going like if I'm on the treadmill or something would be pretty cool just like talk into my phone to Telegram and then like get it going still. Um I haven't tried it out yet though. I did my meeting calendar is like getting insane though. So I I at least want to set it up for that kind of organization around like meetings and Gmail kind of stuff. I think as like kind of like a personal like assistant I think could help. Um but I haven't thinking a little bit about it. I don't know if you've messed around with it.

79:48 product driven companies and barrier higher data DS because the execs themselves do see clear tangible impact from having team of data scientists in the company or is it sometimes more about having enough budget to take the risk? I think it totally depends on the company. I mean, I think definitely like people realize that like um to make better and better decisions, we need to have someone like work with the data to like show us some proof, but we also need someone who is like understands the product and is a business has like some business mindset so they can like you know basically create all like the correct metrics like how I talked about today. Um, I do know companies that straight up have just been like, "We're at the stage where we have enough money to buy a data team now. Let's go get a data team." And they have no idea how they're going to use data, but they just know like people say data driven.

80:53 We should do it. And then they like hire the wrong people, get a kind of shitty team. So like there's definitely that vibe going on too. Um I mean I I think at the end of the day though people are like I want to make decisions and I want to reduce the uncertainty as much as possible so I can make the the best decision and I can prioritize my decision- making and that's a lot of the function of the data science team uh will be interested to see like I don't necessarily think like just like a data scientist has to do that. I think like uh if people like have access to data that they can trust and they know how to like make the right metrics and how to like apply the framework then um other folks could do that too. I feel like that's kind of like the the core benefit of having the team right now.

81:50 Definitely there's people that just do it because they have the budget for sure. What else? Got five more minutes.

82:26 I think we may have exhausted the questions. Okay. If you have other questions, um, feel free to ask them in that Slack community. I'll post the link one more time. Thanks for joining the session. This is awesome. You guys are really interactive and asked really good questions. Um, I had a pretty good time hanging out with you all.

83:00 Okay, so here's the Slack link again. You can DM me there or like drop in the channel. So, this is a pretty new Slack group. We're like trying to wake it up right now. We don't have like a community manager or anything, so we're trying to figure it out. There's kind of two main channels. One's like AI analytics and one's AI evattles. So if you have like questions around either topics like just ask them in there I think broadly then we can try and see if we can get more people talking about it but we'll answer for sure and then yeah I mean you can just uh DM me on LinkedIn uh as well if you don't want to like be messing around with Slack and you just want to talk there that's cool too.

83:43 Sweet. Well, thanks everyone again for thank you all for staying over um 30 minutes. Really appreciate it and thanks for all the great questions and being engaged. Um hope you all have a great rest of your day and hope that we talk again soon and maybe I'll see some of you next week in that uh workshop that Stravi is doing around like interviews or the cloud code uh workshop. See you everyone.