Section Insights
Introduction and Product Launch Discussion
What defines a successful product launch for the new feature?
A successful product launch for the new feature of picking up FedEx and UPS packages would be measured by user growth, engagement, and monetization metrics. Key indicators include daily, weekly, and monthly active users, as well as the signup rate.
- Success is defined by user growth, engagement, and monetization.
- Metrics to track include daily and monthly active users.
- The signup rate is crucial for understanding product feature usage.
Defining Growth Metrics Pre-Launch
How should growth metrics be defined before launching the feature?
Before the launch, growth metrics should focus on defining clear targets for monthly active users and adoption rates, considering whether the feature starts with zero users or builds on existing ones.
- Establish clear growth metrics before launching.
- Focus on monthly active users and adoption rates.
- Consider existing user base when defining metrics.
Calculating Business Opportunity Size
How do you size the business opportunity for the new feature?
To size the business opportunity, calculate the potential revenue based on the adoption rate and user population in the city where the feature is launched. This involves estimating the number of users and their expected usage frequency.
- Revenue potential is calculated based on user adoption rates.
- Consider the population of the city for accurate sizing.
- Assume a consistent adoption rate across cities for projections.
Projecting Revenue Across All Cities
How can the impact of the feature be projected across all cities?
Projecting the impact involves calculating the total revenue by multiplying the number of users in each city by the adoption rate and the revenue per delivery, then annualizing this figure.
- Revenue projections should consider user numbers and adoption rates.
- Annualize revenue based on expected usage frequency.
- Use a consistent formula for all cities to estimate total opportunity.
Key Metrics for CEO Dashboard
What are the top three metrics to include in a dashboard for the CEO?
The top three metrics for the CEO dashboard should include daily active users, total revenue from the feature, and net new users brought in by the feature. Additionally, the number of completed pickups per month is crucial for assessing feature effectiveness.
- Daily active users and total revenue are critical metrics.
- Track net new users to gauge feature impact on overall service.
- Monitor completed pickups to evaluate feature success.
Transcript
0:00 Hi, my name is Victor. I'm a senior data scientist and an instructor at data.com. Lkesh, would you like to introduce yourself? Yes. Hi, my name is Lkesh. I'm preparing for data scientist and data analytics interviews. All right. Excellent. okay. So, let's dive into the question portion. so let's say that we want to launch a feature where dashers pick up FedEx and UPS packages from customers and then take them to FedEx/UPS package stores.
0:37 What would you think of as a successful product launch in that case? Okay, cool. Thank you for the question. let's see. So essentially this is a new product launch or actually a new feature launch where users have an option to call dashers to pick up their FedEx and UPS packages for and then take them to the store for them.
1:10 Correct. Cool. Now to measure a successful product launch I think I'll look at some of the metrics that can be helpful to define how the product launch is is going on right so I think in this case the overall goal is for high level goal will be for users to like use the app to find dashers who can actually pick up their package and delivered to UPS or FedEx store.
1:50 Now I will look at three buckets in this for product success. high level the three buckets will be the growth of the users, the engagement and then finally the monetization. Right. in the growth bucket I would look at some of the metrics like maybe the daily active users or weekly active users or monthly active users depending on how frequently the users do this. Yeah, I'll divide into three buckets. growth which is the daily active users, monthly active users and then I'll use the other metric I'll use is the signup rate. So essentially in this case it's more of a product feature usage. So number of users who are using the pickup option maybe per week per month as well. So essentially an equivalent of signup rate but mostly feature usage rate right and then the just a question here. Can you delve into just one a little bit more detail about what you mean by sign up rate specifically?
3:09 Yes. So when the user opens the app they have an I'm assuming they will have an option to like either order food or order other items that are available and there's another option where there will be option in in the app to use the feature where they can schedule a pickup of their package. So I will track the number of users who are using that feature per month or per day or per week since like usually the packages maybe they are returning once or twice a month on an average. So it might make sense to you track it as usage per month.
3:55 Right. Okay. Cool. And then in terms of engagement now that engagement is essentially how frequently the users are using the pickup option right so I can keep track of that using total number of pickups ordered by users and then I can also look at the frequency which is maybe on an average how many pickups are ordered per month by a user and then the we already talked about the total number of pickups as well. So that's in in terms of usage metrics by the engagement for the user and then finally for monetization I can look at the total amount spent by users on ordering those pickups.
4:58 essentially if they are yeah so I'll I'll track total revenue generated from this feature by the users. Okay. Now that's the user side of metrics. I can also look at maybe the from the dashers the metrics from the dashers. So how many dashers number of dashers who are who have made the deliver deliveries the maybe things like percent completion hopefully we have a higher completion so essentially number of dashers who accepted and delivered those packages to UPS and FedEx And then also the revenue additional revenue that dashers are earning because of this option. Right? So other than delivering the groceries and prepared food, how many what is the income that dashers are earning because of that this new feature?
6:07 Okay. Additional dasher income. Yeah. And you said one other metric. Can you repeat that metric again? percent completion. So out of number of times they accept how many times they are able to complete. Right? This is a new option and I want to track that they have the necessary tool and understanding of the process to to be able to complete the return. Okay. So how would you calculate that?
6:37 So that will be calculated using number of number of successfully returned packages divide by total number of request accepted by the dasher. So if a dasher accepts they should be able to pick up and return the package to the FedEx and UPS at at close to 100% rate. Okay. Okay. all right. So, I think this is gives kind of a good highle overview. We've got the growth, we've got the engagement, we've got the monetization of defining success, and we've also got product launch success, we've also got the dasher side of metrics. So, let's let's go into a bit more detail about some of these.
7:30 let's maybe I'm curious to hear more details about the growth side of things. very specifically. yeah, absolutely. Yeah. So I think for growth we looked at daily active users. We let's let's pick maybe monthly active users in this case. How many how many users are able to order the dasher to pick up their packages.
8:04 Now when you say actually you want to dive deeper into the growth numbers are we looking at deeper definition of the metric or something else? Yeah, let's talk about deeper definitions of the metrics. You know, I'm also curious to hear, you know, at what point would you consider the growth, you know, successful like how would you define success in this specific case for growth of the feature?
8:37 Yeah. So, this this is a new feature. So, couple of clarifying questions. So, is this a feature launch as in we do are we starting with zero users or do we have some some users already and we we we are defining those metrics. Yeah, that's a good question. So as of 2023, you know, we can say that Door Dash, you know, has already had something like 35 million monthly active users in their other products.
9:12 Yes. And then this is a new feature we have launched or is it before the launch we are understanding how best to measure the success. That's a good question. So let's let's do the first question first which is before the launch and then let's then we'll go into after the launch. Got it. So I mean before the launch I think we have define the growth metrics and maybe we can look at the clear definition of the metrics and then maybe what we expect in terms of setting the targets and expectation for for achieving these metrics. Is that does it make sense?
10:06 Right. Right. Got it. Okay. So now we in terms of growth metrics we have let's say monthly active users and the adoption rate instead of signup rate I I might use the adoption rate because this is users who are adopting number of users using the pickup options per month. Okay. So in terms of monthly active users, I'm I'm guessing that this feature will will probably the launch process will be we launch in one location, maybe start with one city and expand expand by region. and then while we are launching in every region, keep track of these metrics, get user feedback and maybe make changes and improve the improve the product. in terms of success in this case for the monthly active users and adoption I can say that the when the feature is launched I think it depends on how much the the team plans to spend on maybe marketing or advertising to make people aware of that that will drive how the adoption progress this but I'll make some assumptions here that Door Dash will do medium level of marketing and maybe in product notification for users to know about this new feature that is being launched.
11:43 Mhm. Based on that I will say the the the product team can make an assumptions that in the first month maybe and this is my assumption here like 1% of the users will make will make use of this new feature. Okay. And then gradually I think as the users understand the value of this we want to keep track of the retention and engagement. So how frequently they use the feature and hopefully through through continued advertising notification for users who are not using it we can keep keep a linear increase in in the number of users who are using this feature. actually let's I will say like initial adoption might be higher and then the growth will be linear. So maybe in the first month we we have like two to 3% adoption and then every month after that 1% additional 1% adoption of the users and in in this I'm I'm expecting the the growth to be slightly faster sooner and then tapers down and become maybe linear after first few months.
13:08 And when you say adoption here, maybe can you be very specific about this? Do you mean you mean the total number of unique users divided by who have adopted the product who have actually how do you define adoption rate very specifically? Yeah, that's true. The the adoption will be monthly active users who use this package pickup option divide by total number of monthly active users on the app who use any of the dasher service any of the door dash service.
13:44 Okay. All right. That's good. Overuse. Okay. And it's this metric that you expect to be say two to 3% in the first month. First month and then maybe 1% after that and eventually it will taper down to whenever maybe maybe in this case we not everyone will will use this feature but maybe like the goal could be like maybe 60% of the users use the delivery option.
14:22 while you expect it to get to to a much higher rate is what you're saying. Yes. Yeah. And that will depend on the that will depend on maybe the data analysis of how many of the dash of the users of the app actually have the del the need for package returns. It will also depend on how many users like use depending on where they live like do they live very close to UPS and FedEx office if they are super close they might not use the option but if they are little bit far they might use use the option so that can some of these signals can help define what should be the ultimate goal for for this feature.
15:26 Okay. So, I'm curious about this specific thing, this specific idea here that if users are super close to FedEx, maybe they don't or UPS, maybe they don't use it as often. How would that impact the adoption rate? Yeah. So in this case, I think the calculation will be if maybe you can pick a threshold, right? Like maybe one mile is something like a a distance where people can maybe even walk or or do a quick drive to return the package. they may not want to order dashes because this is super close.
16:08 the other factor will be so we can use that to like exclude the number of users who may not use the option. the second factor in calculating the adoption would be the open hours for those locations. So maybe the the the closest FedEx office is close by but it's open only from 9 to5 and that's the typical time when people also go to office. So I can use that as another factor where even if some of the people who are living closed they may also still use the service. So I'm not going to exclude everyone who is living within that one or two mile range whatever my threshold is in this case. And then the third factor I'll use here is something like not only just the location where their home address but also like the their office or their commute. So some people like do return when they go to office or in office there is facility in office to return those. So you take the package to office and return it. So I will reduce that those users as well and maybe use the estimation of how many people actually just return their packages in office or on their way to office. So those are some of the factors which will determine the I can use that as an input in calculating the ultimate adoption rate for this product feature.
17:35 Okay, I like all of that. I like where that's going. so let's say, okay, so what we're at right now is we have an adoption rate, and let's say that we've rolled it out to one city. So we have all the data for one specific city. All right, let's say we have the adoption rate of say and it's what you say 2 to 3% in the first month, 1% adoption otherwise in that city.
18:05 Yeah. How do we now size the the size of the entire business opportunity for Door Dash? So, you know, this is a standard thing. So, the manager comes to you and says, what's the what's the potential impact and sizing for the entire if we roll this out to all users on Door Dash? Got it. Cool. so I have few few questions here. So if you launch with one city what is the population for that city?
18:36 let's say that you know it's a mediumsiz city. So you can you can come up with any number you'd like. Yeah. So yeah, let's say this is a city of maybe like Door Dash operates in larger metropolitan areas which many of them have higher higher number of people than on average US cities. So I'll go with little slightly higher number. Let's say 5 million users 5 million residents in that city. The other question other data will be useful is in that city how many users actually use Door Dash app every month. So what are the monthly active users in that city?
19:21 Right? let's say that so let's let's say that you know that there's like 1 million monthly active users in that specific city. Got it? So 35 million there's 35 million monthly active users on Door Dash and 1 million of those 35 million are in that specific city where this product has been launched. And we and to recap again we've measured that the the adoption rate is about 2 to 3% and we do see that there's a 1% adoption afterwards month over month. So then how would we estimate for the business and and one more clarifying question is we we are looking for adoption like after one year or after after how much time after the launch?
20:12 That's a good qu that's a very good question. Let's say that we want to estimate the annualized impact for Door Dash's business. Got it. Okay. Cool. With with this data, I think I'll I'll start doing the calculation, maybe make some assumptions. then I'll I'll start writing down here as well. Okay. So, we have 1 million BAU, population is 5 million.
20:51 And then based on the assumption, let's say we have let's be conservative here. So 2% in in first month. Mhm. And 1% after that for each month which gives me 2 + 11 around 13% adoption after one year. Okay.
21:27 Now this this is very dough calculation. I think I'm not u I'm not using the other factors but that is another way to calculate where we we look at we look at those factors that we discussed. Now I'll use one more bucket here like maybe some of the users who do not use it for prepared delivery but if they know about this option they might use it just for this option right so like there are 1 million users who are using it for prepared food deliveries and groceries. but then there might be users who need this just for returns because they might live close to the restaurant or they might be cooking at home or or or many other reasons. So there are there is additional bucket outside of those 1 million DAUs who will just sign up for the only for the delivery option and that might be the small set but I'll include that as well in this and that's like that point that's a good point to to make here which is new users who are perhaps you know brought on board Door Dash because of this specific thing maybe they're customers from other competitors but now they've seen that they can get their packages delivered and they perhaps move over to Door Dash because of that. So new users is also a portion of this.
22:48 Exactly. And so I want to let you keep going. I want to make sure that we're So just to be 100% certain here, what I'm trying what we want to do is I'll I'll write out the question maybe very specifically. you know what's what's the overall impact in ter what's the overall financial impact on the business seeing the data for roll out of delivery Okay.
23:31 Packages. So this is the question that we're trying to answer right now that I'm looking for an opportunity sizing on. What's the overall financial impact on the business seeing that the data seeing the data for roll out of taking packages to FedEx/GPS in one city? Yeah. So based on this assumption now we have around 13% users after one year 2% maybe additional users who who are not and they come on Door Dash. So that gives me 15% of 1 million that is 150 users who are using this pickup option.
24:18 okay I'm going to assume that every pickup and return for simplicity of the calculation is let's say it's dollar do we have any numbers like every on average what do we expect or I can make the assumption like it's $7 or $8. Yeah let's say it's a $7 fee sure or a $7 cost. Sure. To the user.
24:50 Got it. And then part of it goes to Dasher and part of it goes to door dash. So can we do a five to split? Dollar five goes to dasher and dollar two goes to unless we have any numbers on that. That sounds reasonable to me. So for door dash I think it's the the revenue will be dollar2 into the 150k users which is dollar 100k revenue and for dash earth this will be additional five are 150.
25:40 Mhm. So, so this would be the calculation of the total revenue to the business for this one city. But now let's say that we have now let's say that you know the product manager saying we want to roll this out to all cities. So how can we project out to the impact on all cities? Got it. So if you make the assumpt again make the assumption that we have similar adoption across all cities then in that case the calculation will be each city's population each city's dasher users so for each city is like sum over each city users multiplied by 15% adoption multiplied by $2 for each delivery. Now we we now this is I I think one more thing here is this is $2 each but we are if we are assuming that it's one per month then we multiply it by 12 to get the full year full year revenue right what's the assumption there so one assuming one users are doing one pickup per month so they will do 12 pickup for the month Okay.
26:59 So that's that will be the per city. It will be 36. Okay. Let's make that assumption really clear here. Assumption users use new feature once a month. Okay. That becomes 3.6 million. Okay. Per city. And then if you do that for every city which will essentially be depends on how many users are in each city. Now total is 35 million right so we can use that so 35 million 15% adoption so 15% adoption and then dollar2 per delivery and into 12 will be my total opportunity size very very high level mhm this should be if I do this 154 45 50 rough calculation will be 50 million into 24 is 120 million annual annualized 120 million annualized profit after door dash or after dash or fees. tax revenue with $2 coming from each.
28:39 Uhhuh. Okay. is there so in this impact sizing calculation that we've got here saying look if we roll this out to every city we're going to get about $120 million worth of annualized revenue. Is there other things that is there anything else that might that we should consider? So that's yeah that's the revenue number there is cost right in terms of to calculate the profit there is cost factor also okay should we look at the cost as well or or what are the maybe look at other factors under cost factors the the other piece is increased now increased demand so do we have dashers to meet the increased demand.
29:36 We may need more dashers and making sure it's not impacting the other other businesses like prepared food delivery. If you don't have enough dasher, it may have a spillover impact on other other businesses. the other we can look at how do we optimize the routing because now if you're doing pickup dashers may be able to do pick up and drop off of the packages as well. So they pick up food, take it to the user and they pick up the package as well. So that way it's the the revenue per user will increase but at the same time it's the more efficient routing algorithm as well.
30:19 Okay. So let's let's dive into that a little bit more in detail. How would you go about you know estimating the impact on the cost side of the business as well because I idea that you're mentioning right which is that there's a finite number of dashers available right so once you start asking folks for FedEx to for them to deliver take your packages to FedEx now there's increased demands on the number of dashers available, right? So let's let's chat about that a little bit more in detail.
31:02 Yeah. So I think the cost there are multiple buckets for the cost. Mhm. There's obviously the feature development, feature de development and deployment and all the cost to create the feature. Then there is dasher recruitment. I think basher recruitment cost might include like recruitment drives additional incentive. Mhm.
31:32 Recruit more drivers and other cost associated with the dasher and the marketing cost of like marketing the feature. By marketing the feature in this case you mean marketing it to dashers. so actually it's actually marketing cost share is marketing to the users and there is additional incentive and marketing I can put marketing here for dasher marketing as well addition recruitment drive essentially includes marketing as well.
32:04 but every all the costs associated with the casher recruitment is here and the marketing this is for user and then feature development is all the NGT cost. Okay. Now you ask me like for the dasher side if you look at the cost maybe like there is a so maybe we estimate the dasher recruitment cost. Mhm. Yeah. So I I think I think this is good. I think you've you've given a good set of ideas of what the cost side would be. We've got feature development engineering related costs. We've got dasher recruitment. We've got the marketing costs. I like this.
32:52 can we chat? So another question for you here is okay so let's say that now let's say that all of those costs are calculated and it still turns out that the revenue is really good how do we so and the feature has now been launched everywhere okay so now we've launched it everywhere to all the different customers all across the United States how do we now define product launch success So we've done the mini mini test on one city. So now the CEO comes in and he asks the question what are the three top metrics that you're tracking to be sure that this product is has launched you know and you have let's say you know two or three months worth of data where it's been launched to all the cities. How do you define what are the top three metrics that you want in a dashboard for the CEO to take a look at?
33:50 Got it. Okay. So I'll look at the three metrics if I have to pick I'll definitely one of them will be daily active users who are using this feature. Mhm. monthly active user sorry you don't return every day. The the other metric I will use is the u engagement.
34:24 Okay. Actually u the active user will be the engagement. The other one I'll use is monetization. Okay. So total revenue. Mhm. from this feature. Mhm.
34:58 And then maybe third one in this case will be net new users. It's bringing. So net new signups essentially to keep track of like did this is this feature bringing new users who might actually then start using other Door Dash feature services like food delivery.
35:41 Okay. how would you measure this metric precisely? Yes. So, this will be number of new users per month using only pickups.
36:14 All right. what about what about a fulfillment feature that might be useful in this case to report to the CEO? Yeah, at at high level I think CEO will probably be interested in number of pickups completed per month and hopefully we see a increase in that.
36:45 Mhm. Right. Because if the the pickups that are being assigned are not being completed, right? The feature is not working right. Yeah. Earlier we talked about completion rate which is equal to successful completion divide by total.
37:25 Okay. I like this. okay. So, let me let me ask the last question here. so let's say that it's been launched and everything. And so, here's the last question for you. let's put it down here to the next page. Okay. So, now you launch the feature. and you see more health tickets are being filed. How would you go about investigating this?
38:06 So, more help tickets filed by by who? What are the help tickets? That's a good question. So let's say that there just new more help tickets are being filed by you know the customer service department says that the total number of help tickets that they've seen is now increasing after the feature has been launched to everybody and the helpers are coming from users. Mhm.
38:42 Okay. I think I think in this case do we know if the tickets are coming for the pickup like do they have an option in the subject line or when they create the ticket do they indicate what is the which feature are they using? Yeah let's say that some of the customers are indicating that it's coming from that they're customers who are using the new FedEx UPS delivery service.
39:17 Got it. I think the I will look at in this case to investigate this I will look at some of the ticket attributes right whenever the user creates ticket probably they are telling which feature it is about maybe they are also telling like type of issue like is it like did the delivery didn't happen or they didn't get the notification is is this the system issue or the process issue The other thing I'll look at is like is it related to maybe one of the u one of the is it related to UPS or FedEx? Maybe it's related to just one of the delivery options and then I'll look at it's related to both UPS. I like where you're going with this. Let's say it's related to both UPS and X. Can you detail what you mean by system or process issue? So system issue would be like maybe the app is not telling users what happened or like it's not easy to like use that feature. So things like they didn't see the option or they didn't know what happened. They were not getting notification. So it's essentially like feature gap in the feature or bug. And then process issue is like they the they didn't get the receipt or the tracking number or not the the driver didn't tell them like did they deliver it it or not or did they they had confusion on what happened in terms of understanding of how the delivery is is done. So that will help in tell me what is the next step. Should we improve the product?
40:57 Should we have a better documentation or frequently asked questions and something like help with the understanding? Okay, that looks good to me. Okay. And then the other piece I'll say is like then we can track like depending on how many tickets maybe this can go into the cost of the cost analysis as well because then we we may be hiring new people to answer those tickets and answer those questions.
41:31 Okay. So now that we are done with the question portion let's delve into some feedback in the feedback portion and then for those of you who are watching this if you want access to the full video including the feedback please visit data.com to unlock the content.
Summary
- Success metrics for the product launch include user growth (daily and monthly active users), engagement (frequency of package pickups), and monetization (revenue generated).
- Adoption rates are projected at 2-3% in the first month, with a gradual increase to 15% over a year.
- The financial impact of the feature could yield approximately $120 million in annualized revenue if rolled out across all cities.
- Costs to consider include feature development, dasher recruitment, and marketing efforts.
- Key performance indicators for post-launch success include daily active users, total revenue from the feature, and the number of new users attracted by the feature.
- Increased help tickets post-launch should be analyzed for system and process issues to improve user experience.
- Understanding user demographics and their proximity to FedEx/UPS locations can help refine adoption rate estimates.
Questions Answered
What defines a successful product launch for the new feature?
A successful product launch for the new feature of picking up FedEx and UPS packages would be measured by user growth, engagement, and monetization metrics. Key indicators include daily, weekly, and monthly active users, as well as the signup rate.
How should growth metrics be defined before launching the feature?
Before the launch, growth metrics should focus on defining clear targets for monthly active users and adoption rates, considering whether the feature starts with zero users or builds on existing ones.
How do you size the business opportunity for the new feature?
To size the business opportunity, calculate the potential revenue based on the adoption rate and user population in the city where the feature is launched. This involves estimating the number of users and their expected usage frequency.
How can the impact of the feature be projected across all cities?
Projecting the impact involves calculating the total revenue by multiplying the number of users in each city by the adoption rate and the revenue per delivery, then annualizing this figure.
What are the top three metrics to include in a dashboard for the CEO?
The top three metrics for the CEO dashboard should include daily active users, total revenue from the feature, and net new users brought in by the feature. Additionally, the number of completed pickups per month is crucial for assessing feature effectiveness.