How to prepare for a Data Science Product Analytics Interview: Brainstorming the KPI map

0:00 hey everyone my name is instasea and i'm a data scientist working and living in stockholm in sweden and today i want to share with you one of the biggest tips that i have for preparing for any kind of product analyst or product data scientist interviews i think that's one of the most single important things to do when you're preparing for an interview for a company that you really want to work for and i think it will help you

0:21 both in product interviews and in technical interviews and will allow you to have more interesting and engaging conversations with the people who interview you as a product data scientist or as a product analyst most commonly you will talk about metrics or kpis kpis are key performance indicators and that's a type of a tool that every company uses for understanding whether they're going the right direction or not with how they develop their products so as a data scientist or a product analyst

0:48 you will have to help the company to determine those kpis or metrics to decide which ones are the right to show to which kind of stakeholders to help the developers build the tracking and the tools necessary to understand and be able to extract those metrics from the data and today i'm gonna work from an example that i wanted to talk through we're gonna imagine that we're interviewing for a streaming platform a company that has a platform where

1:15 users can upload their videos and people can watch them and they're earning money through showing advertisements on top of those videos so we're gonna look today at the kpi map first things first there are different types of kpis for different use cases there are product performance or system performance kpis and there are user experience kpis those are the ones that are usually most commonly used for understanding the product performance

1:45 to see whether a new feature is useful or not and generally to understand what's the track of the product you're working on it's going so we're going to focus on those a little bit more system performance kpis are really important to look at tell during the interview process that you know that they exist it's a really important topic and it impacts also the user experience system performance kpis are important to track it could be a load time for the videos

2:13 it could be general uptime of the site or uptime of the app it could be number of bugs or a number of times the app crashes it can be like on the surface user experience impacting system metrics like the ones we look at now but it also could be something a bit more deep and more technical for example i don't know uptime of the load balancing system if the company has load balancing system to allow for

2:40 changes in user traffic everything that you can think about is valuable and may be valid to bring up during your interview process usually when we talk about system performance it's also important to talk about the rollout of the new versions of the app or rollout of the new ap test it's clearly very important to test how the system will handle the change first before looking into the user and make sure that nothing will crash there won't be any

3:05 issues there won't be any problems with logging like for example google recently had and like certain issues like that but it's not necessarily the most important part to look at when you think about the performance of the product as a whole so we're gonna skip ahead from the system performance to the user experience now there are different kinds of users of the streaming platform that you can think of so i already mentioned that there are people who consume the content

3:32 then there are people who create or upload the content and there are people that bring in their revenue to the company so the advertisers that use the platform to run their ads and promote their businesses honestly in this kind of part of the exercise you can come up with as many different sort of stakeholders of the user experience that you can think of the more the better it could be very really really valuable for you to just

4:00 like brainstorm those out i think those three are the main ones we want to focus on right now because they have a connection with each other so content creators obviously create content and ability for people to watch something on that platform the people who watch the content spend their time and like give their time and energy and mental resources to the content creators but also to the advertisers and the advertisers eventually power the platform with the money

4:29 so that the company that built it could pay their employees and they could continue growing the product and provide better user experience from my experience it's also very common that the companies like that have mission statements and really often for companies that provide like user-centric applications like streaming or social networking one of their most common themes in their mission statements is connecting people with each other and in the case of the streaming

5:01 platform most likely it will be connecting content consumers to the content creators and allowing both of them to kind of synergize in the best possible way where people find what they're looking for and what they're interested in and people who are interested in providing that content are able to build it create it upload it see the metrics see how it performs and eventually create better content based on those metrics so when we're looking at the let's say

5:31 easiest one to start with is content consumer there are plenty of metrics that we can look at most commonly the goal is to keep the content consumer on the platform so you want to retain them and companies like google that has youtube or instagram usually look at the overall volumes of users that use those platforms so here we go with daily active users monthly active users a bit more rarely

6:02 weekly active users in terms of like whatever they define as activity it could be just logging in it could be starting one video and watching it for at least five seconds that really depends on the use case and that's something that you should be able to reason around and explain why certain type of activity could count as user activity and why in some cases it's important to know that the user started a video in some cases

6:30 it's just important to know that they logged in then there are metrics that are more about their engagement with the app that they're using so in this case it will be engagement with the content it could be in duration like minutes of watch number of overall streams of the videos so just like i don't know how many videos a person has started could be that we want to also distinguish number of complete streams so how many time we finish the videos it

7:00 could be interesting also to track how much does the user skip around the video so number of skips or number of like jumps around the timeline anything that is related to how you would use this platform as you think of yourself as a user what would you do you would be active you would watch videos you need to search so maybe here we also want to make sure that the search is relevant the search metrics can be something

7:26 around like how relevant did you find that the user this search result whether you looked at the first one or the second one or you scrolled a lot we also know that there are suggestions of videos so it could be important to track whether you were watching the video that suggested from the previous videos like in the tracking metadata anything that you can think of try to just like brainstorm the hell out of it maybe you read

7:49 some articles about how this company that you're applying for is tracking their performance trying to reason around why they would do it this way maybe try to come up with a better suggestion more more interesting way to look at their metrics that you could shine with during your interview process overall just like think of yourself as a user of the platform and try to figure out all the possible things that you can do writing comments liking videos disliking videos

8:20 skipping around watching a lot binging maybe streaming it to your tv or watching it in the background now when we're talking about the content consumer here it's also important to mention that in case of our streaming platform we have paid membership so that user won't see ads or we have the free membership where you watch videos as much as you want but you will see like overlaying different kinds of ads on top of that so content consumer sort of metadata

8:52 is important to think about because that's some kind of different dimensions that you would want to split your data on it could be whether they're paid versus free it could be which country or region they're from it's quite commonly known that in youtube the advertisers for from different like regions pay different amount of money for people watching videos or like ads on top of videos from those regions so this will really help you to understand what are your key demographics

9:23 key areas in the world that you want to focus on and how much value do they bring to you as a customer demographics yeah of course it's very important to look into like also very valuable to highlight that you care about everyone getting as much similar experience as possible and you don't want to discriminate based on age gender or other kind of user characteristic that you can find through the data when users create accounts so kind of be aware of that and try to

9:54 prepare in your interviews to questions like do you think we can target different few different videos for men and women reason around what are your values how do you care about diversity and inclusion and how would you approach solving this problem in the most like uniform way for as many users as possible really important another thing is looking at the devices that they watch the videos from could be

10:25 operating system or mobile versus tablet versus web versus desktops like high performance top-notch iphone is one thing but a more entry-level cheaper less computing power small android device is a different thing and those users will have majorly different experience especially when it comes to such a heavy thing as video streaming so when you think about how they can experience the platform definitely

10:55 should be able to determine between those two just pretty much try to think about anything that could differentiate different users of the platform and reason around why it could be important for measuring the performance or why maybe it could be admitted now coming back to our sort of user experience kpi map looking at content creators it's really important to have a nice way of like understanding the performance of your videos so first of all you should be able to

11:26 clearly upload the video just write a description write tags make it findable and searchable and relevant so pretty much feed into the algorithm that determines whether your video will be shown and watch later or watch next or in search results even be able to kind of set it up in a way that it works for you and you are findable so video upload shouldn't really take a long time because that's kind of also fades into the user

11:58 experience so even if they are not really great with seo with search engine optimization if they're not putting all the right tags they still should have a chance to show their content to the people who might find it very interesting for them and that comes from like the mission statement of the company usually they should have like transparent analytics around the performance of their videos so analytics and the most common like metric or like the overarching goal for

12:26 a company usually is to same as with the content consumers keep them on the platform to keep them active so there are metrics that you might want to look at during looking at the experience for the content creators like number of active content creators today number of video uploads today number of minutes uploaded today number like duration of time it took for a content creator to upload the video maybe you see a gigantic increase and you see that there is an

12:56 issue in your server infrastructure or the way you handle the upload so you can look into that deeper retention of the content creators how long do they stay active or how long do they stay on the platfor how frequently they upload the videos how much do they answer to their comments how much do they like the the comments that people have how much does the performance of the video in terms of like likes dislikes watch time affect the content creators

13:23 lifetime on the platform all those things again try to brainstorm as much as you can like think of yourself as a content creator and then think of yourself as i don't know a data scientist already working in that company who needs to understand why content creators are leaving and try to figure out how can you measure pretty much why they're leaving last but not least question is what is important for the advertisers on the platform of course they will be they should be

13:52 able to understand how the videos are performing so that they know what kind of reach of their ads they can have they should be able to determine what kind of videos are suitable for them to put their ads on how can they pretty much target different people with different relevant advertisement opportunities everything again that you can think about when you're thinking about like an advertiser that wants to show the ad of their i don't know christmas

14:20 chocolate package to a user that forgot to buy last time gifts and just searching for something that they want to buy for their family for christmas like how can this advertiser of the christmas chocolates can reach that user everything that can connect them we can again look at the metadata that we have about users like demographic countries some countries may not celebrate christmas so they shouldn't see ads for christmasy stuff for example

14:53 like everything that you can reason about when you're talking about various metrics that are important for the performance of the product for the advertisers there are some specific metrics that could be measured for example number of clicks through or the click-through rate for the advertisement from the video so number of times out of the thousand times someone seen this ad how many times did they actually click on the ad then they can have a link that

15:23 is related to the specific video or that the advertisement that they put in the streaming platform to be able to see in the end how many users went through and bought the product so everything that you can think of that is important for advertisers try to list it down and reason around why some things might be really important some things are less important this is a really brief overview of how it would go about brainstorming a kpi or metrics map for a

15:50 product that i'm working on or maybe a product company that i'm interviewing for i think it's really helpful to do that before your interview process because doing this brainstorming like beforehand and thinking through all those problems and potential issues or weird things that you see that might arise will help you to answer questions like how do you measure the success of our product or this new feature that we're releasing or how would you determine whether this

16:16 drop in user engagement is important for us to look at how would you try to break it down how would you measure what parts of the user experience are impacting this user engagement drop like less streams or lower daily active user numbers try to brainstorm that try to write it down maybe do a mind map maybe list those things in the list but just like try to think about the product as you already work there like as if you were already a

16:47 product data scientist or product analysts working there trying to like internalize the problem and think from the inside of course you won't have all the knowledge and understanding of what kind of metrics they have what kind of technical capabilities they have but those are things that you should also stay during entry is your assumptions that you make that for example they have a certain kind of tracking or they have this certain technical capability really important i think it will help

17:15 you to ace your interviews much more and in my experience that was a thing that not a lot of people actually do they're trying to like do it during an interview process and kind of figure this out on the spot which is not particularly helpful for many people so maybe this will be your competitive advantage when you're interviewing for a product analyst or data scientist position thank you very much for watching let me know in the comments what you

17:41 think about this video whether it was helpful or not and what kind of questions you have around like determining kpis and metrics for product performance yeah like it if you like the video please for the sake of the algorithm for me and have a nice rest of the day you