Transcript
0:00 hello mark hey dan how are you i'm good are you i'm doing well thanks for having me yeah thanks for partaking as a mock interview candidate um so for those who are watching uh mark is a um a currently an interview candidate for the meta um data scientist analytic physician along with other um startups and tech companies like chin um and amazon and so um we're gonna have
0:30 mark basically go through some of the product sense questions um this is not gonna cover basically the entire like 30 to 40 minute you know product sense questions that are asked on the meta interview itself rather it's more like a kind of like a sample in terms of you know what kind of questions you could potentially be expected in you know these analytic based um questions that meta but also other products sense interview around the different companies as well
0:58 um and lastly this is a glimpse of you know what are the premium content that's available in the monthly subscription course um currently right now as of right now there's about five recordings of these one one-hour mock interviews with um questions and solutions but also uh this is a glimpse of what a coaching service looks like in terms of you know if you were to pair up with me um what what i could offer in preparation for your interview and um
1:27 you know the prep being highly personalized based on the companies that you're interviewing for uh so without further ado i'm going to go ahead and start the mock interview around um basically the process itself so i'll go ahead and um ask you the first product sense question which is the following um so so mark hopefully you've done some research uh you know prior to this interview about uh meta's portal device
1:59 um could you figure out ways to kind of measure engagement on meta's portal sure so um before i jump into the like a brainstorm matrix and like a plan to take all this business question i have a couple of clarifying questions i have um i think first of all is like uh like what what is like business purpose of measuring the engagement of this mata portal device
2:30 and then like how does that tie to the overall um meta mission or like a business objective yeah that's a very solid question um so basically um we just want to kind of understand whether you know meta is like a viable business product um you know in terms of like user engagement um are like how active are um you know basically customers of um portal is um and so
3:01 uh so we want you to kind of come up with you know potential metrics that can help us you know are users um you know are they engaging on this portal device or not gotcha gotcha and just to make sure i understand the product correctly so as far as i know uh like a meta portal is a like a hardware product that enables users to be to do like a a video course with better sound and video qualities
3:33 and also it has alexa built alexa building which you are able to watch news or schedule remind reminder and also play your music on the portal so yeah on is there any other feature i need to be aware of about this product i think that pretty much covers the core functionalities on the uh on the product itself ah yeah thank you so then uh like a i think i get in order to make sure
4:05 the uh engagement on my portal device uh you know i think they are like a i think there are two diff uh two goals first like uh for the user goals um i can type it here how do you measure engagement uh let me let me first like a um uh supervisor why the major engagement is important because at facebook let me insurance to give the power give people the power to build a community and also bring a world
4:34 cluster so this product can serve as the uh a tool enabling a more user to stay engaged with mata platforms and then so i think like for a years ago you can uh by using the portal or they can like do more um uh do more video clips with their friends or their families which intent which would also increase their can spend on uh on the uh um meta
5:04 products like what's app and message us and then from business goals i think like a engagement like a um we can see like how our users keep coming back using these products and encourage them to stay engaged on facebook and so i think like uh like a potential impact is like once we increase the user test span on our platforms which will in turn uh increase and gross our revenues
5:34 um for meta okay cool um then so i think i can i can bring this down some uh my trick around the uh around the meta portal device i think measure the engagement um i think first of all um i think so i just have a wi-fi question so we are just talking about the user who already bought the product devices right yes status quo
6:06 so i think now like a uh like a couple dimension uh we can look at first first is the like a uh hyperspace activation activation would you like like a how many people really after they purchase this product how many like a login using this device on the device on total on portal and also like a how many like um like a how many
6:38 how many people really like a um login i think login is activation and also start uh start like a inter like a using the feature on the device so maybe like a star number of like a video calls and then number of the uh like a uh also uh what else i think um
7:08 a video calls with on my video we already have number of the video calls um like a what other than video code that like we talked about they can use this portal to like a set up uh their calendar so i also want other like interaction um they can like a like a like a meeting setup meeting setup or like a music play um play some play
7:41 etc so basically it's like it was there like an interaction which would look like and then next next part here i will look at the like a um retention so um when i say uh when i say retention is about like a uh like a daddy ip user weekly ft user and um and monthly active search so so other than that so but we also need to define what what does the what is
8:11 active nameless of the users um like i usually uh if we can look at what's the frequency a user do a video call or if if the frequency is about like a two or three times a day maybe we can look at a like a two weeks window to define whether a user is active or not and then also i will look at the ratio like a dau on wau ratio this will be served as like
8:40 a to measure the techniques and stick techniques on like a how how many users come back to us like after a week or two weeks so that's what i will be looking at and then uh another another dimension i'll also touch on is the revenue because at the end of the day on the the from the user goal we we try to in um uh have more people uh using this
9:13 product so that they will spend more time on the portal which like is the opportunity for us to display some ads or like embedding some sponsor accountants when like a uh when they interact with the user that will also like from a revenue perspective i will look at like a average like a dollar like a like uh on
9:43 monitor producer and also gen every dollar as revenue for users and then also like a look at um like a customer like 10 value etc so yeah and yeah uh i think it's just like a three main category i'll be looking at um is there any uh particular bucket you want me to talk
10:14 more about um no i think that pretty much covers um uh you know your you touch base on uh pretty good categories of metrics activation retention and revenue um now can you kind of pick out maybe like three metrics that you think would be important so suppose that you need to create some sort of a dashboard for um you know let's just say an executive right um who's part of the device and hardware team right and they're gonna be
10:43 the ones that are looking at this along with other product managers in that vertical in that horizontal um you know basically the business of the hardware within meta um can you think about what three metrics you would want to display and then rank order those metrics from the most important to the least important gotcha so uh let me let me reframe what i just heard to make sure i understand the question correctly so basically assuming that we are building
11:09 a negative analytic dashboard like empowering our business holders it can be product manager or executives to measure the effectiveness of this product so on like a uh the question is about like picking three main primary metrics that can like a help them monitoring the metrics over time um so cool um let me take a quick minute to organize
11:39 my thoughts um yeah i think [Music] so i think the first and then ranking in the importance based on the importance level so i think uh the first um i didn't know what um i think the first primary metry i will look at is the video uh video like a call time
12:10 in in minutes in hours um per month so because like as you mentioned the audience might be like a leadership or a security so like they will probably care the more about the like a high level overview how much i get total time uh they spend on this product over time and then also to see whether um like they do more uh video calls uh with uh this new launch product
12:40 uh so like a um and i get this another reason why i picked this machine is this make sure also embodies all all the other like interaction like we talked about like a we do a video course or or like a set up meeting playing lucy so like the more activities they uh took um with this device no more time spent on so it's actually not that a video call time i will be
13:12 like a little bit um like a time spent on hours and uh sorry type of spam in all around on photos um so yeah and then i think the secondary matrix i will look at would be because like i think one of the core functionality with portal is the video call so i want to like know what's the
13:43 total number of the video calls per month so total number of video calls per month um i think this will on a day so we uh the reason for that would be uh i i would like to know whether uh uh like a whether um like a with these new products uh users are easier to do a video course or they will um
14:16 like jump on the video call with their family or friends more frequently so uh and then lastly uh i will pick one from the revenue so um and for the monetization perspective on like a like a revenue per month so uh i think that will be like a um i'm not quite sorry i'm not quite sure about how the ass will
14:47 embed it on the portal like a fun face from the meta news feed we know like it when you go through the post the no you will you will probably see some like a sponsored ass but i have a portal on like a could you like it give me some uh direction about like a how the from the advertising side how that would be embedding within this product or yeah um yeah so um in this case let's just
15:18 assume that there's really no like advertisement that are displayed in the portal device itself um so i wouldn't so just kind of going back to original questions which is just focusing on engagement um just leaving that monetization aside can you think about perhaps any other metric that you may want to display in this metric based dashboard sure so so leaving a monetization site then i will um like a doula on like retention like a diu
15:50 and user i think this is like for all the uh meta products uh so we care about like a daily activity so weekly i've used them on the active users so like they're having this um actually like having the active users uh on the dashboard will be give give our audience a better understanding of how how our uh how our products being used by the users and then their active new
16:22 needs so yeah does that make sense okay got it got it now how do you so among these three metrics you have outlined um how would you go ahead and prioritize them from ranking from one to three sure so far name ranking one two three on us per month um i think uh let me move the daily attribution to the first one um um so
16:53 i think the first that i think the problem images should be that you have the user with three user monthly active users and then by different like a segmentation by a country region all different uh like a language etc and then like because like um we have to the reason the main reason for that is um this will serve as the uh retention uh metrics like we just we
17:26 don't want just like a launcher product and user just use once or twice and then put that put that away so that this will be served as the measure to know how like it what's their retention look like um so this will be my primary primary matrix and then secondly as i mentioned the time spent is also critical as that it will like tell us about how like a um how much time is spent with on a portal and then like a
17:56 and in a and then what's their interactions um and yeah and that's least total number of video calls yeah just damage okay okay got it um all right now i have a new question problem for you the third question i have is following suppose a new feature is introduced on portal which is called call suggestion this future recommends calls to users how would you design an ap experiment on
18:28 this feature um suppose a new feature introduced in portal uh which is called code suggestion you just design um so uh so the question is about how do we really um like design the uh experiment i think there are a couple of steps we need to follow when it comes to
18:58 the experiment um first like they we need to state the uh hypothesis testing uh oh no i think first of all we need to define what was the success matrix like uh what are um like a what are we going to make sure uh okay um when we launch this new feature so i think the step one will be okay what what will be this assessment should look like
19:29 so matrix and then like a i assume that um with this new features the purpose is to make users more easily to uh start a call start a call and then which will better the number of the calls will be in work will increase accordingly so i will i will pick uh um i will pick up average number of video calls per users
20:03 as the nature we are going to make sure does that make sense or yeah that's good so average like a number of video have calls on per user and then secondly uh like a uh so like we need to define let's state the hypothesis testing so all i think the number uh here the null hypothesis will be uh the average number of the
20:35 core per users um between the control and treatment group will be the same and alternative hypothesis will be the um um the average number of the uh average number of cost per user between two groups out on are different and then like we need to follow by uh we need followed by like i define the uh significance level uh statistical power and then the mde to determine how much uh same process uh
21:07 we need and then um so let me let me put it down like i say the five pounds like a significant level uh power and the mde so alpha
21:37 alpha like a power and like a md and then and then and then also we also need to determine what's the like a randomization unique so i think this is kind of like a um again or like a uh because like i will assume like in this setting the observation you need is the customer so uh like it's better we can we can perform a randomization at the
22:09 customer level and then and then like a another uh also we've just talked well um we just focus on the user who own the portal device right otherwise we could we could like measure these features the effectiveness of this feature so i will be like a customer who own like a portable device yeah and then once we have that like a
22:40 we are able to uh and again estimate the on like a experiment [Music] xp re augmentation [Music] time so um usually like a typical like a experimentation time in every testing on like usually it will last for like one week or two weeks on the reason for that if we if it is a short experimentation then it will
23:12 probably be problematic given there's a um like a day of the week effect unfortunate for instance like a users tend to do a video call on the weekend or on a weekday so uh like the best practices will be usually um do like at least more than two weeks um so here i will use the two weeks yeah and then after that um
23:43 how would you design and then after we gathered say two weeks of the data then we are able to uh use the um um uh two symbol t tests to to make sure the uh the leaf between the leaf in the metric uh across control and experiment and the treatment groups are actually achieve statistically uh significance um so
24:13 that's like and by checking the p value and so um [Music] yeah and yeah and then after say given that the like a p value is less than 0.05 0.05 and like it it will be like a to decide like whether we should launch this new feature or not
24:44 but like it's not it's not just um feature oh um by the way uh like when we uh something i didn't mention about is that uh we other than just pick one success symmetry it's better we can also have some of like a gar rail mattress which is like a geometry which that which shouldn't be degraded in pursuit of this assessment tree um so uh in here i will use the um the
25:16 um i think like as revenue or 10 spend um portal as the gabriel so like a like as a career metric but like a um on the last step is to launch the new feature or not on and other than checking the statistical uh significant result from experimentation we also need to involve the business holder to see what's the um like a practical significance look like and
25:47 also do some uh like a cost and benefit analysis to make sure it's gonna likely take too much resources to launch these new features um so now that would be the next step yeah does that make sense okay sounds good um and you so you define some good structure there but um but in terms of uh are there any other details you may want to populate uh any other detail um
26:17 so i think like a um i think i will add more details around um like this closer i think um that sounds like a we need to also do something like a like a um velocity checks for example uh to make sure that the um like a um observation we gather should be independent uh with each other and also whether there
26:47 sounds sensational seasonal or holiday holiday effect and also another effect i can think of is like novelty effect which means that when the new features uh uh on a product appears uh given its novelty you should probably may respond to this feature positively at first but like but over time uh the novelty of the feature will waste off then the behavior will return to normal so we need in that case we should probably uh extend the like a
27:19 experimentation to get more data to validate that so yeah okay got it i have a question for you so what do you think would happen if this change was made over a let's just say christmas weekend um uh over christmas christmas christmas weekend yeah uh yeah then i think like a uh if we like a uh lunch experimentation during the holidays like christmas then we will probably see
27:50 the uh we um like a the final result is um like a kind of like inflated or overly uh optimistic because like people um especially during the kobe most people cannot go home to like they have a family reunion so like they will probably do more video calls like a during holiday so in that case we will probably gather inflated uh average number of calls per user like
28:22 during this during this time so uh in order to um um um to get a better reliable result we should probably um um like a alright we should probably uh gather more data and like because i'm assuming like um so yeah so probably like extend the experimentation time um
28:53 yeah does that make sense or okay okay all right so this is the end of the question portion and and obviously you know um you know i would have wanted to ask you more follow-up questions but you know once again this is more of like um like a simulation exercise right so if we could do the actual mock interview then there'll be a lot more company sent than this um but i just want to go through um each question at a time from
29:23 beginning to the end and we'll talk about you know what you did uh well and then what are some areas of improvement does that sound good yeah that sounds good to me yeah um and and just kind of before i actually dive into that just i would say just overall i think the way you were approaching each question um your communication was good um it was very clear to unders it was you you expressed your ideas very clearly so it
29:48 was very easy to sort of follow through what your um reasonings are right on each of the um questions that you were addressing so that's really good that you were doing that um the second thing i i liked about the overall approach of how you were pushing the problems were that your solution styles were very structured um so you didn't just kind of ramble you had a framework in terms of you know how you're going to respond to
30:12 each of the questions um so that made things very easy to follow um as an interviewer um and that's really good that you were doing that um but i would say that there were there was one thing that i i would kind of um refine your response because i noticed that there were some moments when you seem a little bit hesitant or unsure about whether your response is correct or not um i would try to refrain from doing
30:41 that um because you want to project confidence and if you're not sure about certain things um just don't bring that up at all so for instance you asked me about you know does meta is there a way for meta to generate um revenue like are there some advertisement display right um uh when you're not sure whether there is a particular future that already exists in the product or not it's best not to just bring it up um because if you bring it
31:11 up then you can expect some follow-up questions on that and that can cause you some troubles later down the road um those who are watching this um that's actually one of the key things that um a lot of candidates often kind of make mistakes on you know they read some techniques off the internet um you know and then they they'll basically just kind of rehash it um and talk about it during the interview and because they don't really know like the
31:39 actual reasoning or they don't know the exact details behind why um you know that technique taking is exists or whether a particular feature on a product exists then you know they they just sort of bring it up but then like when an interviewer asks follow-up questions they're not able to justify um so it's best not to bring up something if you're not really sure okay yeah i think that's a great quote yeah um so so projecting more confidence
32:05 i think that's something you really need to work on um because in each of the three questions i noticed that you were kind of hesitant in terms of your response um and you really want to be able to project uh certainty in your responses um rather than hesitations all right so that's just a general overview of what i was kind of gauging um off of the uh you know off of how you're responding now let's go through
32:31 each question on each question at a time so starting with the first question how would you measure engagement or meta support of device um so i think i i really love how you are approaching this with a good framework activation retention revenue however though you were kind of missing the main focus of this question which is engagement right if i would ask a question about how would you measure success on meta meta portal device right then that might be
33:00 um accommodating for activation retention plus revenue in this case but then i'm more talking about the engagement so it's not so much about the monetization i care about it's more about the retention here right um activation activation being the secondary thing and the primary thing being retention um and so i would have loved to see more focus on these two categories and um it would have actually been fine if you just left out revenue because that wasn't the main focus of the
33:30 question which is engagement um so that's something um to kind of look out for um really understand the differentiation between when um when anybody is asking how do you measure happiness measure satisfaction measure engagement measure success these fours sound the same but to on certain extent they're a bit different um and so make sure you have a clear understanding of what the question is and you're not just listing some arbitrary set of metrics but you're
34:00 actually said listen metrics that actually answer the question okay yeah sounds good um oh sorry can i ask the question yeah go ahead good uh so like a um did you have other suggestions on like a um um brainstorm the make brainstormer matrix when we were asked about the question because sometimes i would be stuck on like okay this is the other
34:31 this is all the measure i can think of on top of my head but i don't know whether um or less sufficient or like what will uh like when with brainstorm make sure it will be like a fair number of the magic which you bring it out in an interview yeah so i would say that just start with actually so that's a really good question um uh it's especially common questions i get from clients all the time but basically the ideal i think
34:56 instead of the way you should be addressing these kind of metric based questions is um think about you know two to three categories of metrics um so you kind of did that here and then within each category it lists two to three measures so this or itself already provided a good decent comprehensive set of metrics yeah thank you um and and when it when it comes to these type of you know questions about um you know measuring um measuring a
35:28 future or product um a natural sort of following question is like you know how would you prioritize which metric to focus on um so that is expected you know um so this is more like a shout out to those who are prepared for these data scientists you know product data science interviews um when you're when you're practicing you know yourself to just kind of make sure um you know think about how to measure a product but also when you list these
35:56 metrics think about how you would prioritize them based on different vocals um so so that's why i asked the follow-up question which is you know how would you pick out these three metrics and then how would you rank order them right um and so considering the fact that you know really the main focus is engagement um i think it made sense to focus on the um active users um set of metrics across these three three different windows
36:22 the only thing i wasn't really too clear on and i think this is something that you sh you should have clarified a bit more is the definition of of active so what does what does it mean for a user to be active and that's something that i i notice um clients will kind of gloss over which is that they will just spew out okay you know i'm going to measure dau uh w-a-o-m-m-a-u right but then they
36:45 don't clearly define what are the behaviors that actually define actions um and a key action can be it can be a it can be a set of multiple behaviors or it can be just one behavior right um and in the in the case of the portal um you know there's all these different primary functions right you make a call um you know you i'm sure you can also uh you know browse on a web application as
37:10 well right um you know you can uh you can you have this calendar app you can set up meetings and stuff like that right uh so these are all these different type of actions and so you know do you want the action the key action to be defined based on um calls only or do you want it to be expressed based on a set of actions right um so that's something to clearly define in this case
37:36 [Music] and i think i think in this case um you know even though portal one of the cool portal's primary function is like the video calls um it still doesn't mean that you know some users might just use it only for calendar purpose right um and so and so in that case um you know i would just have like an inclusive set of okay um you know if users are active in the form of any sort of
38:02 app engagement whether it's on a calendar whether it's checking email whether it's um you know making a call um any set of like in-app engagement um then that that then i would consider that as an active engagement gotcha yeah all right so now let's look at the av experiment part right um so even before you talked about success metric do you think
38:33 there were there could have been one topic uh topic that you could have flesh out with the more on i think one thing uh i think one thing i was missing is probably uh i didn't ask like clarifying questions around why why this is important and how this feature tied to the overall business goal so i just i jump directly into a framework
39:04 without asking yeah that's precisely right yeah so for those who are watching this um this is this is another sort of common mistake which is like um candidates have a tendency and i think it kind of makes sense because a lot of what we do are very technical focus and so we have a natural tendency to produce a technical response from the get-go um but we have to keep in mind that um you know data science project from the
39:29 beginning and also the end right are always business focused um so we have a business stakeholder we're working with they provide a context of what their needs are um and then we basically provide some you know we basically understand it in terms of you know you know what the business context is translate that into some system technical methodology and then we deliver it back into the business right so we also have to think about it in
39:55 terms of okay whenever we need to provide a response to these type of um you know product based questions the the structure is always business technical and then business um and so in this case what hank did was he primarily focused on the technical part it kind of glossed over the business elements in the av experiment itself um and it's really important to just also just cover the business part of it as well because because you know because
40:24 the interviewers sort of gauging you know if this was an actual project that you were actually performing um you know in real life like how would you actually you know deliver this right and so you want to provide a glimpse so it's really important for you to cover the business aspect of this question as well um and so yeah like focusing on you know what is the purpose of this experiment right but also focusing on um clarifying
40:50 what this new future looks like you didn't walk through the user journey of what the current um process is and you didn't also walk through the experience of you know what this new future would do um and you sort of um yeah like you know basically just kind of overlooked it so skipped it over right um so just make sure you kind of you flesh it out and in this case the call suggestion would essentially be like a ui and so
41:15 you know you could kind of imagine you know if you um go if you go to like a call list right it just looks like an endless scroll of all these different numbers right so instead of having having to go through uh scroll to find the number that you want to actually dial um you know imagine like there's like a call suggestion at the very top maybe it's like top three calls um that are suggested to you
41:39 based on the likelihood that you would actually you would your intent is actually to make uh make a call to that person right so imagine what the benefit would be right um maybe it could help save time maybe it can also improve engagement as well right um and so fleshing out the context um are just as important as covering the technical aspect of it as well so once you paint the picture of this from there um you know hank what you did
42:08 right you know what you did in terms of laying out the structures are good yeah yeah i think that's a great uh advice and i want the most commonly uh mistake uh like i i sometimes will uh ignore i'm not gonna lie i forgot so like before we jump into any solution or like a brain like a provider framework we need to uh stress on like what is the business purpose of this question and also uh go
42:40 through what's the user journey look like when they interact with this new feature uh like it provides some of the contacts around from the business perspective i think uh it will make the answer more complete so yeah i really appreciate it yeah yeah of course now i do want to walk through a couple things here um so i think you laid out a good um procedure of running an av experiment but i felt that there were a lot of there were a
43:07 couple details or someone missing there um so okay so success metric the average number of calls per user you know that um i i i like the idea so i think that works um um because this metric wouldn't necessarily violate the statistical assumption that the observation is independent in this case right you're aggregating at the user level and so that's why this metric would work in terms of stating the hypothesis so i have a question for you um
43:37 could it be possible that just having some ui with arbitrary arbitrary lesson numbers like that itself could have an effect um yeah yeah so let me just rephrase that so so basically um if you if you think about the call suggestion right there's really two elements to this right so there's the ui and then there's the algorithm
44:09 right so when you're displaying this at the very top how do you know if there's an actual effect right meaning that people are engaging a lot more because of this um ui right how do you know if it is a ui that is actually driving the engagement or it's the algorithm itself that is driving the engagement or the interaction of both gotcha so we're talking about like like a confounding variable yeah essentially right
44:41 yeah yeah so how do we really um so [Music] yeah so so in this case this would essentially be um kind of like a multivariate testing so your your control is that okay here's a version without any okay and then second and then the second test is basically um you know uh just a ui with an arbitrary like list of numbers uh you know list of calls okay and then the third one is
45:13 uh ui plus the algorithm itself gotcha like a motor so so it's not it's not just like a like a two groups like a game um i see three groups so it could be three groups uh one control um you know one treatment treatment number one and treatment number two yeah and gosh like this is like a ui and the second one ui plus
45:50 you know essentially you would be doing uh like a pairwise t-test so you're comparing the control against t1 and then comparing the control against t2 and then you look at the relative life along with the system significance yeah and then you make the decision that okay hey you know what um this algorithm you know uh basically this call suggestion with the ui plus algorithm um is so significant and uh you know with the relative left being
46:22 positive the positive direction right um and you would uh conclude that you know essentially um you know you would want to rule out the call suggestion to um to a larger population of the portal device users gotcha yeah that's something i did i didn't think of when i when i was asked about this question like maybe like when we when we we uh talk about the ui change we also need to think about uh whether the like a
46:54 confounding effect from other driver like algorithm itself can also make an impact so how do we really validate that so yes it's about observing the sensitivity of the metric right so um when you make certain change like how do you know that the change is coming from um uh you know from from you know changing the metric is coming from basically what like the condition they have changed right um you know or is there some sort of
47:25 like uh extraneous factors that are in the play here right so it's just a lot about being a bit more keen um being able to kind of differentiate you know where is the effect really coming from yeah um and in terms of defining the statistical parameter alpha power md i would have loved for you to actually flesh them out instead of just saying okay i i will go ahead and list off a power name right so
47:55 so you could have cited like uh you go ahead i know i so i asking so uh maybe elaborate more on each of these three uh components like what was the significance level and then how do we define the the statistical power or etc or because sometimes i feel like it's kind of like a common sense that like we need these three to determine what the sample size needed but really then touch on the details
48:26 about each of these yeah um bring it up bring it out anyways yeah cause the interpreter doesn't know whether you actually know it or not right really the main focus here is to prove provide evidence that you do know how to run an ap experiment and so uh and so i would not um you know i i wouldn't i wouldn't i wouldn't avoid like i wouldn't you know like basically like bring up anyways right even if it's common sense per se
48:53 or even if it seems obvious like um you know so it's so really important to bring out these details gosh yeah um so in this case you know you could have just started off by you know citing some industry standard numbers right obviously that can kind of change depending on the type of test that you're running and what are the type one and type two risks um you're willing to tolerate um you know when you're designing and running these type of
49:22 experiments you know but standard set of values like alpha being set at like point zero five power being a point eight um and and md um you know being somewhere between one to five percent in this case um actually let me truncate that value to like one to three percent um one to three percent would have been reasonable um and of course like you know you might expect the interviewer to ask you some additional follow-up questions about you know
49:51 um you know like why do you like okay so putting outside industry center values like you know why can't you set the alpha at point one or power at point seven or md like at five percent or something like that right um in mid case then you know you can kind of follow up um on you know different values that you may want to explore gotcha yeah um yeah yep um yeah and uh randomization unit
50:23 makes sense um estimating the experimentation uh basically the experimentation time set of two weeks makes sense um the only thing you forgot to mention is like how would you determine a sample size um so yeah so you would have you would have had to determine a sample size after step three um right yeah on determining uh yeah and you also
50:54 it should come before determine the time right yeah once you know what the sample size is then from there um you uh you basically can figure out what the experimentation time partners yeah yeah yeah go ahead sorry yeah and and um do mention the statistical types you would be using in this case um yeah so i was saying the two sympathy tests
51:26 yeah yeah because we are measuring the like a mean value across like a multiple groups but in this case right now we have three groups so maybe we should use anova instead no um so anova would only give you um if at least one um at least if at least one pair of the groups are different but it wouldn't give you information uh
51:57 sorry if at least one group is at least if at least one group is different from all the other groups that's what donova will provide it wouldn't necessarily give you um information on um which pairs which pair of groups are actually different right so um so so in that case and so in this case you the type of um status quo test you would actually want to do is pairwise t-test comparing two simple means
52:43 yeah i think yeah i'm mostly familiar with two groups like uh like a pairwise symbol t-test should we should be using this um scenario so yeah yeah um yeah and the last point on launching the new uh future i think you covered the essence of uh you know what what to do on that sense um yeah so overall you know you did a good job uh you provide a good structure in terms of how you responded to these questions um
53:15 but really i think the main focus for you is that just make sure you're providing details that really matter to the questions and you're leaving out the details that don't that doesn't really matter okay um and so as long as you basically just continue to practice um you know come on these mock interviews with me and you know and basically just go through the uh practice questions in the subscription course then you know i think you'll be uh well prepared for the
53:41 um uh the product interview at meadow thank you yep um great well i um i enjoy this mock interview with you and i look forward to more sessions like this with you in the future yeah thank you everybody appreciate your time yep sounds good all right thanks mark bye bye
Summary
- Mark discusses the importance of measuring engagement to assess the viability of the Meta Portal as a business product.
- He identifies key metrics for engagement, including activation, retention, and revenue, emphasizing the need to define what constitutes an "active" user.
- Mark suggests three primary metrics for a dashboard: daily active users, total video call time, and total number of video calls per month, ranking them by importance.
- He outlines the steps for designing an A/B test for a new feature, including defining success metrics, formulating hypotheses, and determining statistical parameters.
- The interviewer emphasizes the importance of clarifying business goals and user journeys before jumping into technical solutions.
- Mark is advised to project more confidence in his responses and to provide more detailed explanations of statistical concepts and methodologies.
- The mock interview highlights the need for structured responses and a clear understanding of how to measure product success and user engagement.