Instagram Data Science Question - How Many Ads? (Full Mock Interview)

0:00 assume you're a data scientist at Instagram who is interested in balancing ads versus follower posts on Instagram what would you do and how would you monitor its [Music] Effectiveness okay thank you so much for being here with us today waywin uh would you quickly introduce yourself for our viewers no problem so I'm W I'm a product data scientist uh previously at house uh I work for a building dashboard uh for business intelligence and also uh

0:31 establishing AB testing uh experiment design for cost inference relationship finding so let's pretty much of it okay amazing so that's a lot of really relevant data science experience that I'm eager to hear about today um okay so let's get right into our question our question today is assume you're a data scientist at Instagram who is interested in balancing ads versus follower posts on Instagram what would you do and how would you monitor its Effectiveness m so the first thing let

1:01 me clarify so by what do you mean by uh balancing followers post and also ads I was assume it's like uh finding a balance between the frequency of post and also the ads is that right yeah so the frequency of how often ads versus posts are shown on your feed MH cool sounds good okay then let me proceed so uh uh what I heard right now and what I would do first is to have

1:33 measurement to measure the performance on the ad side and also the performance on the followers post side so I would say for followers post is more like an engagement of the social media platform so the metrics I'll put for the engagement is like daily active user weekly active user monthly active user depends on different frequency and also the time onside of layer session uh meaning when they open Instagram how long will they use it and also there can

2:03 be some other metrics like likes and also following uh subscri uh following user but let's put uh weekly active user and time on side for now on the engagement side okay and then on the S side um what we want to know is like the impression of the A and but the more we focus our would be like K through rate of the ads meaning how many people actually click the ad when they see it and also the revenue we're making from

2:33 ad I will normalize it back like average s Revenue per user meaning in terms of every user we are having on our platform what's the uh average as Revenue we have in from them so far does it sounds good to you uh yeah that sounds great I like that you were able to discuss a variety of metrics for both engagement and ads cool so with the metrix defined what I want to do right now here is I want to

3:03 set up an AB testing but with more variant because our goal is to uh find a balance between AD and also engagement so what I would suggest is for example if we our current frequency for ad is every bip poost there is an ad then what I would suggest is let's our control uh uh control version of the AP testing and we can have two treatment one is every three post One S and every seven post

3:34 One S we can uh change the frequency based on our historical data analysis so basically here what I'm proposing is increasing the frequency and also decreas in the frequency using this way I'm trying to see how users reaction toward the frequency change cuz uh I'm proposing the hypothesis that user might found different frequency have different reaction so I'm using this a testing to uh have metad da reaction toward to

4:08 it good so far so that's compared to the control which is the current level of ads right yes exactly okay sounds good cool so from here I'll try to uh use the Whiteboard to like visualize it a little bit and also uh I can also draw some chart this sound okay yeah that'd be great uh feel free to share your screen so on the engagement side uh right now we have like uh what I say I'll try to

4:40 simplify so I put weekly active user for now and on the S side let me use the uh average as Revenue per user right now so what I'm trying to do uh meanwhile when we do the AB testing is that um ideally in terms of the sales uh if we only talk about the sales in ads

5:12 it's basically like weekly average user weekly active user uh times average as Revenue per user that's our like weekly Revenue so I'm assuming like for the xaxis is the engagement MH and on the y axis is like the ads so ideally when we test it we will have like different result like um for um a version B version and C version you might have

5:44 different uh distribution so basically this is helping us to learn more about users reaction and how we can balance between AD and also engagement so ideally through this AB testing we can know like users reaction and analyze the result afterward so using this way um I think how the team is trying to do with AB testing is we need to find a way that we can grow uh from the top left uh bottom left to the

6:15 top right uh cuz um this is the uh direction we want to grow the business because we get more user meire getting more uh as revenue from the our user so we don't want to uh sacrifice the as too much and we don't want to sacrifice the engagement too much using this way we can have right direction how we can improve and also balance between them okay yeah that makes sense cool so um now with the experiment setup like we

6:48 have current version of the uh Edge frequency and also decreasing frequency and increasing frequency um randomize the user into the ab test and also set out a hypothesis we can also have some AB testing uh parameter setting up like Alpha Beta And minimum detectable effect uh if you want me to elaborate more I'm that we can talk about it later but overall setting up the AB testing we test the results

7:18 see the users reaction and then based on the uh uh AB test result uh data we can analyze how's the impact on the weekly active user and also the as uh Revenue per user and seeing um like the P value and uh confidence interval how we can uh share the results with the team analyze the result and providing suggestion like user sensitivity when we decrease and

7:49 increase the frequency um so we can have more sense to understand like if we want to increase the as uh frequency uh how much weekly active user is actually tring negative or is remain the same or when we decrease the frequency how is impact to as reach as Revenue okay and it sounds like you're doing this over a fixed period of time right how would you measure the impact over like a longer period of time to

8:18 make sure that it's not like um a novelty effect or some sort of network effect yeah so once we have sense how to balance and we decide to launch one of the version uh what I would do is that I will hold up like 10% of the traffic and put it into the like still putting into the control and while the other 90% are in the treatment using this way I can observe for longer uh period of time and

8:47 meanwhile because we think the uh version we are launching is actually bringing more uh revenue and also more benefits and the holdout group we can uh keep observing the trend so what we I'm trying to do here is that um for example uh if the X AIS is the time and the y axis is the let's say the Matrix we are pursuing let's say the sales like the weekly active user times

9:18 the average ads um what I'm assuming is that for the treatment version it might Trend like uh better while for the control version in my Trend a little bit below the trend because in the between um in between this is the uh average treatment effect we are observing uh during AB test let's say for example a trend positive for 5% so this 5% should

9:49 uh be consistent throughout the time like for a longer time and if we are seeing the uh average treatment effect is actually decreasing that means there might be some nov effect that users start to realize in there are seeing too many ads or um there we get more active user where we make less money on the average as Revenue so we can using this uh kind of like holdout method to observe if this trend is uh persist

10:20 while we put it into a longer time okay sounds good and and what if um maybe in this like long-term holdout group you do observe this novelty effect where like over time the users do notice that they're uh seeing more ads and then they engage a little bit less what would you do then so with that let's say for example if we observe this uh like decreasing every treatment effect from let's say 30 days or 90 days using this

10:52 way I can realize okay so 30day might be the like how long the novelty effect will persist then I can use the users data from like 31st day to um like forever using the data with less bias and also less noise like the impact without novity effect we can do the analysis again to see um how's the impact right now and also like using the

11:23 same methodology to analyze um this new frequency of ads how's the uh usage reaction in terms of the engagement and also the revenue of ads so you uh basically it's like to use the hold on method to identify how long if there is a novity effect and also how long the novity effect uh it is so using this way even though uh the previous average treatment effect we identify is not uh

11:54 validated as before now we need to update the data but we know know how long uh how to remove the novity effect from this data and using this way we can update the numbers and coming back to the team and we can decide to run an AB testing again but this time running for a longer period of time to see for uh decreasing the frequency and also increas in the frequency uh without the nov effect how's the real impact to the

12:25 user okay great yeah so it sounds like you would then basically you're running a another AP test but longer term this time yes so it's like the first AP testing because of the novative effect uh I would say what we are um assuming doesn't really work as we expect it but we do learn how to like how long the novity effect will uh persist and then we can uh take this learning to do the AP testing again to see how we can do

12:55 better this time okay perfect so let's switch tax a little bit now assume this time that you're working for Netflix instead uh and your team is trying to negotiate the renewal of um an exclusive licensing deal for a popular TV series so how would you use your data to decide whether a that TV series is worth renewing and if you are renewing it how much is it worth no problem uh let me stop sharing and take a minute okay

13:25 so what we are doing right now is to decide like using the data to decide uh how we can um renew this license with proper pricing or how to evaluate between different option so um first I want to check if um like renewing the license without like nonexclusive license is it an option I'm asking this because this might uh affect

13:56 the how I using the data later yeah that's totally an option okay cool no problem so what I will do first is that um I analyze some data based on this TV series I want to know uh from our current like Netflix subscriber how many of them are watching this series how many new user come in and watch this series as the first time so I'm assuming these new user are coming to Netflix because of the TV

14:26 series because the first series uh they choose is this TV series so this is my assumption behind it and uh we can expand it to like first three TV series new user see or something but I'll keep it simple so use the first series uh user choose right now so with this two data point I'm trying to have an estimation like if we renew this uh if we didn't reew this uh TV series license

14:56 how many user might turn after this so having an estimation how much money this TV series have help us made and also how much money we might sacrifice if we don't renew this uh TV series so with this data point I can have a sense of the benefit I'm getting and also based on how much we pay previous for the license we can have a procent cons evaluation first so with

15:27 this um ret like evaluating the first option if we renew the series exclusively uh the situation might uh proed like the user will stay and we are paying uh how much money and the use um how much money we are making from these uh users and then the option two um if we renew this license but it's not exclusive it's actually uh the other streaming platform I have the series as

15:58 well then how much money we might want to pay uh because the user won't ch um and also we won't probably uh earn more users at theya having different set of um estimated revenue and also estimated cost on each option and then the third one is we don't renew any series and how many user we might lose and how much money we might lose at car so having all these data point um uh three different option I'm pitching that we have

16:31 distribute uh like distribution and also confidence interval for each of the estimation using this way I'm doing like the low bound and also High bound of the estimated revenue and also estimated cost so because this question mentioned we need to negotiate so with all these uh numbers and also confidence interval I'm trying to pitch the low bound of the cost we are uh we might uh spending so

17:04 start from this number we can see the uh TV series providers reaction and we can negotiate from there uh if they want more we can justify with the data point and if they um like coming back and forth we have more data point to uh pointing out that maybe the the Fe layer requesting doesn't what uh the benefit we are actually getting we have more data point to uh negotiate during the process MH

17:36 okay so the weap then you're basically pitching three options like one of them being to renew an exclusive license to renew a non-exclusive license or to not Renew at all and then you're trying to provide an upper and lower bound of both the revenue and the um potential costs of each option right yes yeah okay okay perfect I think that's a great place for us to the pause for today thank you so much SW for answering these questions uh I'd love to

18:03 kind of switch into our like analysis portion now where um maybe you can try to pretend that you're the interviewer and tell me a little bit about like what do you think went well and uh versus like what do you think you would have done differently no problem so uh let's take a step back to the first question about the Instagram's uh like balancing between ads and also the uh follower post I feel like uh I pitch

18:35 uh like good solution throughout the process it's just the way I deliver it uh it can be more organized I can uh section my answer into different part like the first part I uh like the metrix definition is okay but in the middle of the experiment design I feel like I can have the hypothesis set up first and then why I'm doing this and then the end uh how I'm going to do it and what's my

19:04 expected result I think in this way the interviewer might be easier to follow uh my thinking process even though the whiteb might help but uh my answer might come in like to one part and then jump to another part a little bit too quick so it might be a little bit harder to follow so I think like putting into different parts and also like oh for uh the part of experiment decide what I

19:35 will do and then the second part I will do the analysis and what exactly I'll do like using this big chunk of different part of conversation might be easier to follow I will suggest myself to do better next time yeah that makes a lot of sense I think organizing your answer especially since you had the Whiteboard at your convenience can always be helpful uh one additional like section that I would suggest adding to that is like uh let's say you can section off

20:00 like the results of your ab test or whatever statistical testing you're doing based off of each possible result um you can have like a flowchart of like okay what this is the action that we're going to take that like oh maybe we'll do like uh renewing a um non-exclusive license or an exclusive license and so on so I think um having that like flowchart of your like decision process is really helpful but actually there were also like plenty of things I

20:24 thought you really did well in this interview so for example I like that you propos a variety of different metrics it wasn't just one metric and you proposed it for like different axes so for the first question you propos metrics for both engagement and metrics um and I also like that you Justified why each of them is useful and then also talked a little bit about like the effects in the short term versus the long term of how

20:44 like you might actually need to repeat these AB tests and you talked about the scenarios in which you might need to like run another AB test um yeah and lastly I also like that for the last question the Netflix um licensing negotiation deal question it was like very open-ended and I like that you came up with very concrete options of like okay well here are the three possible options that we could negotiate for and here are the uh calculations that we

21:08 would run for each of them to try to decide um what is like the confidence interval for each of them and how that would guide our decision- making process so I really appreciated um how well thought out that was and the attention to detail so thanks so much for being here today with us way twin uh and thank you everybody for watching if you have any upcoming interviews good luck bye everyone [Music]