How to ace big tech Analytical Thinking PM interviews (50-min)

0:01 all right so uh as you'll hear I am getting over uh a cold so I'm not um I'm feeling well but you might hear me a little stuffy and I hope that doesn't get in the way of the talk at all um but I appreciate everyone uh taking the time to uh join this lightning lesson uh today we'll be talking about uh analytical thinking PM interviews uh my name is Ben and I'll be uh be hosting this uh this session

0:34 um I'll start by talking a little bit about my experience with these interviews um and where you know why you might care about them so if you're you know interviewing with companies like meta um and many other companies like attentive stripe data dog uh Google you're going to run into some version of analytical thinking interviews and a lot of people don't really have a good framework for approaching these interviews and um my

1:05 my goal with uh these sessions is to give you a framework and help you build some confidence uh around how to approach these interviews uh in the future um I uh prepared for these interviews uh when I was interviewing for meta I got the role as was a product manager at meta and um got uh trained up as a interviewer at meta and ran um over 20 analytical thinking interviews myself as I was there and then have done uh

1:33 probably hundreds of mock interviews um over the last 3 years or so uh with people helping them prepare for these um these interviews as well so I've come up with Frameworks templates a lot of heris that I think can help people bootstrap um what it is that's going to work for them and my goal is to share a lot of that with you uh today um okay so kind of getting into it um some basic housekeeping before we

2:04 start talking about the the actual subject matter um so this talk is going to be recorded uh it's going to be emailed to you afterwards I think something like 24 to 48 hours from now uh you could keep your video off uh if you prefer um that's totally cool with me it might actually be distracting if you turn it on so probably prefer you keep it off um and then I I will be uh actually using the recording of this as

2:32 a pread for my course uh starting which is the next course in January which we'll talk about later um so for uh recording quality I'm just going to be keeping people muted and uh ideally having uh the chat be where people drop questions at the end um and then uh I would prefer if you held your questions until the end so we can get through all the content and then talk through it um there is going to be

2:57 quite a bit to cover and I'm going to go go quickly because this U this actually was a full lecture in my um my first and second cohorts that uh took somewhere around 60 to 90 minutes to get through at a more leisurely Pace um my goal was to try to take all of that and give it away for free in this uh this context which is requiring me to go uh faster than it would otherwise but um one like

3:25 I said it's free so hopefully you're not too upset about that um and then second um you could re-watch the recording on a slower speed and absorb so hopefully you know maybe this will be the first time you you um encounter the content but if you want to take your time with it you will be able to do that for sure um and then the last thing I'll say as we um as we kind of finish the housekeeping part

3:49 is that my materials are going to give you a really good foundation uh to practice but I want you to know that there's no substitute for actual practice and the more people I've helped get ready for these interviews the more this has been true um there is a direct correlation between the amount of reps and the amount of practice that you get working through real questions um and success on these interviews so um I think that's important to keep in

4:17 mind okay uh the way we're going to spend our time today it's going to start by walking through the evaluation areas for analytical thinking interviews and how I recommend um just kind of picturing the the interview flow working and then I'm going to walk you through a template that I have that captures how I uh recommend structuring your response for this kind of interview in a um in a in a Google sheet setting so I'll walk

4:44 you through that and that one is going to be blank and then I'm going to walk you through a filled out version of that template that has real content for a um for a question so you get a sense of what it might look like when you're you know properly filling that out um and then we're going to go through a bunch of advice that I have in general about these interviews um have a little surprise

5:08 waited uh that I'm keeping for the end and then we'll open up uh for Q&A like I said in the chat so my ask for you is as I start go as I go through this content if you have questions uh please write them down for yourself first don't drop them immediately in the chat because I might answer them later but if you have any questions that I did not answer until the end um that those would be

5:31 great to drop in the chat at the end so just write those down for yourself please and then uh talk through them okay so what is an analytical thinking question and then what are the basic you know evaluation areas that that matter for this so um a typical analytical thinking question is something like how would you measure success for some kind of product or how would you set a goal for this product or let's say you're the PM for this product

5:59 how would you measure success right various fla flavors of that question and what they're looking for is your ability to take a very ambiguous prompt like that in in a structured way chip away at it um by walking through typically the product rationale so that's you know what problem does it solve and why is that important there's a little bit more that goes into product rationale which we'll get into in my template shortly second um measuring impact so what are the set

6:27 of the metrics that represent the ecosystem that this product exists inside of third setting goals um basically making an altitude shift from the ecosystem level to like goals that a specific team could impact and then usually there's also going to be some kind of tradeoff question where um you might have to make a difficult decision between two options but they want to know that you have a framework for making those kind of trade-off

6:57 decisions now um time management is key in these interviews usually these interviews are 45 minutes long so um you're typically going to have about five minutes in the beginning of these interviews where you're doing you know introductions Small Talk those kind of things five minutes at the end where uh the interviewer might let the candidate ask them some questions and what you really have left is about 35 minutes to go through the exercise itself so um my

7:29 frame works are all assuming about 35 minutes of available working time and then um the way I I recommend breaking down that time is starting with around 5 minutes no more than 5 minutes on product rationale spending a lot of time on uh measuring impact um spending uh so 15 minutes there and then a little less than 5 minutes on setting goals and then leaving at least you know 10 minutes or so at the end um for

7:57 trade-offs um you know it's not cooking cutter like every interview is different but I'd say as a baseline this is probably a good a good anchor okay so the first thing I'm going to go into here is the um interview template all right for those joining late I am getting over a cold so I apologize um if I have to mute myself and cough a little okay so um the basic structure of a

8:28 template that I recommend using for these interviews and for practicing for the interviews by the way and I will give you all this template for uh for free as part of the course uh resources after this talk so you can make copies of it and use it for yourself uh to work through as many questions as you want and I'll give you a good really good question Bank as well that you could use to find good questions to practice but

8:50 it starts with being asked the question and then um I like to uh recommend making some assumptions up front um we'll talk about what kind of assumptions make sense but basically you know um if there's anything that will help you get started it's good to state that there things like narrowing in on you know a certain country uh or certain region a certain platform those kind of things and then I like to say this thing uh exactly so I'll say that I'll say

9:21 that now because everyone should get comfortable saying this um I want to start by reviewing the product's landscape um and reason for existing touching on where I think it is along the maturity life cycle key competitors the company's unique advantage and how the product generates revenue and then we'll finish by setting a placeholder mission statement for the product then I want to identify the product's key stakeholders um and player ecosystem players along with each of their primary

9:51 motivations for participating in the ecosystem and then Define key actions that indicate the motivations are being met and trans these into ecosystem Health metrics once we have ecosystem Health metrics try to identify a Northstar metric that reflects value creation for the ecosystem overall and I'll critique that metric to Define one to two guard rail metrics to ensure we're growing the ecosystem sustainably once we have the Northstar metric and counter metrics I'll identify a small set of goals that an individual

10:22 team could focus on within 3 to 6 months time frame and at any point I'd be happy to talk through tradeoffs does that sounded like a good plan for spending time together okay um starting the interview after you get asked the question and state some assumptions in this way is going to set the tone for the interview and it's going to demonstrate to the interviewer that you have a game plan for how you want to spend your time together uh which is why

10:45 I think it's worth taking the 30 to 45 seconds to say it okay otherwise you're just going to jump right in I I would not recommend that uh also gives them a chance to tell you whether they have different expectations for how they want to spend the time to get so um I would start there then uh if they say cool you say great let me take a minute to gather my thoughts so this is where we get into the product

11:10 rationale here you want to like I said describe the basic product functionality where it is might along the maturity life cycle how it makes money you can t touch on current Alternatives or competitors and then why this company's uniquely positioned to deliver the best experience or product um if there you could also talk about key gaps uh that might exist with current options and then you want to um set some kind of mission statement for the product that

11:36 you uh will possibly and maybe even likely uh need for a trade-off question later so I'll show you what that means so you set those and then um once you're aligned there you check in with the interviewer you share uh what you came up with in a structured way um say does that all make sense if they say yes great next I'd like to spend a couple minutes thinking through ecosystem players motivations and Key activities for realizing value

12:08 does that sound good they say sure so then what you want to do is you want to list out the key players in the ecosystem um the key value proposition to each of these ecosystem players for participating in the ecosystem what are the key activities that allow each of these ecosystem players to unlock the value of being in the ecosystem and then you want to wrap each of those up with a nice metric so what's a health

12:35 metric with a time frame uh that allows you to capture the Key activities associated with unlocking that value this is the basic framework for ecosystem player uh Health metrics okay then you want to you want to take all of your ecosystem Health metrics and try to identify a Northstar metric um a Northstar metric could be one of the metrics that you ended up identifying for you know one of the ecosystem Health um ecosystem Health metrics but it could

13:07 also be something different so the idea here is that you want to try to come up with a metric that's going to capture some kind of behavior action that um represents what's good for the largest number of ecosystem players that you thought about um and then you want to critique that Northstar metric so you want to talk about what's good about it and then potential drawbacks of of that Northstar metric and the reason that's important to do is because your critique of the

13:34 Northstar metric is going to allow you to then get into um looking at the guard rail metrics and counter metrics um B the basic way to think about guard rail counter metrics I really do think of them as the same in a lot of s like uh these terms mean mean the same thing in a lot of sense um is that if your Northstar metric is supposed to go up like let's say your team is crushing

13:58 what that and in your star metric is a is doing a good job of capturing this thing that you want growing so if it's you know door Dash maybe it's the total number of uh successful orders delivered per week or if it's Uber maybe it's a total number of successful rides per week or if it's Airbnb total number of you know trips booked per month or whatever that metric is that really just like describes like great things

14:20 happening for the product ecosystem the counter metric is the thing that you want to stay within a healthy range as that growth happens so um you know let's just say you're building some kind of messaging like an SMS messaging product uh and you want to grow the total number of messages sent you want to do that without increasing spam reports or uh you know as a certain percentage of messages so it's basically a Health Quality metric um in a way to

14:50 prevent um bad things happening by the Northstar metric going up okay um that's the basic concept and then once you have those um you want to work backwards from the Northstar metric in keeping the guardrails in mind to think about how you might translate uh the Northstar metric into goals that your team could focus on on a let's say 3 to six month time frame now um the way I like to approach this is

15:20 set one ecosystem player to focus on so you know we had in this example four rows it could be five it could be three you know this is just a um it's just a template that you could use for whatever you actually uh run into but you pick one of those ecosystem players and then you um you work on a basic user Journey that leads to that key action okay so let's say you pick the first ecosystem

15:52 player the key activity that allows that ecosystem player to unlock value as it pertains to the Northstar metric you work backwards from that um and then you want to identify some goals along that Journey that you could potentially choose to focus on so these are not decisions these are options um and what you want to do is you want to be able to pick one of those to focus on by um scoring it on your ability to influence it and the impact

16:21 on the Northstar metric so if something is really uh within your control to influence and it has a very high impact uh or you you can make a case said it has a high impact on the Northstar metric that is the perfect goal to set ideally um so that's that's the basic idea here and usually that'll get you through everything other than tradeoffs that that'll allow you to give the interviewer all the signals they need now um again like I said we're going to

16:48 walk through one of these filled out in a moment um and before we do that I just wanted to touch on the trade-off side so so usually with the trade-off question you're going to get asked about something that um you know we'll talk about the different kinds of trade-off questions but basically there's going to be like what is better this or this or you can only put one of these things in this part of the product at the top like

17:15 should it be this thing or this thing it's going to be um you're going to be given a couple options and they're going to want to know how you'll pick one okay that's the basic idea and the first thing to do after you get asked to trade off question is um you want to pretty much describe the trade-off back to the uh the interviewer so you want to put put it in your own words and make sure you understand it

17:43 then um I recommend thinking about the common goal so I would take a minute and actually think about this what's the thing that both of these options would accomplish or what's the thing that they both tie to what is the common Mission maybe um so you want to try to take the um apples and oranges and turn it start turning it into an Apples to Apples um is kind of how I how I describe this um

18:07 and then you want to take the two things and you want to describe the and I I like to use this template because I do think you should be pausing and thinking and writing throughout these interviews do not be talk you should not just like word vomit the whole time you should take your time and and be thoughtful and how you um structure your thoughts but you want to you want to take some time and think about what are the pros and

18:27 cons of each of the things so if we do thing one what are the advantages of that what are the disadvantages Etc and once you go through that you should be able to kind of boil it down to what's like the fundamental tradeoff here is it between breadth and depth is it between um engagement and growth right like there's going to be some that's happening that is going to be fundamentally different um and that's the thing you really want to be able to

18:52 boil down and that's not going to be necessarily evident the moment they ask the question so you want to you want to boil down and then what I what I recommend doing is um really going back all the way to the um the mission statement for the product and if you get asked about um like a feature within a product within a company so let's say uh the example we're about to walk through shortly is uh Instagram reels so

19:19 Instagram reels um you could argue it's a feature it could argue it's a product but it sits with within what we would all agree as a product which is called Instagram and then that sits inside what we would all agree as a company uh called meta okay and if you are being asked about a tradeoff uh about two different features within Instagram then what you want to be able to do is pick which of those features you know

19:43 deserves the better placement which is the example we'll go through by tying it to uh not just the uh mission of the feature that you're working on but you want to actually zoom out even a layer further and think about the mission of the product and the mission of the company okay and that's ultimately what they're they're testing for is your ability to make these trade-offs by um tying to some clear rationale um you're going to be operating with a with barely

20:11 any data in these interviews so it's not going to be something that you'll be able to like data your way to but it's got to be more um principled um in that than that okay so that's that's the basic framework um in my template for how to approach these interviews um now I want to go through a filled out version of this sorry I'm take a siple

20:45 water okay so my window right so let's say you get asked a question um let's say you're a PM at meta how would you measure success for Instagram reals okay a lot of people I work with have told me they've gotten this question the last few months so this is uh pretty standard okay um so the first thing you want to do is State assumptions so um what you want to do is

21:17 start by saying something like you know I'm going to focus on the core reals product within Instagram I know it's also available on Facebook it might be um and then but I'm just going to focus on Instagram itself and then I'm going to primarily focus on the US market um if you want you could also make an assumption you know around like mobile um which you know at this point I think everyone probably says obvious uh for

21:39 something Instagram reals but you don't need to then you say that whole Spiel um that I said earlier and then you start to get into it so um if you think about something like Instagram reals it's meta you want to describe what it is first um like I said so it's a meta short form vertical video product it allows creators to produce and share engaging video content um up to 90 seconds long you can kind of

22:07 be that specific if you want users can discover reels through their feed they could explore um a page um that's associated with reals uh with the reals Creator um they could even look at a dedicated reals Tab and the product generates Revenue because ads get inserted between uh reels and shopping features integrated in into videos something like that uh you could say real is in a growth stage of maturity uh even though it's achieved some significant adoption since launching

22:38 it's still expanding its creator base and competing for market share with Tik Tok YouTube shorts and other short form video platforms and meta is uniquely positioned to succeed with reals given its massive user base across Instagram Facebook uh strong Creator relationships recommendation systems advertising infrastructure that kind of thing um and then I think it is important to talk about why the company cares about this product or why this product ma is important so you know you

23:05 could think um it's it represents a response to a fundamental shift in how people like young people consume content uh the success of Tik Tok is applying pressure if if meta doesn't adapt it's going to risk losing user engagement um fault behind comp competition two it's a little more aligned with advertising uh with for advertising with modern consumption habits um you know I'm not going to read through every single um word on here but the basic idea is you

23:33 lay out why this product matters um and then you want to set some kind of mission statement okay so maybe something like Empower creators to share engaging short form videos that entertain and connect the Instagram Community right you're supposed to make this up on the spot it doesn't have to be right it just has to be reasonable then uh we get into ecosystem players Okay so if we think about who are the highle ecosystem players for

24:01 reals you've got your viewers SLC consumers you've got your creators you've got advertisers then you have meta when you think about the value proposition for each of these ecosystem players um you could think of the consumers as you know entertainment Discovery um accessing engaging personalized short form video content the Key activities are watching reals engaging uh with with them through these different you know Act actions they can take um and then you could think of the

24:31 ecosystem Health metrics as like daily active reels viewers average watch time per user per day number of reals liked and shared per viewer per day per week um so this is um this is uh you know I'm not going to go through every cell on the table but the basic idea is you do that exercise for every one of uh the rows here and again I will share this with all of you so you you're not going to

24:57 you're not going to lose this um and then you try to set some kind of Northstar metric so um maybe it's you know total reels watch time per week okay and the reason that that could be good is because um you know it indicates that if you can grow total reals watch time think like the total number of minutes people spend watching reals per week okay what what's good about that well it indicates people are finding the content engaging enough to

25:28 spend time with it it signals that creators are producing content that holds people's attention it gives more inventory for advertisers to reach audiences because people are spending more time watching this stuff and then it's driving Revenue opportunities for meta and creators what are the potential drawbacks of of this metric um that watch time could increase even if the content quality degrades okay that's one two it could incentivize longer videos even when shorter ones might be more

25:57 engaging and it doesn't directly capture Creator sustainability or Advertiser Roi okay so if we wanted to address some of these potential drawbacks we could say something like hey maybe the average completion rate per real um shouldn't drop below a certain Baseline so this ensures that we're growing the watch time through engaging content not just longer video completion rate being like a total number of like the percentage of the Reel that gets watched right to to its end um and then the second one is

26:28 the creator uh retention rate so number of percent of creators who post at least one real in consecutive months this ensures ecosystem sustainability so if we can ensure that we grow total reels watch time without hurting the completion rate of of watching uh reels and uh we don't start churning creators then that is going to be a good thing that's the basic idea of how to use this framework and if you think about uh turning this into goals

26:59 so at this stage um you could make the argument that creators is going to unlock the most growth for the weekly watch time because while viewers and advertisers are crucial if you don't have a robust supply of engaging content it's going to limit the total amount of watch time that you could have okay so if we think about um the uh the journey you can say it's about creating and editing reels posting reels um reaching viewers distribution generating an

27:29 engagement understanding performance and ultimately maybe monetizing the content that's kind of like the basic Creator user Journey here and so if we look at that we could set some potential goals you know one could be um increase the real creation completion rate so percentage of reels that get um started reels that get published through better you know inapp editing tools and templates something like that second could be increasing the Creator posting frequency through improved scheduling and planning tools

28:01 um and maybe the metric there could be something like percent of creators who post at least two reels per week and third is improving new Creator retention uh by enhancing the first week performance Insight something like that and then you could look at each of these and say okay you know you in a real interview you will need to talk through your rationale for why you give each of these things the score you give them and

28:24 like I said these interviews are going to be 35 minutes long so so I cannot do a full um proper mock on this lightning lesson while also covering content and answering questions so that's kind of why I have to be a bit succinct here um but I can make the case that um you know the Creator posting frequency is is very high U it's the highest ability to influence and it could have a very high impact on North Star metric which is why

28:49 it would be the goal I would set or the goal I would pick um okay now let's say I asked a trade-off question so you went through all that and I said let's say you're the PM for Instagram overall you're deciding whether you want to put reals at the top of the Instagram app instead of stories is it better to have reals at the top or is it better to have stories okay so um this is a real question that

29:17 one of the people I work with got asked and the fundamental trade-off here is what goes at the top there's Prime placement there's only one thing that's going to go at the top of the Instagram app which one is it okay not that not that complicated um when you think about the common goal you can think both reals and stories contribute to Instagram's mission of connecting people through Visual sharing but they do it in different ways stories enable authentic

29:44 ephemeral sharing between friends and followers reals focuses on entertainment and discovery of content from a broader Creator ecosystem okay so if we look at reals at the top or stories at the top and we look at the pros and cons of each you could say reals could you know put if we put reals at the top maybe that accelerates the growth of our Tik Tok competitor maybe it gets higher engagement procession due

30:14 to the video format maybe it gets more monetization potential because it's going to have more immersive video ads something like that the cons is that it could reduce authentic sharing between close connection it could shift Instagram's identity further away from social and it could risk reducing uh stories engagement which is already a uh format and a feature that people are very heavy heavily invested in if you think about a world where

30:46 stories is at the top so this is the you know status quo it maintains Instagram's core social value prop it preserves a familiar user experience projects engagement on a format with High user adoption and um then the cons are slower growth for reals maybe less aggressive competition with Tik Tok and uh lower potential for more entertainment driven engagement in the product so the fundamental trade-off here is between strengthening the social

31:17 connections versus pushing towards more entertainment and Discovery and if you think about uh which direction to go with if you consider consider the current maturity and mission of Instagram um I would say keeping stories at the top makes more sense even though reals is strategically important stories represents a core differentiator of authentic sharing between people moving stories could signal that Instagram's deprioritizing social connections Etc um and then this

31:49 is the kicker I I I always say it's important to include a what would need to be true for me to change my my opinion thing um when you share your rationale so if we see stories engagement Plateau while reals continues to grow rapidly or if user preferences shift more dramatically towards entertainment I would revisit this decision okay so I'm leaving the door open to changing my mind all right I I told you we would all

32:16 go through that really quickly um hopefully you can watch the recording if you want to I know this supposed to be a 45 minute session I'm going to plan to go until the hour mark and uh if you have to drop um I totally understand but uh I think uh we should be able to get through everything so to summarize review product rationale describe ecosystem players and set Health metrics um Define Northstar metric critique it define guard rails

32:43 and set a goal to focus on and then evaluate a trade-off okay and you want to be thinking before each of these things and you want to share back with the interviewer and check in with them to make sure if it makes sense okay I'm seeing some questions come in the chat reminder um I'm not going to answer those till the end but feel free to drop them there if you want okay um quick plug and then I have some

33:05 um a shift into advice okay so um I'm going to be teaching my next two cohorts of the course on January um and February the dates are listed here um people who join this lightning lesson will get 10% off my course with light at uh promo code um and we'll be just getting a lot of practice uh with the template with the framework um you'll get access to me uh with all your questions um and I'll

33:34 share something special on the AI practice co-pilot at the end of this talk um but I just wanted everyone to know I do have a full class if you want to get more uh Hands-On practice with this stuff okay um advice all right so first thing I'll say is that um interviewers are evaluating your thought process and your rationale and not your domain knowledge so you want to make sure you don't overly rely on uh knowledge of existing products um

34:03 metrics so uh for example if they ask you about a product that is in a space you've worked on before don't try to like take shortcuts make sure you actually walk through it um from square one the second thing is that what makes for a good counter metric is that it's a good balance the Northstar metric so um this is a really good chart that I found from product HQ that talks through some examples of North star metrics with counter metrics

34:29 but basically you know um it it's it's a way to make sure that you're growing the Northstar metric without taking a hit on something else as I talked about but um I'll be sharing this this material so you can look at this after the fact tradeoffs uh as I mentioned in our example was you know about whether it's better to have IG reels at the top or stories um some people go through this interview and they get asked uh is it

34:57 better to reals for reals to have you know 100 reals with a million views or a million views a million reels with 100 views each um there's no right answer for these questions but it's all about your ability to frame it and then to have strong rationale or at least reasonable rationale for why uh you believe what you believe and and like I said mention what needs to be true for you to change for you to switch your

35:23 stance in the future um I mentioned this before but it's really critical um do not dive in right after this is you know people ask what are some of the biggest pitfalls of these interviews I would say one of the biggest pitfalls is just jumping right in and I would recommend not doing that so that that framework that I shared um I would recommend sticking with um I do really recommend embracing those you know arguably awkward silenti

35:51 to think I think they're a good use of time don't spend more than like 2 minutes pausing and thinking I think that can get a bit much and eat into your time but up until 2 minutes I think it could be pretty pretty good um I would also recommend not sharing your screen so if you are using my template or you know if you're just writing stuff down um sharing your screen can become a real distraction you're going to start

36:13 like fixing typos in real time just wasting your time on things that don't really matter so my general advice even though they tell you you could share your screen is not to unless they insist on it okay usually they shouldn't um there is no right answer to any of the questions you're going to get asked uh you just need to be to be reasonable make assumptions walk them through your thinking just don't contradict yourself along the way and if you happen to um

36:42 get asked one of these questions from an interviewer that has some preconceived notion of a right answer you can't anticipate that I would treat it like weather conditions okay um there's no planning for someone already knowing exactly where they want to land uh nudging so you might get nudged um throughout these interviews and the way that that can manifest is them saying hey like why are you uh help me understand why you want to go with like

37:07 weekly versus daily for your Northstar metric right that's an example of a nudge um they're not trying to trip you up they're just trying to get the signal they need so when you get nudged just kind of roll with it um they're trying to help you okay um next thing here is um time management is critical I've mentioned this before but it's worth saying twice um the flow that you follow in the interview should be entirely focused on

37:33 giving the signals within the allotted time and if you if you're not providing them signals um as part of your plan uh for the interview I would uh I would cut that part of the interview uh part part of your plan so an example is uh let's say you're also going through product sense interviews as part of the same company's interview process um product sense is going to get into like a different set of signals like problem

37:59 identification and solutions um don't bring those into this interview okay because those are not going to give them the signals they need for for analytical thinking um I get asked like hey is it okay if I if I make a mistake is it okay to like go back and redo a previous section I think if you feel like you took a wrong turn or you did something that you're no longer feeling good about like for example if the ecosystem uh

38:24 Health um if the ecosystem player that you chose to focus on for goals if you're kind of like regretting the one that you picked um you could pick a different ecosystem player right or if the um if the Northstar metric that you chose is uh wrong and you're you're kind of working on you wish you chose a different Northstar metric um let's say you picked daily and now you realize it should be weekly um you can go back and

38:48 fix that okay um the key is just communicating that thoughtfully while being mindful um of time okay some some of the links that I'll share with you are the ones we talked about I'm going to give you a link to what is arguably the greatest question Bank Of All Times um you can so this this guy Lewis Lynn uh wrote a book and has a big uh PM Community around uh interview prep and people can submit uh

39:15 real questions that they get asked from places like meta stripe and other companies and you can see that as of yesterday there were brand new you know five new submissions here um you know analytical thinking analytical thinking so if you're trying to get more practice with um with questions the place to decide which questions to work through I would highly recommend um Lewis Lyn's question bank and I'll link that to you after this I also have some supplemental

39:44 reading around how to figure out which uh which life cycle stage of product is in and how to think about defining success metrics um I promised in the um lesson page that I would also give you a homework assignment I uh probably won't have time to grade uh these for everyone that attended this lesson but um you could use my template and work through a question like how would you measure success for Facebook events okay so

40:12 that's a homework assignment for you if you want it um and start preparing your questions okay and um before we do that I want you to know that as part of my course I'm also introducing a mock interview prep AI co-pilot okay this is a very interesting um it's it's a claw project that I trained in my course materials and um it is incredibly good at providing both um mock roleplay as an

40:47 interviewer so if you give it a filled out template if you follow my template and submit it it will actually grade your assignment and it will give you room for improvement so um i' I've done that with a few questions and a few templates I filled out and um it's it's it's really good um and then the second thing it's good at is it's good at playing a candidate so you could ask it a um a

41:14 question and it will walk you through how it would respond as a candidate including things like checking in making assumptions um responding to nudges and then it scores itself at the end as well scores the the submission of its own uh response so um people that joined my course uh I will help you set that up on yourself and train your project on on all my materials so that you could get way more practice um than you could get

41:43 with just me or with anyone else in a one-on-one setting okay um I mentioned this earlier but I'll mention again my my course uh is starting the next Co cohort is starting in January um 10% off with light at is the promo code and then I'm going to leave this up while I start looking through some of the questions um if you want to join the cohort here's a QR code in case you have your phone nearby um and if you are

42:13 willing to take two minutes um to complete my NPS survey for this talk it will help me do a better job in future talks and improve my material so I would really appreciate that should be really really quick okay I'm going to start reading through these

42:44 questions um okay so there's a question around can I go over an example about AB tests that we would have to set up to make decisions at the end of an analytical interview question where does that fit in the template and what do the structure to go over it so so um what we talked about are um you could argue that it could be AB tested like stories at the top or reels at the top um if it

43:08 was a trade-off question between uh breadth and depth like you know 100 reels with a million views or a million reels with 100 views um that would not be AB testable so I think the the the key is that if you get asked a question that could uh be AB tested that you describe what the goals are that they would have in common so if you imagine um if you imagine uh two different features that each move a different

43:36 metric the questions are they both going to roll up to like some kind of Engagement metric um you know or are they both going to improve overall time spent on the product something like that and then you want to be able to describe if you were to run an AB test and if if you choose to decide to run an AB test I don't think they'll tell you that you need to run an AB test um you'll just

43:56 want to walk them through what you would measure and what you would track and then you um if you want to use an AB test as the way you want to respond to the question then you just need to be able to communicate also how you'd consider things like the mission um and your answer so it can't be just based on metrics you want to also make sure that you're talking about um things like Mission alignment um there's a question on my

44:25 prioritization criteria I don't know what that's referring to um but if that's still relevant after what we went through feel free to Reas that please um what differences do you see between early career PMs and product leaders and how they go through the framework I'd expect leaders are less framework in response but still touch on all evaluation areas um if you're being asked to go through an analytical thinking inter rview uh that follows the basic structure of what we talked about

44:54 you know like me how would you measure success for this Etc um and it's a you know large-ish companies uh company and you're not being interviewed by like the exact hiring manager you know it's not a behavioral interview where they want to hear like tell me about a time in the past where you measured success right but if it's more like this is a structured interview for analytical thinking and I'm going to ask you a question I want you to walk me through

45:17 your thought process I would follow this exact framework um I don't think there should be any key differences between you know like level like senority of people and how they use a framework like this but generally speaking in interview processes there will also be additional questions additional interviews I should say that get that get asked if you're joining like level seven and above versus like a level five um so I don't I don't have any reason to believe that leaders need to

45:45 be less framework um use less Frameworks in their response um how does the upcoming cover everything needed for meta um my course if if that's the if the question is about my course um my course covers everything needed for meta's uh product sense and analytical thinking interviews um my course does not include leadership and drive um which is the third structure type of interview that most people going through the meta

46:15 interviews need to pass um from an interviewer perspective is it better to handwrite your thoughts during an interview as opposed to processing your thoughts through a spreadsheet um I've heard that people using AI have been a problem so typing might be misunderstood oh that's an interesting question um I don't know if that's I don't know if that's something that I've seen come up like I don't think people have been told or not allowed to like type their

46:42 thoughts during interviews um I can I can see why um that might be concerning to people but out of everyone I've been coaching um talking about like dozens of people in the last like six months who have gone through these interviews um no one has told me that they've been not allowed to type okay uh so that said when I went through the meta interviews originally um back in 2020 I wrote I like to like write stuff down and I

47:07 didn't have all these templates back then uh I kind of like figured out a lot of stuff for myself um when I Was preparing so um you could print out my templates too um I tell people they could totally print them out and you know write write in populate the the templates and writing if they prefer with a pencil or something so you could totally do that uh it also slows you down a little if I'm honest allows you to think more um

47:33 Tim yep uh good to see good to see you to Tim um pause thinking breaks what's an acceptable number of these pauses uh I would really so I'll go back to like this this flow uh here so when I say one to 2 minute pauses it's really at each of these moments where you're transitioning between um one section to the next section so like you know if you follow this like strictly four four times um

48:03 something like that thanks yep um does the upcoming boot camp cover everything for meta okay I think I answered that and then yeah um so one someone's asking what about a question like hey we launched this thing the Northstar metric was neutral down but this other metric was

48:35 up what do you do um th those are interesting questions um they they could they could come up and they are ultimately trade-off questions like let's just say this metric is up and this metric is down what do you do it's another way of saying is that good is that bad um and what what I do know is that the debugging questions are on the way out so there used to be questions like this metric was up this other metric is down

49:03 what happened and they want to know that you can kind of like ask the right questions to land and some kind of explanation or get to the root of of how you might understand what caused that that dip in the metrics um but in this case if the Northstar metric is neutral down but some other metric related to engagement retention is up what do you do you want to find the common place where both of these metrics contribute

49:27 to something in common so taking apples and oranges and creating Apples to Apples um we don't have the time to go through this live right now but you want to try to find the common place where these things contribute to something similar and then you want to use again you want to tie back to the mission uh of the feature or the mission of the product and figure out what would be best for the product overall given its

49:48 current stage in the life cycle um that's that's the basic framework um are January and February continuous or two different sessions there are two different cohorts so the the the course is two weeks long so there's two a two-e cohort in January and a two-e cohort in February um generally speaking I tell people they should take my course probably uh in the month leading up to their interviews so that it's fresh um so if someone's going to go through the

50:18 interviews in in March maybe it's better for them to be in the February cohort our estimation questions still asked um I I've been given mixed responses on this I know for uh for meta no um but I do I have heard that Google has continued uh with some people to ask them but it's not uh not every I I don't spend as much time helping people with the Google Google interviews as I do the meta ones but I I do believe it still

50:49 gets asked um okay I have time for uh maybe another question and then we'll we'll wrap this up if people have anything you want to ask right go once twice I'm seeing people already starting to drop so I'll take that as an indication that we are done here so um thank you everyone for joining uh I hope this was helpful and

51:19 uh let me know if you have any other questions about the course uh after the fact and keep an eye out for the email with these materials cheers but