Unpacking Meta’s New "Product Sense with AI" interview

0:00 You know what's funny is is the way that we are scrambling right now with like our Zoom link and stuff and figuring it out live. It kind of feels like how companies are evaluating AI fluency and their interviews today. Like given all the people I'm talking to and how they're figuring out how to adapt and evolve their interview processes to evaluate AI fluency and the PMs that they're hiring is it's a bit of a wild west situation right now. So, I've been tracking it really closely.

0:29 And so, when this news broke, it felt kind of like someone took an opinionated stance at some some large scale to figure out what it might mean for them to get started. And, anyone who tells you that they've have they have this whole thing figured out, like they know exactly how to ace, interviews that are designed to evaluate AI fluency, I think, is kind of full of it. because it's evolving faster than anyone can keep track of it. So I think the truth that is that the best we can do as people that are trying to help candidates navigate the market is report on what we're seeing and what we're hearing from real people at the companies as well as people who are going through the actual interviews.

1:10 and the way we the way I like to do this stuff is to maintain confidentiality. So, we're not going to divulge specific details that will help you anyone listening to this know exactly who I spoke with and you know I don't want to compromise anyone's confidentiality but hopefully at this point you know we just have a honor code between us that we trust that this stuff is coming from like real people on the ground telling us real things otherwise we wouldn't do this event. and so the way we want to spend the time today is talk about kind of what seems to be changing with the PM interviews that is you know manifesting as this specific update to Meta's interview process. I want to clarify a little bit of nomenclature differences. what's the difference between AI product sense and what's product sense with AI in this context and you know I'll talk about that a little bit and then we'll shift into discussion on what what's going on. So as far as like chat you can ask questions in the chat. It's going to be hard to kind of moderate the discussion and speak while also monitoring what's going on in the questions. So, personally, I will probably not be able to to like answer anything in the chat until we get to the end, but we're leaving time at the end for Q&A. but if Tal and Aman, feel like chiming in on the chat, they can do so.

2:29 okay. So, a little bit of backstory. Tal and Aman and I got together last summer in New York, and, you know, it was our first time all kind of hanging out together. And one of the things, hey, Aman, welcome. Glad you you made it. one of the things we talked about when we got together was basically what is going to happen to PM interviews in a world where AI can do so many of the things that are expected of candidates to generate live, you know, in their head by thinking like basically what's going to be the way companies evaluate people if AI can do that job just like you wouldn't evaluate someone's analytical chops by seeing how well they can do like you know long form division on paper because calculators can do it. And maybe that's a bad analogy, but like in a world where the skills to do a specific thing can be done faster and better and more cheaply by technology, why would you evaluate a human's ability to do those things? So, and our our discussion ended up leading us to a place where we knew without without a without a doubt that the current state as of a few months ago of PM interviews is is definitely a snapshot in time and it's a the end of an era of how interviews are going to be running. And then it's the beginning of a new era and and how PM interviews are going to be evaluated and conducted and entering this wild west together of figuring out well how you know almost like speculating how is it going to evolve and sorry I'm getting doorbell ringing with a package which I'm going to ignore. so we we knew that there's going to be some evolution coming and we didn't know when we didn't know who's going to take the first mover advantage.

4:03 certainly didn't speculate that the company that that I used to work at which is Meta which is exactly the company that teaches I'm sorry the exact company that runs the interviews that I teach in my course would be the first one to take a large scale opinionated stance here. So it just happened to be that that happened. and so you can think of this as like a developing story. and it's a developing story because we don't know how it's all going to shake out. Think of us like trying to report live on what we're seeing and trying to give you the best information to make informed decisions.

4:39 so the a few weeks ago I made a post about this. and basically the the news the headline here is that Meta added a fourth interview to its PM loop which you know, if you're interviewing for level seven or higher at Meta, there's a slightly modified loop, but for the vast majority of people going through meta interviews, there's product sense interviews, analytical thinking interviews, and then there's leadership and drive interviews. So, three interview types and that's been the case since before I even interviewed at Facebook back in 2020.

5:08 So, the fundamental format hadn't changed in nearly 6 years. And then just a, you know, a few weeks ago, I started to hear from my students that there was a new interview that been added. So, it's the first time that there's been an evolution to that that fundamental interview loop in six years. And the specific interview that got added is called Product Sense with AI. And then I started to gather some intel about what it means from people that both work at Meta, people that have recently left Meta and people that are candidates who are in the process with Meta at the moment because across these sources, you know, you're lit, I guess three data sources, we're literally triangulating the truth, right? So we're trying to figure out what's going on here. And the first thing I want to kind of call out before we dive into the specifics is when we talk about product sense with AI versus AI product sense. I just want to do a bit of a nomenclature and make sure we're all on the same page about what we're talking about here. So product sense with AI that is a new interview format. It's a specific thing. It's like a noun and it's specifically an interview that's being tested at Meta.

6:19 Given how many companies run interviews titled Product Sense, Stripe being one, Door Dash being another, you can think of companies like even Uber and their Mini Jam now do many product sense exercises. there are many companies that run Product Sense interviews. So what I'm looking at here is basically an evolution of a traditional Product Sense interview to incorporate AI usage into the Products sense interview. So that's what we mean by product sense with AI.

6:50 Okay. Now when we talk about AI product sense, this is something that I believe is going to converge into more of like a real set of on the-job skills and intuitions for how to build products with AI. And that's something that Tal and Aman cover in their course which we'll talk about later. But that's not what we're going to be talking about today is not the on the-job usage of AI and and developing a better on the job product sense for how to use AI.

7:18 We're talking specifically about the the evolving interview landscape here. >> Okay, >> Ben, if I'll jump in and say what really helped me like understand this distinction is >> just like the until now for last decade like the product sense interview is completely different from the concept of product sense that we use on the job. So, it's the same thing. >> Exactly. And there's going to be elements, you know, you could think of the Product Sense interview is almost like a like a caricature of real product sense that you need on the job.

7:51 It's trying to kind of get a basic feel for the bones of those skills being there in a very condensed simulated environment. And I suspect that this interview is following in the same the same paths based on what I'm what I'm seeing so far, which we're about to get into. So so this is the official guide. So this is not you know my content. It's also not proprietary. It's something that's you know being widely shared by by Meta as a company. but the the explicit kind of expectations are that an interview that focuses on your ability to leverage AI tools to enhance product knowledge, market analysis, creativity, problem solving, specifically using meta.ai. So not chat GPT, not Claude, not Gemini, Metaone in-house product. for a product case in a way that demonstrates product sense, ability to collaborate with AI, critically think and make decisions when using AI and expecting some kind of questions about a product with encouragement to use AI tools transparenc transparently and what that means is really like screen sharing and getting eval getting assessed on how you can integrate insights from AI with your own judgment and basically critiquing selecting and kind of like building on top of AI outputs and knowing what to ignore, what to incorporate. a lot of that stuff is things that you you could consider at a high level AI product sense, but I think you like I said, you always have to figure out the exact interview angle on these things, not just like what they mean outside of an interview environment.

9:25 >> the specific things I wanted to create this slide because I want people to remember that even though these things are new, they're incremental. So they're an evolution of existing triedand-true interview processes and and frameworks that are being figured out and tested. Okay. So like the explicit guidance from the guide is you're going to look talk about product motivation and context, target audience, identifying and prioritizing a problem, developing creative and impactful solutions, and collaborating with AI. If that sounds familiar, it's because those first four bullet points map exactly to my my guest post on Lenny's newsletter from April of this last year. So like seven months ago, eight months ago now outlining the key steps that you want to cover in a product sense interview flow, motivation, segmentation, problem and solution. So the the core bones of this interview have not changed. What's been added are a couple things. Specific bullet point modifications for the first four points on that list as well as a fifth dimension that did not used to exist. Okay, so the idea is that they want you starting at the top to integrate AI insights with market realities to figure out the context around the product. to be able to potentially incorporate AI data to help you inform the audiences that you want to go after and your segmentation decisions. Then to be able to use AI to deepen your understanding but not outsource your judgment around the problems that you might want to choose to tackle. and then ultimately evaluating some kind of product decisions through the lens perhaps of AI AI AI driven solutions, AI powered solutions, maybe AI even for evaluation of the quality of the solutions that you come up with. So there's a a lot packed in that last solution section.

11:10 And along the way, just like historically, you'd be evaluated on your ability to to communicate in a clear comm in a clear structured way during an interview process, now you're going to be valued on how you collaborate with AI through this process. So, there's kind of like an overarching dimension to the way that you incorporate AI to to this interview process. Okay? So, this is the explicit guidance, okay, from from Meta's guide, right? and I'll kind of like, you know, skip this last part, but they kind of tell, you know, interviewers might ask you questions and, you know, sketch out what you might look like. So far, I don't know how much of this is is actually happening, but the the main piece is you do want to get practice with the meta.ai product instead of chat GPT and other ones because it does have nuances. Okay, so the don't ever let the tooling in these interviews be the reason you fail. Okay, so and I've played with them. They're very familiar. to anyone that's been using chatbt cloud etc. Okay, so with all that said and I know that was really fast and you'll get the recording and you could watch it more slowly if I talk too fast but the basic idea is that we want to spend the ne the the rest of this session talking about what we're seeing on the ground. So I want to share some insights from both interviewers and candidates about how the current like reality is a bit messier and different than what's in the guide. I think the guide is to be seen as kind of what they would like people to do in these interviews and I I think that applies to both interviewers and candidates. So, it's kind of like how should the interviews go if they go exactly as prescribed? And then there's what's actually happening. And I think these are very different things. And I think every month we have this conversation, we're probably going to continue to see divergences between the reality and and the explicit guidance.

12:59 And then we want to talk about so it's going to be mostly me talking hopefully interesting to people. And then we want to open it up for Tal and Aman to also kind of have a three three three person kind of like round table to talk about what we think this means both for the broader interview landscape but also what people can potentially be doing to get ready like again to the to the point that AI product sense will help you get ready for these interviews but it's not the game you're playing what you're you're playing is a very different game.

13:32 and then just like a thought experiment. I think it's always fun to speculate given that we're in the Wild West. Like if we were to design some kind of interview that evaluates AI product sense and PMS, how else would we potentially think about structuring it? What are some of the the aspects of it that we could we could consider introducing? So before we start the discussion, Tal and Aman, anything you guys want to add on what what I shared so far?

13:57 I just want to reemphasize I admire Ben and and I want to emphasize how much he's acknowledging the uncertainty that this is a developing story and just like this is going to continue being uncertain. we're just going to keep updating and yeah, like Ben, I like how you said, anybody who claims certainty on something like this right now is is full of it. So that's it. >> Yeah, I I'll add on. I think the fact that this interview loop is early, I think we'll we'll get into some of this a little bit more in the discussion, but we've kind of all a few of us have definitely been having conversations in the background with other product leaders at other companies. And I think there's a general sentiment that Ben is highlighting as well here that the way that interviews are changing is early as well and across companies, not just Meta. And so I think what is interesting about dissecting Meta's interview process and dissecting how they're thinking about how AI is a skill in the interview process is sort of like one of the dimensions that you know a leader in the space is sort of thinking about how interview formats are changing generally. And so there's a lot of like applicable kind of learnings and skills across different interview loops you might be entering as well if you're a PM going into you know interview and recruiting season a little bit here.

15:21 so just want to kind of drop that point that interview we're seeing interviews changing across the board. Metas is so far the only one with like formal guidance on how. And so that's why I think this is also really interesting. >> Totally. and to to the point that I think Tal you may have made this point earlier. I I have multiple people who are in the position of hiring managers at companies that are trying to figure out how to evaluate AI fluency telling me they're explicitly waiting.

15:51 They're waiting to see some best practices emerging so that they could copy and paste it. And the reason for that makes sense, right? Like if you think about what your your job is as a company, it's to solve your customers problems in ways that create shareholder value and grow the business. And you know, that's ultimately the job. So every minute you're spending reinventing the wheel on your interview process is a minute you're not spending serving your customers and you know solving their problems. So there that that's why there's so much I think writing on how how the industry chooses to to to to do this.

16:23 so with that said okay so a couple of the the things I want to start with. So let's start with the company side. I've reached out I mean I I've I've worked at Meta so I have a deep rolodex there. I've reached out to many people there and I've asked them basically what do you know about this interview right keeping it open-ended just like I do any user research and what I find is that every like the majority of people I reach out to don't know exactly how to run it because they haven't been trained on it. So people are like I I heard about it not really sure how it works. Okay. that's a lot of people.

17:06 So some people I actually had one person find out about it from me. so even though they worked there, they didn't know about it yet. And then I'd say the the second piece is some of the people that I spoke with have played a role. I'm trying just not to like, you know, throw anyone under the bus from a confidentiality perspective, but but helped to shape the interviews or at least shape the early conversations that led to these interviews. And from these people, what I've gathered is that it's it's very likely that at this stage the interviews are are not failable. And what I mean by that is if you don't like crush the interview, you could probably still get the offer. and the reason for that is they're testing it.

17:47 So my my understanding is that given the current state of where this interview is, it's not nearly at a maturity point where they will disqualify a candidate from getting the job if the candidate doesn't knock this interview out of the park as long as they they crush their other interviews. So I'd say the job is not there's not as much on at stake at this very moment in time as I thought there was where candidates should be like freaking out about this interview.

18:14 and at the same time, I think that this interview is an opportunity to strengthen the candidacy. So, if all things are true and someone comes in and does an incredibly strong job at this interview, I think it will only give them more leverage to to both get an offer or even maybe negotiate harder. the other thing that I'm hearing is that this interview is most likely catered at this point just for machine learning and AI u focused roles. So basically, if someone comes into the recruiting pipeline and they're heading towards some kind of machine learning or AI heavy role and they're coming in at a level six or higher, that is probably the people that are likely to run into this interview. If you're coming in less than level six or you're coming into something that has no path towards AI or machine learning roles, it's it's unlikely you'll encounter this interview. Okay, so that's on the company side. on the candidate side again protecting anonymity I have multiple data points that tell me that the interview in practice is a lot more like a traditional product sense interview than a new interview. What I mean by that is that the nature of the questions that are being asked are the same kinds of traditional product sense questions which people can look up in like Louis's question bank or my Lenny post. I go into those. so the questions themselves are the same. the the element of instead of where where in the past you might have been sharing your screen with Excalibraw or a Google doc, which I often tell people not to do historically because I think it gets in the way of structured communication and verbal communication. Now the expectation is a nonoptional screen share of meta.ai on your screen during the interview. you don't have to use it at every point in the interview. It seems more like an like a thing that's there, like a resource. Okay.

20:03 Now, yeah, I'm not going to say much more on this to to because only only not that many people have actually been through the interview. So, the fact that I have multiple data points could if I say anything too specific, I could throw someone under the bus for for giving away who they are. but that's that's what I'll say. It's not it's not mandatory every step. It's optional. And the structure of the interview flows a lot more like a traditional product sense interview than than another interview. The last thing I'll say is there seems to be a moment like an opportunity for reflection at the end of the interview where the candidate gets a chance to say if they had more time, how would they have potentially used AI more heavily throughout the interview and for which which purposes?

20:47 okay, I'm going to pause here. Talon, feel free to like, you know, add anything or clarify anything that that your curiosity is pulling you into, but I wanted to make sure we started by by really ironing out that difference of the current current reality versus like what the guide says. >> Love it. I have I have one comment and then Ben, I have a a question that we might get into more later so you can punt it, but the comment for me is it kind of makes sense, right? Like AI is such a huge pillar and focus area and so much of these like large tech company market caps are riding on AI driving growth that it kind of makes sense for you know people especially strategically coming in thinking about AI products to be able to try and assess that candidate in some way of like how would you take this existing surface area meta and change it and prove it based on some you know kind of in like an interview type of way like like how product interviews product sense interviews have traditionally right then like you know how do you design a group chat product or something like that you know or something along those lines or a product for personal shopping. so thinking about that from like a meta AI standpoint okay now you have this foundation what would you go build on top of it is that sort of what the interviewer is trying to get at a little bit more from the from the interviewe.

22:07 Yeah. Well, I would say the interviewer is trying well at least like the explicit guidance the official guidance is for the interviewer to evaluate so if you think about the traditional product sense interview it's it's a it's an interview that's guided by the candidate typically taking up let's say out of 45 total minutes for an interview slot it's like 35 minutes of of case because some small talk in the beginning some questions at the end so 35 minutes for a case which means the candidate candate has to drive the conversation. Meta is one of the companies where they do expect the candidate to drive. Most companies will expect the candidate to drive.

22:45 starting with why we might care about this. What's the motivation? How might we go from the super high level motivation, maybe even a mission statement for this product to a specific audience and how can we go from that audience that we want to because you know in the world you can't serve all audiences in parallel. You have to like pick pick your bets, right? And then how do you go from that audience to like problems you might choose to solve and then how do you pick one of those problems and ultimately like demonstrate to the interviewer that you can get to market with with like a compelling solution to that problem within 35 minutes. Right? So if that's the if that's the job, I think what the interviewer is supposed to be evaluating in these interviews now is okay, you're you still need to do all that, but at every decision point along the way, you now have access to like the entire body of human knowledge at your fingertips or at least up based on what the the models have access to, right? So if you do have all that access, how do you then bring it in to these decisions that you have to make, right? So when you're trying to figure out which segment to go after and you've got like all of the world's information right here, do you use it? Right? Like are you are you going to are you going to like make a better decision now that you have it or are you going to not make a better decision? when you're trying to figure out which problem to go after, are you going to use all of the world's knowledge to figure out which of the problems you might want to go after or or are you going to like not use that?

24:10 Right? So I think there and then if you do use it right, how are you going to figure out how within such a time a tight time box, how are you going to take this massive body of knowledge and then create some like immediate connective tissue to like the exercise that you're going through, right? And and when it when you if you do make that connection and it gives you slop versus really insightful stuff, how are you going to know the difference between the slop and the insightful stuff that you want to incorporate? Right? So, I think that's really what they're evaluating is each of these decision points, will you be able to leverage are you naturally drawn to leverage this stuff? And because there's only so much time, do you know how to like laser in on like the essence of the way that you would leverage it so that you could still complete the entire exercise in 35 minutes?

25:07 >> That's great. So it's like using AI as a thinking part partner, but in a smart way to get to better, higher quality answers and considering the speed as well. Like if you go click off a deep research, it's going to take you 20 minutes. That seems like kind of a waste of a thread, right? Or something like that. Yeah. And I and I and I think based on what I've heard from some candidates so far, your your your ability to move forward in the interview without using AI is not a bad thing, right? Like if you have a strong enough intuition and you have a strong enough mental model for the product you're talking about, the space, the context, the audience, and you don't get stuck and don't need AI, it doesn't seem like they're going to penalize you for going.

25:53 but if you do choose to use AI during the interview process you will be it's like you'll be evaluated if you use it or don't and then you'll evaluated you'll be evaluated based on how you use it but but not using AI is not in itself a for for like segmentation for example is not going to ding it's not a failable offense from what I can gather which is kind of how life works right like we all have the option to use AI at every moment of the day we obviously can't you'll never get anything done if you're just using AI all day so you have do some things and take action in the world. So figuring out when to use AI, when not to use AI seems to be what what they're really trying to get at is when do you use it versus when do you rely on your own intuition and then how >> I think I would imagine if I was interviewing somebody like what a successful use of AI would look like or even if I was sitting over somebody's shoulder at work it would be somebody who has it running constantly trying things throwing stuff at it throwing stuff against the wall taking the good and not getting stuck on the bad and that that's It's like the just enough just you know like the ju just enough AI not like getting stuck in it and like taking it too seriously but when it does something good and creative that your your mind you know wouldn't have done on its own on the spot then like you know gleaning that quickly and and making use of that.

27:12 >> Totally. Also, another thing that used to happen historically, I don't say used to happen because it's still the traditional interviews are still running, but if someone gets asked about a qu a product about a space they're not familiar with, like they get asked you know the the one that I talk a lot about in my course for example is like a product for volunteering. So, if you get asked about that, you just don't know too much about the volunteering space. Historically, that would actually be a bit of a handicap in the interview that you'd need to ask the interviewer to tell you more about, you know, what volunteering means, which is kind of a pen, it's a penalty on the candidate.

27:44 because they don't know, like, you're expected to know basically what something like volunteering is. Whereas now that you have AI, you could be like, cool, like I'm not I want to make sure we have a shared definition of volunteering, so I'm going to go and I'll ask Met AI to give me a good working definition of volunteering that I could use for the exercise, right? so, so I think the ability to like replace gaps in your knowledge and shaky assumptions with things that you could just get some like real time placeholder content from that that's trustworthy. Like I suspect meta.ai will give you a pretty good working definition of what volunteering means and then you could anchor to that for the exercise. versus just feeling stuck. and I think that's probably something that maps to like the real world too. Like you're we run into ambiguity all the time with stuff. So I think trying to figure out what something means and using I mean it's basically using the internet is just like a new form factor for using the internet. Ben, can I can I take that analogy to the limit a little bit and say, >> "Yeah, >> in this interview format, what's stopping someone from just using transcription or or like dictation rather or just saying, "Hey, I'm in the meta AI product sense interview. Here's the prompt that the interviewer asks me.

28:57 How should I answer it?" And then using that as a guide, like what do you think an interviewer would think about? like maybe and I know not this part is not ground truth but just speculation like what do you think would that be an acceptable approach like why wouldn't that be an acceptable haven't seen that attempted yet I think it's it's risky because don't forget you're you're playing a time management game too so the question is how much do you want to rely on AI for it and if you could you could potentially find yourself in A I mean my answer to this is basically why not use AI to cheat in interviews.

29:35 Okay. So like the the the reason not to use AI or like my co-pilot for example I tell people don't use it in live interviews use it to practice. The reason I say that is because you're almost like crafting an argument one step at a time and you need your argument to be defensible. So the interviewer should be able to ask you probing questions, follow-ups. It should not crumble under a basic line of questioning. Okay. So, if you're like me, when you use AI, oftentimes it it at the first hint of disagreement with what it came up with, it will immediately go to the other side of of a of an argument and tell you why it was completely wrong and why it should do it differently. It's just trying to appease you. Okay? And if a if a candidate does that in a real interview with an interviewer, the interviewer will be like, "This person has no backbone. This person has no conviction. This person is just trying to please me. This is not going to be a strong PM." Right? So, for those reasons, you actually do want to drive the structure yourself as a human. And you every time you make anything that even resembles a decision to move forward in the interview, you want to feel like you could defend it. It might not be bulletproof, but it's like wellreasoned. And I don't think you can get to that level of conviction by entirely outsourcing your thinking to AI today.

31:03 >> Aman, there's something in your question that I think and Justin also put this in the chat. I think there's something really cool or really skilled about working AI into conversation by using dictation like you mentioned text speech to text. So, you know, something I like to do when I'm sitting next to somebody or on a Zoom and talking to somebody or doing a demo is I if I'm already saying something to the person, I can hit the hotkey. I use Whisper Flow now and it's just a does a great job. I used to use better dictation. There's like a billion of these. and I just hit the hotkey and like I'll just keep talking and it's also, you know, dictating that. Maybe I'll then I only need to type in a few more words to like kind of make it make sense in the prompt box and I can hit enter. I think it's also kind of like the I would I would again speculation, but I I can imagine like what does like a skilled person, you know, using this fluently look like is it's it's you know, working it in so that they're not doing work twice.

32:00 treating AI like a teammate. I also think because AI takes LM take time, there's kind of like a cooking show element to it. Like you put something on the stove and you're waiting for it to sizzle and you like keep talking and then you go back, right? And like and and being able to juggle these things I think would also has at least very good optics. I I also think it's very effective. I think and I I love I love that answer from Ben because I think a lot of it it almost seems like on on First Order this interview is meant to gauge how candidates use AI, right? It's like here's you know you have this tool like how are you going to use it? But I think that the inverse of that is just as powerful too which you're alluding to Ben which is like are you too deferral? Do you reach to it too much? Are you letting it do all of your work for you which is we're still PMs, we're still product people. we're still interviewing for this job at this company and if you're just going to outsource all of it to AI, that's actually a negative signal. And so there's this like sweet spot in the middle which I think I think is a little bit of the definition of like building AI product sense is also you know >> giving you that intuition for like how to like demonstrate that in an interview format and and I know when we interview people at Arise like the 30 second story here is we we really look for like technical depth in in our PMs because we sell a technical product that's eval and observability for AI agents and you know a ton of companies out there that have a AI in production and so we would ask people to like try our product which means that you'd have to like go and code an agent as a PM try out our product and then come back with ideas for how to improve it well now you can go and vibe code an agent in like 10 seconds so that part of the the interview loop like has actually gotten easier which is great because now we can ask we can ask people to to show us their reasoning and thinking for why they built an agent a certain way. And if they're like, "Oh, I don't really don't know. Like, the AI did everything for me." That's a negative signal. But we want you to use it. We want you to be effective and efficient at your job. We just don't want you to not know how things work at the same time.

34:06 >> Exactly. Exactly. you know what's coming up for me is like I wonder why this this transition hasn't happened and when just like with the internet, right? Like people have been able to Google stuff for so long. like why why did we not allow people to use Google or to use the internet during product sense interviews for a really long time or just computers in general? and I think there's something interesting happening where the LLM like the evolution in AI tools seem to have just changed people's paradigm and how they think about thinking. and I think that's the reason is because you ne you never looked at Google as like something that could complement your thinking. it was just like a resource for research.

34:46 Whereas now with so many people outsourcing their thinking to AI, it seems like that's and because think you're you're on the spot thinking skills are what's being evaluated in these interviews that is what seems to be causing the landscape around interviews to change. It's because it's not just resources, it's it's thinking firepower. So >> yeah. So like one way you could use this tool is you develop this rhy this line of reasoning or thinking to to or structure to a problem or or a part of one of the interview problems and you can you know you could potentially ask the the system what are some some pitfalls with this push back. What are the things that you know you can almost like simulate the interview during the interview a little bit of like what are some things I should probably bring up or that I'm missing in this this story potentially.

35:35 >> yeah and and the thing to be mindful of if I was a candidate going through this interview and let's say I chose to not use AI for the motivate the the first part of the interview the motivation and u mission and let's say for the purposes of this this point I'm making let's say I even chose to not use AI for segmentation for the second part but then I get to problem identification and then problem identification is where I think oh I should check my thinking on these problems with AI guess what if you just ask AI what do you think about these problems it will have none of the context that it needs to give you anything that's thoughtful. So, the skill that might be evaluated in that moment is how do I bring the AI up to speed on what we've already covered in this interview to give it the context that it needs to then help me potentially come up with some with some useful critiques of these problems that I came up with. And can you do that fast on the spot? And I do think voice dictation would be the way to do it. But if I was an interviewer and a candidate got to that point and then they like super succinctly were able to like fully summarize what we did so far in that to that to that point in the interview and then introduced like a thoughtful question to the AI and just that was all the prompt and the AI came back with something and then they like talked out loud through how they would reason through what to do with the response from the AI. I'd be like, "That seems like a very successful session, like a a successful exchange with with AI that demonstrates that they know what this what the stuff does and how to use it.

37:00 >> We're getting close to the hour. We started a little bit late. So, I I'm going to pick some of the some of the strong and high emojied questions I've been keeping track of in the chat." Actually, there's a a bunch of questions that I put under like lightning round. They're like very yes or no or close to yes or no, Ben. >> okay, let's do it. >> So, I'm going to throw them at you and then we'll get to the more nuanced questions. So, from John, anyone seeing or hearing requests to prototype an actual solution as part of Product Sense yet?

37:34 >> Not at Meta. >> I've been hearing about >> Yeah, I've been hearing not at Meta as as Ben said, but I've been hearing about other companies testing out this interview format. So, >> I can see more of that if that's interesting. Yeah. And >> if you want to go and dig in. >> Yeah. I just made a post on LinkedIn last week, I think. someone I know, again, not going to divulge the company or anything identifying, but told me they had a product sense interview.

37:58 it started with a data dump. So, they were given a bunch of rows of data about business metrics. I won't say which ones. And then they had to basically there was a there was a problem like there was basically a problematic trend that they had to diagnose in the first part of the product sense interview. And then they switched gears into the second part of the interview to they were given they were given a resource that kind of shows them what the product flow looks like in that part like basically there's a drop off conversion problem and then they got screenshots of what that looks like and they were basically had you know 15 minutes to vibe code an improved version of that flow based on the problem that they diagnosed. So that was kind of definitely something that was happening in that interview which seems kind of interesting actually like an interesting way to to to evaluate this stuff.

38:47 >> from Lee is Meta also assessing how much a candidate knows about LLMs like are they looking for you to use AI lingo a prompt unstructured way? I would say that at this point seems like it's going to fall to the interviewer to determine how deep they want to go, but I don't think that the candidate is from what I've been told so far needing to get into anything super technical about like you know eval like how eval work behind the scenes. they might be expected to comment on how they might use evals to det like to mitigate risks associated with like the solution that they go with. But yeah, that's that's probably as technical as it will get from what I've gathered.

39:33 >> I hear if you're interviewing for an AI PM job that that will probably be more part of the loop than like meaning like if you're on a team that is incorporating AI in as a product or feature that that would probably be expected. I would also guess that would be like a cross functional interview, not the product sense interview. >> Correct. >> Which I met all of the interviews are run by PMs. So or at least like the the core PM loop is run by PM. So I think that would be that would be an interesting evolution of the process if you started to also have like engineers interview PMs like that would be a pretty dramatic deviation from the historical structure.

40:10 >> And we have a question is it we covered this is that the meta.ai AI or another internal model or any model. >> so so it's a product. Meta AI is a product. so yeah, people can go to like meta.ai and I think they'll see it. >> Great. and okay, yeah, that that's for the lightning round. so some bigger questions, how can we address hallucinations with AI in the interview? For example, should we be asking the interviewer to confirm the output of the eye? Should we verbally address that we are making the assumption the output is not a hallucination?

40:54 I would definitely not ask the interviewer to be the arbiter of what you choose to proceed with and what you choose to throw away. in a lot of ways, the discernment between the baby and the bathwater is, I think, what you're being evaluated on your ability to to do. so definitely don't outsource that to the interviewer as far as how you can tell what's what to do about hallucinations and stuff is look like it's kind of where the rubber meets the road on a daily basis for for every one of us is like we're going to get some stuff from the AI that is good and we're going to get some stuff that is not good. And I I you know, one of the reasons I'm excited for this course you two are about to start teaching is because I think that day-to-day discernment of what to use and what to what to ignore as part of your workflow on a individual productivity level, I think does scale to like the types of decisions you you need to make about AI and the intuition you have about AI in like an interview context or like as a team or on the product you're building.

41:56 so I would say yeah the the failure state here is just like taking what the AI gives you at face value and running with it without applying like critical thinking to it is I think like just like the far end of the extreme of what you need to avoid. And if and then if the other side of the extreme is like the perfect the the person who makes the perfect you know usage of AI generated insights at every step of the way and uses it at the exact right time you know like the standard is like the ideal here is unknowable. It's almost impossible because none of us even know what perfect usage of AI looks like. So I think it's mostly trying to avoid the failure states of of outsourcing your thinking in a lazy way to to to AI >> or or never reaching for it when it's like something that makes sense to reach you.

42:45 >> Would you say that? >> Totally. >> Yeah. >> Yeah. And and even if you don't reach for it during the interview itself, if there is that kind of reflective check-in at the end of the interview where the interviewer asks you, hey, like if you had more time, like how would you have used AI in this context? at least have something insightful to to say about moments when you thought about using AI but you chose not to for which reasons, right? Like because that that's that's interesting too. Like if someone thinks about reaching for for AI but doesn't, right?

43:13 Tell me that's not interesting. Like what what was happening in their head? You're learning something about that person by that decision of not doing something. Right? So that's that's what I would go with too. there's a really really really important point. You've you've said this in kind of different articulations and people are asking it in other phrases. So I just want to really ask this again to make it super clear. product sense with the PM using AI in their job or product sense for AI native products and services. Thanks Lee for for that articulation.

43:54 Which one more resonates with what you're hearing? >> >> well, the the interview is always going to ideally be designed to evaluate some proxy for on the job skills and qualifications. That's at least like a definition we could work with. So, if someone is watching this and they're like, I'm not interviewing. I love my job. I'm staying in my job. I'm not taking calls from recruiters. I'm just trying to get better at product sense on a day-to-day basis, building AI products. then, you know, all the stuff we talk about with interviews just doesn't matter. Okay. and what they need to do is they need to basically take everything we've been talking about and apply to their day-to-day life.

44:40 Like, how do I choose when to reach for AI, when not to reach for AI? How do I choose what to do b based on what I get from AI? How do I prompt my way through all that stuff? in an interview setting, you are the the the difference is it's like a time boxed thing that you're driving. It's a time boxed session you're driving. There's a clear place you need to land and you have a life raft on the side that you could choose to reach out to or like sorry a little bit more dramatic analogy, but it's like a loaded gun. Do you want to reach for it or not? Right. and if you guys remember that game like the Who Wants to be a Millionaire, like that old like game show on TV, like you had these lifelines and like you were I I used to be really interested in when people chose to use the lifelines because there's you only get three of them. I think you could call the audience, you could call a friend, you could, you know, have a 50/50 elimination, whatever those were. I'm just remembering those on the spot. I don't know how that's still in my head. but the idea is that when when people chose to use those lifelines was interesting and it was felt like like a like a deliberate decision. So in a in a real interview, you can only reach for AI so many times based on what we talked about. So your your decision of when to use AI and and when not to has real implications on a specific result of like an interview, right? Whereas on a day-to-day basis, like most of us don't have like an immediate feedback loop on whether we like failed today or whether we aced today. like we don't have this kind of like value judgment on on a on a on a daily basis. So that's kind of like the difference I see.

46:08 >> I want to throw in one of my questions. Ben, can you share that slide with the Lenny diagram? The diagram you had on when you were post. so when I was in engineering school one of the things that like was shocking to people who weren't were were in other departments not engineering was that our exams were not just open book but they were also open laptop with internet. and that was just because it's not going to help you, right?

46:39 to to pass the the exams and you know for a lot of engineering classes like you really have had to internalize and really internalize that way of thinking and and no amount of looking anything up on the spot is going to going to save you and that a lot of time. and yet right my degree was not called chemical engineering with internet. It was just called chemical engineering. So, why I want Ben to bring the slide up is the on the left is the product sense with AI. On the right is product sense.

47:08 They're exactly the same except for the section number five on the bottom left, right? >> And and some of the specific call it like sub bullets of the of the first four points, but yes. Yeah. >> And I I'm going to bet I'm going to make a you know, I'm going to take the risk of predicting the future. My g my guess is that we look six months ahead, a few years ahead, there's not going to be a product sense and a product sense with AI interview. There's going to be a product sense interview. Right now, they're probably just making a distinction because there's two different rubrics and it's a slow roll out. That's my my guess of what's going on hearing Ben share insights. And I I I think this just hammers home how much it's going to be very much the same thinking like to to learn the same way of approaching it. the AI is not going to change it dramatically. It is going to be a nice thing if you know how if you are thinking the right way and you are approaching problems the right way already as a skilled professional you know like then it's it's only going to help. but that's step one.

48:12 so I I also want to emphasize if if you're here hearing our voices live or recorded and you are interviewing or have an interview on the horizon for Meta and you anticipate a chance of having this interview like what the resource you're looking for is Ben's course. That's like as you can see that's like what's going on here. it's this it's 90% if not 95% or 99% like a product sense interview. >> I appreciate that. I'll just take a chance. I know we're at the hour so I'll just plug plug my course. you can just go to maven.com and look for my name, Ben Arez, or you can just Google Ben Arez Maven. You'll find this.

48:46 it's also on Lenny's List on Maven. And, I recently made a big update to make this available on your time as candidates and students, not mine. It used to be all live. Now, it's largely, pre-recorded and has access to both limited access to to me for one-on-one questions as well as my my interview co-pilot, u, which you can learn more about on that page. And if you use the Lenny's List promo code, you'll get $100 off the course. So, I appreciate that plug, Paul. can we also plug your guys' course just while we're at this because I think it's it's very relevant to what we were just talking about about how these skills are. Yeah. Actually, I I think I'm in the driver's seat, so I don't know the exact timing of these bullet points, but you want to just tell me next bullet point and I'll I'll hit it.

49:31 Yeah. >> Yeah. You can you can just let it rip. We'll just we'll just talk talk through it. I'll do like a quick quick quick 10 seconds on it. which is we're running it this weekend right now. Tal and I don't have any other workshops planned like this. So it's very much a something we've wanted to do and work on together for a long time. So we like to think it's like the culmination of where we both excel, which is like how to use productivity tools, how to use AI tools in your day-to-day, and then also learning about how they work in the process. if this conversation resonated with you, if learning about when to reach for AI, when not to, how to get better results from AI, whether you're interviewing or you aren't, or you just want to apply it more and more in your day job, that's something that's interesting to you. This could be relevant. so yeah, I don't know anything else you want to add there, tall, >> I think it just ties back to the analogy at the very beginning, right? Like just like in for the last decade there's been the product sense interview and then there's also been product sense on the job and they sound similar. They're related. They're different things.

50:37 what am I working on here is more the the long-term like the actual product sense for using AI as an enabling technology on the job. >> Yeah. And and eventually I think like you're saying like the interviews will probably end up converging speculation prediction predicting the future. I think our jobs are going to start converging more and more around how to use this technology and it and so we wanted to take a little bit of a peak ahead with some of our own systems that we've built independently and share those with you all. So >> awesome.

51:09 >> Cool. I think is maybe with that we stop screen sharing. Do do we have time for more questions? How do we want to do this? what we'll do I think we have a lot of questions in here. I try to cover I think a lot of them I was scanning a lot of them you've answered on the way. a lot of the ones that I had on my list that got up voted so we've we've covered like 80 to 90%.

51:29 >> >> and I dropped I dropped all them in a Google doc so we can you know we'll take a stab at >> answering them async and we'll send out you know Google doc with some of the answers along with the resources that Ben shared in the recording tomorrow. tomorrow, exactly in 24 hours. Aman and I are also doing just a chill hangout AMA both for our previous learning lesson or this learning lesson. If you want to ask questions, bring your questions live. I'll send out an email tomorrow with all of the resources, all the things we mentioned.

52:02 I've been keeping a list throughout this entire session of things people have been asking for. links to a lot of free resources that Ben, myself, and Aman have have put out there, so they're all in one place for you. Well, thanks for for hosting this, guys. And also, I apologize to the audience for the the initial Zoom link shuffle, but I appreciate it that that all of you chose to to follow us from one Zoom Zoom room to another, which is a big ask.

52:27 So, thanks. Yeah, plus one to that. Thank you all for joining and and participating. This was a lot of fun. Ben, we got to do this again. this is this is >> it's an evolving situation. So, if I'm sure we're gonna have plenty of stuff to do in in January and February to talk about. So, yeah. Cool. >> And the recording will be sent to everyone. So, thank you. Bye bye everybody.