How to interview product managers for AI fluency (Ben Erez, Tal Raviv, Aman Khan)

0:01 All right, Tal, while we're listening, how do you discover music? What's your How do you find these bangers? >> Good question. >> I love Shazam >> and I love even more discover on Shazam >> like in the real world using Shazam. >> And my favorite my mo my most satisfying moments are like a car pulls up next to you playing something and you can like Shazam what's going on in that car. That's like, you know, if I Shazam things that I'm like >> should not be easy or like just before the light changes or while I'm walking by a coffee shop or something. So, >> I rate I rate my coffee shops by Shazams per minute.

0:40 >> Yeah. And I I'll give one more quick tip here since we're talking about tech as well. You can add a Shazam shortcut. So when you when you like pull down from the top of your phone and it's way more subtle to just pick up on on songs without being like I have this big blue app that's like listening to everything. Yeah. >> Where's the speaker? >> Where is it coming from? >> You know what I haven't tried to do yet is I haven't tried to use like chat GPT advanced voice mode and have it and like play music in the background and ask it what song is playing. Like do you think it it would detect it?

1:17 That's like basically there's a new Google one too. >> There's like a Google Shazam thing where it's like you can hum apparently >> they would bake that into Gemini. I would imagine like the Gemini >> pace of change is too fast. One one Shazam every 15 years is enough for me. >> All right, man. The song ended so sad. >> Yeah. >> Well, welcome everyone to our our session today. We will not be talking about music the entire time. that was just to get us get us warmed up. But I want to welcome my friends Tal and Aman. How you guys doing today?

1:54 >> Really good. >> Yeah, >> good. >> I'm glad that like our our Lenny Post didn't set people's cursor on fire and like No. >> Yeah. >> Yeah. For those who >> cloudbot, >> for those who have not checked their inbox this morning, assuming everyone's probably subscribed to Lenny's newsletter, a beast of a post just went live today from Tal and Aman. I had the distinct pleasure of seeing very early versions of it and I I I think it came out incredibly good. so I'd encourage people to go check it out.

2:26 And for those who have been, you know, one of the things we'll talk about today is obviously the the way PMs are being evaluated around AI fluency. And I think that getting comfortable in cursor in in developing what what Tal and Aman have been calling kind of like this AI product sense is table stakes for for a lot of people in this market. So I won't we're not going to spend too much time talking about that specifically, but I would definitely encourage people to go check that out and congrats on on shipping it.

2:55 Yeah, my my my promo for it in one line is like it's like the in the 90s there was like the put on your 3D glasses now for movies like when that came that's that's like the Lenny newsletter equivalent and I won't say more than that. >> Yeah. Or like the Goosebumps choose your own adventure. >> Yeah. >> Cool. well with that said I think we have quorum to get started. So I'm going to go ahead and share Oh. Oh no.

3:26 Previewing. Okay, here we go. >> Nice. >> That's That's what I That's what I get for trying to be prepared. okay, so yeah, today we're gonna do for those who have never joined a lightning lesson, these are kind of like free sessions hosted on Maven. this is scheduled to be about 45 minutes with 15 minutes of Q&A at the end. we're going to you know, we think of it as interactive but more like back backloaded interacted interactive. So that means like you know everyone's muted by default. if people post stuff in the chat through the session, we're not really going to see it. We're going to try to be really present in the conversation and then at the end we'll open up for Q&A. So if you are sitting on a question you really like us to get to, my my advice for increasing the chance that your question gets answered is to wait until the end.

4:13 and then ideally if if people can upvote or you know react to the questions that they want us to answer, that'll help us prioritize if we're tight on time. for those who we'll do a little bit of intros on each of us, but I just want to start with some housekeeping. So, the talk's going to be recorded. So, you know, feel free to do this in the background or to listen in the background or just like pay attention.

4:36 You don't have to worry about taking notes. You'll get the recording after and be able to spend as much time with this content as you'd like to really soak it in. we're also going to share the slides. this happens to be a pretty slideheavy talk today relative to some of the other ones that I've hosted in the past. but that's just because we really wanted like there's a lot to say here. So, I think you'll you'll get a ton from re-watching that or consuming the deck. And then, we will, like I, like I mentioned, keep people muted just for recording quality.

5:04 and you know, Aman, if so, we realized like I I had no way to make them co-host. So, if they just need to mute for certain parts, they're going to have to like ask me. So, just people be aware that that's going on in the background, but hopefully it doesn't mess with the flow of the conversation at all. yeah. All right. With all that said, let's get started. So, AI fluency and specifically evaluation of PMs when it comes to AI fluency, why does that matter? Okay.

5:36 I feel like if you look at every job description for product managers these days or you talk to hiring managers that are looking to hire product managers one thing is clear at least like I spend a lot of my time in this space talking to candidates and hiring managers and also doing like building interview guides and AI is kind of like ubiquitous throughout what people are thinking about. They're thinking about the implications of AI in the products they build. They're thinking about AI in the implications of their culture at work and how they how they get things done.

6:08 And then they're definitely thinking about AI in the in in the context of like how can they better solve the problems that their customers are experiencing and what they build and ship in the market. So no one is hiring a PM right now without thinking about AI in some sense and ideally they want PMs that are very AI native. And that's going to mean a lot of things. So, we'll talk about we're going to unpack that. That's kind of like the point of this whole session. but theoretically, the more the talent the better the talent a company can accumulate, the faster they'll be able to move. It's no longer kind of like a game, a bottleneck about technology or the ability to like get as much code coding done or make a technical breakthrough as as it used to be.

6:50 given that a lot of the hiring managers themselves have never been through an AI focused PM interview because maybe they've been in the job call it pre-Chat GPT or they they've been in this job for more than two years even maybe the first year of chat GPT no one really was evaluating any of the stuff. So you could really think of most people kind of in the hiring position figuring this out as they go. right now there's a lot of experimentation and tinkering with how they might evaluate something they themselves never had to actually prepare for as a candidate. and the biggest takeaway that I think I have so far is if if a hiring manager lacks clarity on what they're evaluating, it's it's unlikely they'll just kind of like happen to stumble into a good rubric for evaluating people. So it really does need to be an intentional practice for people to to kind of get where they want to get. So with all that context, Talon and I hosted a session a few weeks back about Meta's new product sensor AI interview. We're going to come back to this, but I want people to keep in mind we're going to do a little interactive session later in this in this call. And I want people to start thinking about which which bucket of interviews this will fall into. And we're going to give the answer to what that counts as, but it's just an example of something Meta is doing that we'll want to give as a concrete case.

8:18 Okay. So, who's this session for? Hiring managers that are designing PM interview loops, recruiters who are screening for AI fluency in the PM candidates that they're looking for. PMs themselves who are preparing for interviews at AI forward companies and want to get a bit of a pulse on like what people are thinking about and what's happening in the market and ultimately also leaders like founders, CEOs that are thinking about building their AI native team. So I think there's really something in here for everyone. but I'd say the most actionable bits will probably be for hiring managers.

8:49 >> Ben, by the way, Aman is muted and can't unmute himself. We could we could leave him there. It's up to us. He doesn't Oh. Oh, damn. Okay. All right, Aman, I guess you can talk. >> Yeah. Well, okay, I guess I'll go first. awesome. Thanks for having us on, Ben. it's a pleasure to be here. Just by way of background, my name's Aman. I've, been been lucky in that I've been building AI products for most of my career. starting on self-driving cars and eval cars. Worked on Spotify's machine learning platform.

9:17 And now I work as head of product at a company called Arise where we help with eval products and agents. So very relevant to the building AI products part of this and I'll hand it off to Tal. Hey >> everybody, I'm Tal. I up until a year and a half ago I was either a founder or very early product manager at Hyperrowth PreIPO companies. And a year and a half ago I I went on Lenny's podcast as a guest and we did not nothing on the agenda had to do with AI.

9:50 and I found myself like sharing something that I was doing at work and I thought that was like, you know, late to the party and like Lenny spent a lot of time on it. And then I after the episode I was like interesting. and that kind of made me realize that there's a no matter how far AI had come, it was actually like a really mo so much of the value was like how we use it as PMs. And so I've spent the last year and a half helping with that and focusing on that. Most recently, Aman and I started a course together where we realized that like how you use AI actually leads you to having much stronger intuition for building with AI and all comes together really nicely.

10:26 So, yeah, we all work together. >> Yeah. And then we all connected the dots on on this collaboration last summer where I was sharing with Talin what I've been thinking about when it comes to PM interview prep. I teach a course on Maven specifically for product sense and analytical thinking interviews like the ones used at Meta, but a lot of other companies use them now like OpenAI is copying that playbook. Stripe Door Dash have variants of the Product Sense interview. and I kind of like posed a question like how's this all how's this all going to evolve?

10:59 You know, like I feel like these interviews have still really not changed in any way since AI came. So like what's going to happen? and that got us just like talking pretty continuously leading to you know we did a talk about the meta product sense one with AI and then I think that really helped crystallize a lot of thinking that like I kind of this session is going to share a framework that I've been just like working on but Tal and Aman have really helped me shape my thinking around this this way of structuring interviews and this is our first time talking about it in public so I think we're you know I'm personally just like really excited and if I'm honest a little nervous to share it because I think it's it's not going to be fully baked and but I think it's it's got a lot of good bones.

11:45 So how are we going to spend our time? so yeah, I think we're already deep into this. we'll I'll quickly talk through the high level of the framework kind of not burying the lead. and then we're going to spend the bulk of the session going through the framework. Tal and Aman are going to be subject matter experts in a couple different areas that I am not a subject matter expert on. and then we'll wrap up and do Q&A.

12:09 Okay. So you know a sensationalist slide to get started. this is all developing story. Okay. So I I don't want everyone to take what we're doing. We're like try we try to be super intellectually honest about everything we do. But I think in this one, it's particularly important to not pretend to be some kind of like definitive authority on the right way of doing these interviews or evaluating candidates. This is probably going to evolve every 3 months at this pace, maybe more. but I think people should feel like they're getting a really solid on the ground reporting snapshot in time of what's of how these these these dimensions for evaluation are shaping up.

12:49 Okay. So high level framework you got to think about interview design as as a product design problem. That's like the fundamental principle that you know kind of everything else we talk about is going to be a double click into. And every decision you make in how you design your interviews should be intentional and it should aim to gather the most important signals for your specific job while being really explicit about trade-offs. There's no perfect interview process. It's all about trade-offs, just like product decisions.

13:20 There's also no single correct way to structure a loop. It really depends on the role, the team. you also don't want five of the exact type of PM on your team. You probably have some spikes you know, in some areas. So, like it's all very team and role dependent. So, I want to make sure people are very clear with that. And the goal of using a framework for designing interviews is to reduce the cognitive load like the amount of you know cycles you have to spend designing the interviews so that you could really maximize the time you spend on your real job which is solving customer problems. Not saying your job as a manager is not to grow your team. That's definitely part of your job. But but ideally you're growing your team in service of of solving customer problems and creating creating business value.

14:03 So, I just don't want people to think that design like interviewing people is it should be a full-time job because it really shouldn't. okay. So, within our framework, the way the way we want to talk about this is there's two highle AI fluencies. fluency competencies is the way to think about them. And we want to spend a minute hanging out on definitions up front because, you know, we're really going to again flesh out the rest of this in in the talk. You could think of the first bucket as using AI to build products. This is the bucket that Tall is kind of going to be our, you know, subject matter expert on in in today's session. which is really all about like using the LLMs and the agents to get stuff done. Strategies, data, prototyping, research. Imagine all of the things that you have to do as a PM just to get product out the door. How do you use AI to do all of that? Okay. And then I would say the second bucket here is building AI products. And this is where Aman's going to be the subject matter expert in this in this session. And this is really about like getting you know Aman has taught a course you know prototype to production.

15:11 It's kind of everything that comes with actually deploying large scale production AI applications to real customers in the market. Not every job is going to need that skill set but I'm sure a lot of people are thinking about hiring for that skill set. So I think that's the area where AI Aman's expertise around like eval observability and a bunch of other stuff is going to come in. Ben, the second subject matter expert has muted himself once again.

15:37 >> Thank you. >> Yeah, if you if you want to remain muted, you know, you're welcome to. you could always raise your hand, too. I guess that's a Zoom feature. But anyways, thanks Tall. You're going to keep me honest. >> Cool. And just to further visualize how we're going to structure this because you know there there's a good amount of slides I want people to be clear. So we're going to basically go rowby row. So we're going to start by talking about using AI to build products and we'll go to building AI products.

16:05 Now what are these columns? Okay, so when you're designing interviews, you know, and this is just like me talking to candidates like there's basically three highle buckets of what people, you know, do in interviews. There's behavioral interviews. So these are your like hiring manager calls, your recruiter screens, your XFN interviews. there are live cases. So this is your you the stuff I teach in my course for example like product sense interviews, analytical thinking interviews. those types of things. And then there are takehomes or presentations like you'll get a homework assignment and then you have like a week to complete it or 24 hours or whatever it is how the company does that. And sometimes there's a presentation sometimes there isn't. But at a high level, these are like the three main types of you know ways that companies are trying to get signal from candidates. And I think it's helpful to fit you know the way you think about interview design into this general framework. Doesn't mean you have to have one exercise or one interview per cell but it's you know you got to be explicit with which of these cells do you want to use. it's like a like a menu you could order from.

17:15 okay let's keep going on. anything so far? All right. just while we're here, before we get into it, I want to let people know if you end up wanting to get help while we talk about the live case style interviews, you could use AI Fluency to get 10% off my course. You'll get an email after this with the link. I don't want to spend too much time on it, but it's been a real kind of like passion project for me to teach this those specific interview types. They're not going away, as far as I can tell.

17:45 They're evolving in interesting ways. but I but I think if you're trying to get ready for those, would love for you to check out my course and potentially go deeper, with me. And a lot of the insights that, you know, helped me shape my thinking around this came from, something that I've been working on with my partner Mark, which is called Insider Loops. And Insider Loops, we basically interview PM candidates who are successful and not successful at top companies as well as hiring managers and calibrated interviewers from the same top companies. and put together kind of like a like a like a master guide on what people should know about the interview processes at the companies including pitfalls and things that work well don't work well. The four companies we have so far Stripe, Figma, Uber and Door Dash, but we're working on more. So, just to kind of let you know these these insights that that that we'll be sharing are kind of super well informed by many many hours of looking at these types of kind of interview guides and building these types of interview guides.

18:47 Okay. So, the the matrix walkthrough. So, all right, Tal, I'm going to set a timer. Let's try to bang it out in 13 so we have some more time because I think we're a little behind. >> Okay. >> so, where are we? We're in this top left cell of the matrix. Okay. So we're going to talk about using AI to build products and we're going to talk about what you can do here on the behavioral side. So the high level question you're ask you're trying to answer here is has this candidate actually used AI tools to ship faster and better. That's what you're trying to evaluate.

19:25 typically you could think of this as you could probably get this done in 45 minutes and you'd want a hiring manager or a senior PM or someone like you really trust as a cross functional partner to evaluate this kind of signal. So that's like the high level about this. We're going to go into, you know, some example prompts here. So the types of questions you could ask are, you know, like walk me through a time you used AI to solve a product problem.

19:50 make a decision or improve a relationship or tell me about a time that, you know, AI tools changed how you think about approaching the work. Like what did you try? What worked and didn't work? Tell me about a task or a workflow that you couldn't have completed without AI. describing a situation where you chose not to use AI and why. What's something you found yourself doing with AI frequently and turned into a workflow and how have you used AI to improve your team's performance? These are the types of things that you can ask someone to get them going and really probe as a hiring manager to understand how they use these types of tools to get the work done.

20:27 >> Tal, I'll just cover this quickly and then I'd love for you to to to chime in and share some thoughts. So if you think about like a rubric like an example rubric you could use like a very simple one you could think of like three different dimensions here. One is the depth of usage the second is their judgment around the AI tools and the third being the impact like how you know like the actual results that they've been able to achieve using these tools and strong signal can be things like you know really being specific about the tools and the workflow. If you can probe deeper it kind of like the picture be gets more texture is kind of another way to think about that. they and they can really articulate when AI was helpful versus hurtful and how they adjusted their approach and they don't just take the first result that they get from AI, but they kind of demonstrate repeated refining of prompting and to get the outputs that they want and then just being very concrete about connecting that AI usage to to specific outcomes and results.

21:21 >> So T, I feel like that's a lot of me talking, but I'd love to get your your thoughts on on this section. one of one of the important things I think to say here is that we've all I'm sure many of you have heard of the term the jagged edge of AI. Like the you know, it's like really smart in some ways and like really dumb in other ways. And and that's changing. It's like it's getting smart at a higher percent of things. but it's still like surprisingly dumb at other things.

21:46 And I think it's important to take that into account when you're evaluating somebody as a candidate or even setting expectations of people in a role. that what you want to look for is like you want to have these stories these like let's talk about a story let's talk about a time when like try to focus people on a on a peak moment let's not evaluate them on an average or like the you know a bad moment let's like say like like a moment that you know AI really did its best like let's look at the best situations and like first of all let's try to hone in on on those cases I think it's way more fair to people and you can kind of filter out a lot of noise around like you know what is the use case. and yeah, peak moments.

22:30 it's also I find myself saying this a lot like fancy isn't a value. So if they're using cursor versus chat GBT project, I don't I don't think somebody gets points for that for in its own sake. it's not like oh they they've been you know refreshing Twitter more or or something. It's more like how do they you like if they're using a tool, let's take that as a given within that, you know, how are they how are they surfing that tool, right? Are they like over overly you know, doing this like theater of like writing this incredible prompt for five hours and then hitting enter and it's supposed to get perfect result or they just like diving in, throwing some context in, you know, seeing that it wasn't great, iterating.

23:15 that can happen or not happen in any tool, right? Like are they kind of like frozen up and paralyzed? Are they kind of like leaning in and and happy to happy to just try and see what works and not worried about the downside? Are they treating it as, you know, like if it doesn't do what they wanted? Are they treating it like a like a failure of some kind? Are they treating it kind of I think the best analogy is driver's assist or like mobile eye where you know if your when your car beeps and something it's about to hit something or something's in your like blind spot, right?

23:45 and your car beeps and nothing's actually there. Nothing bad happens. You don't freak out. You don't you don't swerve your car into the median just because it beeped, right? There's no downside to that. and if there is, it's amazing. There's incredible upside, right? Infinite upside if if it keeps you alive. So like the the same approach and the same mindset for for using AI, it's like can you focus on the only the good, don't worry about the bad. like there's not like oh it failed completely at this. I think that's like the kind of mindset and you want to you want to see people talking about when they when you go through the the story.

24:25 yeah I think another by the way there's also another dimension to this. This is kind this kind of gets like very multi-dimensional and then we'll stop talking is like there's as a as a person and then there's like context or AI as a team sport. So like what what do they systematize like Ben was mentioning? What do they turn into something repeatable? like are they thinking that way? So, but that's that's a few a few levels. So, I'll pause there.

24:50 >> Sounds good. Also, thanks for whoever drew on this. I just figured out I can disable annotation. but I don't know how to wipe it off the current screen. So, all right. So, it's just going to stay there permanently in the record as we keep going. I think you can clear white. Does anyone know how to potentially if you click on whiteboards, try to clear whiteboard potentially. >> Whiteboard. >> I I'm in favor of keeping it. It could have been so much worse.

25:17 >> Let's see. Annotate. this is I have an eraser. >> Oh, there you go. >> There it is. >> Yeah. >> All right. We're tinkering live. I am so happy that that wasn't like one of those erasers that you have to drag like in Paint, you know, like back in the day. I thought I was going to have to wipe it. all right. Let's >> Great job, Zoom Zoom PM for that feature. >> Yeah. All right. So, T, I think, I didn't. This is the next.

25:47 >> This is so Yeah, as we go into the next way of evaluating, which is the live case, right? a vivid image to keep in mind is, Daniel Conaman and Amsterki won the Nobel Prize in economics. And I think the one of the greatest lines here is like separately we're okay, together we're genius. And like that's that's kind of the relationship with AI. It's like this gets every gives everybody that opportunity, right? It's like it's not going to be replacing you. It's not going to be doing it for you. It's not like you're just delegating. It's like one of the highest levels you can aspire to is not a utility, not not a good Slack update writer, not a good Google.

26:32 but it's like something that's making you smarter. And yeah, and go to the next vivid photo. and the really again when you think of peak moments, this is not this is not for me what using AI looks like all the time, but sometimes when it all works out and I've set it up correctly and I've invested in the context and all that stuff, I I have moments where I feel like this where I'm not even looking. I feel like I can pass the ball. It's going to be fine. and hopefully we'll have more and more of these. So, yeah, I just want to make this really vivid like what success looks like, what you're trying to get examples of from candidates. go for it, Ben.

27:10 >> So, you're saying don't hire LeBron James. is is what what I'm hearing. >> I mean, if he fits the rubric. >> Okay. >> Right. If he if he does well in the interview and and and especially the take-home. >> Okay. I'll add one more just like I'm actually curious what you guys think about this, but like it it seems like a big part of this part of the interview is judgment on when to use the stuff and just as importantly when not to and that that's sort of what you're trying to sus out and feel out. Is that like a fair assessment?

27:37 Yep. >> Yeah. Yeah. I mean, it's really like just there's just table stakes about how people are using AI to get stuff done right now. And I feel like people that have been reluctant to like really embrace it to get the basic job done, I think, are going to be at a severe disadvantage in interviews moving forward is is my read. Okay. so as we talked about, there's kind of like a progression from behavioral interviews, which is like tell me about a time kind of stuff to now let me see it. Okay, so like we're in a live case. you're going to maybe be in a room for 45 minutes or like a Zoom room most likely with maybe a PM or an engine lead or a designer, maybe it's the hiring manager and they're going to want to see how you use AI to get stuff done live versus just wanting to hear you talk about it.

28:30 if you're like me, then when you when someone is screen sharing these days and you could watch how they get something done, you are learning so much about how they think and how they work in ways that are probably better than they could articulate it themselves because it's like the implied behaviors and the real behaviors versus kind of like their stated behaviors. and I think this is really where I think there's a lot of magic to be had in in these interviews. And I think that I'm I'm personally like excited about the kind of signal that can come out of these. so some examples are, you know, hey, like here's a product problem. You've got access to claude or whatever the LLM is.

29:19 Walk me through how you might approach this. Okay. So it's like when to reach for AI, when not to reach for AI, how do you prompt, right? like all that stuff. it could also just be like, hey, like how would you draft a PRD for this feature? Use whatever you want. I just want to hear see how you think. Like I want to follow your process. if someone can't draft a PRD for a new feature given obviously like sufficient context within a 45minute time box, that's definitely like a red flag these days just given the state of the tools.

29:44 and and doing so in a way that feels defensible or at least like they're it's not just taking what the AI gives them. I do believe that allowing people to use their own laptop I think is really important. because they're going to be in their own their own like home court, you know, they're going to have the tools that they're used to using. So you don't want them to trip up in an interview because you're forcing them to use a tool that they haven't used before. if the tools that they have they they are using are, you know, sufficient, you know, meet them where they are.

30:16 >> Yeah. >> okay, tell I I won't talk through these like last time. I think it ate up too much time, but yeah, I'd love to get your kind of input on these this this live case way of getting signal. >> Sure. I I'd also say for live case, you know, especially when for this topic more than like any other skill set you're evaluating for because there's so much hype online and there's so much theater when it comes to AI and there's so much everybody feels behind all the time, you know, like every all the Yeah.

30:50 No, no more needs to be said. Everybody always feels behind. Everybody's always feeling like somebody else is doing it better and they don't know about it. and it's probably going to continue for a while. There's a there's like a very big temptation or very big risk of people not doing what they would actually do, but doing what they think you want to see. and I think it's really important to like set the tone like that you don't care that you don't care about the hype. You don't there's no like there really is no right way.

31:18 there's no like magic spells that you're looking for to see somebody use or like start your prompt a certain way and structure it a certain way and you have to use markdown. Like all that stuff is is noise. I think it may be true in any given time, but long term it's noise. And what you're looking for, right, is is like reassure people like I want to know how you would honestly go about this. Like actually solve this if as if I wasn't here.

31:42 because I'm not looking for what you think I'm looking for. And and then you're you're you're looking for like mom was saying like do they know when not to use AI? Are they also investing in themselves and building their own context like not moving too fast that they're not they don't actually know what's going on in the thread and getting sloped right as a result? when when they're communicating with AI, are they like leaning more into conversation, right? than to some kind of template that they saw online 2023 and haven't let go of and and very very engaged like can, you know, are are treating this like they they got a junior intern, they're reading everything. and can speak to everything like they haven't lost the the plot of what's going on in the thread is like because yeah you can imagine there's there's a lot of different ways to use AI you're not evaluating for for speed you're you're evaluating for like just good judgment of a professional using a tool but it's still you're evaluating the professional >> yeah anything else I would add about this yeah man anything else you'd add >> I liked a a there's a good question actually like could you give an example of this like it might be helpful to do a quick turn of something, you know, if you have one off the top of your mind.

33:01 >> so like there's example questions, there's example behaviors. I'm curious like what kind of example we'd be going for. >> Maybe the one on the previous slide we can do. >> Mhm. >> Yeah. So like drafting the PRD for the >> drafting a PRD, right? That's a good example. you know candidate A might go in and literally write hi Claude draft a PRD for X feature right that's that's an extreme case another candidate might be a little bit better than that and say tell Claude I need to draft a PRD for this let's let's have a conversation about it ask me questions get context out of me right let's think through these things a third candidate might take that even further and and do that but before they do that they might say you know what I I'm going to do some research on my own first and maybe I'll structure a document of like what I'm going for and maybe I'll ask the interviewer if there's some like you know like stub fake documents or something I can use maybe online like to to to put into the chat as context along the way. Maybe I'd like spend some time being thoughtful of how I want to engage just like I prepare for a meeting with a person I'm going to be a little thoughtful of how I'm preparing with for AI. so there's different levels, right, of like and none of these are, obviously there's like a spectrum here and there's a direction, but like what I said is not literally the answer, right? You want to see thoughtfulness, >> right? Yeah. Like I I can think of like even two more signals here of like you might someone might think that they're being really clever by like firing off a bunch of clouds at the same time. Let me ask it like a bunch of questions at the same time. There's some cleverness to that because you're accounting for the latency of these systems. So you're demonstrating that these systems take a while to respond. The problem is that you're in an interview and you have to then parse all of the output from Claude at the same time. So like this is probably an example of like someone who might try to take it too far and say, "Oh, I'm really good with these systems.

34:50 I use them all the time. Let me do five different questions, come back with five answers." But how are you going to synthesize that to actually get through the prompt? I think is another example just just to give a couple of examples for us to riff on here. Also, I would personally be very curious in whether the candidate is choosing to type this all of this or they're using voice dictation because I feel like voice dictation is going to be way faster. So, you've got a time box, you got to get a lot cover a lot of ground. Are you going to go slow and type it or are you going to have like whisper flow or willow voice or one of like the those apps I think is going to be really critical.

35:21 You could also talk to the AI and tell it, hey, like I'm in an interview right now. I need to come up with a PRD. I want you to help me create a really good pro prompt that that so you could even like use AI to help you come up with the prompt that you use to to like work through the PRD and give it the the context of like how deep you're trying to go for this context which is what you do in a real interview as a human too.

35:41 You're there's a lot of assumptions that you're not going to treat it as like an actual thing given that you're in an interview >> and maybe just Oh, go ahead. >> I was going to say two things is like the the u as much as I want to say like who cares how they do it. Some people like to type, some people like to dictate. Let's be honest, dictation with whisper just looks a lot better. like it's going to be it's going to influence how people evaluate you or it's going to if you're interviewer, it's going to influence you. the second thing is the people putting in the chat like customer use cases. Like if you're talking about you want to evaluate when not to when not to use AI, one thing you can do as an interviewer or hiring manager, you can say draft a PR for this feature. by the way, here's like a CSV with MPS results categorized for this or here's a user research transcript to see are you it'd be really impressive to me somebody does not use AI for that at first if somebody first reads it and and reads a good part of the raw data. that would that would be really impressive of high level.

36:41 >> It's a good point. The hiring manager has a bit of a fiduciary responsibility from a process perspective to give what what they deem sufficient context to to the candidate for the exercise. is. And again, if they don't want to give context and part of what they're evaluating is navigating in complete ambiguity with no context, they can do that, too. But just be mindful of the trade-off of you might be less likely to get something interesting, >> right?

37:04 >> How are we on time? >> We we got we got to keep going. Yeah. >> Okay. >> Sorry, Mom. >> but yeah, that was awesome. I could talk about this for three hours, no problem. okay, so the last the last dimension here is going to be kind of what we've been talking about, but in more of a take-home. Okay. So, I'm writing 2 to four hours here because like honestly it bothers me when companies ask candidates to do like 20 hours of of homework for an assignment.

37:31 It's like really disrespectful in my opinion. So, I think if I was asking a hiring manager to like design a process that at least had acceptable candidate experience, I'd say two to four hours and time boxing it would be my my preference. note that some companies are actually asking c this I actually really like this. Some companies are asking candidates to tell them when they will have like a three-hour block to work through an assignment and then they will literally send them the assignment at the start of that 3-hour block and they will ask the candidate to submit it at the end of the three-hour block. So that way you kind of like have to spend 3 hours on it. You can't spend more.

38:11 you could spend less but you probably shouldn't. but that doesn't give people the open-endedness of some people spending 50 hours on the assignment and some people having like a bunch of busy life stuff get in the way. and and only being able to spend five hours. Okay. So, I think that that's one thing that people can think about there. >> and then examples here. just because I want to be mindful of time tall. Should we should we just like >> Yeah, let's run through the >> Keep going.

38:39 >> Yeah. so like you could do a friction log which is basically like what would you improve about this product flow? Stripe is a good example of a company that uses the friction log homework assignment. they have a lot of candidates go through like the the Stripe one of the Stripe flows like Stripe Atlas sign up or Stripe checkout or one of those we we cover that in a bit more detail in the the insider loops guide for Stripe but they are explicitly asking the candidates to talk about to to include a note about how AI used assist them in the process.

39:09 So they're certainly expecting people to use AI they just want to you know know how they did it. and then there's a couple other here. You could ask people to vibe code something. Vibe coding can certainly tell you something about how someone approaches solving a problem. It's not production grade AI, which is what we're about to talk to next. but and then you could ask people to to derisk maybe like a new feature launch by giving them some context and have them do an assignment.

39:36 But the idea is like each of these things would probably be like very very hard to go through without AI in in the previous world within call it like 2 to four hours. But the expectation is now that people should be able to complete these kind of assignments if they use AI within a small time box. >> One one I'll say for take home you want to be giving like assignments that bring out their PMing as well. Like you know you vibe coding is not PMing. being really really good just at the vibe coding part is like that's not that's not the win. You want to see somebody who still shows really good PMing, order of operations, good judgment, maybe vibe coded something that is like a very small angle of it because that was the biggest risk, right? And it so basically you guys get what I'm saying like it still has to fit into framework of of of functional excellence.

40:26 >> Totally. cool. Yeah. And here we're looking at the quality of the output. how they've speak about using AI and their judgment along the way like we just talked about. okay. So, just to close the loop on what we teased earlier, Meta's product sense with AI interview, if people want, they you could go watch the full session I did with Tal and Aman from a few weeks ago. that would be an example of a live case that is gauging how you use AI to build to build product and work through a prompt. although internally some people are being told it's meant to gauge your ability to ship product AI products but it's certainly not the case yet. So there's that that's what I mean by this stuff is messy at the moment.

41:06 okay cool Aman now we're going to hear from you. I'm so excited and sorry for taking so long to get to this. Okay. so now we're in the second row of our matrix. Okay. Okay, so here now we're talking about building AI products and we're going to go through those columns just like we did with Tuls. So like behavioral live cases and takehomes. I think in these cases we're trying to gauge have they shipped AI powered features to real people, real users on production. a lot of people have not, right? I'll be honest, I I I never got to that kind of experience when I was doing full-time PM work. So I don't have that. and a lot of people don't. And I think a lot of people are trying to figure out how to tell a story if they do have that experience.

41:51 so this is an example of storytelling to to get that through. examples here, you know, talking about an AI feature you shipped and what made it hard, how you evaluated it was working well enough, how you handled AI behaving unexpectedly and walking through trade-offs that you navigated along the way. And then Aman, I'm going to stop talking here. but the kinds of things you might evaluate are these types of dimensions. And I would just love to get get your take on kind of like how you think about behavioral interviews when it comes to assessing some of these things.

42:28 >> Yeah, this so I think this you know the rubric we think about here is basically it is very similar to the behavioral interview rubric you would expect from a you know product sense interview. In many many cases it's like except you know instead of it being a case it's someone actually walking through an example where they had to make a decision like you know when to ship or whether to ship or not ship for example is is a really classic one. you know the this metric moves up but the other metrics move down. How do you make that decision? And what you're really gauging for is their thought process and the context and how they how they PM this project. What who were the stakeholders involved? what was the goal of the product in the first place? One really powerful signal I'll I'll call out in the AI specific trade-offs, did the product need AI in the first place or not? was this just something that the CEO wanted like put AI in the product and that's the motivation or was there a real clear reason why they needed to add AI into the product in the first place?

43:29 That's usually like right off the bat like you're trying to understand the motivation for how they decided to get to that final state and then what were the challenges along the way. So that's a that's that's one where I would definitely be looking for you know can they understand when to use it when not to use it similar to how when we're gauging in in the case of like Tal's interview as well you know when to reach for it or not reach for it in the product. so yeah, that's that's like a bit of a high level. And then I I'll add one more just on the eval side. Like this is, you know, as Ben kind of pointed out right in the beginning, this is definitely an emerging field. I don't think hiring managers you know, even if you're at the cutting edge, they're not looking for eval experts to to join the team. What they are looking for though is someone who can connect the dots between what eval are and how they relate and ladder up to product metrics.

44:17 and thinking about eval metrics and AI eval metrics in relation to product metrics. Remember, they're hiring you as a product manager or if you're a hiring manager, you're looking for a PM that understands this stuff and can communicate around it and can get into the weeds, but you're not looking for someone necessarily to go and like fully implement an endto-end eval system in most cases. That's a great point. Okay. So, continuing to think through building AI products, but considering the way that you might be able to use a live case to to get signal in this dimension, you could have a PM or an engine lead or some ML or AI specialist in house evaluate the candidates's reasoning on this front.

44:58 so here you know some of the examples that we came up with are maybe asking someone to actually design one of these features AI powered features and walk through their approach or talking about how they might solve a problem with a given AI feature and diagnose and address it and how how they might evaluate that trade-off of whether adding AI to a given product is is the right call. yeah man, anything you want to add here?

45:26 Yeah, I think on this one it's again you're gauging pretty heavily the communication skills around the technology. So it there is a little bit of like crossover with having that initial sort of product sense of what this tech can do and what it can't do. So you're gauging that. That's definitely a really important signal here. I think on top of that, you know, the other things you want to weigh are what should the product experience look like? so how should this how should the technology fit into the product in the first place? Is it a chatbot? Is it a background agent? Is it something else? when should like basically think about how does this PM map through the user's journey and where can AI layer in on top of that? And you get to see this like really beautiful sort of interaction where you get someone thinking through a problem journey just as they would in a standard sort of product sense interview, but now you're seeing how AI can change that user journey. And that's really the signal you're looking for.

46:26 >> Totally. Yeah. And then I think this is probably just a summary of some of what we've what we've talked about. Okay. So the take-home I mean, Aman, would you would you agree like at a high level, the take-home is almost just like larger kind of like investment of time from the candidate and an asynchronous investment of time to kind of try to get the similar signals to what we just talked about with with the live case or is there anything that you would look to kind of evaluate or you you think can be evaluated with a take-home in this dimension that is hard to kind of get a signal on in the in the live case? The ground truth I can share here is that for the most part the take-homes that I see in this space tend to be driven around feedback and what changes you would implement. So that's like the critique the AI product or product spec or here's a broken prototype. Like that's a great question.

47:21 We literally will throw candidates at our product which we sometimes know has deadends or is broken because we haven't gotten there yet. We want to see if they can probe and find that themselves and then try to figure out what to do next. So it is a it is a lot of like gauging again can this can this PM think about product experiences think them think about them all the way through with real products take you know time box this spend two hours on in max or maybe a couple hours max and then think about what they would go do next if they were the PM here that that's been pretty consistently what I've seen across the board so far.

47:55 >> Yep. Okay. >> Oh let me call out one more really quick thing here actually which is the use any tools part. That's really important. This also does have that bleed over with what you were describing earlier which is like when do they reach for the tools and how do they use them? Are they vibe coding everything but do they understand at least how they got to the end solution in the first place totally. Okay. I think we covered and people should probably are seeing some patterns and kind of like the rubrics and and I think the idea is that if there's a set of signals you want to evaluate like building AI products, you know, the dimensions that you're evaluating should probably not be like super different across the different interviews that you have for that competency. You're just trying to get a signal on that competency in different ways. Okay? Some of these are going to be better than others at getting the signal that you need. so this is kind of like more of a superset of what you might end up wanting to do.

48:50 >> You definitely want to don't want to throw your candidate everything here because you still want to gauge their like core product skills, you know, like collaboration, all the things you would care about from a PM, but these are the ways you can layer in these questions, I think. >> Yeah. Okay. Thanks. And sorry we had to to go a little faster through your your section of this, Aman, but hopefully we can hear from you a bit more in the Q&A. yeah. Yeah. So, I'd love to give you guys the mic and just tell people a bit more about your your course in case they want to kind of go deeper with you.

49:21 >> I'll take a first first pitch. first off, the the I I just noticed the dates are actually the dates are next week. So, it'll be on Thursday and Friday next week, two-day sprint. so I think we we reused the slide from last time, but this is a lot of fun. This is actually like a passion project from Tal and I. basically what we were doing is we were sending like WhatsApp voice notes back and forth of like as we were both using coding agent tools and just seeing how amazing they've become and learning about them at the same time. We thought that it would be a lot of fun to just make that a collaborative environment for us to spend basically two whole days going from start to finish learning how coding agents work and how you can implement some of these learnings in your own company. So, it's meant to be really focused on people that want to compress this knowledge in two days. And it it it's very I think I think like the main things we hear back and the feedback so far has just been it's rewarding to like get hands-on in the first place. So, you can come with you don't have to be technical, you can come with nothing.

50:20 We'll get you set up from start to finish. that's that's our goal. yeah, go ahead if you wanted to add anything else on top of that. No, I think I think the the yeah the insight that we had was was like I'm learning so much about how a full-fledged AI agent works when I use cursor cloud code a little bit but actually cursor is a little bit little bit better for this and its transparency and how much it shows its work and I'm suddenly all the concepts that you're hearing about all the time being used in production. Yes, you know it's not the same, but it's actually like very similar. U cursor is an agent harness. It runs on your computer and watching it work, messing with it, using it for things you actually care about as a tool, like things you need to really need to get done. Suddenly, like without a flowchart, without infographics, without slides, without anything, like you turn around, you're suddenly like you you like I understand a lot more than I realized. and things start to make sense. And then, you know, Aman's way more experienced than I am. You know, I' I've stumbled into AI world with all of you. you know, most of you at the same time, right? Like, and and yet I I really felt like I was able to anticipate or by solving problems for myself, I realized that's the same problems that people, you know, building product AI products at scale are solving. And then suddenly I'm like, "Oh, I I'm feeling really in in sync with the industry and announcements before they happen." I don't know how else to explain that. So what we do in this workshop is like we spend a lot of time using cursor for like the two categories that we saw like to build products and use it as product managers, but we also connect everything we do to how an agent works in production.

52:07 >> Yeah, you guys have taught me so much. So I'm I'm pumped that you're bringing this to more people. And that that Lenny post today is just a banger. Okay. so to summarize, you know, I'm not I'll include this in this in the slides, but I don't want to spend too much time talking through this, but you know, there's basically some mistakes you want to avoid. there's reasons why those mistakes, you know, kind of can kind of like hurt you in an interview process, and there's ways to mitigate those. So we we'll include the slides after this.

52:33 and again, just like wrapping up with takeaways, and we'll do the Q&A. we we we marked some of these questions. tall, thanks for dropping those in our in our WhatsApp chat. So, we'll just kind of go through those. the So, the takeaways, the candidate experience is a product decision. So, just how you design the loop is going to, you know, sometimes you might get more signal, but you lose good candidates because they just don't have the time to deal with you. they have a lot of options. So, just be mindful of how much time you ask people to spend and which trade-offs you want to make. Using AI is not the same as building AI. as Tal and Aman just mentioned about their course, using AI a lot will give you a lot of intuition about how to build better AI products, but is it is not the same as having that experience of shipping AI products to production. and you have to decide what's more important to you.

53:20 probably can't be two most important things. Everyone here's PMs, you know, you got to make those trade-offs. you can think of the matrix as a menu, not a checklist. So, you know, you want to have a focus loop that maybe covers three to four cells. You don't have to cover all six cells. I think that would be potentially overkill for an interview process. And maybe some of the cells you want to have two two different interviews to cover. So you just got to figure out where you want to put more wood behind the arrow. And then I would strongly recommend optimizing for your must haves. So where where does someone really need to spike to be successful in your team with that given role at this point in time? And just test those things that actually matter.

53:59 yeah, and every interview does have a job to do. So, basically, if there's two interviews doing the same job, I'd be like, "Do you really need both?" Like, you should probably cut one. and if you can't tell me what job an interview is doing, I'd also say cut it. you're just wasting your time in the candidates's time. So, just be very mindful there. And Q&A. All right. so I'm going to stop sharing my screen and let's see. We've got the first question.

54:28 Yes, there there's a I saved some questions from from we're asking the chat at like different points so we don't lose them. and >> okay, I have to pick one because we >> I see people raising your hand. We're we're going to we're going to prefer the the written ones. So, I just prefer not just given time. I don't want to let people kind of like ask their questions live if that's okay. Feel free to drop them in the chat, though.

54:53 >> yeah. Does either of you have one that you see in our WhatsApp that you guys want to pick up? there's a question I would love to hear Aman's thoughts is when you talk about like why a feature AI feature got built what if different versions of the question came up which is like what if it was the CEO told us to experience that we can all relate to that right like what if you had to push back on an AI feature from an executive some versions of like the hippo highest paid person is the reason for the feature >> I've never heard that term before but that's that's a that I think that's a great follow-up question I saw that in the Zoom and I was hoping We could cover it. Look, I mean, we have to reflect on the reality of today, right? Like CEOs are getting pushed by their board to ship AI features whether or not they want to in themselves. The point that we're trying to get to though is can this PM understand and articulate the why. And that's what any PM should be able to do. Replace AI feature with any feature, right? The CEO wanted this feature. If that's the answer, that's not a great signal versus why did the CEO or why did the board or why is this right for the customer to do? And if not, how did you push back or how did you handle that objection? So, I think that's really what you're looking for in the signal. It's not really the like, oh yeah, we have to ship this AI feature, you know, because, you know, we just had to and the VP wanted it. Let's talk about it. Like, I want to I want to agonize over the decision. in this case that if that's the case I want to get to the why how did you know give me the five wise exercise on this so anything else you would add there Ben or or another question no another question that came in is is there usually a live case or and a take-home or is it either or or look there's companies like like Door Dash that just don't do a take-home assignment and then there's companies like Uber that are known for for their jam, which is like a very time- inensive one. They say spend a few hours on it, but a lot of people spend like 20 hours on their jam prep, which is nuts to me. but that's kind of what it takes to really stand out right now. So, I think you you can and then Meta, you know, which is what I coach a lot, does not do any takehomes and is arguably one of the most efficient interview processes I know of. It's like five case style, you know, four case style interviews and one behavioral.

57:10 all live, no no take-home, no presentations for for most of the PMs with some additional behavioral interviews if you're going into like L L7 plus. So there's certainly not a one-sizefits-all way to design these these interviews. >> >> yeah, I see some good a few questions around the same topic of like if how do you interview for the building AI products? part as a candidate if you haven't yet had the real like company experience doing that right like if you haven't done that but maybe if you've done that as a side project somebody asked right like like if I have AI experience in my own projects but lack it at work like varants of that question are coming up and I think I'm sure a lot of people thinking about that yeah how do you >> you can't lie right like so you're not going to make up experience you don't have and like I think the hard pill to swallow here is like if someone's trying to hire someone that has that experience and you don't have it, you're probably not the right fit for the job. Like there's a reason that's a requirement.

58:16 But Aman, you tell me like if >> if that's a if that's a non-negotiable and someone doesn't have it, there's really not a lot of ways for them to kind of like make it up. >> Totally. If this is a company on the cutting edge of shipping AI products, they're going to want PMs that have in some way, shape, or form shipped a version of an AI product that they can speak to. If this is a company not at that edge, but you want to get a job there eventually, there are companies hiring that don't require that experience and where you can build and develop the chops. They're looking for someone who's passionate about this area and wants to try to build something in the space. So, you can reflect and and I it's actually a question I ask even now.

58:51 you know, what are you working on on nights and weekends? I'm not looking for like someone who is actually working on nights and weekends. What I'm looking for is what this person is building outside of work that is interesting that they can speak to. In fact, that's usually more interesting than what they're being paid to work on many cases. And I'd like hear the craziest, most interesting examples that I still remember. I met a candidate about a month ago that had interviewed for us like 3 years before and he told me about this whole project that he worked on for, you know, predicting certain players in the NBA for his like draft his like fantasy team and like how he built it. And I'm like, and I told him back I told it back to him and he was like, "Wow, I can't remember. Can't believe you remember that." But the thing is, those are the things that are memorable and are going to make for a good interview for both yourself and the interview interviewer. And so I I just highly recommend like thinking about how to use this stuff outside of work. It's going to help you stand out as a candidate.

59:44 >> Yeah. And just one thing I'll add is like talking to hiring managers like hiring managers are acutely aware there are just not that many people who have like strong production grade like large scale AI deployment experience as a PM. So those who are choosing to make that a must-have in their interview process just need to be mindful. You're really narrowing down your pool of who you could hire from and you're competing with basically all of the companies that have infinite amounts of money to get to get that very limited talent. So, I would encourage some hiring managers that think of that experience as a mustave to really ask whether it's a must have or a nice to have as you design your interviews.

60:20 >> And one thing I I actually found out about this week is I have a friend who was going through a process that the company was it was for a role that would be building an AI product. but they were they were not limiting themselves people who had previous experience. They were looking for people who were using AI at work and had really good behavioral cases around that. And so they were it's one of the companies that Ben was like kind of that that case where like they they're aware of that that not everybody's going to have that and maybe the best candidate is not going to have that but but there are signals that somebody would be building their own good AI product sense that way.

60:57 >> Yep. Can you guys hang for like five more minutes or you got you got to run? >> I can I'm good. >> Cool. If people need to drop, thanks for joining. But I I'd love to add as much value to people as we we can in the session. okay. >> There's a question I think you that's I think you might be the best person to answer. which is like how do you suggest we answer questions about reimagining a product given AI? so like how to reimagine an experience? The example they gave is how would you redesign Corsera in the age of AI?

61:28 >> Yeah. So, you know, this would typically fall into either a take-home or a live case. and I would consider that to be I mean it's interesting. Is that using AI to do product work or is it using is it or is it gauging your ability to actually deploy AI grade like your experience deploying AI production scale AI products? I would say it's probably a little bit more in the first bucket because my guess is as an idea the best thing to evaluate is how you might approach that. so I'd say maybe like a live case here, could be something like, you know, your typical product sense flow of setting the context for why you might want to think about doing something like that.

62:09 What's your role? Are you a PM at Corsera tasked with cannibalizing your existing business? Are you an upstart trying to kind of, you know, eat Corsera's lunch with an AI native lens and why that might be interesting and some commentary on the education landscape and why now is the right time, etc. and then you want to get into like thinking about segmentation. And so like who's the different audiences that might benefit from something like this? And how how might you pick a segment to focus on because you can't serve everyone to get started at once. and then how might you pick a problem or maybe like a a set of pain points for an underserved segment that's worth focusing on within, you know, the audience for something like this and kind of design an AI first or an AI native solution to that that problem for the underserved audience that is strategically important to the company at this point in time and just kind of create that nice arc at the end. And you know that that's like the way I would generally approach at a super high level a question like that. The specifics are things we're not going to have time to get into right now, but things that I like my my co-pilot would have a really good fun time, I think, like chewing on that if people want to check out at benerys.com/copilot.

63:16 I think that is a great use case for seeing what it comes up with and I suspect it would come up with some interesting stuff. >> And by the way, >> I'll just plus one Ben is is has been so helpful from helping to think about interviews. so I just wanted to like endorse and plus one that how to add structure to you know a company that's growing and trying to add more interviews to it and adding adding interview loops that you know kind of measure the product this core product skill set. I think Ben has just been immensely valuable there. So I just want to plus one. Ben is super helpful here.

63:47 >> Cool. Yeah. Just because someone's asking for it I dropped it but not trying to overly promote that. although I'm very proud of the co-pilot to be clear. >> okay what other question? really well. What worked really well about that, by the way, is the question was actually about like as a candidate, but I think you you nailed it because it actually ties back to the Facebook product sense with AI, which is what they're evaluating there is not about AI products. They're evaluating product sense.

64:13 >> Product sense. >> That's that's the idea. And and the answer there, even though it has the word AI in it, it's a PM interview, >> right? So there don't Yeah. like just you can you can take the the suggestion from the interviewer to use AI as a constraint for what you come up with in the solution space. but I think you really do need to ground to to Aman's earlier point like you know if the CEO says put AI in it, you still have to be able to come up with a really compelling narrative on why AI is well positioned to be the form factor for the solution. And those kind of live cases I think give you the chance to demonstrate that you could do that.

64:53 >> because you don't choose what to work on in the case, right? Like it's like the interviewer is like the CEO in the in the simulation. So the there's like build it and you're like, "Okay, I guess I got to like think from first principles like why would I build this and how would I approach it and who is it for and what does it do and why is this the right form factor?" >> So it's kind of like a condensed way to get all that signal.

65:11 >> Yeah. If you're going to say build an agent or you're going to build an AI product, justify the why and be able to stand up to the like how it's going to work. >> Yep. any other questions here that we wanted to tell from our what's up? I >> I think there's a few more questions around like how do you build things in your spare time? Do you put that on your CV? Do you start your own thing? I I think I think what's I I'll say what I'm seeing from the side and I'm on Ben tell me what you think is like it's kind of like AI kind of reset a lot of things and it's you know reminds me back when you know before I had anything that like any kind of like formal experience and it was like okay how do I get that and we're all kind of experiencing that too right like no matter how much experience you have when it comes to AI we're all being reset so what would you do or what advice would you give somebody coming out of university right or switching careers side projects, volunteer, help out, startups, early stage employee, start your own thing, right? All these things where you you you can a little bit more strategic but quickly build up that leverage that you have for other areas.

66:16 >> Can I add one more note there, which is that applies to experienced product leaders too by the way. I follow a ton of experienced PMs. I know a ton from the fang companies of the world. They are all doing the same thing. they're all posting their weekend projects, like at least the ones that are adjacent to AI or want to be kind of known to be adjacent to AI. And so whether or not you're at the cutting edge, I just want I just want everyone to know like it is it is a big reset for everyone in the space even if you're in AI and sharing what you're working on actually shows that you're thinking about it versus you know and it doesn't have to be like fully big thoughts. It could be like hey I tried to set up CloudBot this weekend and it didn't work. Does anyone have tips on how to make it work? Right? That's that's the type of stuff that is at least showing that you're like in the dialogue thinking about this stuff. So whether you're experienced or inexperienced or just trying to get the job in the first place, I think that advice applies to everybody 100%. And if people want more tips on kind of like how to get noticed by hiring managers, I hosted a really good session last week. I did not do the majority of the talking like in this session with Nikki Scarstad from Dualingo and Hillary Gridley who recently left Whoop as the head of core product but they shared like tons of nuggets on like how to get noticed and if if you're interested more of that that's also on Maven. You can just look for that session. I will link that too in the follow-up email.

67:35 wow nice tall you are two steps ahead of me maybe three. cool. I know we are already 10 minutes over. So any closing thoughts Talin from you two as we kind of wrap up this session and anything you want to leave people with we didn't maybe talk about as much. >> I can I can share like a quick note that it definitely feels like at times that there's so much happening and it's happening so fast. hiring managers on the other end feel it too. What's I think important is just how you frame and show up to the interview as well.

68:13 Like being excited and ready to think about this from like a first principles perspective and viewing AI as just as any other technology that is really transformative and going through a moment doesn't mean that you throw everything at the at the wall and say it's AI, right? So I just wanted to kind of like underscore that note. it it's you know we're still I think people are still looking for those core skill sets that make us human and we're still building products for humans and agents but we need to think about this from like a product first principles perspective. AI is only going to be as good as the data in it and we're curious what you bring to the table. What's your unique perspective? So that's all I'll say about that.

68:52 >> I'll say everybody feels like they're behind. Everybody feels like somebody else is doing it better. everybody feels FOMO. So on one hand like you're not alone. On the other hand I would say the just focus on like diving in get hands-on use it for real things in real work in real life as much as you can. try to build things as much as you can. and even if it's not like the latest fanciest thing, that'll be so much more meaningful and interesting and valuable in life and definitely in an interview than than any anything in your Twitter feed or or LinkedIn feed or anything like that. Just there's no there's yeah, everybody's making it up and the more hands-on you are and the more real stuff you're doing, the closer you are to the asphalt. Like just be confident in that.

69:42 >> Cool. Well, thank you both for taking the time and I know it's been a really big day for you with the Lenny Post going live. So, hopefully you really, a lot of people get the value I think they're going to get out of that. And this, is related but completely different. And, yeah, we will share, or I will share the, recording for those who couldn't stick for stick around for the whole thing. We'll share the slides. I saw a request for the transcript. Now you're thinking about how to use AI. I love it. I think the transcript should be fine to share. I don't think we said there's anything in the transcript that was not already said in the recording. So, I'll share that as well. And and yeah, we'll link to Tal and Aman's course, my course, Copilot, and any of the other resources like the getting noticed lightning lesson last week. So, with that said, I hope everyone has an awesome week and yeah, we'll talk soon. Thanks for joining.

70:37 >> Thanks, Ben. Thanks, Ron. Thanks. >> Thanks, Ben. Yeah. Thanks guys.