Masterclass: Interview Homework for PMs

0:00 This deck, back in '23, it actually got through. It was a top company. But today, this would probably not even survive the screening round. Their push is towards making the candidate think rather than just providing AI slop. >> You need to demonstrate your unique experience and expertise, not how well you can prompt AI. >> For builder PM roles, this is the most commonly sought-after skill. This is what we will see the most even in the coming 2 years. It worked really well because of the systems thinking, audit trails, so on and so forth, phase roll out. Especially enterprise PMs, we live within these constraints almost all the time.

0:35 >> It sounds like overriding lesson from both of these decks is quality of insight matters more than formatting. >> Most candidates land just 80% of the way. That 20% is the whys and the economics behind it. And and that's not something any LLM can help you do. This is something that either you have or you don't. >> Really quickly, if you want help with your PM job search, including one-on-ones with me, my job search OS completely powered by AI to apply to jobs for you and find connection requests on LinkedIn for you, be sure to check out my cohort program at landpmjob.com.

1:15 I run these every 3 months. The next one starts November 23rd, and you will get everything you need to land your next dream job, earn more, and have a more successful career. Now into today's episode. >> Samanth, why should PMs even care about interview homework? >> first of all, thanks for inviting me on this podcast, Akash. Really looking forward to our conversation this one. Let me answer your question. In the past and even in the current times, companies they want to assess candidates, but because there's an influx of way too many, there is of course a shortfall of hands on deck to assess all of them.

1:56 pre-interview assignments and homework assignments or take home assignments. They kind of give companies an opportunity to put the candidate to work before they actually assess their qualities and skills in an interview, right? These assignments, they allow a company to see whether the candidate possesses the right skill set in terms of technical capability, problem solving, even presentation and keeping an eye out for typical pitfalls, right?

2:30 Stuff that most companies would have to train people upon, they can test all of that right here within these take home assignments. Now, the new thing that has come up, especially in today's scenario, is because AI has made it just way too easy for people to go ahead and conjure stuff up, companies are kind of tightening the noose around what kind of submissions are asked for, and their push is towards making the candidate think rather than just providing any sort of AI slop. So, for this new age, the candidates have to become a lot, lot, lot more smarter. And personally, if you ask me, the ones that I have coached, I've always asked them this question that if I give the same question to your younger cousin, will they be able to prepare the same kind of slides that you have prepared, or would you have done a better job? If the answer is even 50% yes, then I tell them to go back and prepare again.

3:28 >> I love that. You need to demonstrate your unique experience and expertise that you've developed in your years as a PM or whatever background you have that qualifies you to be a PM, not how well you can prompt AI. So, who actually asks interview homework, and who doesn't? >> That's actually a great question. Many companies like OpenAI, Anthropic, even Razorpay, CrowdStrike, Retool, etc. These companies which have, you know, maybe a smaller pool of people, less of hands on deck to be able to take out hundreds of interviews.

4:04 Also, companies which have specific processes and use cases for which they want AIPMs to come in and demonstrate their abilities on using AI products. They are usually the first ones to push out take-home assignments. Larger companies, established companies like Google, Meta, and others are, you know, FAANG companies more often than not who anyway want to establish themselves as at the frontier of best hiring practices, they usually don't do this. At least for the, you know, higher-ups, they never really give take-home assignments. There is an exception, I have mentioned that as well, that their early career programs, they might have this round for some cases, right? But usually those companies you don't really expect take-home assignments.

4:53 >> So, is it fair to say, just based on the companies you have listed here, that sort of AI companies or more tech-native companies are more likely to give out homework? >> You can say that. A lot of AI-native companies, they do want to assess the real AI skills of candidates because it has happened. A lot of people that were product managers earlier, they have become AI-enabled product managers, but they are still writing up AIPMs. So, AI-native companies, they do want to kind of differentiate between that, and that's why a lot of them they do like to give out case studies, specifically to discern whether the person has real functional knowledge about building AI products or not.

5:37 >> Okay. So, where does this show up? Like, is this typically in the middle of the process, the final round? Is this before you even interview? What is the most common setup? >> Sure. It actually goes throughout the process. So, there's of course the pre-interview screening. This is one of the most common steps. You can assume that a lot of companies they push for this because it allows them to filter anyone, you know, before they actually spend time iterating through that candidate, having multiple discussions internally, and then bring that candidate itself. But, they they just don't prefer that. They'd rather give them a pre-interview screening. If they qualify for that, only then they are brought in for an interview. It can, however, also be taken forward. For example, there can be a mid-loop deep dive as well, wherein there would be some maybe third or fourth round of interview, wherein technical or perhaps behavioral round is converted into a deep dive round, wherein they will ask questions about why did you choose this particular product only, what made you prioritize this feature, which metrics are you suggesting and why, etc. So, that can happen. It can also be, and this is not unheard of, the final round presentation, where the depth of the candidate and their problem-solving skills are tested live in front of a panel based on the case study that they have submitted.

7:05 >> This one in particular feels like very common, even for senior roles, right? So, I guess what it be at least what I've experienced is that juniors, they tend to get it sometimes even early, like first step. Like there's some associate product builder programs, associate product manager programs, where it'll even be pre-interview screen. And then I find sometimes for executives, like when I was a VP of product, and I interviewed for a couple of VP of product roles, my final round would be like a presentation to the CEO and CPO. Is that what you've seen? Is that fair to characterize broadly, or is that just my experience?

7:42 >> No, that's actually absolutely fair to categorize that way. It is, like I said, not unheard of that even senior folks are brought in front of a panel, and they are asked about the most strategic aspects of their decision-making. Why exactly are you choosing to go with this particular setup? What is the AI economics behind this decision, etc. Those are the questions you do expect from a VP, right? >> Love it. So, I think we both have experienced a similar thing in coaching people. So, I think it's fair to say that's the universe out there. So, what types of homework do people get if we have to put it into broad buckets?

8:17 >> Sure thing. So, I've categorized the five types and these are more prominent ones. I'm sure there would be some specific to analytics, something specific to just building prototypes and all, but the ones that I'm broadly categorizing, the first one has got to be PRD. I mean, product roles in AI. You've got to ask them to generate a PRD for this vague and ambiguous problem statement. Yes, typically recruiters are looking for the candidate's composure. They're looking for whether the candidate asks questions back, states their assumptions properly.

8:51 The same old standard practices that regular PMs also have followed. Just that this time the assumptions will be a lot more around AI. Same goes for strategy and compete related questions. It can also be competitive benchmarking and based on that suggestions of what to build, what to improve, what to integrate, or maybe what to buy. There can be roadmap and prioritization related cases. There can be regular case studies and problems wherein the focus is going to be more around the user empathy, the problem solving, the judgment and taste abilities of a candidate. And the thing that we are seeing new these days, that's the AI prototyping. Yes, this is the latest one and for builder PM roles, this is the most commonly sought-after skill. This is what we will see the most I think even in the coming two years for builder PM roles.

9:46 >> Okay. And what might prompts look like? So, like I just got a prompt. I was helping somebody with Netflix homework and it was like submit no more than two pages >> >> for their strategy memo. Is that common? What are the like most common prompts within these buckets? >> so, there can be a number of ways in which people attempt these particular case studies, but more often than not prompts can include a structured markdown wherein they divide a long process into context-aware decomposed prompt. I'll take an example of a PRD.

10:28 Typically, when people are building out product requirements document, there are multiple phases, right? It's not like we can write a PRD in one night. A functional PRD, there are various phases through which it evolves through. So, if we go to any of these LLMs and directly ask them to build a PRD, sure, they will generate something, but will that be useful? No. So, what we do typically is, you know, we suggest candidates to go through the process, write down step by step, and showcase that process to AI so that LLMs understand what is the context and how to break it down in the same way. And then they go through it in a sequence.

11:11 That's one big god prompt that probably the candidate will write, and it will not just stop at the generation. It will go through multiple recursive iterations wherein AI probably have a red team evaluating, probably have sub-agents checking out every step, and marking it up or maybe providing feedback, and then based on that that evaluation, a new generation is being created and so on. The final result, it actually is pretty good because it has gone through so many filters, and we also suggest our candidates to do a quick human check. You know, basically basically a human in the loop. We tell our candidates to constantly keep human in the loop so that AI doesn't, you know, get astray too many times or hallucinates. And if it does, the human is there to guide it.

12:01 >> Amazing. And I feel like with the rise of like PanGram and AI writing detection and Anthropic even watermarking their text, Google Gemini even watermarking their text with SynthID. You can expect probably OpenAI will follow. So, basically, AI is just making itself known. And so, it seems like the better model for people doing homework is AI as a thought partner, not as a writer. >> Absolutely. >> Okay, what does good look like? Can you walk through some real examples for us today just so we're not just talking, but we can really see it?

12:35 >> Absolutely. Let me take you through some examples. I have taken real snapshots of presentations and slides and documents that were submitted. The good and the bad examples both come from real slides after due consent, of course. Every data here is what was submitted. There is no mention of the person or the company here anywhere. If there was, it has been redacted. So, let's just go through these examples, okay? The focus will be towards understanding what are the good examples, and then I'll show you some bad ones as well. This first one, this essentially was about an LLM incident triage proof of concept. And the candidate had to make a case for production architecture compliance, etc.

13:21 for the same, right? So, what this guy did was he essentially showcased a POC versus a production scope himself, right? So, >> Can I interrupt you here because I feel like the question itself is kind of tricky. Can you explain like if somebody is seeing this, they don't know what this is. What What exactly were they asking for here in this question because I think even there you can learn a lot. >> Absolutely, right? So, think of it this way. This is a incident triage tool, yeah? And the proof of concept was already there. The candidate was expected to go through the proof of concept and then make a case for the production architecture compliance, rollout, success metrics for this particular POC. So, to take a POC from there to production, what are the various things that they were supposed to do? That is what the candidate was supposed to suggest.

14:14 >> So, this is one of the most important slides then that they would have is like, what does the POC currently say versus what's the production scope? >> Absolutely. Yep. >> Got it. >> So, I'll just take you quickly through the various things that the candidate wrote. The actual details are not as important as the logic behind why this works, okay? So, I'll talk about that in a bit. Like for example, the candidate mentioned that currently in POC, we have the incident JSON structure coming in from static files, right? But in production state, we will not really have static files because it just doesn't make sense. It's production. It has got to be real time, right? An incident cannot just keep waiting for it to be, you know, kind of read through a static file. Hence, in the production state, it has to be a real time event injection, maybe via APIs. Yep.

15:09 Other lines also mention the same kind of a logic, right? Now, why this works is because even though the candidate has mentioned places where there would be a one LLM call per incident to categorize it, or there would be suggestions taken from an AI, the candidate has not taken the easy path. I mean, the easiest path here which a lot of people arguably will probably suggest is let AI handle it end-to-end, right? Just let the AI handle the incident, let it write the summary of it, let it even suggest what it should be the priority with, sorry, what its priority should be, and what should be the next steps, etc. That's basically throw it in ChatGPT, get a response kind of a mindset.

15:54 >> >> But the candidate actually went one step ahead, and he had this throughout his his submission, he had this hard rules that was set above the model. The AI's role, like you only said Akash, it was supposed to be only there for suggestions. Yes, it was only supposed to be the thought partner. There was always rules that dictated, you know, whatever AI is suggesting, does it cross this threshold? Only then the decision can be taken forward and even then they had a human in the loop. The final decision was taken by a human only. That showcases maturity. That showcases real-time AI build experience. And that was why this particular presentation was actually shortlisted, right? Similarly, this was a very honest table wherein he showcased component by component including what is still only in mocks. All of those things he included here. Despite that being a risk because it was untested, it was after all just a mock. Despite that, he showcased his vision about how it is going to look like in future. I'll show you one other slide. He showcased the value propositions in a very simple manner along with the impact clearly mentioned, right? For example, the actionable recommendation I was talking about, he mentioned that it will provide operators with a verified starting point for remediation. Instead of just letting AI handle it end-to-end, which potentially can have a lot of dollar value loss in case something gets hallucinated, here the impact would be that operators' life will become a lot easier, yes? And then he mentioned those impact points in real numbers, yes? He even had a slide specifically around how his proposal was following all the comp compliances. And he quoted the exact lines from the compliance rules that were being followed based on his proposal. This level of depth, this level of clarity, and honestly this level of transparency in presenting what you have in mind, that's quite rare.

18:01 That showcases maturity and that's one of the biggest reason why this AI product and this presentation actually cut through. Make sense? >> Yeah, so it's almost like you're thinking about you versus the other three to four people doing the homework. And you're trying to come up with what am I demonstrating that potentially the other people may not demonstrate and maturity in terms of what I can and can't do becomes a very powerful vector and it feels like this person really triple clicked on that and another thing it seems like this person really focused on is they haven't built like super fancy slides or anything.

18:39 They focused more on the quality of the content. >> Absolutely. >> Got it. Okay. >> Let me demonstrate one more. Yeah. So, I'll just give a summary of the previous slides. The previous slides and in general the whole presentation, it worked really well because of the systems thinking wherein the candidate had showcased audit trails, governance, phase rollout, etc. Right? So, they showcased their experience in a regulated enterprise. But the thing that actually separates you know, the regular good from the elite is most of the candidates when they showcase their AI builds or their ideas, they you know, simply push for some numbers, some big top accuracy numbers or precision numbers. But there was no real mention of eval methodology or any cost model per se, right? And because of those reasons most of the good submissions, good ideas, good content sometimes also doesn't really make the cut because well, they missed out one of the most important things. Yes? So, this is where most candidates land. Like just 80% of the way. Yeah.

19:52 That 20% is actually evals and the economics behind it and that's what many candidates just either forget or miss out on completely. This deck actually showed that. That's one big reason why this guy got shortlisted as well. Right? yeah, go ahead. >> Yeah, so what I'm taking away is if we're building an AI feature, let's talk about evals and pretty much any feature, let's connect it back to the business impact. Okay. >> Yeah, sounds good. >> All right. Let me show you one more example.

20:24 The ask here for the candidate was he applied for a AI data company and they wanted him to unify sprawling portfolio of five separate data products. These were products that they already had and they had a bunch of other smaller products in their kitty as well. But they gave him these five and they asked him to propose new data insights. Yeah. And they gave him some sample tables, some structures, etc. The expectation was that the candidate himself will kind of figure out what to do with this. Yeah. So, he essentially went through the huge amount of data that they had already shared. Went and did some research around the competitors, figured out from his contacts. He actually went and talked to some of his contacts and understood you know why the users specifically use you know some types of products like I'm talking data products here. So, what do most companies, especially in the insurance sector, they use that data for? Mostly reporting templates insights. They have separate teams that are doing just this. Then there is a larger team that is doing analytics.

21:41 They require the data exploration tools. And then there is the tech team which simply wants the dump of data. So, based on the users, this person segregated this list of products. There were other other ways as well. I remember because I was the one who kind of helped him through this. I remember there were at least two more ways in which we could have kind of sliced these products or bucketed them into different ways. We could have simply gone with the let say a free tier product and paid tier products and all, right? Instead, we chose this because he found the way users use a product to be a better way of classifying, not just in the current times, but even looking at the future because and it was his vision that later on when we enable many of these products to be AI native, let's say AI native exploration, AI native inside building, etc. And as these products scale up and let's say there are even newer products that are built internally like audit, even they will fit within one of these buckets. That was the level of envisioning this person was able to do. And that's not something any LLM can help you do directly. This is something that either you have or you don't. Yeah?

23:02 This was one big reason why his presentation really got through the cut. Yeah? And he didn't stop there. He actually mentioned all the rationals behind the structure and even the tradeoffs, even mentioning the opportunity cost, something that is very rare these days. Lot of candidates, they never really talk about the opportunity cost as a tradeoff. This guy did. You see? And yeah, go ahead. >> So, first thing you need to do is you need to really you have a deep understanding of the users before you just go and start to work on a solution.

23:41 How did he get that deep understanding of how users are actually experiencing the products? Because let's say like even just using this Netflix example that I was helping somebody with, it was like the user of that product is studios who make movies, but neither I nor the candidate had made movies before. So, how do you really get inside the head in a homework assignment? >> Fair enough. so, in this particular candidate's case, he was from that same industry.

24:08 He had some experience building similar products or rather data products in the past and he was from the insurance sector. He had about three or four years of experience working there. So, he understood that this is how the typical market in the insurance sector works, right? Right? You have a platform, you have partners, the TPAs, then you have employers that take services of these and the employees' money is what's actually invested towards getting insurance, right? So, he already knew this. He had a lot of contacts, especially with partners. He used to work with a platform earlier. So, he knew this value chain. All he had to do was kind of find out his contacts that were employers and some employees.

24:52 Employees are usually the easiest, yes? And based on that, he was able to fetch insights. These were real insights. I mean, it's always better to get it directly from the horse's mouth, isn't it? On top of that, he supplemented his own insights that he got directly with secondary research, too. So, this guy I got to give credit where it's due. He uses a very interesting approach wherein he doesn't depend on just one set of deep research, yes? He would get, let's say, Perplexity doing deep research. At the same time, there's a Gemini deep research running as well.

25:28 There's probably a couple of agents on a different platform doing deep research as well. And at the end, what he does is he accumulates the whole context and gets it summarized so that in case one agent missed some specific points, the other one didn't. And because he got everything accumulated, the proofs that were, well, more real, they would have multiple signals coming in from various researches. So, that way he becomes extra sure that you know, what is being fed into his research is actually backed by multiple sources. So, it actually reduces the amount of time that he has to go and figure out, okay, what was hallucinated, what was not, what's real, what's not real, etc. So, his way of research, it saves him time, makes it a bit more accurate. More importantly, he took the signals that he got from the secondary research, which were quite coherent with what he found in his primary research.

26:30 That's how he went about this. >> Makes sense. So, it sounds like this person succeeded in part because they did have the domain experience. Is there a case to be made that maybe you shouldn't always do every homework assignment cuz these can take a lot of time. Maybe you should only do the ones that you can actually differentiate yourself and you actually have experience or should you take on every homework assignment? >> so, I have a pretty subjective take on this because it applies to me as well.

26:59 Back in my university days, I was faced with the same choice because we were applying to a lot of companies for our job applications and I made a call that I'm going to pick up some specific case studies and assignments in which I have the best chance of getting through, right? Because I don't want to miss out on an opportunity which looks custom-built for me. Yes? For the others, I can take a call depending on how good my chances are because if I know that there are candidates out there in the market who have better experience than me, who are a better fit for the role, and who will most probably be doing a better job at this case study because they just have the right contacts and I don't, then probably it's in my best interest to save that time, effort, and not incur the opportunity cost, isn't it?

27:49 >> Yeah, what I'm learning from both these presentations is that the person themselves could bring some sort of alpha to the question being asked. And so, there's sort of a homework selection meta game that you need to play. And you need to say, "Can I really even get this role? >> >> If so, then I'll spend the time on this homework." It sounds like. >> The caveat here being if you push yourself for something, then you go all in. It's not a half-measure kind of a scenario, right? Because you are not taking up other case studies so that you save time for the one that you really want to go for. That's what I did and that's what I recommend to all my students, too.

28:34 >> Okay, makes sense. >> Yep. All right. So, now when he submitted this, I'll just show you one more slide which he submitted. This was one slide where he was supposed to show an insight that can probably extracted from the tables that are already there. So, and it is quite text heavy because he knew he had to write in a lot of content would be read by AI at the other end as well. So, he was quite okay with the format being like this.

29:05 The larger point why this thing got selected was that every insight here, yes, it's not just surface level. Yeah, it's not just vague and ambiguous or dependent on what we can build later. No, it is actually being built based on the data that was already shared. So, these insights are being extracted from real tables and columns that were present inside the initial data set. So, that way the candidate is telling that yeah, I can use the data that you have provided, and based on that I can bring out insights that probably aren't even there right now. There were at least three more insights like this. This is just to demonstrate that if a candidate wants to showcase depth, and they want to do it with a bit more transparency, this is the way to do it.

29:53 >> Makes sense. So, for It sounds like overriding lesson from both of these decks is quality of insight matters more than formatting. >> Absolutely correct. Yep. After all, it is not unheard of that companies are also leveraging AI to go through a lot of assignments that are getting submitted, and only the ones that get through the first pass of AI get through to human reviews. Anyway, so, why did his presentations work? Because every claim that he made, every insight that he suggested, everything was backed with real data.

30:27 Even the stakeholders that he mapped, each one had their risks allocated, and that's essentially the same muscle as setting autonomy tiers for an AI agent about what each agent can do and where each one is at a risk. How do you handle those contingencies, etc., right? The strongest AI PM homework that we just saw in these slides, if you notice, there was not a real mention of AI in it at all, even though the insights specifically mentioned that yeah, the insight will be generated through generative AI coming into the mix and if you know, evaluating patterns based on that, telling what would be the right target. He didn't really go deep into talking about the models and all, not at all. The rest of his presentation also talked only around what are the ideas and insights he has. And that actually brings me to a very important point about constraints. Yes?

31:23 A lot of the times, presentations are showcased in a way that people like to go beyond the suggested number of slides or things that have been asked for. Candidates feel that it's a better idea to just go ahead and demonstrate I can do more than this. Yes? What do you think, Akash? Should candidates do that? We are going to discuss this in detail a bit later, but I want you to have an opinion on this. Okay?

31:51 >> Oh, I don't know. It feels like you're setting me up to be wrong, but I think if you are disciplined enough not to use AI and it doesn't smell like AI at all, >> Mhm. >> and you want to like in the Netflix example, what we ended up doing, they said give us two-page document. >> Yeah. >> So, what we did is we had links >> >> in the two-page document. We, you know, here's a link to the prototype.

32:17 >> Yeah. >> Here's a link to our data study in case you want to learn more. And we just didn't use AI through it, but I'm curious what your take is. >> Well, that's actually very much aligned to what I have in mind, but we'll talk about that later. Yep. Now, that I have shown you some good examples and reasons behind why they work, let me also demonstrate what are some bad examples. Now, I understand I'm calling them bad examples, but you got to really remember that yeah, even one of these slides got shortlisted.

32:48 >> >> So, this is first of them. The ask from the candidate was to create an MVP for an app that revolutionizes the in-person live music experience. That was the broad idea, okay? Now, this is an actual slide of what this person submitted, okay? If you notice, this is a slide that is showcasing the features of an MVP and they are all over the place. I mean, I don't know about you, Akash, but I personally don't feel that this app and the way it planned to revolutionize the in-person live music experience, it really required all of these. I I don't think this is the right definition of an of an MVP.

33:29 It just looks like whatever came into the mind, nobody said no to that. Everything just got accumulated and got thrown into the MVP. Yes? Recruiters looking at this, when they see, okay, it's going all over the place and it's not really testing the one core idea that solves the main problem, that moment, that's usually the red flag. We are one of the first people who would be like, okay, this guy, no. All right? That was a big big big challenge with this. Even though it has been written in extensive detail and a lot of information has been provided, yet it misses out on the core idea of an MVP.

34:09 Yep. There's more. In one of his final slides, he actually mentioned a very unique take on how to make money. He mentioned aggregating anonymized user data, which is to be sold to vendors and event organizers. So, again, I don't know about you, but I don't believe that selling user data is a great idea to be mentioned at all, especially during case study submissions. So, yeah. The funny thing, this deck, in '23 when it was submitted, it actually got through. It was a top company. Yes, again, it was only for an APM role. So, I guess that's just them trying to give this person a chance. But, today, this would probably not even survive the screening round of the presentation. Yeah? The homework round itself will have at least three or four red flags, big P0 red flags, attached to it. And, the moment it gets to more than two red flags, anyway, the can- -didates are out. So, yeah.

35:12 Perfect bad example. What's your take on this, Akash? >> So, it sounds like make sure that you're demonstrating that you're making tough choices, stick to the scope of the question, and if you're going to propose something risky, like hey, selling user data, at least maybe put the risks or put the flip side on there, that hey, there could be some brand reputation risk. We need to examine if this is standard practice, if this could be anonymized.

35:37 maybe this could come up as something that the public doesn't like, so we need to understand if it's actually permissible. Maybe if they had put that, it would have been better. >> Correct. >> Got it. >> Absolutely correct, right? So, that was the point I wanted to highlight, that yes, there might have been examples where people got through even with this, but that doesn't mean they're going to run today. Today, with the AI doing first-round screenings, catching these becomes a lot, lot, lot easier. I mean, there's not even any limited scope of human oversight at all now. You know, AI is going to flag it. The moment humans see it, you're out. I'll show you one more bad example, right? This is a whole different presentation for a different topic altogether, from a different candidate, okay?

36:26 this candidate was supposed to perform a basic competitive analysis for a top tech platform, and then suggest features to overcome their competition, yeah? A very standard exercise that a lot of PMs or PM2 candidates get for homework, right? what this person did was he mentioned two specific AI-powered features. Yes, very clearly. And he essentially named their pricing, their models, put that on the road map, etc. But nowhere in the 14 slides mentioned anything on the specification front. Like what exactly is the data that is feeding them? What kind of accuracy must these AI products actually have? What is the cost per request for them? What is the feature economics going to look like? Is it actually going to hit any success metrics? Are there any particular success metrics for this?

37:22 Nothing at all. Yes, essentially it just became a quick I would say advertisement of AI products without any real depth of thought around what those products will actually be like, whether these are good propositions for us or not. Yes? And it was made worse because he went ahead and I would say he committed a blunder because he simply mentioned that I'm going to go ahead and add this into the priority queue as an initiative which will have a realistic delivery under 6 months. He mentioned everything, telemetry, attribution, heuristics, fine-tuning, everything. But if you ask them, "Okay, is there a eval plan behind this?" Nothing. No mention at all. In fact, even this was written with bare minimum detail. Yes? Now here comes the question. If you're not mentioning evals, if you're not even including that as part of your realistic delivery, then are we going Are we not going blind-sided into this game? No?

38:23 What if your AI is actually not that great? What if you are just shooting blind, right? That's the part which breaks the case for product managers. And that was the problem that this person's presentation had. Yes? This, unfortunately, is not even the first time I've seen this. There are other cases as well which I have not quoted in my presentation, where candidates have made similar mistakes because like I told you earlier, Evals and the economics, these are the two E's that most candidates seem to miss out on because there's just way too much excitement on showcasing what an AI can do. Who wants to care about the boring numbers, isn't it?

39:06 >> It almost feels like executives a couple years ago who just wanted to sprinkle AI on top of everything. You can't really do that in a PM homework assignment. >> Absolutely correct. Yep. That is the problem. So, yeah, these were just some examples, both good and bad, and that too from real instances. Hopefully, that added a lot of insights to our conversation. >> And some of these bad examples, they might have even been formatted better. And so, it really comes down to the insight. So, what's the repeatable process for PM? If I give given a homework assignment, how do I go about executing on that homework assignment?

39:42 Let's say I have 10 hours or 6 hours. How do I go about executing that time properly? >> I mean, 6 hours is guessing it's more of a guideline because many of these assignments might require some more time than that. Some of them, they even want you to go out and research manually, right? But even if I have to just give you a broad distribution, I would say focus a lot more in the initial couple of hours on understanding the problem statement in detail without worrying about how can I throw more AI into the mix. You are building an AI product. Yeah, it's a assignment about an AI product. Focus on problem solving is still going to be there. Ultimately, it's the role of a product manager.

40:31 Yes? If you don't have product sense, if you are not getting to the crux of the problem, then no matter how much of AI skills you possess, no matter how good your understanding is of Evals, still not going to cut it. Yes? So, focus the first few hours only on understanding the problem properly, understanding its nuances, figuring out why something is going wrong. Yes, as much research as possible. Take help of AI. Throw it all in.

41:03 Yeah. The next couple of hours is what you push for execution. This was the planning phase. Now you push for execution wherein you go ahead and start building out the structure of what your response is going to be like. What are the instances you're going to quote, what your assumptions are, what are some open questions, what would be the outline of your answer, what would be each step which will feed into the next one. Yes. personally, if I tell you about one of the candidate examples that comes to my mind, he mentioned the same structure.

41:39 he put an outline of his presentation and we got it reviewed by me. I gave him the feedback that this just sounds too nascent right now. Yeah, because he was kind of jumping the gun. He had done his research, but he was too much in a rush to showcase the solutions he has thought of, right? >> Mhm. >> gave him that feedback that you got to go back and simply lay it down like, "Okay, this is the problem. This is the top priority problem." Because you still got to do a prioritization. And then based on that priority only, showcase your solution. And even then, mention the trade-offs. Yeah. You you don't rush into solutions just headlong, right? You go ahead, mention the trade-offs, and keep it flexible. You don't just say that you know, with an arrogant tone that, "Okay, this is what it is going to be, nothing else." Yes.

42:33 In presentations or even in general when we are proposing any ideas, it's always a better idea to have an air of modesty about it. Yeah. And then finally, if you have time remaining, which hopefully you will these days using AI, go ahead and format it. If you have pushed for good content, if you have some time remaining, push for formatting. Make it readable though. Yes, the first focus has to be on readability. If it is not readable, if you are goofing up outlines, if you are goofing up alignment, it just looks unfiltered. It looks uncouth, honestly.

43:12 Yes, you are a product manager. You are expected to understand basic design. Yes, you don't want to create a very bad first first impression because this will be your first first impression when this slide reaches someone, right? So, have good enough formatting to keep it readable. Keep it in a format that is readable even by an AI. Yeah, don't over complicate it. Don't make it too heavy by adding in too many images, etc. Links, yes, you can add. Yes, but other than that, keep it minimal if possible and focus a lot more on readability and solving a problem step by step. That is how I would have suggested a student to distribute their time. One caveat here is there will be cases where even the allotted time is not sufficient and many candidates do send out emails that okay, we need one more day because we were busy with something due to which we couldn't submit. My suggestion is that it might have worked a few times in the past for you, but these days because even recruiters know that AI is available. My suggestion is don't. Figure out a way to complete things on time. So, here we saw slides.

44:28 Should a lot of people I've been talking to, they almost want to do a AI prototype for every homework assignment. Should you have an AI prototype in every assignment and when does it even make sense to make a full-fledged app beyond a prototype? Makes sense. This brings us back to the question I was asking you, right? Should candidates go ahead and step towards something that has not probably been asked? Is it not a great idea to just flaunt our prototyping skills? Yes, so here's the answer.

44:57 If the role is a builder PM, if the prompt or the question that has been given to you in your day home says that you need to demonstrate it or prototype it or or and rather, if the company's own product is also Asian Tech, by all means please go ahead, build a prototype. Yes, that's as simple as it gets. But, if they have only asked for a PRD or any just just document like competitive benchmarking all right. Or if you know, it's an app that they didn't really ask for Sorry, what am I saying?

45:35 If you still go ahead and build an app that they never really asked for, then it has a tendency to backfire because it will be perceived as you can't follow instructions that you're trying to unnecessarily spend time building something which nobody is asking you for. Basically, it becomes a question of prioritization. They would feel that you will probably try to flaunt off the same skills when we hire you even though you have other things to do. That's basically extra cost to the company, not extra credit. That's why.

46:08 >> So, don't always just add it unless like really the role or the company calls for it. >> Absolutely, right? And even if you do build, right? Just maybe build a thin slice just to showcase your idea, just to supplement your ideas, right? It should not be a huge broad mock, very elaborate structure that you created. Just just show the eval, right? Not just the demonstration part. The idea is whatever is being asked, can you follow that instruction and create something really full of content, the real value for the recruiters? Because after all, it's part of the interview process. They are testing your skills and they have other questions to test you on your other skills. Don't demonstrate everything that they didn't ask for in the same question.

46:58 >> Stick to the prompt and talking about sticking to the prompt. A lot of times they'll ask for just like one page or two page and we want to add an appendix or we want to add more links. What's your take on that? Should you stick exactly to the length requirement? >> Yes. If they ask for one page, that's it. No no negotiation that that one, right? Even when we were discussing this earlier, you mentioned it yourself. You submitted two pages. You did not really added on too many pages after that, right?

47:30 You used a smart way. You had links present in those pages and you redirected people to those links, right? But the submission was two pages only, right? That's the point. Following the constraint is also part of the test. Can you cut down the detailing to that level that you can make it so concise that it can fit in one page or two pages? Sometimes that is what as PMs we might need to do. We might not have all the space in the world on a small mobile phone screen, but we will have way too many features to accommodate. Can we do that? That is what is being tested.

48:10 Especially enterprise PMs, we we live within these constraints almost all the time. Yes? Akash, you have had your experience with enterprise products. What do you think? >> Yeah, I think so. Although I think that the links maybe is a little bit of the sort of hack. If you think that I've done all this work, maybe like the first successful candidate you actually went and talked to people in the four buckets, partners, ecosystems, customers, and you want to show that off. I think you can use the links strategically. Like maybe you recorded those calls. Maybe you went even further, you put them into Dovetail and you tagged the insights. Maybe you went even further and then you aggregated the insights in the quotes. If you really did all that work, I don't think it hurts you to summarize the key point, the key takeaway, and then have a link.

49:05 And so, that to me I think is the one hack where maybe sometimes you can go beyond it, and hopefully not too many people will think, "Oh, he's just going outside of constraints." >> Absolutely agree, and that is what this point is. You send one page if they ask for one, and you add the link for anyone who wants it more. That's it. >> So, what are the biggest mistakes you see people make with homework? >> Oh, there are a lot, actually.

49:29 I'll just quickly jump to that. Even the greatest PMs, sometimes they end up making these very common pitfalls. Like I mentioned earlier, sometimes you know, they will have very silly mistakes like a typo present on the first page of the deck itself. Interestingly, there's a statistically higher probability of missing things out in the early half of anything that you're about to present rather than the latter. You know that. It happens because we typically have a bias towards the middle and the end.

50:03 But, we feel that the first part we have already had a good look at. It's never the case. So, this is a very common problem. Candidates, they end up polishing slide nine where they have written the conclusion, etc. But, on the first page itself when the presentation is about to start, the intro slide itself, they might goof up a spelling, misalign a text, etc. Yeah? Very bad very very bad impression to create, especially when you're just starting out. You kind of build a perception in the mind of a recruiter that you lack attention to detail. Not Not the thing you want to create. Yes.

50:40 Next, you know, kind of answering the prompt they wished they got. It's like again, wishful thinking that oh, I think the question means this without going into good depth of what the question is really asking you for. Without understanding why a particular problem is being tackled specifically, right? Instead, if the candidate just imagines that, okay, oh, this is what the problem must be asking. That's the problem. A lot of smart PMs, they also get into this. I remember one particular case.

51:14 the candidate was asked about taking a decision around adding a few features, and he had to decide which of these three suggested features would be the best fit. Was I would say not best fit, rather the best bet for the company. Yes? They were supposed to do market analysis, arrange statistics and numbers for it, and based on that, create a calculated call.

51:45 Yeah, that was the assignment. This person, in all their wisdom, they perceived it as a benchmarking and parity matching exercise. And that's all they did, and that's all they submitted. They did not really go through and mention which feature should actually be promoted further. They simply compared the three features with what other competitors were offering, and simply showcased a benchmarking. Yeah.

52:15 Similarly, many people, especially the ones that are into getting AI to write things for them, you will see this a lot in their submissions. There would be sentences that say nothing, like leverage AI to revolutionize product line. Revolutionize that itself is like right out there, that is written by AI, right? So, nobody talks like that. And yet, you will see these terms. Whenever I see something like this, whenever I see m dashes, whenever I see those not this, but this, these are absolute AI-isms, and they can be spotted on from a mile away. So, when you see that, even with great PMs, you will see sometimes they will just be like, " just it's just a conclusion. Let's just get AI to summarize it." No, please. Humanize it.

53:00 Spend some time, okay? missing out on numbers, or worse adding numbers anywhere you know not adding a number where a number belongs and vice versa wherein you don't want to put numbers at a place where you're drawing conclusions and yet you will unnecessarily throw in numbers there. Both are wrong. Understand where a number is really supposed to be and then only put it there. Usually it's about storytelling. Yes. A well stationed number it supplements your story. You know it just makes sense to see that number at that particular point of the story. But if you miss out on that you're telling the recruiter that you're not really that good at telling a story. And for product managers like I said you can attest to that storytelling crucial skill. Very very crucial skill especially when you want to align your stakeholders. Man, not the perception you want to create in front of a recruiter that you're bad at storytelling. Okay. And finally I think one of the most important points and where I have seen personally a lot of people making mistake they skip the defense.

54:06 The defense requires prep. You can expect to be grilled upon the decisions that you have mentioned in your presentation. How can you not have really solid points to back your claims? Yes. It is not sufficient to just build a good presentation and crack that round. You got to remember that many times during the interviews you will be asked questions about it and you got to remember the statistics you got to remember the format you got to remember every decision that you made that made you prioritize a particular feature above something else. What was the reasoning behind you know putting this particular slide before the other etc. Yeah.

54:48 There are many other more mistakes like this that a lot of candidates do as well but these are the ones that I've seen the most so that's why I thought I'll just present these first. >> Yeah, preparing the defense is so important. And that's I think where AI could actually be helpful. You can have AI read your document read the prompt and say, "Hey, what are the three biggest problems with this? What What are the three questions they're most likely to ask?" And then practice. And maybe even check in with your recruiting buddies or coaches or This is where coaching can be helpful, I feel, also.

55:23 Doing that rounds of iteration. And so, if you let's say given 4 days to build a homework assignment, try to finish it with 1 or 2 days left at the end, not rushing until the end and emailing, "Can I have another day?" But actually finishing a little early so that you can spend time on this defense. That is very high ROI. >> Absolutely correct. >> All right. So, if we put this all together, everything people learned, what's the system to succeed on homework?

55:51 >> I'll mention three broad points, and that will summarize the system. First, be in a rush to understand the problem. Don't be in a rush to just send AI slop, okay? Point number two. Evaluate everything that you have written, and get it reviewed as many times as possible. First through AI and yourself, next with someone experienced, someone who has actually done hiring or evaluated these in the past. That will give you, my god, so much concrete evaluation, concrete feedback that adds just way too much value to your presentation, right? And thirdly, in order to succeed, focus a lot more on the content. Yes?

56:42 Don't worry too much about the formatting, just keep it readable. Don't go into fancy templates. Yes? Focus a lot more on delivering value. Don't again try to portray fancy keywords about AI. A lot of people are getting into the habit of throwing in embedding and this eval framework and whatnot. Yeah, all that is great, but focus more on the problem at hand and how are you going to solve it. Don't forget the evals. don't forget the metrics, don't forget the economic side of things.

57:14 Mention those parts. You are applying for a AI product role after all. These are essential parts. Forget about the formatting. That part you can cover later. Just focus, laser focus towards just the value part. And that's how you do it. >> And that was a ton of value, all freely provided. Thank you so much for this deep dive. If people want to find you online or work with you or learn more, where should they go? >> so, well, interestingly Akash, I have also started my YouTube journey.

57:48 Yes, so they can find me on my YouTube channel. I'm also there on LinkedIn and I can also be found on Topmate for one-on-one consultations and other plans. >> All right, guys. I've been watching Sumanth's content for a couple years now. He seems like somebody who's super dialed in and if you watched the end, you've enjoyed his insights as much as me. So, go give him a follow, go subscribe to his YouTube channel and we'll be bringing you more PM job searching content on this channel. So, make sure you have subscribed here as well. Comment down below what you want me to cover next and I'll see you in the next episode. Thanks so much for watching to the end. You can check out my newsletter for a deep dive on this video at news.akashg.com.

58:32 And I wanted to take a second to talk about my land PM job program. There is no other program like this. In this you are going to learn how to create a candidate market-fit-driven job search. Create inbound leads from LinkedIn and actually land jobs even without applying for them. On top of that, we will create an automated cloud-driven job applier for you that will apply to jobs, create connection requests, create networking meetings, drive referrals for you on autopilot.

59:05 And it's not just job search. We're going to teach you AIPM with Ankit Varmani who is head of AI product at Uber. We're going to teach you PM fundamentals with Prasad Reddy, who has been a chief product officer at El Neutra and senior director at Danaher. So, you are going to learn absolutely everything you need to land your next dream PM job. It only costs $5,000. If you were to buy this stuff individually, it would cost you $3,000 for the job search core, $3,000 for the AI PM fundamentals course, $2,000 for the PM fundamentals course, $500 for the resume review, $500 for the LinkedIn review, $500 for the portfolio review, $500 for the GitHub review. On top of that, the mock interviews themselves each would be $500 of which you get three of them.

59:48 Plus the one-on-ones with me, which would cost $600. If you add all that up, it's way more than $5,000. You're getting all of that for just $5,000. And if you are not in the United States, we offer regional pricing. It is only 1 and 1/2 lakhs in India and 2,000 euros in Europe. So, be sure to apply at landpmjob.com. Cohorts run every 3 months. The next cohort starts November 23rd. I'll see you there.