AI PM Technical Interviews: Complete Guide

0:00 So what questions are you actually asked in the AIPM technical interview rounds at top companies like OpenAI, Anthropic, Google, Meta? >> The fastest way to end a PM interviewway is to freeze on one question. Do you use LLM or MLM? >> Meet Prasad Ready, former CPO at El Neutra, senior director at Danar and someone who has coached over 150 PMs over the last 12 months in the job search. Almost all the PM know how to use chat GPT and card code but none of them understand how this works underneath the system. And understanding these technical concepts is the key to nailing your AIPM interviews.

0:42 >> Most people think this is just a prompt because they've been trained by custom GPTs. But really it's prompt plus tool skill. >> I've been coaching one of our cohort members Brias who landed a role at the VP of product at a AI startup company. The questions she was asked were how do work? How do work? How does work? And these are the key concepts in nailing AIPM interviews. One of our cohort members interviewed at NVIDIA and Glean and said she was asked these types of questions. So today Prasad and I are not just talking about these questions or listing them out. We are going to teach you the fundamental concepts you need to understand and teach you how to answer these questions.

1:29 So what questions are you actually asked in the AIPM technical interview rounds at top companies like OpenAI, Anthropic, Google, Meta, Sierra, the list goes on and on. We used the database that we've collected from our cohort members to come up with some of the top questions you have to understand. How does an LLM actually work? How does function or tool calling work? How does lane chain work? How does routing work? How would you define the agent skills? So, if you stay till the end of this video, we will teach you how to answer all five of these top AIPM technical interview questions starting with the first principles. The typical way people want to answer this question is kind of the reality, which is, hey, I do use AI for that. The much better way to talk about it is here's the model, here's the call, and here's how I'd route it. You want to be as specific as possible. Actually show that you don't have just a quote unquote business PM understanding, but actually you have a technical PM understanding. You need to be able to speak to model choice. You need to show that you were actually in the room. You don't want the interviewer to write that you were just adjacent to the AI decisions and not driving them. And so that's the key. And to do that, you have to understand how these things actually work at a fundamental level. We're going to break this video down into five key concepts for you and we're not only going to teach you them, but we're going to demo them. So, we're going to start with LLMs. We're going to help you understand what next token prediction is. We'll go into tools and function calling. We'll cover skills and agent capability. We'll cover orchestration with lang chain. And we'll cover routing, how you choose the right model for the right job. Yeah, looking at this concept, they might be overwhelming and sound too technical, but trust us, we're going to actually help break it down and simplify these technical concepts in the next 30 to 45 minutes. So, let's start with how an LLM actually works.

3:22 Fundamentally, LLMs are predicting the next token. So, when you ask it the capital of France is, it's not working like a dictionary or a Google search. It's figuring out which of these Paris, France, located the A is the right thing. And so the model is sampling from this distribution. It looks at it and it gives Paris a 0.92 rating. That's the highest rating. That's the next token. Now, how does it decide amongst these? There's a temperature setting. So temperature sets how risky the pick is.

3:52 It's not really retrieval. It's just a probability. That's the very first thing to understand about LLMs. And there's a couple things I want you to understand beyond that. So temperature is one, right? Low temperature means it's going to be safe and repeatable. High temperature means it's going to be creative and varied. That's essentially the sampling dial. Then there's context window. Fundamentally, we need to understand that LLMs go through a training process which is comprised of pre-training and post-training. And at that point, the weights are frozen. So once you actually get to a user querying, there's a certain context window that an LLM has access to. A lot of modern models, take Opus 4.8, has a 1 million token context window. This was made really popular by Gemini 2, which released with a 1 million context window nearly a year and a half ago. So within a context window, that is fundamentally on top of what the training data has that an LLM has access to. It defines the boundaries of what it's looking at.

4:58 The final concept to really understand is hallucination. LLMs are doing that next token prediction that we just talked about. The capital of France is France even gets a 0.02 probability rating. In order to reduce it from choosing the capital of France as France, we add in things like retrieval and tool calling. These prevent it from making confident decisions that are actually wrong. So that's the theory. Let me actually demo this for you live. All right, let's get into Google AI Studio and see temperature in action.

5:27 We're going to ask it to name an animal with temperature zero. It's tried capibara. So theoretically, under temperature zero, if this is true, we should get capiara again. Now, under temperature one, we should be less likely to get the exact same animal because capiara shouldn't dominate the distribution. Let's see. We'll try it once. We got temperature one. Capiara. Now if we do it again and again and again theoretically under temperature one it should be more likely to change.

5:59 And if we go even further beyond temperature one all the way up to two here we see it's chosen axelottle. So what's happened? This is the intuition you need to understand. At temperature equals 1 you sample from the model's native distribution. At temperature less than one you sharpen it. The top tokens lead grows and we saw capiara repeat. At temperature greater than one, you flatten it. The gap between the favorite and the rest narrows. And so one time you actually get a non-deterministic response like axelottle. Another way to prevent hallucination like we talked about is adding tool calling. So a tool calling example might be grounding with Google search. And so even at a temperature of two, if we do something like the population of the USA is less than five words, we're going to see the population is approximately 349 million with four sources. And if we do it now, we get approximately 349 million people with six sources. So we're getting fundamentally the same fact, but a slightly different answer actually. And so that's where tool calling comes in to reduce hallucinations. Finally, let's understand the power of context. So here we have an entire book war in piece by Leo Toltoy. What happens when we paste this into Google AI studio? Well, it's less than a million tokens and so we get so we get a really good response and that is the power of your context window. Now what happens when you overload your context window? I'm going to go grab crime and punishment another huge book. So we've now given it more than a million tokens. As you can see an internal error has occurred. That's because we've overloaded the context window of the model. So now you understand the three basic building blocks of LLMs. Now let's move into tools and function calling. The model doesn't actually touch your systems.

7:48 What it does is it emits a JSON call. Your code runs the function. The result goes back to the model and your model writes the answer. So let's see this in practice. You're up prasad. >> Yes. Thank you Akash. Now we're going to do tool and function calling. I am in the OpenAI platform platform.opai.com. As you can see here, I have the ability to pick the model. For the demo purposes, I'm just going to pick a mini. it's going to be faster. So now we're actually going to build it a function for get weather. Again, don't be alarmed with all the code. Let's actually use AI to generate the JSON function for this. I'm going to say create a a JSON function for get weather use location. Boom. It actually generated a JSON function for us. Again, as I said, we don't actually have to write any code. And I'm going to say add. And now that we have created a function, I'm going to say what is weather in San Francisco.

8:59 It's actually going to return us the get weather by location because we it's actually throwing an error not calling the function. So what I'm going to do is now I'm going to go give it I'm going to manually add a tool desert for example temperature 63° FY. Now let's see what it does. So now it should write an answer using the data that we provided. See the location we gave earlier was San Francisco. Now we're saying you know temperature is 63 degrees. actually returning us saying San Francisco is 63° and foggy. Right?

9:38 You hand the model a list of tools each with a name, a description, and a JSON schema for its arguments. The description is what the model uses to decide when to call. So vague descriptions cause wrong calls. As you have shown earlier, right, we given a vague description saying what is the weather in San Francisco it actually return. when the model decides a tool is needed. Its output is a structured tool call. The function name plus arguments as JSON. It does not run anything. Your application code executes the function. Then you pass the results back into the conversation and the model uses it to finish the answer. As you can see here, San Francisco is 63° Fahrenheit and foggy. The model can get the arguments wrong or call the wrong tool. Your code validates and handles errors. The intelligence is in deciding to call and with what not in the execution. MCP is the model context protocol an opensource standard from anthropic for connecting models to tools and data sources through one interface.

10:47 So think of it like the APIs in the old days of SAS. That's what MCP is. So you're not hand building a separate integration for every tool and every model. You just use the MCP essentially like you know write it once and use it for hundreds of tools available. >> I want to take a second to talk to you about the fourth cohort of LAN PM jobs. I trained 30 students in cohort 1, 50 students in cohort 2 and 75 students in cohort 3 and I am bringing back a program for cohort 4. It starts in August and it lasts three months where you're going to have intense sessions. A Monday morning session where I go over your resume, behavioral interviews, LinkedIn. On top of that, Bart Jorski is going to be teaching you the PM fundamentals in 2026, how to write AI PRDS, how to AI prototype with cloud code, all of the key skills you need to freshen up your knowledge for this market. And Ang Vermani is going to be teaching you AI product management. He is an AI product manager at Uber and he is going to teach you how to build AI features that actually work successfully. On top of that, Prasad Ready is going to be doing one-on- ones with you for mock reviews, LinkedIn review, candidate market fit review. So, it is a full package. It is three courses in one for one low fee. So, join at landpob.com.

12:03 Now, let's move into the third topic you need to understand agent skills and capabilities. The most important thing that you need to do is define how your agent works. Most people think this is just a prompt because they've been trained by custom GPTs, but really it's prompt plus tool plus skill. This is what you need to understand to really define an agent. So, what is the instruction or the system prompt? Not just the one-off instruction. What are the tools that it can act on, the APIs or the MCPS it can call? And then what are the skills? What is the package knowhow it has for a whole job? Defining the agent skills is one of the highest alpha areas. So let me show you exactly how important agent skills are. So here we are inside of my PM operating system within cloud code OS. I'm a hypothetical PM working on Slack. And so I've defined all of that information in for my PMOS inside my context library. Inside my context library, I further have all of these skills that I've written. One of the skills I've written is PRD draft.

13:08 The skill is a package set of instructions and it's optional with scripts and resources. It has all of the routing, the context priority. It might go check the context health, what we know about a feature, when to use it. It dictates whether it should ask you clarifying questions, how to structure drafts, and so it goes on and on. And this is really the alpha. So let's go ahead and see what happens if we don't use this skill. We're going to ask it to write a PRD for an AI feature that summarizes unread Slack channels. Now, normally it would go ahead and just use the skill, but here we've asked it to not use the skill. So we've actually put that into there. There's going to be some questions that it asks you, which is a good thing. That is some of the intelligence that's already built into a high quality model like Opus 4.8.

14:02 Now, we're going to try the same thing, but we're going to ask it with the skill. You can just ask it to write a PRD draft, but the most reliable way to call a skill is actually to use the slash command. We can see there's already a difference between how the two chats have gone. The no skill chat here, it asks some questions. The yes skill chat first read the context library, and it's already found that I have an existing PRD. So, we're going to go ahead and evolve that PRD. You see, it actually asks less questions with the skill because it has a better routing of my content. So having a skill in this case actually reduced the amount of human input I needed. And now we've gotten an output. This output has the hypothesis, the problem, the strategic fit, the solution. So it's created its own sort of outline. Now look at how differently the version with the skill is approaching it. It understands I'm 2 weeks from the June 18 GA. It understands we're moving from solution review to launch readiness and it's creating a launch readiness version.

14:57 It's worth understanding exactly how all this worked under the hood. So, our skill version PRD is ready. Let's take a look. We've got the right stage. We've got the TLDDR for the go no-go, what we're shipping and is locked, the success metrics, the launch gate, the go no-go checklist, the rollout plan, the roll back and kill criteria. If we were to compare the two PRDs that we got, it's obvious that this one is closer to something you could actually send to your colleagues and it wouldn't be considered AI slot. This one was missing a lot of that. And that is the power of having a proper skill. A skill does not retrain the model or change its weights, but it is instruction and capability that's loaded at runtime. So the prompt is the one-off instruction in the moment. The tool lets the agent act on the world or fetch data or do something in the world. And the skill is the reusable knowhow for doing a whole class of task. Well, when you find yourself in an interview, make sure you touch on all of these components. And you might even touch upon how the tools are creating memory for your agent, which is another thing that generally is going to enhance your agent. And so, when you're thinking about answering interview questions, don't define features. Instead, think about defining agent capabilities. And now you understand how agents work under the hood and how important skills are.

16:15 If you want a PM operating system like the one I've shown here, check it out on my website at www.news.akashi.com. You can either purchase it one time on Gumroad for $49. Or you can become a founding plan member where you get the $150 version of my newsletter for a whole entire year which includes my bundle with free access to tools like bolt new arise dovetail and many more. Plus you get access to this PMos, my job search OS, my PM prompt library and a couple upcoming products. Let's move into orchestration and lang chain.

16:47 >> It might be overwhelming for you to think about what orchestration and lang chain is. Let me actually break it down. Think of as an orchestra. You've got violins, cellos, a precussent section. Each one is capable on its own. But without a conductor, they don't know when to come in, how loud to play or in which order. Lang chain, think of lang chain as the conductor. The model is the orchestra. Lang chain does not make the model smarter. It coordinates when each piece plays and in which order. The model is still doing all the thinking.

17:23 so with that what I will do is I will step into the demo to simplify how orchestration and lang chain works. Okay here we are in collab research.google.com. So easy one you can just say collab google and the first URL the welcome to collab. This is where we are. What we're going to be doing is we're actually going to start creating our own orchestration. So when you go here, you can actually insert new cells here. let me do this. Insert a code cell. I've got a bunch of them. Before I run this, I will tell you everything about in terms of what you need to install. The only thing we need is to get to this collab one and then I'm going to tell you what we need to do to install lang chain. Here I am going to copy paste some code here. So we're actually going to be installing bunch of things. Again, the good news is you're actually not physically installing anything on your machine.

18:23 It's this instance of Google Collab. We're going to install. We're going to install lang chain. We're going to install OpenAI wrapper and fiss vector. We're going to talk about what that is in in few seconds. All I'm going to do is I'm just going to hit run and let it do its magic here. Now that it's almost done, then again, ignore the dependency for right now. But if you're actually doing it for the first time, you'll actually see everything green here. For right now, I will ignore this. And what we will do right now is we'll actually go ahead and create a one of the smallest chains possible. You know, we introduced what orchestration is here.

18:58 Let me do this. Let me actually copy paste this and actually walk you through what we're trying to do here. as I introduced orchestration, what we're trying to do is actually three things here in the chain. prompt, the model, the parser. The prompt here is as you can see was saying tell me one interesting fact about and the topic here is the black holes. So the prompt is tell me one interesting fact about black holes. The second one is the model here we're actually saying we'll use the chat open AI we'll actually use GPT 4.0 mini and then the parser is how do you actually want the output for right now I'm actually saying the output is going to be a string but you can say JSON anything else to simplify things I'm just going to say give me a string so the other thing to notice is you need to because we are actually saying we want to use an openi model we need an API key let me actually get my API key here and then I'm going to replace my API key here so now this is good. What we're asking is the prompt is again tell me one interesting fact about black holes and let me hit run. It should print us the output in a second. First as I said first a prompt template it takes your input and formats it into a sentence. Second the model it receives the sentence and generates a response and the third is the output part for sure. So we asked what give me one interesting fact about black holes. As you can see here, it actually went and you know brought us back something saying hey one interesting fact about black holes is that they actually grow by observing. Again I'm not going to read it but you get the essence of it.

20:41 So this is exactly what orchestration is the prompt the model the par. This is a simplified version of it but you get the gist of in terms of how to go about it. now what I will do is I will actually touch upon another one. If you have heard something called rag, what we will do is what is rag? We will actually add retrieval right now. This is the pattern that shows up in almost every AI product and the one PMS get asked about most in system design. It's called rag. It's retrieval argumented generation. Instead of letting the model answer purely from its training data, what we're doing is we are argumenting or fetching relevant text from a store and inject into the prompt. A number of reasons why we want to use a rag. We can reduce hallucinations or think of something like if the information changes very frequently, you don't want to go to the and retrain the model every year or every num x number of months. That's where the rag systems are going to be helpful. In the introduction package, we talked about the the lang chain and all the installation done. I also touched upon fis here. We're not going to have to go back and reinstall it. It's think of it like in a vector database that is it's a vector store. That's where rags really come into picture. So what I'm going to do is I am going to copy some code and and demonstrate how this rag system works. The good news is we already given our API keys, right? So it should already do it. We're still using the open AI and what we are actually providing as part of the prompt is right now I'm actually providing some snippets here but think of it as your enterprise knowledge. It could be PDF documents, your database, your customer support knowledge, whatever you can think of.

22:36 That's what these snippets are for. To simplify this, I'm just giving some text, but essentially you can sort of replace with anything else. And now I'm going to go hit run. The see it actually responded saying black holes form when massive stars collapse under their own gravity. again the good news about it is the the response is grounded in tooth, right? The truth is we actually give here black holes form when massive stars collapse under the gravity. so now the response is the question is how do we how do black holes form? What this has done is it actually read the snippets from the rag system is again I said this is just a string but you can use PDF documents other data sources and their response is actually grounded in truth here and black holes form when massive stars collapse under their own gravity. So with that I will move on to the next topic. Again now right now we're using the rag system. We can actually do this the same output with less code and raw SDK no lang chain no framework. This is the tradeoff made concrete for a single call. This is completely fine. Again a number of things but right now what we'll do is the same thing. Here is the code. I'm copying the question we're asking is tell me one interesting fact about black holes and the response is one interesting factor of black holes is that they actually evaporate over time.

24:13 So in this case we are not giving it a rag or any grounded reality but then directly asking questions of open AI tell me one interesting fact and that's what it is again this is one call using the SDK a system with many moving parts consider this framework this is where you need to understand in terms of trade-offs about making simple calls versus using rag systems versus using orchestration with tools like lang chain or otherwise >> one more thing I'd add here is that langchain isn't the only option and Prasad talked about that. But what's worth noting is that Langchain itself split in two. There's lang the core library and then langraph the agent layer from the same team. So if someone says they're building agents in lang chain today, they're probably using langraph. If your problem is mostly retrieval chatting with documents, you might consider using llama index. That is retrieval first. And so the rag part of it comes out in about half the lines.

25:10 For multiple agents working together, let's say a writer, a reviewer, all these agents that you put together, most people are using Crew AI or the OpenAI agents SDK. And if you're not doing coding at all, visual tools like NAND or Lindy are also options for this. It's the same takeaway as the SDK point. Pick the lightest tool that solves your problem. So the next topic we're going to go on to touch upon is routing. You know what is routing? This is one of the most practical things you can understand about how AI products are actually built and almost I'm pretty confident no PM candidate can talk about it correctly.

25:49 Not every query deserves the same model. So let me explain what this mean. Sending what is 2 plus2 a frontier to a frontier model is like hiring a surgeon to put on a band-aid. Sending a nuance reasoning question to the cheapest model is a quality risk. And this is exactly where a router comes into picture. A router makes the decision automatically. If we are asking a question, what is 2 plus2? It understands and says, "Oh, you know what? This does not require sending to the best model. It actually sort of routes to a cheaper model where it can be answered." That's what routing is.

26:25 Two patterns that come up constantly. model routing, cheap small models for easy work, large model for hard work. that's what the intent of the routing is. Classify what the user wants and send it down an entirely different path. One agent handles billing question, for example. Another agent handles technical support. They never overlap. The classifier doesn't need to be fancy. It can be cheap LLM call, an embedding similarity match, or hardcoded lines. So I will actually explain this with a code and I have some code here and so let's see how this works. So what we're asking is to answer this question here and then we have the large model we have a cheap model the as we talked about earlier the prompt we're actually saying you know go down the path of cheap model here and if the classifier says you can use a cheap model it actually routes or simple it actually routes to the GPD4.0 mini. Now, as you can see, right, the output is we're asking it to say, hey, which model did it route to and what is the answer because the the router figured out and said this is a simple question or a query to answer it actually routed to GPT 4.0 and then as you can see right we can actually printing out and saying now tell me which model it was routed to and what the answer is the model is the 4.0 mini which is because it's a simple question that we're asking and then the answer is 2 + 2 equals to 4. And now let's actually go to the the router function here. The router is a function label comes in decision goes out the right chain runs. simple gets the mini model complex gets the large model.

28:16 Two lines of conditional logic and I'm going to show that right now with the demo here. Again I go back to my code cell. I have some code here again. the large model is J GPD 4.0 and then the answer prompt is answer this question. So actually the next topic we'll go on to is two queries and then we'll do a cost comparison. And now we make the cost gap visible with two real queries. so with that what I will do is I will we are going to be okay we are actually asking the question here what is 2 +2 and with a model GPD 4.0 of mini it's actually 14 tokens and the approximate cost is as you can see it's pretty negligible right the same question in terms of what is 2 + 2 were actually going through a GPT and then the tokens consumption is 574 again the cost is still negligible but think about compare it to the cost here it is and then you add that up over let's say hundreds and thousands of times or millions millions of times the cost adds up. As you can see in this clear example, routing to one model to a cheaper model versus a more expensive model, it might seem the cost is negligible. Yes. But then when you do this millions of times, that's where really it comes into the cost. As you can see, that's exactly what the router is going to come into picture in terms of figuring out and saying which model do I route to help answer this user query correctly, accurately, and more importantly cost optimization.

30:05 >> So, we've covered four layers. Interaction, which is the app, chat, voice API, then orchestration, then orchestration, which routes between agents, and it manages state. This is where the router sits of course that we just demoed. Then the agent layer. These are the named agents with non-over overlapping job. Then data and infrastructure retrieval vector store model APIs and memory. You guys just got a master class in the basics of the technical concepts you need to understand. Let's pull this all together now and talk about how to answer these technical interview questions actually in the room. The meta move is to answer a technical question by showing you actually understand the system. Naming the mechanism and choice, not just reacting with a definition. A line that often does well is we don't need an LLM when an ML model will do. Any sort of tabular prediction like churn or fraud forecasting, these are usually better served by gradient boosting trees.

31:03 Things like XG boosts, which are cheaper and faster. They're also easier to interpret. You really want to use an LLM for language, reasoning, and generation. Hybrid systems are normal. And saying that actually is going to signal your real experience. When you genuinely do not know, name what you would check and how you would find out. For example, which eval metric you would look at, like recall or F1. That reads far better than bluffing. And bluffing on a technical detail is what ends interviews. Now, we're going to do a couple practice interview questions so you can see what a good answer looks like. Akash that was a great session. So let's actually do a mark right now.

31:40 Let's say you're building a churn production. What would you use? Would you use an LLM or an ML model? >> That's a great question. The answer is ML model. But let me explain. Churn is tabular. So you have structured features with a clean yes or no label. I would put gradient boosted trees on it. XG boost because they're cheaper, faster, and more interpretable. And the first question after I ship is why didn't an account get flagged? trees answer that well. An LLM does not. Where an LLM really is going to earn a seat is in the language layer. Turning messy support tickets into a feature or drafting the save email once an account trips the threshold. So the real system that I would build is a hybrid system. So overall it lands hybrid trees predict the LLM reads and writes.

32:24 >> Cool. That that's fantastic. I love the fact that you actually broke down and also made a comparison between what an LLM is great at and why ML is actually a better way in terms of solving this problem and then also said hey there are certain things where we'll have to do a hybrid approach. Yes, sometimes folks the question is not an eitheror but I love how you framed it saying for the majority of the things that ML models actually work but for the output you just use an LLM. Awesome.

32:57 Fantastic response there. So, so let's actually get to the next one. Your AI feature calls a tool. Walk me through what actually happens under the hood. >> Okay. So, as far as I understand, the model never runs the tool. That's what most people get backwards. I hand it a list of tools, each with a name, a description, usually a JSON schema for its arguments. The description is what it used to decide when to reach for that tool. So, a vague description is a real bug, not a cosmetic one. When it decides its output isn't an answer, it's a structured request. The function name plus arguments. My code runs that, gets the result, and passes it back into the conversation. Then, the model writes the final answer using my data. That's the key distinction. So the intelligence is in deciding to call with what the access and execution live in my code which is also where I validate because it can absolutely pass the wrong arguments.

33:53 >> So great response. So what one follow-up question would you think the orchestration we talked about earlier comes into picture here. >> Not for this no a single get weather call like we talked about earlier that's just request and response. The model asks my code runs it the result goes back that lives in my app code. Orchestration shows up when the loop stops being a single step. When we have multiple tools to sequence or a failure to retry, maybe state to carry across turns or of course more than one agent to route between. I think that coordination is really what the orchestration layer own. So one tool, one call, you don't need it. Many steps with states to manage. That's where orchestration earns its place.

34:34 >> Awesome. Great response. Yeah, you are absolutely right. This is in this context we're only doing one tool call but in the bigger picture if there are other things this particular system is doing that's where orchestration comes into picture let's do a couple more short ones Akash so how does lang chain work and how would you and would you use it here >> okay here's how I describe it lang chain is the glue around the model prompt templates output parsers memory retrievers it's the way to pipe them all together for a single call as we said we'd probably skip that. A simple SDK call is all you really need. It's also simpler and easier to debug. It really is going to earn its place once I've got many steps, swappable pieces, memory, and tool use to coordinate. I'd emphasize that abstraction out loud. I'd say that trade-off out loud because the abstractions can actually hide what's happening, and that makes debugging harder. For this feature, if it's one call plus a tool, I'd start on a raw SDK and add a framework only when the orchestration gets real. I'd also flag that lang chain split into lang chain and lang graph. So for building agents I'd use lang graph when it gets real.

35:42 >> Awesome. Fantastic. Akash. So why do models hallucinate? And if you are a PM, what's your move? >> It's really the model's working as designed. It's not a glitch. The model predicts the next most plausible token. It isn't looking anything up. So with nothing to ground it, it'll produce something false with complete confidence. I don't treat that as a prompt problem. When the answer has to be right, I ground it. Either retrieval to pull in real content or a tool to fetch real data. Then I measure whether the answer actually traces back to what I retrieved. A longer or sterner prompt doesn't fix it. Grounding really does.

36:21 >> Great answer, Akash. One follow-up question. How would you measure grounding? >> Okay, that's a good point. Honestly, I'd probably confirm the exact metric with the team before I lock it on one. But my instinct is two separate checks. So, first faithfulness, which I kind of alluded to. Does the answer trace back to the retrieved text? Second, retrieval quality. Did we even pull the right snippet in the first place? Those are different failures, and I wouldn't really want to chase one when it's the other.

36:49 >> Awesome. That's great, Akash. >> All right, so those are the mock interview responses, guys. You can see my responses are probably like a eight and a half or a nine out of 10. You could go deeper. You could say more. And that all comes from actually using the systems. Go execute the demos we talked about in this video. Now you have some hot questions you probably watched this video and you're asking. So let's answer them.

37:12 >> These are some of the questions we actually get from our LAN PM cohort members. So let me fire up a few of them for you. Akash are PMs expected to know this or is it just a filter? >> If we're honest, it's mostly a filter. Nobody needs you to derive how attention works. They're checking one thing. Can you sit between two developers and not freeze when they go a level deeper? The freeze is really what's going to fail you. A wrong detail you can recover from in the next sentence. I've coached PMs who have found out that they made factual details and passed and other PMs who were really weak in their responses and they got dinged. So, it's about showing that confidence in your depth.

37:52 >> Okay, fantastic. Let's move to the next one. You know the we talked about lang chain and all but you know the question we actually get asked is is this a real tool or is just like a buzz word on the resume. Honestly it's both. Real teams do ship on it. That's important to know. And it's also the thing people list to sound technical. The tell is pretty much one sentence. The name dropper is going to say we used lang chain and stop. The builder is going to say we used it here ripped it out there because the abstractions were hiding what broke. So, it's the same word with the opposite signal depending on how you actually reference it.

38:28 >> Awesome. I'm glad you actually said it's both a tool and a resume bit. Yes, this is such a common thing we keep hearing. Hey, you just slap in couple of key AI tools on the resume. Okay. So, you know, we have seen a lot of improvements in AI models. Let's say at least in the last 6 months, right? every every week, every month, exponentially improving in terms of accuracy and everything else. Why does this all matter if models are keeping proving? The capability moves, but the shape doesn't. You're still sending tokens in, hitting a context limit, still handing tools, still grounding the results. If anything, routing matters more, not less. More good models means more decisions about which one handles what. And someone has to own the decision. Usually, that's the PM. So, one last question before we wrap it up is what do you think is the one technical concept every PM or at least majority of them get wrong with AI?

39:26 >> It has to be that the model knows things. People talk about it like a database they're querying. Let's be clear, it doesn't look anything up. It predicts the next token. That's the whole reason hallucination isn't a bug you prompt your way out of. When you need it, right, you ground it. retrieval or a tool, not a sterner prompt. That reframe separates the people who get it and those who don't. >> I like it. I like the way you actually said it's not a database that is has all the knowledge. All it's doing is it's actually predicting the next next best token. I I love how you simplified it and actually explained it. Awesome.

40:02 >> Hopefully you guys enjoyed this deep dive. This is literally like 5% of what we teach both in my newsletter at news.g.com pg.com and more importantly in the cohort that Prasad and I run. We are currently in cohort 3. We had 30 students in cohort 1, 50 in cohort 2, 70 in cohort 3. We are going to be starting cohort 4 soon in August. If you want to get this type of coaching and instruction, but even deeper, more foundations of AI, more mock interviews about how to succeed in these, join our cohort. We're not just going to show you AIPM. We're going to show you how to create a PM resume, how to create a PM LinkedIn, how to create a PM GitHub, how to create a PM portfolio. We're going to give you reviews of all of those products. If you put together everything we're doing, $500 resume review, $500 LinkedIn review, $500 portfolio review, $500 GitHub review, plus interview coaching plus AIPM coaching, plus PM fundamentals in the AI era. It is worth way, way more than the cost it will be for you. So, go ahead and apply at landpob.com so you can work with Prasad and I, including several one-on- ones with Prasad that people really love. And with that, we'll see you in the next video.

41:12 >> Yeah, >> I hope you enjoyed that episode. If you could take a moment to double check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube, all these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career.

41:37 Finally, do check out my bundle at bundle.ac.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobin, Linear, Reforge, Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.