Google Product Manager Metrics Interview: GPT-5 Launch

0:00 Product success metric interviews are now a part of more than 50% of product management interview processes. So if you want to land a product management job this year, you have to master this product interview question. In today's video, we're going to walk through an A+ example from a former Google product manager so that you can see the structure, the steps, and ultimately how to deliver a great product success metrics interview. So without any further ado, let's get straight into it.

0:31 Mark, how would you measure the success of GPT5? >> Okay, GPT5. super interesting question. actually a couple quick clarifying questions. am I responsible for all of GPT5? I mean, that's a pretty big product. >> Let's assume that, you know, the OpenAI product team, it's actually surprisingly lean. Like 90% of the company is researchers and engineers. So they do have, you know, CPO Kevin while you're reporting up to him. He has a pretty flat or of like 20 PMs and yeah, you're responsible for the GPT5 roll out because they're releasing a model almost every 3 months. They have a PM on each one.

1:07 >> Okay. All right. Cool. So, let's go there. We'll go pretty pretty broad-based. So, the way I'm going to approach this problem, classic analytics problem. The way I'm going to talk about this problem is one, talk about OpenAI, what their mission is, and how GPT5 might roll up into this mission. the next thing I'm going to do is I'm going to create some product goals. You know, what are we going to try and do with CPD5 and how is it related to the mission? Give us some context. then what I'm going to do is take a look at the product. I'm going to really break it down in terms of you know who the stakeholders are, what are the actions they take, what are the key metrics that they take. we'll spend a little bit of time here and then finally what I'll what I'll end up with is I'll end up with a north star which is what's really really important to target to and then we'll also talk about some things like some countermetrics and downstream metrics as well and we can also do some trade-off questions at the end if we want to. Okay, sound good so far?

2:02 >> Love it. >> Okay, awesome. Okay, so let's start with let's start with actually the mission of of OpenAI and what's important. So the mission of open AI is to ensure that artificial intelligence benefits all of humanity. This is a really really really big mission. And so when we think about chat GPT right this is really the this is really the face of the company. This is how people interact with the company. this is their exposure to the company. And they've had a number of models in the past. I think they started with one, two, I think when they got to three is when it got interesting and then three and 3.5 and four and 4.5 and five is their latest hallmark. So obviously this is a big phase of the company and it's and it's super super important. also I want to say the use cases that is going on is well we'll get into use cases but we'll just say it's used by consumers, professional users, companies, governments probably now at this point, right? So a lot of people are using chat TPD5. I think it's actually the most downloaded app on the app store in general.

3:07 >> Yep. So, let's as we kind of drill into the importance we can see why this is important to open AAI, let's create a little bit of a mission for chatbt5 and I think it's going to be an extension of the mission of products before it. But I think it's to make AI accessible and useful and aligned to everyday problem solving. Of course, it means very different things to different people, but it helps with creativity and productivity. So, again, we want to make AI accessible and useful. Sounds a little like Google in a way, but but anyways, let's go on. Okay, so now we've got this we've got this broad goal.

3:45 I want to break that down a little bit. because when I think about OpenAI, I think there's going to be four kind of sub goals underneath this. so on the sub goals one is that as a brand we're trying to acquire more and more users to try GPT to use GPT. So I think goal number one is or sub goal number one is acquisition. sub goal number two is for all these new users new users using chat tpt and all of our existing users who are on three four and five we want them to upgrade or three and four to upgrade to five. We want them to do we want them to use the product. We want people to use this on a regular habitual basis and we want to demonstrate a really strong engagement and then there's a couple other layers here that I think are important to J GPT specifically or open AI specifically. So one is around quality, right? So this is about AI, right? This is about making AI useful and accessible. And so it's got to work. The first versions of it worked pretty well, but I think they inspired people, but you know, people had problems with hallucinations and things like that. and so we're going to continue to invest in in quality and our road map to to get to to get to AGI. And the last piece we want to take a look at is probably some layer of monetization. What you should notice here is how quickly Mark came to a framework that he could use to guide the rest of the conversation. This is really important in case interviews.

5:03 >> It's a very different model than a Google. Google's model is spend more time on site, click on blue links, make money. chat TPT is more of a subscription model. So, it's more about jobs to be done and it's really about helping you do your task and getting people to upgrade to their models and pay them subscription revenue. And I think I believe actually subscription revenue is the number one driver of revenue at OpenAI right now. So I think this is a a really important layer.

5:29 >> One question for you. How are you going to think about the consumer versus the enterprise business? >> good question. in fact let me I'll take a step back a little bit. So when I think about users of Chat GBT actually I think there's a number of users and there's a number of surfaces. So the users I I'd put in consumers. I think it'd put in companies actually I think it'd also put governments right and and even chat tpt openai is kind of a platform itself. So I think there's both consumer users and corporate users. and I think corporate users are are basically like larger consumer users which are and I see lots of companies doing doing this all the time which is they are subscribing to chat GPT as a platform so that their employees can be more productive right.

6:17 this is also an interesting use case for how GPT is used in surfaces right so obviously there's a web- based app which everyone is familiar with and there's a mobile app which everyone's familiar with there's also have a desk desktop app which less people are familiar with and there's also APIs right so they have an API layer and this is where companies are using it as well too to use CH GPT as a service to make intelligent offerings and we see things in everything from customer support to chat bots and things like that. So it's powering it's powering all sorts of use cases just around the circumference. So if I think about chat GBT, I think about all of that usage. In fact, this leads to what I'll call a primary goal, which is around engagement, right? So we talked about four goals here. Acquisition, engagement, monetization, quality. I would actually put engagement kind of the top most important thing. So let's actually break this down a little bit further and we'll talk about actions and metrics at this point.

7:18 So, in the actions game, what I want to take a look at is what are the primary ways that people interact with this and and then and then we'll break it down into a number of metrics to to really kind of geek out this product from a metrics perspective. So, from an actions perspective, and this applies to, I think, consumers and businesses alike, right? They're going to visit the homepage. they're going to visit the app. They're going to download it. Step one, start two, they're going to they're going to chat with it, right? They're going to start engaging with it.

7:44 step three, they're going to, you know, have this chat experiment experience. They're going to provide feedback, right? And from that, they might, you know, upgrade to the paid plan. They might say, "Oh, this is really great. I might start using the API." and then, based on those outputs, they might also maybe share their output with friends or colleagues or share through email or things like that. and then the last two actions I'm going to look at are really going back to our goal but I want to look on more broadly are on key metrics around quality and monetization and we'll talk about those in a little bit more detail. Okay. So let's talk about monetization metrics.

8:20 So again since monetization is a really really really big part of this product. So I think the things we're going to look at are standard metrics like Arpoo and I think this product has a $20 subscription rate. I think it also has advanced rates for users as well too. And then corporate users are in a completely different bucket because they start using the API and the API is kind of a per token type model. So we want to have a number of models where we're doing you know per user and also per token to basically see which is really kind of almost a per query model to see how we are monetizing. And then we want to take a look at conversion rates which are you know converting free to paid and then paid to corporate and then paid to API and corporate to API as well too. So there's a number of kind of mini conversion funnels getting people to use more and more. So monetization metrics really important part of the equation.

9:12 also as part of that we might also have people who are churning out people are like paying $20 a month and going I don't want to do it or maybe they signed up for the API and then they're not really utilizing it. so we want to take a look at those users part of our overall monetization ecosystem. Okay. So we talked about a lot of metrics. Big question now is what's our north star? What's the most important thing? So when I think about the stakeholders here, you know, I think about consumers. I think about companies. I think governments are also involved and interested in this. I think open AI certainly interested in this.

9:44 What's everyone interested in? So I think consumers are interested in I want amazing AI responses which are you know fast and inexpensive and useful. If I think about companies I think they want the same thing. I want AI responses which are fast and and useful and you know and costefficient right. If I look at if I look at governments and stakeholders like that I want to make sure that responses are safe and that safe things are happening right now. I think safety is a a layer that that probably borders the whole thing. And if I'm open AI, I also want to, you know, pave this world towards a safe, responsible arc towards AGI, right? That people are using it in a in a way that's that's useful and it benefits all of humanity and it benefits all of humanity is also interesting to governments, which I think they they have a vested interest in that. Not sure about consumers or or companies, but I think certainly open AI and governments do. So based on all of those things, I think the one northstar that would kind of bring all these together is we talked about this concept of jobs completed. I think our best northstar here would be the number of jobs completed. And let me define that just arbitrarily as when we have a job, someone has an intent to use AI and say for today I have a specific intent.

11:02 If I'm able to do that job, it's done. It might take me one query or two queries or three queries, right? but I really want to get that job done. so it's really about accomplishing jobs. And if I can accomplish that job more efficiently, better for the consumer, probably less expensive for OpenAI because I don't want to spend them all day quering. It's just going to be very expensive, right? So, I want to be able to help people get jobs done efficiently. And I know that if I can help people get jobs done efficiently, they're going to come back and do it again and again and again. And and I I think we want to use guard rails like things like safety and quality as a ways to make sure that we're doing the right jobs for the right people. So our northstar number of jobs completed. and the last thing I want to talk about are a couple of countermetrics and maybe some downstream metrics.

11:54 >> Let's take a quick break to notice how Mark answered the question, but he was unfazed and quickly kept moving on. You need to keep a forward pace in your for this north star. I think it's interesting in that >> it's definitely getting towards what we want out of users. I'm just worried how are we going to operationalize this? How are we going to measure this reliably and effectively in a way that you know when the product team goes and talks to a researcher about some specific model, they're not going to get overly into the minutia about how this was defined.

12:30 that's a good question. So I think one there's a lot of work from data science here. I I think it's important for Northstar to be fairly understandable in terms of number of jobs. Now how we actually define jobs we can probably break that down because you can also have a job which has specific intents. Let's say like I want to build a website, right? That's a big job, right? Which is going to have lots of little lots of little subtasks. So I think it's about defining right what's a level of subtask per session. And I think I'd use work with data science to kind of figure out what's the right right granularity of a job. I guess you could say the right granularity of a of a task to be done. but I want to break it down so that it's it's like the it's the right task size. It's not so big like I want to build a website which has got thousands of things, but it's also not so small so that every query counts against it, right? Because really what I want to develop is an efficient artificial intelligence system that when I ask it a question, it can give me a reasonably good answer that I think like wow this is a great answer and it may invite me to be like I've got everything I need or it may invite me to explore more based upon this information. So I think in terms of you know what's the right fidelity I think this is something that we really need to work on with with the data scientist team to figure out what that right fidelity is but I think the highle job of getting the the number of jobs done I I think it's the right I think it's the right high level metric to focus on I think the company can get around that >> okay so marching forward a little bit here on countermetrics so you know if our if our you know if our key metric is jobs to be done. I'd say actually a key countermetric is job's not done. Right?

14:19 Imagine people coming and going, " I couldn't get it to do the thing I wanted to do it. I give up." Right? This is not on the path to AGI. Clear countermetric for us. also, I think some other good countermetrics overall for us are things like latency or API error rates. Right? We've all had scenarios where we're using this and typing questions along and it's just like, I'm full. I can't answer. I'm busy. Right? These things are all countermetrics on our on our arc towards ATI. And then also I think thumbs down rates, hallucinization rate hallucinization yeah hallucination rates are I I think those are also great metrics to use counter metrics to understand the overall quality of the product. So we've got good guard rails to make sure we're getting the jobs done. We're doing it fast. We're doing it high quality.

15:05 we're doing it in all these ways. And then the last thing I'll take a look at here is downstream metrics which is you know what happens right if people are using this product downstream what's the what's the consequence of this so I think big consequences of using this product are obviously the the funnels that we talked a little bit about right so you know going from a user to a subscriber to a corporate user to using API for downstream companies right those are all great downstream metrics to look at also I think extension ions into the ecosystem, right? So third-party apps and API calls, things like that and then really kind of enterprise adoption across the industries in general, right? Using this, I think those are all, you know, going from AI today to much broader applications with third party developers and companies. I I think there's a lot of ways we can we can measure this on a on a downstream basis. Okay. So so what have we done today? just that quick summary of where we've been. we said hey how would we measure chat GPT5 and what we did was we came up with a mission statement for what we want chat GPT to do. We created a set of goals across acquisition engagement quality and monetization. We walked through, you know, kind of the user journey from querying to responding to answering to sharing and all these kinds of things and a bunch of metrics on that. And we ended up with a a really solid northstar goal on jobs done which I think would be a great way to measure chat TPD5 as well as some counter downstream and countermetrics. Okay. let's wrap it up there. what kind of trade-off questions do we have?

16:42 >> Notice again how Mark just answered the question and moved on. he didn't over fixate a little bit more on the north star. Let's say >> one of the things that's happened in OpenAI actually is that we've really gotten a preference for more I would say harder metrics and we specifically want to look at kind of like CFO level metrics. How would you define kind of a CFO level northstar metric for GPT5's launch?

17:13 So, a CFO, if so, if I'm a CFO metric, putting on my CFO hat, which I don't play one on TV. putting on my CFO hat is I want to build a financially sustainable company, right? So, on a CFO hat, I'm looking at what are my costs of of all this infrastructure, right? I've got servers and and usage and energy utilization and things like that. So, I've got, let's say, I've got a billion dollars in cost, right? And then I've got the revenue side of the equation, which is what are all the revenues coming in? That's from API calls. It's from subscriptions, things like that. And obviously, I want to make sure my revenue is greater than my cost.

17:53 Otherwise, as a CFO, I am making a a a long-term investment in a short-term company. So, I want to make sure from a CFO perspective that my, you know, that my revenues, my gross margins are financially sustainable. I would probably break that down on a per query basis, which is across the services. You take that billion dollars. If I've got a billion queries, that means it's a dollar per query. And it's definitely not that, right? and then I look at from a cost perspective. And I look at my revenue perspective. I'm going to look at my my users, what their average utilization rate is, and how much they're costing is consuming, and how much we're doing. So I want to basically make sure that our our marginal you know query costs versus all our sunk cost align so that I have a sustainable business moving forward in the future right so that I want to know that I can continue scaling my business as I add that next 10 million users and next 100 million users that it's actually going to increase the monetization of my company and not and not and not and not make it go the other way.

18:58 >> So this is really interesting. Mark went back to stakeholders. Originally, that was earlier on in his framework, but responding to the flow of the interview, he decided to pull that back in before his chose his northstar metric. It's a really good idea to reorganize your framework as needed. Love your response. I love how you also kept a forward momentum going, which I think a lot of people struggle with doing, you know, after somebody asks a question, kind of like getting zoned in on that question, even letting that question take five or six minutes, but just continuing to move forward. And having that balance of enough pausing, but then also sticking to your framework. And it was interesting how stakeholders I think it was originally meant to come before actions and metrics, but you kind of adapted to the questions I answered and you brought it in before Northstar metrics. So, I thought that was also really clever.

19:47 All right. And here's my analysis of your interview that you did, Mark. I think that what you could have done even better is a couple of different things. The first is have a little bit more on the Aie Eval style metrics. I think you really stuck to product metrics, but didn't necessarily get into AIE valves. The second thing you could have done better is have more responsiveness to some of my interview questions. For instance, when I talked about the API or when I talked about having a less fuzzy metric, I ultimately needed to come back to those things once more afterwards. It would be better if he could be even more responsive to those things and pre- guess, okay, so here's how I'm going to think about it financially. Here's a less fuzzy version of the Northstar if we really have to go for it. If you answer questions in that way where you have a backup, then people see how flexible you are in your thinking. And then a final thing that could have done even better is potentially a screen share of some sort so that we can really see as you follow along in your thinking. Mark is one of the most humble people about this. He's helped literally hundreds or thousands of people get interviews. If you haven't seen him on YouTube already, welcome to his YouTube back catalog. There's plenty more amazing things he has there as well if you want free resources.