Transcript
0:06 All right, we're here. And I warned them backstage what we were going to start this talk about because I know there's a lot of questions about OpenAI. There's a lot of questions about the future of product development. But what I want to talk about is >> what do you want to talk about, Claire? >> The toggle. M I want to talk about the fact that every AI product has two faces. It has left toggle and right toggle. And there's been a lot of conversation today about iterative product development and shipping things that are imperfect and getting things into the wild. And so Tara, I just want you to talk to me about shipping imperfect things instead of a toggle and how do you make these decisions about putting things in the wild that you know may not be the perfect thing right now but are the right thing ultimately to build the right product.
0:55 >> For sure. I will say this to start like you know I'm the daughter of Asian parents. I'm not used to doing imperfect things. You know you have to output something perfect every time. I also worked at Stripe for like seven some odd years and everything at Stripe is like deeply polished and deeply considered and did you think through it 100 times and did you name this enum the exact correct thing and I think what has been a big change about shipping things in this era is actually urgency really matters in terms of being get getting the functionality out there and also there's a lot of like theorizing that one can do before you ship something but nothing compares to like the actual empirical evidence of seeing users try it, use it, and then iterating from there. So, I've really had to shift my mindset from how do you get it perfect a priori versus how do you ship something and iterate on it as quickly and effectively as possible. So, with the toggle like is it the perfect solution? No, it is an imperfect solution. It is certainly not like the ideal thing to ship. But what was really important was getting this agentic harness or like the ability to use an agent to get things done in the hands of like the over billion users that use chat GBT. And so we needed to ship something that allowed us to do that without disrupting developers workflows >> as such an imperfect solution. And you know what I do believe that in a year maybe we won't be having the same conversation about the struggle and non we've had this we've had this chat before where we've talked about right now you have to build and throw away a lot of stuff whether it's code or product and I'm curious how you think about deciding what to deprecate and what to keep around and keep alive. You know, the toggle is a good example of this where it I think there's an obvious sort of next step which is like no toggle, right? Like there's a there's a story that evolves from step A to step B. And if you can take users on the journey with you, then they're a lot more accepting of like the change and the turn and the the the product evolution. especially if they appreciate that, you know, they can they can have an understanding and some transparency over like what's, you know, what's actually going on behind the scenes. So, I think as long as you have a coherent story that users can follow along with and, you know, with the amount of attention that they're paying, you know, like there's a there's actually a lot that you can do and not, you know, and not not really burn any kind of bridges with them.
3:23 >> Yeah. And I'm I'm curious, you know, that the toggle is maybe imperfect and we may build a lot of stuff that we have to throw away, but you you do exercise good judgment and you do exercise caution and reason inside the products that you've worked on or are about to work on. What constraints do you hold really strongly when building product that's moving so fast? What what what keeps that quality bar high for you? I think there's a few different elements of it. One is certainly are we doing something additive for users? Like are we unlocking additional value? Is there something genuinely useful in this product? I think the second is that we have this internal bar of taking a product shipping it internally trying it out having everyone use it and if there isn't enough uptake internally and that's not a specific you know number or exact bar but there is is it retaining well do people find something delightful about it or something surprisingly great about it is it unlocking a new use case of the models like is it you know doing something novel I think that is that like sense for this is adding to their experience. This is something that is unlocking a new set of functionality.
4:41 And then probably most importantly in this era is this aiming for 2 to 3 months of like where the models will be rather than being overly anchored in the present or being overly like futuristic and unusable. Is this like at that perfect zone of being I don't know sufficiently AGI pill that I think is like the bar here. So is it valuable? Is it do users care about it? Is it surprisingly great for users? And is it like in line with model capabilities?
5:08 And is it getting out of the way of the model of doing a great job? >> Nan, what about you? What constraints do you think help you build great product? >> The the biggest constraint is honestly like people's understanding and ability to sort of absorb what you're what you're giving them, right? We talk a lot about like capability overhang but like but mods can do so much and just like you know people aren't necessarily taking advantage of all those abilities. I think that's where that's where the gap is right. So it doesn't matter what you put into it, if the if the ability for people to absorb those changes and absorb that capability isn't there, then that's the that's the real kind of bounding like kind of process that you're dealing with, >> you know, and we're talking right now, you two are working on codecs and work and there's a lot of business users including myself that have built businesses around relying on this tool and I think you know I spent a minute and a half in enterprise and a lot of people say that B2B customers cannot not absorb change. They I I mean like customer success people tell me they can't absorb change. Sales people tell me sales people can't absorb change.
6:11 Like we cannot absorb the rate of change. Do you feel like that still stands true in the AI era or do you feel like that statement is is broken right now? Some of it is certainly true in that I think the current pace probably to most enterprises feels beyond breakneck like something beyond breakneck like a la palooa of features and advancements like it certainly feels maybe to some overwhelming but at the same time like if you're not shipping the latest and the frontier to these enterprises you'll end up getting leaprogged. So a great example of this is that most of our enterprises were using chat. So they were merely you know asking questions and getting answers and whilst that was happening like the agent revolution was going on and we needed to get agents to these enterprises as fast as possible otherwise they might consider alternative products. And so while they have their own maybe in the day-to-day pace of absorbing changes, we needed to ship this giant revolutionary change that broke all of their processes and broke all of their, you know, conceptions about how enterprises should receive new updates so that they could actually unlock more value. And so it's one of those things, I mean, it's like a truism of building product, which is like, you know, do what your users need, not what your users say they need. And I think that applies here almost more than it ever has before. I want to talk about agents since you brought them up because I do think we're in the age of agents and I have certainly been in the age of agents for a little bit and I want to have a debate with you.
7:47 >> Oh gosh. >> Which is or I want your opinion. Maybe we won't have a debate. Maybe we'll agree single identity agents versus multi-identity agents. my beloved codecs docked on my computer that I talk to all day versus my 40 Grock bots that are loadbearing across different parts of my business. I do think as product builders people who are building agents, we're seeing this divergence of do we centralize on a central kind of generic agent identity that can do anything, let the models get, you know, get out of the way of the models or do we like artisally craft sole MDs and have little like micro agents for the job? But I'm just curious, you know, coming in fresh eyes, where do you land if anywhere on that spectrum from a design principles perspective?
8:34 >> I I I think when designing this like the the most successful efforts have really tried to map on to human nature, right? And like we have millions of years of evolution that like help us understand how to hold different things in our heads. And you know 40 agents is quite a lot, right? And like if you asked anybody like would you manage 40 direct employees like some you know Jensen Huang might do it, right? But like like a lot of people would kind of look at that and be like I don't know I don't know if I could actually keep up with that many threads ongoing. so I I I think there there's there's something in there where we need to group these things into sort of bundles of activity right like if you look at like the u you know you ask people hey like how do you manage your your agents?
9:14 Usually what you'll hear is like well first I have this chief of staff agent that manages all the other ones for me. It's like okay well like you you've just cheated right you're not managing seven you're managing one that's like you know kind of running your team. So, so I I think that that that kind of dynamic kind of immediately kind of you know comes comes to the front right as soon as you let people do this kind of stuff.
9:32 >> What about you? What do you think? You know, my mind with this like certainly there's a philosophical element to it of like you know do you have this one unified entity? Do you have these like smaller entities? But actually there's a lot of really tactical questions that go into it like how do you think about data access and permissions? Like if you have one entity is it like switching between using like a service account or your account? like how do you manifest that to the user in an easy way to understand if you have a single agent like does it have segmented memory like every times it joins a private slack channel is it almost a different agent is it like severed in the slack channel in like the severance sense and so actually to me answering this question goes back to really specifically the use case and probably thinking through all of those edge cases of like how should the memory behave how should it like how should it talk to each person should it be using their credentials or should it be using a universal set of credentials? It's actually like there there are philosophical things one could have and opine on in this topic but to me there are just so many practical product considerations like the use case matters so much in deciding this. What I love about these two answers is I think for product folks everybody in the audience plus designers there are these two sides of the craft that we do. There's this very human side where you're trying to like in there's inception and severance I guess. there's like the inception of like how do I get in the mind of a user and what are their mental models and how do they interface with the world and what is the most natural way for our products to touch their problems and and solve them and then there is like systems and architect thinking which is in order to make this like manifest this into the world in a realistic way I need to think through all these systems all these platform components all these edge cases to make sure that comprehensive we've can even deliver this to the user in a sensible way. Do you think those are the two kind of like skill sets you need to design and build great agents in this moment? Are there other hard skills you feel like you're relying on now more than ever as you design or think about agentic experiences >> you're describing you sort of like systems thinking plus you know user empathy. These are very classic kind of product management types of ideas right that you know we we apply to you know a different you know universe of technology in the before times. but now you know we have to kind of start from scratch and and you know from those principles kind of rethink what does you know how do these products want to be shaped right so I I I think those are very important but ultimately they're just the old principles in disguise. I would maybe add a thing to that which is again I mentioned like I used to work at stripe which is very academic company and my parents are very academic and I love thinking about those things in abstract but I'd say the third thing that you just really need in addition to user empathy and systems thinking is like a relentlessness to like try it out and keep iterating and you're going to take a lot of pain while you do that and there's just an element of like relentlessly trying and trying and trying and trying and trying to see what works y >> that and the ability to learn maybe from those feedback loops that certainly has always been necessary, but just the clock speed has gone way.
12:36 >> Yeah, you're throwing a lot of spaghetti against a lot of walls all the time. Actually, your chief of staff agent is telling all the other agents to fling spaghetti at the walls. >> You know, talking about systems thinking and platform, I am really curious because all of us are looking to tools, certainly ones that you build for productivity and efficiency. And I was telling somebody backstage, I just love computer use now. Like don't even give me an MCP. I want to point computer use at your UI. And then, you know, I don't care the the token. Congratulations on me selling me tokens. but you know, I think about what is the intersection between the tools we had and the tools we have now, what you build, what you platform for. I'm curious how you just think about the broader ecosystem as you're trying to enable productivity for people at work. So I think I think we think a lot about platforms and chat GBT as a platform and what is the functionality that like natively goes into the platform. What is functionality that like we might allow folks to build in whether that's like first party or third party as as plugins or like additional interfaces on top of it and that's kind of how that relates to the ecosystem. So in that platform what are the hooks that we need to expose to a third-party like meetings app so that it can be most effective for you use in codecs using codecs using computer use like using all the these tools but I don't think it's so binary as like what should be in our platform or in our productivity system versus in one of these apps. I think we should offer an array of different tools like for perhaps like a plugin to be able to like get all your meetings data and understand you know what you need to do next. But if that doesn't work and those systems don't interface well then there's always computer use as like a next layer or as like a fallback to be like well if the user wants to get this task done and you know the pre-built tools or like ecosystem tools aren't available for it then there's like always something else that they can fall back on. And so I think it's more about structuring these capabilities in the platform in terms of layers, exposing them as like a set of hooks that third-party developers and first-party developers can like hook into and build cohesive experiences, having the right system for those experiences to be composable with one another. And then ultimately all of this is just serving the user's goals of as they try to get their task done, if like the first layer doesn't work, there's like a second layer for them to be able to accomplish their task as you like traverse all those capabilities. What do you think about kind of like integrated experiences like do it all in one platform, let's live in our house versus kind of expanding into this ecosystem and how do you build I mean the thing that I was thinking as you were talking is like you have a non-deterministic model on top of a like natural language interface functionally with a computer that can click and do anything with third party tools and integrations you don't control and you're still trying to build like a comprehensive of highquality good branded beloved experience like how do you even climb that mountain as a product leader? Do you have any what's what genius are you going to bring in to help us here?
15:44 >> look I I I don't I don't know if this is a the a new or unique idea but there is a a huge difference between getting like all of the job done versus getting everything except for the last mile. And the the last mile honestly feels sometimes worse than just like it's a non-starter. If it's a non-starter, you can just like look, I'm just going to do it myself or I'll try another route or whatever it is. But if it gets like 99% there and then it just like barfs, then you you're like, well, this this is this is actually a worse sort of like unfulfilled promise. And I think what feels great about computer use a lot of times is that it will always work, right? It might be a little bit like slow or like, you know, it might cost a lot of tokens like you were talking about, but like it will going to get the job done all the way. And you know, if as the ecosystem kind of, you know, we wait for it to catch up, right, and to provide the right hooks and the right MCPs and all that kind of stuff, it will 100% do the job for you. And I I think that ultimately that's that's what makes it feel super great. I think the right experience amidst all those complexities and changing things, the ecosystem, the interface, the model itself, the like tools like computer use, the unpredictability of all of it is that like the user should almost feel you know like the Charles Emmes quote that your that your user should almost feel like a guest in your home like you've anticipated all the things that they want to do and you've like provided for them gently. Like so should you know our user in our app feel like we've anticipated their needs. And in some ways it's like the model increasingly doing that.
17:08 >> Well, I have to tell you, every time I go into your home, it tells me it's time to update. >> Yeah. Yeah. Yeah. Yeah. Get get the latest get the latest build. That is the welcome, Matt, that I get. It's like I walk in, it's like, you want a Renault? I got a Renault for you. Well, I I'm curious. You know, you're you're designing this product while it's flying off the shelf. You know, a lot of the things, Nan, as you said, are are old things. user empathy, systems thinking, but there are some new things, right? There are some new skills and t I'm curious about working with research because this is something I don't think a lot of folks in the audience have done in their careers because I think it's very new for product people or product leaders. What have you learned from going into this new role where you really have to collaborate with research? Yeah, I will say for me working with research was entirely new coming to OpenAI. So I'm certainly not the like expert on this topic. I will give you like the you know novices's view like Connecticut Yankee and King Arthur's Court sort of view of this.
18:11 my take coming in working with research was oh working with research is different than working with engineering. Like that might be obvious to everyone here but I was like oh this you know it's it is quite different. in as much as like I can add add something to working with research, it is for me to come with very very specific use cases, a really clear idea of what like users want to do and their goals to have done like sample readings. So understand like literally what was the like session and what did it look like and why did the model not do what it needed to do.
18:43 If I can write evals that is like I'm there like learning how to write good evals and writing evals as much as possible is like the most important thing and in as much as I can say with this prompt and these skills I was able to achieve the outcome with the model that is like the ideal thing I can take to post- training and be like look now how can we postrain this capability into the model from the beginning but I think it starts with first understanding that this is like a wholly different way of working that I really didn't know how to do before and then getting to the point where you're good enough at like bringing these the specific user data and then if they trust you sufficiently you can then write the eval to like get that iteration loop going. How far in the future do you feel like you live as a product leader? Like are you you know because I I do think there's this time horizon of like I could fix what's right in front of my face and there's stuff to fix. There's I know what's coming next and I can anticipate where the product's going and then there's like the models next year are going to be out here.
19:45 Where do you feel like you have to live as a leader building on top of that foundation that's moving? You know, as soon as you said that like do you live in the future? I was like I live in the now. Like I don't live in the future. I live in the now. no, I do. I think I have to live a little bit in the future. To me it's like the ideal is two to three months. can't be too bigrained and be like this is the future in years because I think that inevitably our predictions of that are almost always wrong. Like I have a personal dock of in incorrectly predicted things that I thought were going to happen and didn't happen. just to remind myself of like my inability to predict the future years out and then at the same time if you build exactly for today you're going to get left behind. So it's like I constantly repeat this but it's like two to three months out. I I do want to just let's pull the audience again. It's the end of the day. Let's get interactive.
20:34 Who right now is doing 2027 annual planning. And who just heard her say you can't go out more than 60 or 90 days? I think this is the fundamental tension. We're all It's like so hard to even guess. But but it's not the same for every business. Y >> like again just to use an example I'm familiar with like Stripe's business is very different than OpenAI's business. The dynamics of the payments market look largely similar to how they looked before. certainly accelerated in other ways, but you like know the forces and you can kind of model like, oh, here's the bull case, barecase, base case in a way that for Open AI's business, you just like cannot do. Well, maybe Sarah Frier can and she's the expert and I'm not as good as her for sure. But like I it's hard for me to predict OpenAI's business the way I I can feel like oh I understand the levers and in stripes and so maybe it is appropriate to plan you know many many months out but I think it involves understanding like the pace of the market you're in and understanding given the pace and the dynamics of that market at what you know cadence should I be thinking and planning in the future.
21:38 So, so what you're hearing is Stripe Stripe PM still have to do their annual planning. >> Oh, for sure. Good luck to you all. Like, yeah, for sure. >> All right, let's talk a little bit. Let's close out with then and now. so I want, you know, we've all been in product for a minute and a half as fresh 26-year-olds. What What still matters from a classic PM craft? And what do you think is like newly table stakes? you know, I I I think the the emphasis like kind of changes, right? Like for a lot of reasons, but you know, we talked about, you know, just like absorption like rate and like capability overhang.
22:14 So I I I think something that probably matters way more now than it's done before is is onboarding. I think we've like talked about, you know, people people have like talked about on board is an important thing, but it's also something where it's a little bit specific and like a lot of engineers don't love working on it and things like that, but I I think really to to kind of get a lot of success like thinking about like what is the initial experience? How do you get people to really understand what your offering is like constantly is is probably something that matters way more now than it does before.
22:41 >> Tara, what do you think? >> Yeah, I think I think users understanding like privacy, their data, how that's used is like way more important than it always has been. like users understanding norms of your product. Like before any sort of unpredictable behavior you could get away with because there was some like settings menu or some popup that could appear. But now if the you know agent is doing things semi-autonomously like you actually really need to make sure that at any given point a user can like predict what is probably going to happen with a feature or a primitive that you're introducing. And in terms of like just product thinking or getting up and running as like a PM in some ways it's going to sound like a really boring answer but like all the usual stuff still applies except the importance of testing things empirically and moving faster has only increased. So it's like yeah do all the other stuff you used to do which is like get in the details have the high level and the low level at the same time like really understand the design really understand the product but also do it all faster and and test with users has just become way way more important. I have one more that naan is going to become a victim of when you are in the open slack which is the open AAI product and engineering team is highly DMable highly accessible highly visible. I mean, they're getting like feedback UYU IDs from me on the regular like I didn't like what Astra said. I had an attitude like and so I'm curious, you know, certainly I have tried to do this in my own life, but do you feel like direct user relationships right now are becoming table stakes for product leaders? Like this idea of the DMable PM I think is one that I see a lot out of OpenAI and I'm curious like staying close to the ground if you feel like that's becoming more and more important.
24:25 My background is in like developers and enterprise. So I've always been a DMable PM in the sense that people could always message me and ask for features. but I think what kind of feels new is that for like a large consumer product or a product that everybody uses to do the same level accessibility. so in some ways like this it doesn't feel normal to me like of course you should be in direct contact with your users. And as I mentioned earlier like your value to research is like knowing specifically what the users want and what their use cases are. And so the more you can be a DMable PM or engineer the better. like if you didn't like Astra's output. I was actually thinking like hm Claire should send me those outputs.
25:05 it would be really interesting to like know what's happening there. Like that that stuff is like the important thing in part that's like how you add value. But I'm curious Rene's take. >> Yeah. I I I think you know you worked in like dev tools, right? And you know engineers are extremely picky. They're extremely detail oriented. Often you're not going to get good feedback until the second or third follow-up question. And I think that's like what a lot of PMs are feeling like a change about, right?
25:31 Because if you're designing an agentic experience with like natural language interfaces, the problems that people encounter like Astro Game is attitude like they're pretty subtle, right? That that is that that bears some explaining like what exactly, you know, did it say? What was the context? Did you make it mad? Like like all sorts of, you know, kind of follow-up questions that you need to have. So I think the the having a a more kind of direct deeper relationship bears a lot of fruit there, right? that that maybe you wouldn't have been able to to take advantage of before.
25:58 >> You neither of you have gotten my DMs. So, I'm very excited for the next 90 days for you all. We'll put us in a group chat. It'll be really fun. Okay, last question. So, when Dan was up here with the anthropic team, he asked like, you know, what in a year about the product craft, about the role will be different. I want to I want to talk about the stuff. So, like what do you feel is going to be different about products like the what we touch? what the buttons on the site, the experiences, like where do you feel like this is all going? Because I feel like every week, every month, we come up with a new form factor, a new thing that matters. It's curious as product thinkers. What do you think is going to be big in 2027? I have an answer for you and it's voice.
26:41 >> Voice. >> I love voice. You try out GPT live. I love voice. Like we're yappers. >> Yappers. Hillary Fidley. The yappers API genius. Like voice is a like has changed the way I work. Like I just want to talk to computer. Like I do not like it's so much better. It feels so much more natural in some ways. Like I'm sure all of you have been tech support for your parents. Like the existence of voice has reduced the amount of like tech support I've had to do for my family to like an incredible degree. Like I was onboarding our own people team at OpenAI onto a new product and the existence of voice like changed the game like it is such an intuitive interface for so many people.
27:24 It's like the natural way they interact to the thing Nan said earlier about like you kind of have to think about human nature and the way we are used to like the way we've evolved over you know millennia to interact with others like speaking feels so natural like voice is the models are finally getting good at it. It is getting faster. Voice I'm like so bullish on voice. Voice. What about you? I I think I think probably like something that looks a lot like self-driving is probably going to be the the thing that really you know becomes much more important and prominent in the next year. You know, I think a big problem people have is like, you know, empty input box problems, right? And and it doesn't really matter if you're using a, you know, general tool like ChatGpt or something like more industry specific like you you have the same it's like, okay, I have the product. Now what? Well, now you got to learn how to use it, right? And then so having these products are smart now. They're like literally intelligent. So having them you know kind of use themselves and sort of to help you out and and make that a really really gentle on-ramp I think is probably gonna going to be a very normal thing that we're going to look back on be like why why didn't it always work that way?
28:31 >> Okay we need the notetakers to be our accountability buddy. So we got voice, we got self-driving. I'm going to say hardware that replaces carrying around our laptop open. so hardware. Yeah. Like I want I want like a little baby baby bot over here. A little codeex in a box where I don't have to do She's nodding where I don't have to do this. That's my prediction. So whoever took notes, we're going to see which of us were right on what 2027 hot product of the year. Everybody thank these lovely leaders from OpenAI for coming.
29:06 Claire, thank you.