Transcript
0:10 Without further ado, I'd like to officially welcome you to the session. I would like to introduce our speakers today. We have with us Amin Saberi, CEO who is a professor of management science and engineering at Stanford University. Amin was the co-founding CEO and Chairman of NovoEd, Inc and does a lot of other amazing things as well. Welcome, Amin, so lovely to have you. And today, he is going to be in conversation with Farnaz Ronaghi. Farnaz is Co-Founder and Chief of Product and Technology at NovoEd, an AI-powered talent development platform for the enterprise, and NovoEd, started from her PhD project at Stanford Social Algorithms Lab in collaboration with Professor Amin Saberi, where she worked on collaborative learning at scale.
0:56 Welcome, Farnaz, welcome both of you. So wonderful to have you here. And thanks so much for being with us today. We're looking forward to your conversation, and please take it away. Wonderful to be here. Thanks a lot, Robin, for the introduction. I'm really pleased to have the Stanford alumni, Farnaz Ronaghi, here. She has built this team from the time it was just me and her at the company, and now she's managing a large and global team.
1:32 And over the years, I have learned quite a bit from her, and I think this is going to be a fun conversation. So before, I think, where are you at right now, Farnaz? Tell us a little bit about your journey from the time you've just started your PhD-- just starting as a PhD student up to now that you're managing a large engineering team? Yeah, thanks, Amin, and thanks, Robin. Hi, everyone. I'm so happy to be here.
2:03 Well, I came to Stanford just in September of 2008, I think, and it was for a master's degree in MS&E, and my area of focus was information science and technology. And this is, I don't know, many years after that. I'm now based in Seattle. And from the days of starting NovoEd, from meeting of myself and a few master's students that used to work with us to get things going to a team of, I don't know, now less than 100 people-- just about 100 people working on this product.
2:43 It's been a wonderful, amazing journey. Yeah. Where do you want to begin talking about it? Yeah, so I think I would love to hear about your journey. One thing that I should add is that Farnaz and I started this company together, but my involvement ended several years ago. So Farnaz has been running the show with her team for the past several years.
3:14 So many of the answers will also will be new to me. We're going to alternate between Farnaz's experience and your specific questions about decision-making. So let me focus on go back to the topic of decision-making and ask Farnaz about some of the maybe more difficult questions she has had to make as a leader. Can you tell us a little bit about those?
3:44 Yes, of course. Well, I think difficult decisions, they are not the same for everyone. It is a reflection of leaders, both personality, character, and personal ethics, that drives what a difficult decision is. For me, people related questions and decisions seem to be the hardest ones to make, as I care deeply about people, I care about-- they will bring into my team. I feel a deep sense of responsibility towards them, and decisions that impact their work lives in a meaningful way end up being the most complex ones for me.
4:20 However, I think that this is Stanford, we have-- we probably have a lot of aspiring founders. We might have a lot of early-stage folks in our audience here. I would say the more relatable, maybe the one that they might be close to make-- closer to make decisions about pivoting the business. They are extremely difficult. I mean, you have been one-- I mean, you have been through one of them with me together. Most founders, their teams, they're in love with their product.
4:48 Not a great quality to be proud of, just a fact of life. They're also in love with the problem they are solving, and sometimes, as we discovered in our journey of founding NovoEd, while a problem matters, there may not be enough business for the right product market fit to create a sustainable company or meet the growth expectations that your board might have of you. So pivots, for me, create a sense-- like, in where I've seen them, they sometimes create a sense of identity crisis for the team.
5:19 The team has come in to solve one problem, and now when that problem is even changing slightly, that might be very difficult for the team to accept. And they also come up with a hard set of decisions, data points, and bets that essentially one is placing. So I find those to be one of the hardest ones that one has to make in the journey of founding a startup. How much do you bring your team into the decision?
5:45 So how much do you talk to your team about some of these existential doubts that you have about the company, whether you're going the right direction or not? And then when you want to pivot, how do you facilitate this with your team? You just go in and announce, or do you talk to them about the process of pivoting? Yeah, going in and announcing never works well, never works well. So I actually think that the decision-making processes are best when they are not committee-based, but they are collaborative.
6:19 So you need the right voices in the room at the end, but that doesn't mean that the decision is being made by the committee. Rather, it means that everyone's input, everyone's thought process, the background they have, the domain knowledge they might have, the instances, the experiences they've had in the market or with customers are being considered in the process of decision-making. So from my perspective-- not that I did it back when I was just 27 years old.
6:46 Today, what I like to do is to create a room that includes the right set of voices, the voices that have something relevant and insightful to add to the conversation. And they are diverse, but make it clear who is the owner of this process and who is the super vote in the room, because those two people could be very different people. For example, we might be trying to fix up product onboarding, related customer onboarding, related issues in our product.
7:19 A product manager might be the owner of the decision, but the super vote might be a head of support or a head of success, someone who is dealing with the problems as they manifest in the real world one by one. When it comes to pivot, at the end, the super vote is really the CEO. They are the decision-maker with the board on whether this pivot is right or not. However, whoever we have in a go-to market function, with that responsibility in early stage has insights to add.
7:49 Sometimes our early, big, impactful customers might also be one of the voices that we don't officially bring in the room, but we are keeping close, we are consulting with, we are taking their input and insights and bringing it into the room through that, as well as key leaders in the team. Key leaders of the technology organization. You don't want to lose all of your engineers in your pivot. There are a very sensitive topics to keep in mind and make sure that everybody is feeling that they are heard, they also know the rationale for decision-making, and then they can, at the end, say I agree or disagree, but the stand behind the decision and execute because what you cannot do in a startup especially-- in any company, in my mind, in any stage, is to silently disagree.
8:42 We disagree as loud as it is necessary, and then when we are behind the decision, we have to go together and execute. Yeah, that's a very good point. One of the things that actually we discussed also in the class is when there is a decision-- when there's a decision to be made that involves multiple stakeholders, you should all be on the same page who has a voice, who has a vote, and who has a veto. And it's not that you always want to have a vote or have a veto.
9:15 Many situations, you just want your voice to be heard, but you don't want to have the burden of the responsibility or you don't want to have to aggregate the input from all the different sources to make the decision. So it's good that you get that alignment on the roles in this group decision that's being made before the final decision is made and executed. And I think this is-- I've seen this in a startup, and I've seen it in other contexts, too.
9:45 The worst thing that can happen after decision-making is for everyone else-- for everyone involved to go back and do something between what was decided as a group and what they think is the right thing to do. You really want to bring in everybody so that regardless of even whether you think this is not the optimal decision, you're committed to it, you're committed to communicating it with your team, and you're committed to trying it out. And even if the decision fails, you can go back with the evidence and try a different approach, but having different parts of the company or different parts of the team moving in a slightly different direction could be a killer.
10:25 If not a killer, it can slow you down significantly. It's one of those things that you really want to build alignment from the beginning and, when the decision is made, to increase the speed of execution. So that was also a very good-- a very good point. So you talked about the people you bring into the room. How about the kind of data you want to look at before making a decision? So I'm going to say something that's a little bit controversial.
10:56 I find data to be a little bit of a proxy question. The first moment I bring up something new, people are like, oh, we should go collect data. But, well, we are not aligned on the problem yet. We haven't reframed the problem yet. You and I don't understand the problem the same way. So where I like to begin, the data sources in my mind, data sources make themselves very visible as soon as your questions are relevant, high quality, and insightful, and everybody is aligned behind them.
11:25 I find the first set of questions that people want to ask from data to be a little bit low on the insight index. So what I personally like to do-- and we teach this in MS&E, there are courses we have on creativity, we have courses on decision-making process. And there are books written in these things. And I have my own version of all the things I've learned together in a process, but reframing the problem. Let's sit down, depending on what is the area that we are focusing on sometimes, I actually find it always to be helpful to think about the customer journey, the user journey.
12:06 Like, you're looking for an outcome. OK, great. Well, where does this person, this buyer, whoever they are and what universe they are, where do they begin? How do we come into their mind? And then what happens? Then what happens? Then what happens? Until they have their hand on this one feature, one problem area, one product area, one service area that we want to offer. And then when you map that out, then you can actually attack-- put different lenses on the outcome you're trying to drive and what the problem is, turn that into opportunity statements, and then just sit in that room that has all the voices present, different people, outline the different risks in different parts of this customer journey and risk statements.
12:51 I really like them because you can pivot, you can just reframe them into opportunity statements, and then you can reframe an opportunity statement to, OK, which ones of these are real do we really believe in? And you have a few of them. And from there, all right, what are my discovery questions? What is it that I now know? Like, I know because it's sitting in my Salesforce, it's sitting in my product usage tracking system. It's sitting in, I don't know, my HubSpot whatever it is, or it's sitting in some report that I'm looking at, and what is it that I don't know?
13:22 And I need to come up with my own new data source to assess it. And my own new data source, what I mean by that is usually some form of customer interviews, some form of prototyping and running experimentation that will provide answers. So to me, putting the effort first in finding the right questions and being aligned around them, that those are the right discovery questions, will just make the resources visible to you. Yeah. And for me, they have ranged anywhere from paying for very expensive, executive interviews with people in the market that are not yet my customer, to putting in $100 gift cards on LinkedIn and talking to whoever I can talk to, to things that if-- we have a lot of founders here, what I recommend you-- I look at my book shelf immediately when I want to refer to books.
14:18 If you have a lot of founders here, I recommend books like The Mom Test, how to ask questions and how to submerge yourself in where your customers are. Those are the best data sources in my mind. Yeah, I absolutely agree. A lot of times, collecting data is a very expensive and time-consuming. And I have seen in many organizations, this is used as a way to avoid decisions or to delay decisions.
14:54 So let's say if you're a leader and you have to make 10 decisions every day, if you make the decisions with the information that's available to you to the extent that you can, maybe some of them are wrong and then tomorrow you can fix them, or once you get the data, you can fix them. But if you are of the extreme type that for each one of them, you propose an experiment and you ask for more data, then you don't make those 10 decisions, tomorrow you have another 10 questions that you know presented to you, and then the day after and the day after they start accumulating, and you start becoming a bottleneck in your organization.
15:34 So now everybody is focusing on running the right experiment or bringing the right data to convince you, and this is because you're not doing your job well. So of course you want data. Of course you want to make sound decisions. But asking for more data just to avoid taking responsibility of taking decisions, that's a bad idea. Yeah, and I want to add something to that. So I also see data as a proxy for forgetting that hard decisions are just as much of an art and a creative process as data about data.
16:07 There is an overreliance, overconfidence, and overindexing on if I have the right data, I will make the right decision, and that's absolutely not-- like, absolutely wrong. You may be looking at the right data, but someone with a different strategy sense, product sense that is just-- like, their LLM, their brain is better trained to navigate the strategy space. They will look at data, that data in a very different way than someone else might.
16:38 So decision-making is as much a creative process as a-- strategic decision-making is as much of a creative process as it is a data-oriented process. So we should also not forget about that part. That just we cannot say that, oh, the data will tell me where to go and I will go there. That may not be the right answer in all situations. Why don't we make this a little more concrete. Do you have examples in mind about product decisions or about people decisions that were difficult and then you had to get input, you had to look at data or talk to your team, talk to your board, talk to your CEO to make that decision?
17:21 Something that you can share. I know many of them could be sensitive for a variety of reasons. Yeah. I think that something that I think every early-stage founder is going to run into is a decision about redoing their product effectively. In early days, you're just trying to look for product market fit. You're trying to look for an evidence of traction somewhere in this market. You're building fast. You're ignoring a lot of principles.
17:52 You may have bad UX, you may have ugly UI, and those are all fine. You're looking for traction. Optimizing for those things is not right. You may make subpar technology decisions that can't scale in the future, but you have to make them because you're trying to get something out and, again, see if there is any traction for it. So I find that to be one of the common ones that everybody has to make, and we had to make it as well.
18:19 At NovoEd, it's a very difficult one because you're saying for x number of one month, I don't have any new feature development effectively with my small team, and I'm going to be sitting and redoing this user experience and this UI and some on the backend so that I can scale for the future. I find those to be really hard to make sense of, and I personally think ultimately they are really a bet, a bet that, for example, I believe that my better UI and UX is going to sell on its own, and the new features that I will be able to add in these nine months are not going to sell just as much.
18:58 So I find that when is the right moment for that? There is no science for it. It's really like based on this-- the strategy sense of the people involved in the decision. OK, let's just pull the trigger now. It's really hard to look at data and say, oh, this is the right time. So that's one of the obvious ones that we go through. Every time we have to start a new kind of like a product line in a way, not in the sense that it's a product line as in it has a separate P&L or it will be sold separately, but is a major area of product that we know we will have to continuously maintain it, add more into it, and it will grow exactly like other-- exactly in parallel with other areas of product-- of our product growing, we have the same set of-- similar sets of trade-offs to work with.
19:53 And again over there, it's a bet being placed. That, OK, if I add this new piece of product into my portfolio, it may not be the first-order driver of sales, but is it a retention driver? Is it a conversation-opener? Is it a-- so you place where it is and based on that, decide, OK, I can invest X percent of my resources into it and I will be fine, it will be a good decision overall. So I personally-- I sometimes use actually-- like, we had a decision-making course that Ron Howard used to teach for us that I-- sometimes even use that.
20:38 I sit and create my decision tree and I try to come up with, what do I put as a probability of success over here? And not just based on what data is telling me, but also what is my own sense, what is my other senior colleagues' sense and try to make sense of our decision space using that. Yeah, so since Farnaz mentioned Ron Howard, Professor Howard was a Professor in Management Science and Engineering. He passed away a little while ago, and it was a great loss.
21:10 My co-instructor in the Strategic Decision-Making course is actually his PhD student. And a big part of the course, especially the part that actually my co-instructors are teaching, is based on his methodology on how to make decisions. We have actually a couple of very good questions that I see here. One is about documenting decisions. What are your practices for documenting the decisions you make?
21:43 And I can imagine, it's, again, different if it's a product decision versus if it's a personnel decision. How do you document it? Do you go back to the documentation that you have made and think about whether you made the right decision? Do you go back and try to learn from whether the decisions you have made were the correct ones or not? So talk a little bit about documenting. Yes, of course.
22:14 Well, I might be one of the more rare breeds in my-- and in the C-levels that believes in this. I don't believe anything is concrete unless it's written. So if it's not written, everybody has a different impression and everybody for sure will do something different. So in the universe that I operate in, and depending on who I'm working with, some form of remembering is happening.
22:46 If I'm working with executives, that remembering is happening in the form of memos. I don't like PowerPoints, I like documents that outline the rationale, the insights. Not everything we looked at, but the insightful pieces of information that we really counted on. What is it the decision? And then what are the downstream implications? What work needs to be done? Who are the owners? What is the time frame that they have committed today on that? And I also think that each of these things, when they become cross-department and they are new areas that you're launching on, they do need some form of project management, but I personally don't subscribe to the project management role.
23:26 I believe that at the end, there is a driver behind every initiative. There's a driver behind every one of these decisions we are making. And that driver should figure out within their own staff how are they going to stay on top of whether we are taking the actions that we need to take across departments within the time frame that we were supposed to be taking them. So in my world, either product marketing manager or a product manager or a client success manager, they become the responsible party for keeping everyone else in line for taking the downstream implications of a decision we have made.
24:02 If I'm working with product managers and designers, in my universe, we live in FigJam, we live in Figma. At this point in our journey as a company, we have our own defined templates for how do we go through client interviews, how do we record insights, where do we keep them, and things are very traceable. So if we want to figure out that, oh, for our, I don't know what product-- for our AI learning assistant product, why did we decide that it doesn't answer this type of question, but it answers that one well?
24:39 And when you go back to our meetings and the questions we ask, the insights we agreed on, they are right there. And I also heavily work with my team and coach them to make sure that they are also relying on writing for communication. For product manager, this becomes, OK, if you have made a decision, great, then what are the user stories? Write them down. If you have not written a user story and tomorrow you come back, your designer is designing something else, your engineer is working on something else, well, you know where the root cause is.
25:16 You did didn't write it down, so everybody took whatever they heard, they have taken away. So I'm very much so-- but I don't think of them as documentation. I think of them as making said words concrete. And I think of them as writing what is necessary to make sense, not document everything that we said. No, it's funny because I still quote Farnaz after so many years.
25:47 Once she told me, if a meeting doesn't end with action items, it wasn't a meeting. You were just hanging out. So I still tell people, did you have a meeting? What were the action items? I did do that, yes. I used to be very blunt, yes. No, but it makes sense. The meeting has to end with a decision, with the summary of the decision, the action items for the people who were there, otherwise, again, as she said it, you're just hanging out, you're not having a meeting.
26:20 OK, good. Let me take a look at the questions or-- we actually have a ton of very good questions. Yeah, so I guess there's a question about multiple choices at one time. Do you have this process of first generating alternatives, like brainstorming and creating many options and then choosing one of them? Must do when we talk about reframing problems. You see, I like to have methodology, it helps me think through, and one of the methodologies that I subscribed to in that area is from Tina Seelig and her crash course on creativity that we actually offered on our platform, and I had the pleasure of both implementing it with her and then learning from it, learning from her, and she has a book on that area as well.
27:17 I learned about framing and reframing problems, the Crazy 8s or whatever you call them, coming up with different solutions. But then later on in my career, there are other people like-- in product management, especially, Teresa Torres, who have written a lot about the process of continuous discovery and opportunity, this-- what does she call it? The opportunity trees. But at the end, the whole point is that you frame your problem in as many ways as you can and then break it down, break it down, break it down until you get to the leaves.
27:50 And the leaves are really atomic, small changes that you can make to your process, to your service, to your product that drive one atomic outcome. And you use that framework to think about, OK, which one of these should I prioritize? Why, why not? Then you look at your drivers. What is the effort? Is it helping retention? Is it helping expansion? Is it helping my new sales? Is it a conversation-opener? At the end, you also need to have a certain set of parameters that matter in your business that are not necessarily your North Stars, but they are making sense of your universe for that you put together to see between all the options produced, all the ways one can look at the objectives here, which one might be the right one in this particular decision.
28:43 So there has to-- but to me, then there's no-- to me, the funny thing is, in this smallest thing that we want to do, smallest feature, I don't know, give our clients the ability to disable a certain generative AI capability because there are compliance problems, just take that, it's really surprising, when you actually want to do good work, there always are multiple options, and they always produce a slightly different outcomes.
29:13 So I actually find it empowering to always push to produce different options and not take problems simply because the moment you take a problem, simple-- oh, this is it, obvious. You have probably chosen a subpar option, or you're really lucky and you have such outstanding product sense that it just made sense in that moment, but you likely have missed out opportunity behind. I see. So that actually brings me to this question about bias. A lot of times when you just make a quick decision without spending too much time, you may bring a lot of your own biases.
29:52 And maybe early on, when you have a small company, when there are two or three people in the room, of course, the product is an extension of you. Of course you bring in your biases. But then when you have multiple stakeholders across the globe, you want to make sure you're avoiding these biases so that you can continue to serve multiple customers, multiple stakeholders. How do you avoid biases? How do you try to-- or at least reduce them as much as you can in your decision-making?
30:25 For myself and for team members who are in leadership positions-- and leadership, to me, it doesn't mean that they're managing people. It might mean that they are managing a service or a product or making decisions that actually impact what we are offering at the end and in the market, actually, first, start with awareness. I think that every leader has to take the time to understand wt are at least the most common decision-making biases in the area that they are in.
30:57 What is their definition? How do they manifest themselves and impact decision-making? In my universe, there's a lot of confirmation bias, like hearing what you want to hear because you have a predisposed notion of what would work out in mind. There's halo effect, optimization or overconfidence bias, planning fallacy. Sunk cost, very big one in product development. And what you see is all there is, essentially like not exploring beyond the box that we are in.
31:28 They are the set that I commonly run into. I personally think there is no magic bullet, really, but there is an art to training the brain to ask the question. If you know how halo effect looks like, if you know how sunk cost looks like in decision-making, you can ask yourself, you can ask your team, are we-- is that impacting our decision? And ultimately, I think that there is a product manager-- like a leader in the product area, Shreyas Doshi, that I respect a lot, and he talks about it in modern-day words, train your LLM.
32:14 It's look at other people's decisions. You're using a product-- I don't know, Netflix, whatever it is. They change the pricing, they send an email. Look at that. Ask yourself, what do you see in it? What brought them to that decision? What is working about the email, what is not? What are some of the biases that they're trying to combat by communicating in that way? That actually makes your brain stronger so that when you get to your own decisions, you're way more aware of how the different things show up.
32:47 But, I mean, you were talking about the form of bias that was more different points of view or different areas of the market or just competing-- for me, for example, I know that has always existed because we actually serve two very different markets. We serve the training companies. Very different, but synergetic. We also serve corporations. And in my world, what I have come to is that if you're defining a problem, you have to define it for one of these guys and optimize it ultimately, for one of these guys.
33:24 Being a jack of all trades or just optimizing for somewhere in between makes it so that you are a somewhere-in-between product. You don't solve anyone's problem sharp enough. That it just connects and they will just get it, use it, buy it. When you are just in the middle, in the average, there's always a translation between the customer problem and your product, and that will show up in a slow sales cycle, that it shows up in mysterious marketing, that shows up in customer not understanding, what is it that you are and what is it you do for them.
34:03 So ultimately, as difficult as it is, we have to put a lens. That, OK, we are optimizing for this subsector of our clients when we are solving this problem and here is our reasoning. For me, that's really important. Well said. So Farnaz now has actually mentioned a whole bunch of common biases, like the halo effect or confirmation bias, and these are some of the things that we do discuss in the course and extensively.
34:35 And I agree, awareness is the first step to be able to check yourself and then go over the list and say, am I making the wrong decision because of these biases? That can be often quite helpful. Let me make-- I had a set of questions that I wanted to ask you. Let me take a look to make sure I'm covering all of them. There was also one question about communication of the decision to the team, and especially people because they have different communication styles, different expectations, different expertise.
35:16 How do you think about that? Whether you want to communicate a decision to a customer who may not be happy about hearing the decision to your team or to your board. How do you manage that? Yeah, I think I was reading that question as well, some of the relevant questions, and it's like a question about the needs some people have for more data and rigor versus the ones that are more intuitive. I love intuitive people, and I am an intuitive person, but I put the onus on the intuitive people to actually articulate where their intuition is coming from.
35:58 At the end, intuition is a combination of domain knowledge, patterns you have seen before and you've collected from different products, from different life experiences that are bringing you to advocating for one decision versus another. So yes, I do have intuitive people around myself and data people around myself as well, and if an intuitive person is insisting on an idea or a direction, what I ask for them, to enable others. That means both they're more early career team members who might be early career just generally or new to this market, new to this type of problem, new to this type of user or persona, to articulate their whys, and allow others to challenge that.
36:41 OK, we don't see it in our product use data. Why do you say that? Or we don't say-- we don't see that in our website or webinar interactions or our top-of-funnel interactions, why do you say that? So that helps it so that we are answering the right questions, but then when it comes to communicating, I always think that what brought the insight that might be a data source that might be looking at a product usage in Pendle and comparing this to that.
37:11 That insight should be included in the communication. So the ones who are just receiving the decision and they were not part of making it can actually benefit from the insight and not go back and ask the same questions that we all asked while we were trying to come back with the decision. Yeah, very good point. Yeah. You also mentioned the other side of it, that you have different customers and their needs might be competing. Some of them may not be happy with the changes you make.
37:42 That's a tough one. Honestly, I just had one of the most traumatizing experiences early on at NovoEd. I just added a simple navigation button to the bottom of our pages that would help a user understand where to click Next. And I had an army of upset clients who were expecting users to navigate this from random linking in the pages. And I stood by my decision because it is just horrible UX, and my client may not understand that, but I understand that.
38:20 And it's OK. You fight the fight, you explain. Sometimes you lose a few, but you win more with the right decision. So having upset people or customers and not pleasing everyone with what you're choosing to do is one of the hardest parts, actually, in fact, and one of the most important parts of making a decision. If you try to make everyone happy, then we are not really deciding, we are pleasing.
38:51 Yeah, so two very good points that resonated with me. One is about this question of-- it comes up especially in the context of B2B products, because you're designing a product for someone else that's going to offer to its users and clients. And then after a while, you see enough data that you may know about the needs of the users more than they do, and it's very subtle to be able to communicate that in such a way that so even though they are your customers, you know the-- what they are-- how they're going to use the product and how that's going to help them actually have a better-- create a better user experience for their end users.
39:40 I think it's now a good time to switch to the Q&A. If I can invite Robin again to the-- hi, Robin. Hi. I've been really loving this conversation. Farnaz, I love your strong opinions about when to add a navigation button, when to-- and when to maybe people aren't happy with that, but I really appreciated that. We got a ton of amazing questions.
40:13 I tried to answer a few of them. I'll ask Farnaz to answer those, but I think we have around maybe 18 or 19 questions. So maybe we can go over some of those and see if we can discuss those. Yeah. Yeah, absolutely. So many wonderful questions coming in. And I've seen this question that asked a couple of different ways, and I wanted to know, Farnaz and Amin, what you think about this.
40:44 Someone has asked, what do you do when you have to make an impactful decision and there's not enough time and there's not enough data? A crucial and impactful decision, what do you do if there's not enough time and enough data? Amin, do you want to go-- Go ahead. You have to actually make those decisions a lot when you have a small organization, a small team.
41:14 New initiative in a big company or a small startup, somebody explained it to me, it said, big companies typically act like cruise ships. You make a small adjustment because you know in a year or two where you want to be. But if you're in a small organization or a startup, it's more like, surfing. So you have to make decisions quickly, otherwise you're going to fall on your back. Farnaz, go ahead.
41:45 I agree with that. I don't have a-- I don't have a ton to add, that's a good answer. I think that's great. And sometimes you just have to make a decision with what you have and you have to call it if you have a time limit. As you said, you probably have to do this a lot. You have to make the best one that you can. It's normal life. It's actually normal life. I think-- I'm going to say this again.
42:06 It might be one or more-- my more try not to say it comments, but at the end, decision comes with responsibility. You could be wrong. It comes with the fear of failure. We have to also take into consideration all of the emotions we are feeling when we are trying to make a decision, and depending on our level-- like, small decisions could feel very hefty on the soul. Part of becoming a good decision-maker is also working with yourself to be OK with failing.
42:37 It's OK to fail. Failure is an opportunity to learn. Just reframe. You're not-- and then what I tell my team is that you do not need to make the big decision when you don't have the data for-- the data, the insight, the-- whatever is necessary for the big decision. Just make the smallest decision you can today. That gives you the next step. And when you get there-- and you will have a lot more to work with, actually.
43:04 When it comes to product development, that's just the magical rule. Something-- you can do all you want to in interviews. Prototyping gives you a little bit more, but nothing, nothing compares with having an actual touchable piece of functionality that people can actually play with. And the real insights come back-- come out to you right then. So you can also think of it as reframe it as, I will make the smallest decision I can make today that is directionally right, and I will reevaluate my input in the next step.
43:41 Yeah. Actually, I need to-- it's a very good point. I need to mention this. A lot of the things that we say are at the cognitive level about how you can think better about your decision, how you can frame your decision better, how you can generate alternatives and choose the best alternative eventually, hopefully with the aid of experience or the data. But there are emotional components to the decisions, and I have seen a lot of people, even though they have all the right information in front of them, they can make decisions for emotional reasons, partly because of the fear of failure.
44:23 And that's something that's equally important and it's part of everyone's personal journey, to identify those and try to fight them and try to mitigate them. Well said. And I appreciate, Farnaz, earlier you said making decisions is an art. It's a creative process. It's intuitive. And so it's a lot of right brain, it's a lot of left brain. We have someone that asked, can you share a framework or do you have a thought process and how you can prioritize?
44:55 Which reminded me, Amin, of-- you mentioned Ron Howard, and you mentioned your co-instructors in the course, and maybe you could say a little bit about the framework that's in the course because someone here is asking about a framework. Absolutely. So actually, the course is built around this framework that I mentioned, some of it from framing the decision, identifying the stakeholders, generating alternatives, potentially collecting data, even building mathematical models.
45:30 I saw a few people asking questions about Markov decision processes or other decision tools like linear programming, integer programming, where you have all the data at some point, but to feed it into one of these frameworks, like a decision tree, to eventually identify the best choice available to you and identify what were the assumptions that went into that decision.
46:04 So in case if the decision ends up not being the right one, you can go back and look at those assumptions and find out which ones were incorrect. We discussed these, and in addition to those, we bring in a few cases, a few examples. And in addition to those, I actually have talked to quite a few other experts in the field whose interviews are incorporated into the course.
46:35 Wonderful. We've talked about intuition and we've talked about analytics and rigor. And someone asked if there are any tactics or tricks that maybe either one of you have adopted to accommodate both types of people that you may work with when communicating out about a decision or when you're in the middle of the decision process. So you might have those people that skew analytical, you might have those people that skew, hey, I'm making a decision more intuitively.
47:09 Is there a way that you found throughout your experience to communicate with them to accommodate both types of people? Farnaz, go ahead. Yeah, I can, I can because we always work with a diverse set of people. I err on-- like, both of those are superpowers. Intuitive people have one superpower and analytical people have a different superpower.
47:40 So when we talk about, let's try to map out our customer journey, let's try to map out the service journey that we are trying to offer at the end, what we are trying to accomplish, and let's try to map out what are the risks that we have in between. What are our questions that we need to have answers to? That provides an opportunity for analytical people to actually point out, hey, here's an assumption here that I want to challenge, we don't have enough data for it.
48:10 And it helps to make the room in such a way that the intuitive person can come in and say, well, we have this pattern, this is study from this other vertical, this other market, this other area that helps us answer that question, and helps us have a conversation whether that question needs to be asked and answered in a strictly analytical way for what we are trying to accomplish, or do we have enough that we know?
48:41 So I try to make the room flat enough that everyone's voices are heard. And I'm sorry, part of doing that for me is doing it in writing. I really don't like voices to be heard as the way I'm speaking right now. So problems are laid out on a board. We are all putting Post-its. Our questions are going quiet so we all have time to think. We all have time to negotiate with ourselves. Many times in decision-making rooms that we are just like starting, the first person asks the question, and then you have the analytical person in the need of data, intuitive person, and all of a sudden you have a crossfire between the two that is completely unnecessary, energy being lost, and decisions not being made in a methodical way.
49:25 So I like to do is to get everyone to have alone time to think in the middle of the meeting. 10 minutes off-camera, everybody put your questions in Post-its and comments in Post-its on the board. And the owner of the decision reads them, chooses which ones are the most important ones, and we try to have a very balanced room. And I think Google Ventures people from around our universe at Stanford have written a book on this called The Sprint, or-- it's about a design sprint, but I find a lot of good tactics in general for managing the room in which decisions are being made, managing all the opinions that fly.
50:12 Making sure that the C-level voice is not the loudest. The room is actually flat. And so I recommend everyone to try to find tactics that will help them respectfully make the room flat because that, by itself, is an art, and it will help cater to both the analytics person and the intuitive person. Thank you for that. I also hear a lot of, as we're having this conversation, I've heard prototyping, I've heard framing the problem. The methodology of Post-it notes, very much design thinking.
50:45 So the customer, the empathy map, what the customer journey is, how did they find us? So a lot of overlap there, wonderful. A question came in I think is interesting about, do you have a methodology-- or how do you approach reviewing previously made key decisions and about whether or not they're still relevant? Do those get put back into the decision-making cycle or relooked at with regularity, or is if it comes up?
51:17 Is there anything that you do within your organization or with your team that is about revisiting relevant key decisions to make sure they still stand up? See, I think that technology organizations have done us all the favor of coming up with the concept of roadmaps, priorities. Is it a sprint that helps you organize work? Is it a Kanban board? Whatever it is, I really recommend other areas of business follow the same mindset and I try to do that with everyone.
51:49 For example, in marketing, in sales, your marketing go-to-market bets. I made a bet that this particular message with this particular sets of events and webinars is going to produce so and so top-of-the funnel numbers for me within two quarters. There you go. That should be an item sitting on some form of a tracker, and workstreams are happening within it. And in that realm, for example, your revenue operations managers become the owner of coming back, looking at the data, and sharing with everyone what happened over there.
52:22 In product management, it becomes, we are releasing this thing, we expect that X number of clients are going to adopt it, and as a result of that, we're going to have X number more, X percentage more license usage, for example. That-- call them like-- depending on what system is being used, they become goals, in a way. They're not just decisions, now they're goals for the organization, that an organization needs to have the hygiene of going back and evaluating, are they working out?
52:53 And always, they have to be time-bound, otherwise you have no room for iteration, and iteration-- someone had mentioned rapid framework, iteration is part of decision-making. Yeah. Actually, one of the things that I learned early on was, in the process of either running the company, setting the direction of a company, or setting the direction of a product, a very good methodology that I had training in was the scientific method.
53:26 So think about all the hypotheses that are going into this decision, going into this design, and then ask this question of, how much do you believe in this hypothesis, and what evidence do you have, and what type of experiments can you run or what kind of additional information can you collect to validate the hypothesis that's going into the decision? And how critical is the hypothesis? If you ended up being wrong about the size of the market, if you ended up being wrong about the rate of adoption or your set of competitors, how is that going to impact the success of your organization, success of your company?
54:06 It would be nice to articulate them, as Farnaz mentioned, again and again, write them down, discuss them one by one instead of trying to keep everything in your head. Wonderful. All right. It's getting to 10:55, maybe there's one more question? Or if you don't have time for that, we can switch, too. Yeah, I think we can take one more. We can talk about one more question. I wonder, Farnaz, we've asked you lots of different questions.
54:37 I'm wondering if there's anything that you-- an opinion you hold strong that you feel like we didn't ask you about today that you'd like to share with us. That's an interesting one. I don't think so. I think that I've had an opportunity to talk about what has worked for me, but ultimately what I recommend everyone over here do is to practice decision-making. Take the frameworks.
55:07 Learn from them. Great. It gives you a structure, it gives you a way to think about decisions. But make decisions and put yourself in a position. If you get to play, the best thing that I say as a startup founder is, like, if you get to play the next round of this game, great, success, success. So get yourself an opportunity to make the next one, to make the next one, and that will just make you a much stronger decision-maker, and at the end, executive.
55:37 Wonderful. Thank you for that. Amin, Farnaz, thank you so much for this great conversation today. It was really wonderful. And I would also like to thank everyone for showing up today. I know there's a lot of different ways that you can spend your time, and you chose to spend it here with us today, thank you. And also, the questions! We had such wonderful questions. It makes for such a rich conversation. Thank you again. And if you're an attendee and you enjoyed your time with us here today and you'd like more information on how to make more high-quality decisions-- as Farnaz said, practice, practice, practice making decisions, then check out the course that we have linked in your dashboard if you-- maybe you want to check that out.
56:20 Great stuff in there. And thanks again, everyone, and we're signing off from Stanford. Thank you, Farnaz. Thank you, Amin.