Liz Reid on What Publishers Get Wrong About AI // AI Inside #135

0:00 Sometimes they'll add a payw wall and then they'll say my traffic's down and you're like yes yes yes that is what will happen if you charge. [music] Hello everybody and welcome to AI inside. This is a show where we take a look at the AI that is layered throughout the world of technology. I'm one of your hosts Jason Howell. My co-host Jeff Jarvis is with me today. He'll be here for the interview. And boy, do we have a special interview for you today. Been looking forward to this immensely. Our guest is Liz Reed, who's been at Google for 23 years. She was one of the first female engineers in Google's New York office. Actually spent 17 years building Google Maps. And today, she runs Google Search. Billions of people using the product that she runs every single day. Pretty important person doing pretty important stuff behind the scenes at Google. Now, right now, that product search is going through the biggest transformation that it's seen since the introduction of the search box. And actually, last month, I was in Mountain View for Google IO. Uh they mentioned that AI mode hit a billion monthly users. People more and more apparently treating the search box like it's a a conversational interaction. And Liz actually says in our interview that people are adapting to that faster than even she expected.

1:27 And she shares a few key reasons as far as why she thinks that is. We also dive deep into the personalization of search. a lot of like the personal intelligence features coming into search. And Liz actually makes the case that integrating your personal data into search, your emails, your maps history, calendar, all that kind of stuff could actually be the thing that helps small publishers instead of hurting them. Publishers will want to hear that. She also has a pretty direct take on pay walls that I'm guessing some publishers won't love to hear. She flips the filter bubble argument upside down. Uh Jeff asked her about this and she argues that not personalizing search actually creates more filter bubbles because people just go straight to the one source that they already know. And she's honest about how early all this stuff really is, how they at Google are still figuring out how to deliver the proper metrics that publishers are looking for. And I even asked her where she was when the word ignore broke AI overviews just a few weeks back. She walks us through what it's like when something big like that happens so suddenly. So, we got a lot of ground to cover. Let's get into it.

2:42 Here's our fantastic conversation with Liz Reed. Welcome to AI Inside, Liz. It is so great to have you here. Liz Reed from uh from Google, the uh search team and a name that everyone that watches the show is very familiar with. Thank you for being here with us today. >> Thank you for having me. >> Yeah, it's it's super cool to uh get to check in with uh folks at Google and we're just really appreciate appreciative of your time, especially because this is such a such an important moment for not only not only just technology at large, but clearly artificial intelligence is such a huge um you know, moment that's happening in tech right now. But beyond that, I mean, you work so closely as you do with the search product and we're going to talk a lot about that today and how how all of these technologies kind of coming together is really in some ways reshaping in many ways reshaping what Google has done with its primary product of search. But um you being the person who really gets to kind of make these changes to the search box, this thing that is so iconic and integral to what Google has done over the last, you know, few decades. What does it feel like to be in that position from from the inside having your fingers on kind of that kind of change? It's a big deal.

4:03 >> Um I think it's both very exciting and very humbling, right? um you know this is uh it's a product that both um you know billions of people use every day but that also means like it's my parents it's my daughter it's my next door neighbor right um and so people really um continue to trust Google to give them great information high quality information and you want to make sure that you continue to live up to that responsibility each and every day >> Liz um I'm fascinated with with the notion that search has now expanded did so much and and it started with a keyword search for this word and find it right and then it became most recently I think questions and people I'm in the habit now of asking it questions because I know I'm going to get answers to it and soon enough it's going to be commands uh which is kind of a literative accidentally but from from keywords to questions to commands um what are you seeing about people's adoption of those changes are is there a high proportion still just doing keywords are people asking questions more and more uh and and the bigger question is is there is there a point at which Google is not a search company because it's something so much more >> um so great question um I think we see that people are adapting um and I would say they're adapting faster than I might have guessed a few years ago right um you know you think you've been trained for years and years on keyword ease um but you know you've been trained for tens of thousands of years on talking to humans right and so when you enable that um with that you see um so we do see people asking more questions questions.

5:35 Um, we also see even the way that they talk about questions and the way they ask the the queries are changing. Um, they're asking longer questions is one way. Um, but they're also asking putting like I in it, right? I am trying to do the following things. Please help me figure this out. >> Oh, that's interesting. >> Um, and so at at some level, you know, I I think of keywordies as kind of like you had this question in your mind and then it was your job to translate to the computer so the computer could understand. Okay. Um, but you know when you talk to humans, you just talk kind of how you think and you assume that the other person can do a lot of the translation, right? Um, and now we're getting to the point where really actually like the computer should do the search product should do the translation. Um, and so you can just express it the way you would like to express it. Um, and then we can do the work behind the scenes to actually interpret it and and pull the information together. And I think that's really exciting um in that both there's a set of needs where like previously people had a very specific question right the question was really unique to them you know you would say okay like restaurants New York is a generic question but like nobody actually just really wanted all the restaurants in New York to know about right like they had a very specific set of here's their price range here's who they can go here's who they're going with here's the type of cuisine like they had all this questions in their mind associated with it um and now people can really ask that question that's unique to Um, and we can pull that information together instead of forcing the user to sort through information after information to sort of narrow it down to what they want. Um, I think you also see that when you eliminate that friction, um, people ask more questions. Um, I think there's there's sort of this naive view many people have, which is they ask all the questions they have already. And this is actually not true. You have questions that kind of flit into your mind and you're like, is it worth it for me to pull out my phone? Is it worth it to do this one? Um and so people sort of make this like very fast mental calculation of should they even bother to learn um to explore that curiosity to get help with that task. Um and the more you make it easier for people, the more they just come and ask that question because now it feels worth it. Um and so that ability to help people both more each time they come but also just help them with more of their lives um is really exciting and and also a big responsibility to help people with.

7:48 >> I want to stick on on the the branding question for a second. Does does search become Gemini? Does Gemini become search? Um, you know, I just went through a dialogue. My wife had a health question and I went through a great dialogue with Gemini and it was a dialogue, right? It was it was additive. It was it it wasn't just a a search to blue links. Um, as a Google is more and more a leader in AI, where does that head in terms of how you present that to the consumer uh from internal brands?

8:15 >> Yeah. So right now um we think of the two products as as sharing a lot but having different north stars right um so Gemini is really focused on that personal productive powerful assistant really being that assistant for you um and search is really focused on helping you with your information journeys um and so we make different choices along the way um in the space now we share a lot of the technology behind the scenes um and so if we build new capabilities in the model we share the same underlying models then we can express it. Um if we build some tools, we share the tools across um but we sort of curate the final product a little bit differently. Um search will put more emphasis on information needs. Um Gemini has a lot of productivity needs, a lot of creativity needs and so it steers it a little bit differently. Um we also see that that that search fundamentally is still a search product. So so AI mode and AI overviews think about um this case about when you want not just AI but you want the web. and you want the best of both together. How do we make that possible so you can get started with AI to help you curate the information but make it easy to jump off whether that's read more about this World Cup event or find the restaurant that you want to book the reservation or whatever else is um now we do have overlapping user needs for sure um we really believe that we should meet users where they are um we don't want to tell people you picked the wrong door to come into Google somewhere else um but we'll steer the products to to really go after the strengths of both H yeah, you know, personalization has been a big um and a very recent kind of priority from what I've noticed of of Google's kind of rollouts and personal intelligence and everything. Now we've got that kind of coming into search. I mean, search really is becoming kind of the Swiss Army knife of a lot of the different technologies. They eventually make their way into search and start sort of superpower kind of how the search product actually works, right?

10:10 not as much the the blue links anymore. It's very dynamic at this point. I'm curious to kind of hear with the kind of more recent integration of personalization into search, how users how you've seen users actually kind of pushing this to the limits and testing this and kind of um representing the the best of what that personalization can actually do to the search experience that we didn't have the ability to do a couple of years ago when it was, you know, just a little bit more of the blue links experience.

10:41 >> Yeah, I think um if it it is actually really exciting. I mean search has been looking at personalization for you know I don't know two decades now but it feels like it's actually to the point where it becomes um much much more useful. Um you know there's the obvious ways which is like you just want recommendations for shopping or local or others and there's all this past context you have um you know here's the places you know the brands you bought that is in your um email here are the places you've been to according to maps. those give a taste that it won't just be like the cuisine, right? Um, of the places you might want to go. And so making it easier for a user to not have to go and specify all of the information they want when they just want a pair of jeans, but for us to tailor. So that's sort of like the the more basic. Um, but I think one of the things that's interesting um with personal context is that you see a lot of cases where it wasn't just obvious, but it brings new delight. Okay, you're trying to figure out what you do when you're stuck in the airport for four extra hours and you ask for some ideas and it's like, well, did you know um you actually have this thing associated with your credit card that will let you go into this lounge and then there's this really interesting store on the floor above you and then by the way, here's a great restaurant that you might like.

11:53 Um, but it surprises you, right? You might figure think it will tell you a good the restaurant you want, but you wouldn't necessarily realize it would tell you that there was a perk associated with your credit card. Um, I have a set of three kids. um uh actually being able to go and say, "Okay, where do I want to go?" And it says, "Oh, well, you know, if you're taking your youngest daughter and she needs a stroller, this is a great park to go to when you're visiting some other area, right?" So, this this moment of delight because it can sort of um tailor you in ways you didn't even think about. Um I think is pretty exciting. In general, there's like amazing information out in the world that's public and then there's amazing information about you. Um and previously as a person, you had to kind of do that joining, right? and that meant a lot of work on your part. Um, and with personal contacts, we can sort of take that heavy lift um, and sort of join the the data more effectively together for you.

12:41 >> Okay. So then you've got that that kind of extra context and um, >> and that's super powerful. What will come up probably a couple of times throughout the course of this interview is is the side of the publishers of course because there's a lot of you know there's a lot of publishers that look at the the search product and what they've been used to that search product being and watching it transform and change and kind of uncertain about like what what does this mean for us sort of thing. Um does the personalization does that personal intelligence information kind of change the math when it relates to the publishers? um because some of them kind of think that some of these changes might make them more invisible to search than they were prior to those technologies existing. How does that impact that?

13:24 >> Um I think there's a few different things actually. Um what happens with personalization? Um there's both personalizations and some additional things we're doing with things like preferred sources that that I can talk about in a moment. Um what we see with some of that is that it allows um sort of a a larger breath of um creators of publishers to sort of surface, right? Um if the only thing you enter is a few keywords and it's unpersonalized um then everything kind of looks the same, right? Um but actually if you're really interested in, you know, eco-friendly brands and you don't necessarily say that, but we go do that.

14:02 Now that brings the little merchant to show up in a new way, right? Um it brings up the reviewers that specialize in looking at these emerging companies to come up in in new ways. Um so I think there's a lot of opportunity with personalization for um creators and journalists who specialize in something and would otherwise have been difficult to be found to now be found. Um we're also looking at doing new things with things like preferred sources and subscriptions. Um, many people with publishers have um an existing connection, right? They're paying money to one publisher. Um, they really trust that brand. How do we make it easy for people to get to that content and so that we can actually strengthen the relationship between publishers um and their audience? Um, and I think preferred sources and some of the work we're doing around subscriptions um, throughout the search page and in AI overviews and AI mode, we're going to continue to do more there. Um, so that the people who really that they build that trust, they build that audience, they can really strengthen it going forward.

15:02 So Liz, uh I'm a former journalist or once a journalist always and was an executive at a big media company before I started to teach and write and podcast and uh I have many frustrating conversations with publishers these days uh which I think you'll sympathize with because I tend to argue uh in favor of AI and the things that I hear I'm sure the same things that you hear and I'd like to hear your arguments back to a few of them. Right. One is the is the I'm losing links. I'm losing clicks. I'm losing audience to this AI thing. That's one. Another is uh well, I don't want them to learn about my stuff. And then I argue back to them. This is the new mechanism of discovery. If you hide yourself from both search and AI, people can't discover you. The AI won't know to recommend you. Um and uh the old um chestnut that people get the full answer from Google, they don't even read anymore. Well, that was always the case.

15:56 if I needed the score of of of a World Cup game, I needed the score of the World Cup game. I didn't have to read 20 inches of a story about it. So that's the context in which they think they tend to to treat themselves as victims of progress and victims of the internet and victims of AI. Uh what would you say to publish Oh, one more point. I'm sorry. Uh in this I try to argue to small publishers that they should take advantage of this moment and they should make themselves available to AI. uh starting with Google that and I've argued for the creation of APIs for news so that we could say a whole bunch of let's say here in New Jersey where I am where we have 300 plus publishers in a in a news sharing network um well it's getting to that number but we could say to Google here here's all of our stuff let's have a conversation about how it's used and branding and so on so finally what would you say to publishers uh about this new the reality of this new time and what do you think the the best relationship is between Google as an AI and search company and them as uh old media companies.

16:56 >> Uh so so great questions. Hopefully I remember all the all the points to to respond to. Um I think to to start with um there are first of all multiple things going on besides AI, right? Um one of the things that we see is that people are often going for new formats, right? Um they want to see videos, um not just text. Um they're often going to social media for content. There was a recent study from Reuters around this really just talking about how people are shifting. And so one of the things I would generally tell publishers is is make sure you're innovating with what what users want and how they want, right? Um you see a bunch of publishers who will be like, "Oh, my video traffic is up, but my web traffic is down." And you'll be like, "Yes, yes." Because you know what? People picked your videos over your your web, right? They're not sort of separable things. Um so there's a lot of dynamics and just user expectation about who they want to hear from um the formats they want to hear from and it's and it's important that they that they innovate on that. Um I think the second thing that's that is really important is that that people um to the extent that you produce really interesting expertise content we still see that people are interested in that.

18:03 Right. Um to the extent that you might just read um an AI review and you wouldn't click you might have previously just read the headline and not clicked through. Right. >> Right. To your point. Okay. If you were going to go spend 10 minutes reading the article, >> you're probably not going to decide that an AI overview is a substitute for the 10-minute article. Right. >> Um, so [clears throat] where you're going to see differences on things like to your point, like if you went to the web page previously to get to sports score even before AI, now you go and you just get the sports score up front. Um, but if you're like, "Oh, let me tell you why this game was like so unusual." and you're going to write this really interesting article that follows the sports score about why this game was remarkable. Oh, people are going to be pretty interested in that and click, right? Um, and so that the more that publishers um produce content that is really um where they shine, what they bring to the table, um that it's unique, it's not the 1,000th copy of the same story, but it's something that has an interesting take on it, um the more I think we'll see that people will continue to collect um to click through and read that. Um, and we really want to um strengthen that and that make that more possible. Um, to like the mention of preferred sources as well before, right? If you're building that that trust um in your brand, how do we make it easy for people to get to that, right? And so that they are clicking through to content that they can read and not that they're clicking through to content to get a pay wall, right?

19:24 Sometimes people they'll add a payw wall and then they'll say, "My traffic's down." And you're like, "Yes, [snorts] yes, that is what will happen if you charge, you know, make people pay on that." Right? Um, and so for those who do have access, can we make it easy for them to to get to that content? Um, I think actually for small publishers, it's an awesome time. Um, because actually the small publishers bring something unique or they can talk about local news and it would have been hard previously to match that user to that content. But because users are expressing much more um their intent, what they're curious about um then it can sort of allow more differentiator, right? It pushes more into the tail. Um, if you build that relationship with your user and then you get them to list your preferred source, then you can show up in a way that you might not show up to one billion people, but you're going to show up to the people that care. So, I think we really want to get to continue to be in this world in which um, you know, publishers do see search um, as a great place to connect to their audience. Um, and I do think that users still really do want to hear from other people, but they want to hear from other people with high quality content. they don't want to hear from people to like check the time zone and come back, right? Um and so build build rich content and then we will really work to connect it. Um but I do think we all have to innovate, right? Um if you build a product that is too high friction for users, they will just go elsewhere.

20:45 >> Um and so we need to continue to innovate and search, but we really do want great content to shine um and continue to connect people to it. >> Hey everybody, buddy in here just real quick. Uh fascinating conversation. We're going to get back to it, but I do want to throw a huge thank you to those who support us on Patreon because without Patreon, you know, we we really couldn't do these interviews. You guys, you support us on a deep level and it's just the driver of everything that Jeff and I do here with AI Inside. Want to throw out a huge thank you to Chris Spackman, Steven Kimbrook, to two of our amazing patrons who help us do this show each and every month just by supporting us on a deeper level. They get access to the AI inside daily podcast which you get if you're a patron and all sorts of other stuff. Discord at free, you name it. So anyways, patrons, thank you so much for supporting us. Thank you for being there for us uh as we develop this show even further and get amazing guests like Liz Reed who we will talk to in a moment, but we're going to take a quick break. When we come back, we're going to talk about what publishers can actually do to make their content more visible to AI mode. That's coming up in a moment.

21:57 What can uh what can publishers do or creators do to make their content more visible to AI? If if this is something if this is kind of the new direction that all of this is headed, which it clearly is, and publishers are concerned about getting the right eyeballs on their content, how can they work with the system to ensure that they're kind of playing by the new set of rules and getting their content maximized in in front of the right places at the right time?

22:28 >> Yeah, I mean, I think um I would probably put it in in in two buckets. The first is make sure we can access your content. um like if you block the content that will not work. Um you know if it makes it hard to discover then that then that's difficult. Um we have various tools in web master console um that give you controls you know so that that that publishers can choose but making it easier for us to access is is certainly the first step. Um but we've also published an updated set of guidelines um around uh for website owners and publishers to think about um how they make great content in this day.

23:01 Um, at the heart it is really this continuation of, you know, if you want people to click through, then implicit in there is you want people to read your content. That means you need to make content that people want to read, right? So the more you build a content that your audience will love, um, the more it will work. Um, the more you build content that you think is just designed for the search engine but not for the audience, then people will learn that over time. Um so the more it is brings in your expertise um that is sort of fresh and relevant content to what people are curious about um that it brings in that sort of experience um that detail and richness um that's really great but we have a long list of um more specific guidelines that we published recently that I would encourage publishers to read if they haven't. If I could follow up real quick on that one thing that I see whether I'm bsing publishers when I say this that I argue that they've got to move past always thinking of the article and the click to it that they've got to become at a local level they've got to become gatherers of data themselves and so that it's known that they're a repository the my hackne example is if we if we uh scanned every restaurant menu in in a in our readership area and and somebody could come in find out where who has empanadas uh or um the I'm trying to work with local publishers to transcribe the video from town meeting so we have that data or other data from a town. So it goes beyond just saying I have an article that's fresh today and fish wrap tomorrow. It goes to saying I'm a source for deeper information uh and that that one would hope then that Google would recognize oh there's good stuff here and maybe maybe the aim isn't to click through. You want to see the menu, you see the menu, you click through, fine. But maybe sometimes it just says, "Oh, this brand comes up. Oh, they've got stuff. They know what's what."

24:46 >> So So am I am I full of it or is there an opportunity here to rethink? >> I don't think you're full of it. I think there's sort of a a couple of things in there that you're coming like if you take the example of town meetings. Often times that information is is like pretty hard to find. Um and so you you're bringing a unique source, right? Um that that people are craving. And you know if you live in that town that content really matters right um I grew up in a small town of 900 people right like that the town meetings decide this is what's going to happen to the town beach decides whether you you know how recycling like these these things really matter to to people's day-to-day life.

25:24 Um and so I think the more that you can bring this content that otherwise is inaccessible then people come. Um I think the second thing though that you're highlighting is one in some cases um the brand visibility does matter right um if so you know and we actually try and think about how do we surface this more effectively in the product right so you know where are we going to say according to so and so right and so we can um not just provide the link but help people understand the brand um and then the next step of it is okay one we can provide that visibility which is real value um for for some class of of publishers Um but also then if if the user builds that sense of the brand then you can actually encourage them to be for you to be a preferred source right and to say actually like the user has an affinity with that and then if you have the same information as somebody else yours should show up stronger right um but it the more your information is special or your take on it is special um the the more valuable will be and the more it is recycled of other people the harder it will be because people are just like well everybody says that who cares Thank you.

26:33 >> So, um, so when you got, you know, publishers interacting, um, with, well, let me back up. Google, you you recently, uh, in the search console broke out kind of AI impressions, which is giving a little bit more kind of clarity on how they're interacting with this. They get impressions, >> they don't get clicks, which I think publishers are are very very curious about. uh how how do they verify that things like personalization and these other aspects of the efforts that you're rolling out, how can they verify that the right audience is finding them through these tools? [snorts] >> Yeah. Um so I think there's a couple of things. Um we rolled out AI impressions.

27:09 I think we're going to continue to look at what are the metrics that are useful like how do we help people understand and you know for um for all all the space on this it's sometimes hard to remember like we're we're very new into this right like AI mode is one year AI overviews is two years. So, so we're still learning and and we are actively trying to figure out how do we continue to improve the the metrics on there. Um but I also think for publishers some of this is um how do you think about instrumentation instrumentation on your own side right of um where do you understand about are the users coming um can you understand what are a what is a high value click versus not right um one of the things we see um with AI experiences is that one of the clicks that tend to that do tend to be affected are what we call bounce clicks right a user went to the site and they're like oh that wasn't what I want and they come back right Okay, versus now with AI reviews where they get a little star and they know why they're they're going to click on a link, then they stay longer.

28:05 Okay. Well, do you know when people land on your site, are they enjoying your site? Are they engaging? Are you able to turn them into a subscriber? Um, so we will both continue to work on um how do we present more more metrics and the right insights on search console, but also would encourage you to think about, you know, what is valuable to you? Okay, is it the subscribers? Is it repeat users? Is it bringing a new audience?

28:30 Um, and how do you detect that on your side so that you can also steer appropriately? >> So in I think it was 2012 uh Eli Perezer wrote the book uh the filter bubble and then after that uh Axel Bruns wrote a book entitled are filter bubbles real and his answer at the time was no they're not. And the that that this belief that everybody saw a different Google was not true at the time and so we weren't being um fragmented. Um but in the era of personalization and in the higher value uh that we get um uh yes we we will see different Googles we will see different results as we go and I think we still have this reflex in mass media that we have to be amassed and we all have to see the same thing and I think that that that to a great measure is a waste of people's time to be treated the same as everybody else when you know more about them but this is a change in I'm going back to to Jason's earlier questions in this context There is a change here in Google's philosophy about personalization and um how often do you think that the answers need to be standard need to be uh similar for everyone and how often is that unnecessary?

29:42 >> Yeah. Um so I think it's a it's a it's a great question. Um I think there's maybe a few different angles to look at this. Um some question the same answer um is just like a waste of time, right? Um so like you're not all going to buy the same shoes. It does not matter whether or not we give you all the same shoes, you're still not all going to buy the same shoes, right? Um so pretending that we know the absolute best shoes for everybody, it's just not true, right? Um and so that seems an obvious case to personalize. Um if you take something like news with preferred sources, um we really tried to think about how to do this to be thoughtful about filter bubbles, but but also to be um cognizant of how people are behaving. And so when you give your preferred sources, we don't not show the the default top ones.

30:25 we show the we continue to show the default top ones and we show your preferred source. Okay, so you still have access to the breath of information. Um but you can see your preferred sources more effectively. Um and so that's sort of important and that you have access to all of that information. Um but you can ensure that you can do that to the extent that the top five sources would have had pay walls that you don't have a subscription with. It's very nice that we put them up there, but you're still not going to read them anyway, right? Um, but also one of the things we see is if if we don't enable people to connect to the information they want, well, you know what they're going to do? They're just going to go directly to the site, right?

31:01 They're actually, you know, this hope that this will somehow prevent a filter bubble backfires because they don't expose themselves to multiple sources. They just go directly to the source they want, right? And so actually enabling people to see the source that they recognize and trust as well as other sources gives them variety. Um, I actually think we also see this with AI with with new opportunities. Um, one of the things we try and do with AI overviews, um, on some of these questions is actually present sort of more of a multi-point of view on these topics. Okay? Here's what some people have to say and here's what other people have to say. Okay? Um, and then we can provide links to both and you can read about it, but that's a great place to get started and actually have people have an overview rather than picking one or the other. they learn a little bit more about the um the breath of what the world has to say and then they can choose where where to dive in. Um and so I think it's really important that we are not narrowing what people have access to but also that we're not telling them sorry you must use this not that right um you know people at the end of the day have to decide for themselves what they trust.

32:06 I'm thinking we can uh dive in a little bit on agents because uh the agentic side of things is starting to also come over to search which is interesting and exciting but also kind of creates some real kind of curious uh directions here. You've said in the past that AI search um that that search is AI search through and through. when an agent, let's say, calls a salon and books an appointment, that agent, you know, nobody's actually searching for anything at that point, right? The agent is kind of doing the bidding for there for them. And I'm kind I realize I'm kind of lifting this from you, Jeff, so my apologies, but you inspired me when we were chatting about this conversation about a new verb like we're used [snorts] to calling search search, but now search is becoming this entirely new product and or or entirely more capabilities. Anyways, and with agents, it really goes into uh interesting directions. Is there, you know, is there a new way to describe what search is becoming um in this kind of new paradigm uh $100 word there, Jeff, you can you can build me later.

33:12 [laughter] >> Um I don't I don't know if I don't know I don't have a new verb um uh for you at this point. Um, I would say, you know, we really do want people to feel like they can come to us um for anything that is sort of an an information need um and ask it the way they normally would and and take action. But I would say um sometimes I think in these discussions we we we paint search as if um until two years ago it was just 10 blue links and then suddenly it becomes AI. Um actually search has been on on quite an evolution. Um it's been I don't know how many years now since you could book restaurant reservations right on Google.

33:48 Um and so it's sort of enhancing over time more of these powers of what is hard for people? What are they asking us for help with and how do we continue to help? Um and in your hair salon case, well actually usually they are doing a search. Usually what they come and they say is like I need a hair appointment with such and such um time and such and such thing. And then we're helping search information that's not necessarily on the web that's highly dynamic, maybe in, you know, their own systems and calling to gather that information and and help you pick and and select. Um, we want to do this in a way that really continues to maintain trust for users that they really feel that they're in control um and have agency and and they build up that that sense. Um, but I do think, you know, we we sometimes treat these things as if they're very isolated in people's, oh, well, this is the this is the search and then this is the action and this is but like for users, it's all blurry. Like I have this job I want to do and it consists of a bunch of questions and then a couple of actions and then some double-checking and then maybe texting my mom about it. Um, and that the friction between those substeps um has been very expensive for people historically. And so we want to make sure in search that we don't sort of artificially narrow it such that we can't actually help people with the task and we make it um too hard. And so we just want to reduce those barriers again and again. Hey budding in here one more time. Just a real quick one. I mean mostly quickly. Anyways, actually it's not really quick but if you didn't know that we have uh all of our episodes in video format. You know now we have a YouTube channel. Go to YouTube, search for AI inside or just find the YouTube channel which is called AI Inside Show.

35:29 You will see this episode and all of our episodes, all of our interviews and everything in glorious video. And so it's all over there on YouTube. Um, and we also love that you subscribe here and listen. So why not both? Okay, going to take a quick break. On the other side of the break, we ask Liz about her day-to-day, what it's like being Liz Reed at Google and what she was doing when the ignore bug hit AI mode. Very interesting stuff coming up here in a moment.

35:58 >> Liz, you the greatest job on earth as far as I'm concerned. Um because you you >> I told my kids that they they have mixed videos. [laughter] >> Well, and your bosses get a raise. um uh when it comes to uh organizing the world's information and making it universally accessible and useful, which is of course Google's mission, >> that's you, that's what you do. So, I'm really curious to hear more about your job, how you go about your job. Um what are the metrics you look at >> uh in anticipating change? What are the measures of success or failure with an idea? I I used to work with about.com many years ago when we had a deal with Google and Google would worry about AB testing. Literally every pixel we would change.

36:39 >> Yeah, we do a lot of AB testing. >> Exactly. Um and and and so your kind of short term daily, what are you looking at? What's your dashboard look like in terms of what you want to do? But also in the longer term, what are some of your dreams for where you think search can go? Search or whatever we choose to call it can go now. >> Yeah. um on on sort of a day-to-day we tend to look at um sort of two things at the highest level. One is like um are users more satisfied with the experience they get when they come to us? Um and are they coming more often, right? Um and and the first is really important and there are lots of different metrics, right? Pe people's behavior is complicated and so you can do user studies, you can do experiments, um you can ask people in the moment for how they're doing, you can watch how they behave. Um you can see if they issue the query and then they like hate it and they immediately try and reask it. Um in the case of the conversational experiences sometimes they will tell you very bluntly that you screwed up.

37:36 Sometimes I will tell the product very bluntly that it screwed up. Um so you get all of these signals and you're trying to understand them about where the product is working well and not and how do you continue to improve and ensure it's going up. Um but the other part of this is that we do genuinely see that if the product is working really well then people will bring more of their questions that that um barrier of like is it worth my time to try and ask that question goes down then they come and bring more questions and so the the real question is are they bringing more of their questions and so then when you look somewhat um more in the future of not just what is today's one but like if you're planning out a couple months then one of the questions is like what are the types of needs that people are still having a hard time with um that they're not asking. We know that they have them in the real world, but we don't seem to be seeing them and and how do we reduce that? Um if you want to do my my longer term vision of that, like I think if you go back to Google's mission, um there was that first part about organizing the world's information and there's that second part about like universally accessible and useful. And like useful is pretty important there. There's lots of ways where um you could technically give people information, but it wasn't really useful to them. And so, one of my um you know dreams with search [snorts] is that you're not just sort of democratizing access to information, you're democratizing understanding that information, you're democratizing using that information. um you're making it such that sort of exploring that curiosity or helping um you learn in the world and experience more of the world is not hindered because of technology.

39:14 It feels open because of technology. It feels like you can explore your own dreams. Um I think many people in the world are feel pretty busy and they feel pretty busy with a set of stuff they don't want to do and they would like to spend more times on the stuff they want to do. Can we make it such that all of those tedious tasks, we can help you eliminate a bunch of that work. Um, and on the other hand, we can help you figure out where to explore with the time you have. How to get out in the world and do new things, how to learn new skills, how to enjoy your time and and follow your favorite team or your favorite hobby. Um, and sort of unlocking that, making that feel more possible. Um, and getting rid of the the tough things for all the power that search and all these products come. you want to go build a new appliance, buy a new appliance, and it's like a lot of stress for people, right? Are they gonna buy one, is it gonna last for 10 years, is the right one for them, etc. Like, how do you make those things feel delightful for people um and easy and then they can get back to the things they want to spend their time with?

40:14 I am super curious because uh what Jeff was you know what he was asking in the last question was really kind of going in the direction of like how you do what you do and your perspective and your place inside of Google and everything. It it reminded me of the story from a couple of weeks ago which was the ignore AI overviews story that um >> where ignore got in there and it kind of changed what AI overviews actually was doing and in my head [laughter] I was like >> commands. Yes.

40:42 >> Yes. And and in my mind I was like, what does Liz Reed do in a situation where you're going about your daily business and suddenly you see this news? Like what like what does that look like for you? What is the response like being on the inside to something like that? Yeah, I mean I think um we have we have the mix of sort of you know what is our overall our data and ideas tell us we need to work on and then um you one of the problems with a billion you know getting billions of queries every day is you don't realize everything that people will do and you don't um >> you know you're not going to get every one of those billions of queries um perfect. And so when we hear that and we see that >> then we put a set of people on and we say okay well what's what's not working right in the system? how do we fix it?

41:26 Um, and how to improve it? And then you get a a group of folks to brainstorm um, how do you actually improve the signals? How do you improve the system? Is it just a bug or is it a quality problem we need to improve? They run instead of a valve. Um, and then off to the races to try and improve it for for people. And that's what Search has done for years and years. Um, search has never been perfect. Um, it always has opportunity to get better, keeps us in a job, which I guess is good. Um but but really this desire to get better and better and so we constantly want to look for where could it be better and how do we fix it and how do we move quickly on that, right? Um the ignore example was a case where um the system thought it was a command instead of a definition. Okay, let's go um and and make progress on on fixing that case. But there's so much opportunity to continue to make better.

42:20 Um, and you know, if you look at that case, part of that was that was a bug and we obviously need to improve, but some of the background on there was opening up sort of um definitions and information um to to a much longer list of of words um and much more languages than we otherwise could have supported. So, we moved that direction and then the system had a bug and then and then we fixed the bug. So, we try and be really dynamic and adaptive um but always focus on how do we help the user every day.

42:49 >> Yeah. and and and obviously, you know, bugs happen, everything and everything like that. Totally understandable. Something like that was a pretty low impact thing. I mean, maybe a lot of people saw it, but it wasn't the the worst outcome in the world. It was just kind of hilarious. >> We do put we do put extra effort um and extra testing on things that we view as higher stakes. So, one of the the things that is important is we do overall testing, but you know, we talk about um your money, your life queries, right?

43:17 Um, so we put extra scrutiny on things like medical or or things involving finances because we know those are high stakes. Um, and you know, to your point, other things may be seen by a bunch, but um are lower stakes, so those are good things to improve as we go along. >> Yeah. Well, we're big fans of of the work that you're doing. We talk about um Google's many AI uh products and and directions that are happening right now because you guys are just so incredibly busy. I've, you know, I've I've met with the team, the the Gemini team a few times before and just been amazed at the rate of progress and the rate of change.

43:52 You all on the inside having to keep up with that and driving that that change at that rate because that's the speed that everybody else is working at is just really uh pretty impressive stuff. And uh I can't thank you enough for spending some time with me and Jeff and the AI Insight audience. Liz, it's been a pleasure talking with you. Thank you. >> Thank you, Liz. >> Thank you. Pleasure talking with you as well. Yeah, we'd love to have you back again soon. We'll uh put out the invite.

44:15 Thank you again. >> Well, we'll work on that. Figure out a path. >> All right. [laughter] Keep up the great work. We'll talk to you soon. >> Thank you. >> All right, that's the interview. Thanks again to Liz Reed and everyone at Google for making this happen and I'm just really excited to continue collaborating with Google to get amazing guests like Liz on the show. Who knows, maybe we'll have her back someday. Uh, don't forget real quick, you know, Jeff is a busy busy guy. He's always writing things. I always love to talk about it and shine a light on it because he's brilliant at what he does and I just have a huge amount of respect for him. So, uh, go to jeffjarvis.com. You can find and pre-order Hot Type, the book that he has right imminently releasing here very soon. Also, the Goodenberg parenthesis magazine, The Web Weave. It's all there.

45:02 jeffjarvis.com. uh hit our site on the web, ainsside.show. Everything you need to know about the show is there. I still need to add the store link here because if you go to ainsside.show/store, we've got a bunch of, you know, things like like hats and stuff for AI Inside. So, you know, some of you super fans can go ahead and purchase those. I got to get that on the site. I promise it's coming soon, but hey, I've been busy.

45:27 But all of our episodes and everything can be found there. And then finally, of course, patreon uh.com/ai insideow. We have some amazing patrons who help us do this show on a regular basis um pretty much, you know, every month and including our executive producers who get a whole lot of extras. They get all the general swag and everything, but they also get a t-shirt. Speaking of swag, anyways, our executive producers are Dr. Dub, Jeffrey Marachini, Radio Asheville 103.7, Dante St. James, Bono, Derek, Jason Knifer, Jason Brady, Anthony DS, Mark Starcher, and Karsten Somatki. And who knows, maybe you too. patreon.com/ai inside show. Thank you real quick to uh to just everyone for supporting, everyone for watching, and for listening, to Victor and Daniel for helping behind the scenes on all the video support. We couldn't do this without everyone involved. So, thank you for your support, and we'll see you next time on AI Inside. Take care, everybody.

46:31 >> [music]