AI in Healthcare Series: Tracking and Trusting AI in Medicine, Dr. Shantanu Nundy

0:15 Welcome back to another uh episode of the Stanford Healthcare AI podcast and we're thrilled to be joined by Dr. Shantanu Nandi who is a physician technologist worked at some of the uh biggest healthcare technology companies over the past decade uh and is now a adviser to the FDA on on AI. Welcome Shant. >> Thanks so much Justin. >> Uh so so many things coming out there there's never a dull moment. uh but one of the places to start actually was something uh we found fascinating which is how people are actually using these AI tools uh in in their daily lives. So it was amazing actually uh OpenAI shared uh conversations and published on uh millions and millions of conversations and on the right of this chart you can see h how people were starting to use this. So I'll call out just a couple things first and then uh Matt and Shantu, please jump in. As we're talking about healthcare, one of the things we've talked on bits and pieces about and heard, you know, rumors about from the companies, but at 5 to 10% of chat GPD conversations relates to health. Now they publish some of these stats. You know, they claim practical guidance 6% here. You know, we've heard claims similarly, Google, other companies, health related queries are top of mind for consumers. And so you're seeing this now here in the data that they've published some of the other things writing seeking information uh is something very top of mind people are doing all the time and then interestingly on the left they you know they also talk about in the work setting and there's slightly different uses that people are using at work lots and lots of writing so the thing I joke about is expect all those personalized emails to be written by AI that's just the expectation as we're moving forward but uh anything else to stand out to to you I mean, for me, I mean, I'll just say like my personal journey, right? Like I sort of joke it's like it started off as like a copy editor for me, right? Then a little bit of like a ghostriter and now it's like a like an analyst, you know, and I think that's kind of reflected in a lot of these things, right? I'm actually sort of like almost like building on top of um you know these tools as obviously the tools are getting more mature but honestly I think most of it's me and just getting used to figuring out how to like create more complex prompts that sort of build up to more higher order tasks. So this this resonates >> I mean from my side like uh again we are terminally online. We are definitely following this maybe more so than we should even I would say but but this is totally what I would expect to see I especially on the writing side to see like you know 40% I you could almost argue that there are behaviors that are are internet behaviors that are kind of starting to transfer over I think in a large degree like seeking information to the model right but the other part that I my my mild critique or it's it's a weird thing but you mentioned you're kind of moving up this ladder or sophistication.

3:14 >> Yeah. >> I can't wait for the rest of the world to get there too because >> every time I open up >> LinkedIn or Twitter or what name it >> I im because what we all have worked with these models so you can immediately spot just the complete cut and paste of an AI and and I have to say it's starting to I I I don't like it. I I don't know how else to say it. I I feel I feel like it's right now and I'm and now I'm turning into someone who's like searching for the post that is like maybe has a spelling error or has a little bit of a you know clearly not an AI written >> cut and paste because like >> he is the double hyphen. If you see the double hyphen, you know that it's not this, it's that. It's like it is it is so bad. it almost like you lose credibility which is an interesting phenomenon that I did not expect true >> I didn't even expect to be saying this but but to some extent I am um I'm wondering there's two things I'm wondering as a part of that one is I think as people move up the sophistication ladder and the models become better whatever that looks like personalization memory that may start to change hopefully maybe this is just a a passing moment but the other part is like who are these ultim what is this content ultimately written to too because if the consumers of that content eventually will be >> another model that's just grabbing that and providing your then do I even am I even ever looking at the source again?

4:45 So those are kind of things that are bumping around in my head as I'm seeing this adoption curve and this again I know we like to we hear a lot of AI slop. I am I am full up of AI if I'm being honest and I'm begging for just a little bit more at least as a reader >> more content taste uh on the internet. >> Totally. >> Interesting. Interesting. Yeah, the the video content again we haven't talked a lot about the video and image generation >> as it's been a little less relevant on kind of the healthcare side. so far. Uh, but that's just another piece. You talk about Matt AI slop that's happening and just how easy it is now to create a video of you with any famous person that you want doing whatever you want. You know, we continuous, you know, uh, we saw OpenAI, Devday, all the SO2 announcements, Google's V3. there's that is moving so fast and >> curation of of content is just going to be something again we've talked about it not not a ton here in the healthcare context uh but it's it's coming and and actually maybe the one healthcare related example just that I saw recently is actually doctors starting to get faked um real doctors that people are starting to use now to use their image and name and likeness so you know I haven't seen that yet of you Matt or or Shantu but that That's how I guess you'll know you made it.

6:15 >> How do you know I'm not one of those right now? >> Exactly. >> Exactly. Exactly. >> I had a suspicion. No, I'm just kidding. But but on but on that topic too, like I one of the things I've noticed on the video say maybe less on the writing side just to kind of maybe pivot from that former point I was making on the on the the Soratu for example. Great interesting where like almost immediately you can sense someone's inherent creativity like some of the things that people are thinking up are pretty freaking cool, right?

6:45 >> But the signal to noise >> Yeah. >> is difficult, right? Like there's a ton more noise now. But the but the signal when it's good is like, wow, that's that's amazing. So I you I don't know where that's going to head, but I have a lot more respect for the creatives of the world >> are able to take that tool and still apply their inherent genius and creativity to to really conjure up something amazing. I don't happen to be one of them. I've I tried my kids called it all cringy and like I gave up, you know. So, >> well, that that that democratization piece, you know, is is something that is paramount on that. And I think the other piece uh and Matt, maybe you want to go through this is just the cost changing.

7:29 You know, these models used to be so expensive to create the videos. You used to only get a couple seconds, but Matt, walk us walk us through this. Yeah, just two seconds on this and then happy state of AI report to all of you who celebrate because a lot of us wait around for this report. Shout out to Nathan for putting this out every year. This is massively consumed by the entire industry. If you haven't seen it before, check it out.

7:49 Stateof.ai. You can see the link on the bottom. And this is just such a great public service because it does encapsulate in a very difficult time what's happened for the past year. This is really just to show the point I think that yes, we talk about the capabilities getting better, but we don't often talk about how much the capabilities are getting cheaper. >> Yeah. >> And and more democratized. And I don't know if either of you think about this, but as we start to kind of pivot this conversation towards the direction of healthcare, and we've talked before about the increasing capabilities, the access uh of this technology, I think, is unlike anything I've maybe ever seen.

8:26 I'm sure I know the internet's a great analogy, but it just seems faster to me and cheaper to me than ever before. >> Yeah. No, I think it's I think it's paramount. I mean, I think obviously like affordability is a huge huge huge huge issue, right? And equates to access, especially in this country. And so I I think that part of it, you're right, is that not a part of the story we hear enough about. Um, and it's I think it's pretty incredible. Like I I I was we had the OpenAI team in recently and just sort of saying like the the model that they made to be the cheaper model is now performing at the same level as like their best model like you know three or four months ago, right?

9:02 And so that's sort of that happening I think does open up a lot more possibilities. So yeah. >> Yeah. and the the access point and right the cost as these are now ubiquitous and you know the state of AI report also talks about the fully open source models and that race that's happening um the world is moving fast and while once oh you can't do that costs are prohibitive you know I have this conversation all the time with folks in our team or health systems the cost will come down that is the benefit uh right of the competition we see in the space is people are racing and when that happens cost costs will drop One of the other things we're talking starting to talk to you about though is is access and we've talked about this some before but then and I know Shanti you we're thinking about a number of of problems here but well what happens when you give people unfettered access and you see a lot of amazing use cases but we haven't talked as much about some of the kind of areas or problems or things that are coming up but uh Shan I know you had a couple you wanted you wanted to to go in and I'm super curious as how you're how you're seeing this. Yeah, I mean to me what's so interesting is like the justosition between like these exciting progress and then sort of some of the stories that you hear, right? So there was one just a couple days ago in the New York Times, I think the title of the article was, you know, you know, person came to the ER with the flu and died a few days later, right? So it was basically about a college age kid, totally healthy, had like flu-l like symptoms, went to the ER, was discharged home with the viral illness, and then a couple of days later came back and then ended up passing away. Right? And there's a lot of pieces of the story, but one piece of it was that there was actually a sepsis alert that was at that hospital and that sepsis alert activated and actually was a positive alert, but it was ignored by the provider, right?

10:49 And I laughed, but like I I've totally been there, right? I see I see patients still I usually turn off the allergy, you know, drug drug interaction stuff. So, I think that just brings up like a whole host of issues, right? And I think like zooming out, I think those issues are now going to become like the gatekeepers, right? It's it's not going to be the accuracy as much. It's not going to be the cost as much to Matt's point. I think it's going to be the sort of social technical side. And some of the specific issues, right, were okay, well, they happen to have the alert set up, which is great, but we know that all alerts aren't created equal, right? And then in terms of workflow, doctors were getting tons of alerts for different reasons, so it was ignored. But now there's other interesting issues that are coming up which is okay well of all the hospitals since it's a lawsuit right the key is what's the standard what do other doctors and hospitals do in that area do they have an alert and then do doctors generally follow the alert and now there's even people asking like maybe I should turn off my alerts as a health system because if we don't have the alert right then there's no risk of our doctors uh ignoring their alert and then creating this problem and it's like that's like the exact opposite I think of what this technology and those graphs apps are showing us we should be doing, but like we have to understand like all those different sort of components of the healthare system if ultimately we want to solve the problem.

12:06 >> Yeah. I mean I I I struggle with this too. I think it's and I sometimes I'm explaining to folks that aren't in healthcare like how something like this could happen because like on its face this is like right you just it seems like the how did the doctor ignore that alert fatigue? I think this has been talked about since you know I think Bob watch wonderful book on this and we've known about this for a long time.

12:28 >> Uh I try to relate this to someone who's not in healthcare. It's like if you ever go to a city that they always honk their horn no matter like even for the slightest infraction they lay on their horn, >> right? you just you start to become like okay they're honking but that's just like that fades into the background a little bit and then you you know if especially then if you go to a place that doesn't do that and then the hon it's like a huge like you know event that someone's honking you it has a little bit of that vibe to me like when do you pay attention and then if >> or maybe the simplest is the boy who cried wolf right >> 100% this is a tales all this time >> it's all tales all this time >> right but to your point about but the worry though I share this worry is that either that becomes the background noise even like it's happened here or it leads to a reaction that says >> no more AI, no more decision support. Uh it's clearly, you know, causing harm and and I don't I don't think that's right either.

13:25 >> Totally. >> Yeah. >> Well, well Sean, like you're you're seeing this at scale. You're interacting with scientists at the FDA reviewers every day. You know, you're seeing so many issues for these things coming up. But like tell us a little more like what are the conversations you're hearing? What are the topics that keep coming up? Is it is it this? Is it is it others? And h how are you hearing those conversations going? >> Yeah. I mean, no, I mean, so I think well, first of all, let me sort of I'll say a couple like preamble things. I think one is, you know, um it's been really incredible to to get to meet a lot of the scientists and reviewers that work at FDA and and made their careers there. I mean, a lot of them came straight from their PhDs or MDs into the FDA and have been there for 10 or 20 years. And so, it's there are really incredible people thinking about this problem already. Um, and and it's it's really inspiring. I have a chance to work with them. Um, the the other thing is like, you know, when I was coming in as an adviser, you know, I thought a lot about I do feel this urgency, right? I think like stories like that that example in Colombia, um, which was the hospital involved like creates urgency.

14:31 I thought a lot about like what do I want to say to have everyone share that urgency because at the FDA like the mission is really to promote and protect, right? But I think sometimes like there's a focus on the protect side and not as much on the promote side. And so like when when I came in I thought a lot about like okay like maybe I could use you know science and and and numbers and I sort of came up with five numbers that I talk about in like every meeting that's become a running joke now at FD which is that you know there's 100 million people in our country with no regular medical care. There's 75 million that live in a you know desert. Medical error is the third leading cause of death. 95% of rare diseases have no FDA approved treatment and life expectancy is largely flat and four years behind other OECD countries, right? And we all have our favorite statistics, but I I just I start with that every single time because I think what sometimes gets lost is like what's the counterfactual, right? So you like this example we just talked about of okay well you know there was an alert and maybe the alert was wrong or right or it was annoying and it was ignored but like the counterfactual is like we've known for decades right since the Harvard medical practice study in the 1970s I think or ' 80s that you know there's a jumbo jet every day that's crashing due to medical errors right and so but it's hard because I think the way that a lot of the folks you know sort of look at it is okay well is this device causing harm as opposed to saying what's the harm that's already existing because people can't get access or that access isn't as high quality as we want.

16:07 >> Yeah. And I think this is the fascinating example we're starting to see in so many fields is, you know, we've talked a lot about benchmarks on tests and exams and >> right >> and while it's exciting, you know, we know none of those are the kind of real comparator as we're talking about. What is that? What is the what is the chat GBT? Is the answer right? Is it wrong? It's pretty good. >> Yes. >> Oh, it it made a mistake and that's going to lead to harm and it and it will. How does that compare to the Google search compared to nothing? And and these are just kind of the fundamental questions we we have to wrestle with. And I think the the interesting field, you know, you all probably be annoyed with me for how much I point back to it, but it's fascinating starting to see the data come out in the autonomous vehicle world.

16:57 >> You know, Whimo's released a lot of data now on kind of their crash events and events. And actually, I've seen a lot of now doctors starting to comment on, >> whoa, whoa, whoa. Like, if you extrapolated these numbers, this would be saving, you know, thousands and thousands of of of lives. And it's not just the accident rates, the fatalities is, you know, you can react in different ways. And uh there's there's not a right answer to be super clear, right? There's there's this fundamental fear which I think is real and myself and I think the country that you start to see on >> an AI related error that causes harm, we're going to count as more than the physician we know harming someone or than the car crash that was caused by, you know, a drunk driver because we're almost numb to it. Yeah, it's a great video of that car, right? It like it sticks it stays on you, which is interesting, right? Um I think the other thing you brought up, which is like like I think you're right, a lot of the testing right now, this is something that we're thinking a lot about right now or the scientists at FD are thinking about is like a lot of the testing is in these very simulated hypothetical environments, right? And so we we think, okay, that's happening. That's great.

18:06 Now we need to figure out like how to do that when it comes to like you know real world prompts and real world scenarios because the other thing I think people think a lot about is there's sort of performance which is sort of like the median or like the mean versus like the long tail like it's like part of the FDA's job is to think about the long tail right and so when you simulate and people like oh yeah we tested 100 case we thousand thousand cases you're like these things are being used by you know hundreds of millions of people so that tale gets extremely long um somebody from the Google team unrelated to LLM sort of shared with me that even today and I'm going to get this wrong a little bit but even today every day of all the Google searches something like and again I'm going to get the number a little bit wrong but like 10 to 15% is a search that Google's never seen before >> even today right I'm not talking about like their their LLM I'm just talking about their normal search right and so that long tail so anyway so that's the second part and then the third part is now how do we actually begin to understand the the outcomes piece and I think the other thing that relates to your point about the self-driving cars is unless we let these things drive how can we iterate fast enough right so if we force everyone to stay here and then we kind of control here and we really tightly control what can be implemented with patients the cycle times to get that learning right and this is where I think healthcare has historically really really fallen down right and not been able to improve quality and things, right? Um I think that's sort of something that I'm thinking a lot about at least, right?

19:42 >> Yeah. I mean, well, and and then like so and for every one of these, to your point, they stick in your head. There does seem to be a different standard. I don't know if I my gut says that I don't know if I disagree with that different standard. It's for some reason. I I don't know whether it's because I know it can scale, whether I know that it's probably just a a blip that is maybe representative of something that we're not seeing. Like, I don't know. it doesn't. So that's I struggle with it too, but I I also feel like there's the other stories that are like, hey, I went to the health system, they told me XYZ, I I put it in GPT and they told me something different and then later it confirmed what GBT said and not my like and I I feel like we have these competing narratives. I don't know what's going to win out, but at some level, again, reflecting back on the beginning of this conversation around consumer behavior, >> some level, and the stats, by the way, that you pointed out about the desert and the lack of access, these things are are kind of two speeding trains heading at each other to me.

20:40 >> And I don't know how it's going to turn out. >> Absolutely. >> Yeah. >> Super well said. you you me you mentioned Sean new this idea of like how do we get started and learning iterate and actually just uh it was last week uh you know the FDA just put something out about you know real world monitoring and so tell us a little bit more more about that you know it sounds similar to the concepts but what what's what can what can you say at this point >> yeah it's really exciting I mean I think look I I think in general right if you just even if you zoom out of software for a second and you think about you know like the the the canonical sort you know drug pipeline right so you have you know phase one two three and then phase four is postmarket right I mean I think that in today's environment where we have so much data right like when I look at a patient in front of me they look a lot different than the patients that were eligible and not eligible in a clinical trial right and that's why you know people say oh half of health care is not evidence-based and half of evidence-based medicine isn't done on some level that's fair on some level it's like well the evidence base has limitations right it doesn't apply to most of the patients standing in front of me Um, so those challenges were already there, but now you add to the fact that unlike, you know, a pill that doesn't really change once it's in the market, right? Like we know that this stuff is changing really rapidly, let alone the fact that the underlying technology itself is advancing rapidly.

22:00 So I think there's sort of this this this new idea that's maybe not super new, but it become maybe becoming more common to say how do we, you know, focus more of our attention on how these things are performing in the real world, right? For all those reasons, right? like we may not have the best test to know if it's valid or not. We know these things are going to change. We know the underlying technology is going to change. We know how it's deployed matters so much. And so why not, you know, focus our attention there and we realize like that's a science too that's evolving. So the whole the whole idea behind, you know, what the FDA put out last week, which is really exciting, is like, hey, we we want to learn in that area, right? And we recognize that that evolving sort of let's call it like the regulatory science or like the science of how do you evaluate and monitor these things um no one's quite cracked the nut on that and that that work is really distributed across academia industry you know um regulatory bodies and so it's a chance for us to really be able to understand what's what's what's working and what's not when it comes to that >> I and I I think this is really important because well first of all I just want to make a meta comment that the level of discourse and and this is again back to the credit of the those that are serving in in these roles the level of discourse has I I would say elevated beyond I think anyone's wildest expectations I mean the the that the fact that we're thinking about these things and the other topic I want to bring up in addition to this is we are I at least I'm hearing uh maybe you can confirm some of these conversations that we could get to a place where some of the trials can be performed digitally. Um we we've we've seen some recent work that's kind of continuing to show us that we're getting to a level of scale and sophistication and this is somewhat orthogonal to just the the chatbts of the world but we can start to think about data and then projecting out with a great degree of confidence matching trials that we have already you know plenty of data on to show that had we done this digitally >> the same outcome would have been reached. So therefore, can we accelerate >> the the the market sort of bottleneck that we have in our traditional structures without sacrificing safety?

24:13 This is science fiction stuff at least, you know, as someone who's been around long enough to know that these have been floated for these ideas have been floated for a long time. It feels tanalyzingly close. And again, that goes back to the level of sophistication, the level of discourse that I think the community is having with the regulators uh in a way that I I I personally haven't observed before. >> Yeah. Yeah. No, and that and that's honestly why I wanted to to to be on with you guys is I think like what's been incredible just generally about this community, not not just the amazing folks at the FDA, but is that like everyone I talk to is like how can we help, >> right? I I think and again I haven't been at FDA very long and I haven't worked in other spaces within FDA but I think you know usually there's like a oh the FDA is going to regulate us and like you know I don't know what they want and but I feel like in this space I think folks that are really at the frontier to use an overloaded term on the frontier like I think have a lot of humility around like how can we do this together like we're trying to answer honestly like the most important health policy question of the century which is like how do we assure access to safe and effective AI Um, and that's been that's just been really, you know, incredible to see. And like part of what I'm shining a light on is like, okay, yeah, there's that that whole post market space. That's where we need help. We need like really pragmatic ways across a very wide range of technologies, whether it's an imaging based AI, whether it's a pathology based, whether it's, you know, purely, you know, conversational sort of primary care or specialty based like we need ways of benchmarking and and monitoring those tools. And then underneath that there's a whole scaffolding. So I'll pick up example of something that most people aren't aware of is so let's say hypothetically we had like all EHR data in the in the country flowing through a place like the FDA.

25:54 Amazing. Well, guess what? We still wouldn't be able to evaluate and monitor any AI. And why is that? Because all all of us are docs. We we operate we work in EHRs. That data is not even in there. Right? So when you prescribe a pill, right? You have the pill, you have the NDC, right? You have it eescribe. It's in the claim file. It's in the EHR file. Where is the fact that I used open evidence last Friday? Where is that in the EHR?

26:21 And so it's like you can have all the data in the world, all the EHR data, but if it's literally not there, and if it is there, do we have a unique identifier? Right? So every pill of every formulation and every generic and every type has a different code. Do we have a code that says that this was I'm not going to keep picking on open, but this was this algorithm versioned on this day and this thing. And without that, how can we use real world evidence to be able to do this? So there's like there's like the methodological sort of sexier stuff and then there's like the plumbing part of this which again I believe that if you put that in front of the innovation community we can solve it but we're not really talking about those things and instead like we're we're really focused on sort of those like hypothetical testing environments as opposed to saying guys what is it going to take to support an entire ecosystem to do this?

27:09 >> Yeah. And just it's just related to one of the uh multiple conversations I had. I won't name names, but just talking with health systems of we don't even know what AI tools we're using. >> Yeah. >> Let alone the who's using it, where and how. >> Right. Right. Right. Right. Right. >> Uh and there's been a couple instances where where even, you know, CEO has been let go because they didn't they were asked this question. This question was presented and it's like how can we not even have that answer? I I I bring it up because I actually would >> most and and it's not even a fault on on trying to most hospitals are trying to are starting to put together lists of where their AI is getting used or starting to put together governance communities >> but software is moving so fast this old tool you bought and provisioned right a health system might have two to 20,000 software systems that they work with as these things are put together um and rightly so many CIOS want to reduce that list but every one of those software companies if they're not sleeping should be implementing AI.

28:13 >> We're still grasping as healthcare how do we track this? How do we understand this? How do we move through this? And uh to to your point, I think this is again part of the reason we started talking about this is how we can kind of bring people together to try to solve these problems. >> Yeah. >> And like and it's got to be you have to know it at the system level, but it has to be timestamped. I want to use an old school term time stamp in the individual patient encounter >> because if you want like let's say you're looking at a clinical decision support tool right and you're trying to understand in a postmarket real world way how it's working well you have to know that it was used on Mary Smith these are the inputs into the model these are what the outputs to the model is and this is the version of the model just like very simply so that you can roll the tape and say okay did Mary Smith actually have that diagnosis did Mary Smith actually benefit from that treatment but if if we don't have that um you know like we even if we timestamp that it was used and we don't know the inputs and outputs how can we know if it was right or wrong right I mean so it's really there's a lot of foundational stuff that has to happen in addition to again like what are the benchmarks what are the what's the tooling right what's the framework and then like how do we get access to the data so there's a lot you know of pieces there which I think is all solvable but you know uh we have to be paying attention I mean with such a broad landscape. I mean again this this again goes back to the prior comment I made which is just that this is the kind of conversation that bringing this stuff to light even hearing from you now like I actually hadn't considered some of this and and the the the stat about the CIO like right imagine like the old model which is still an operation of of running a a tech shop in a health system is you know how are my data centers doing? how is my is my software all patched and up to date and you know what am I licensing but but to Justin's point like each of those solution providers their new version could have nine or 10 or dozens and dozens of AI based capabilities that's one side and then there's this again this consumer part that I keep bringing up that's not just the patient as the consumer but it's also the healthcare workforce that is maybe at using their phone or whatever the the solution of the day is so so this is a really hairy problem. I guess one of the things I I was going to ask if at some level do you have to just prioritize because it seems like there's a lot here, a lot of big rocks >> and is that part of that prioritization?

30:36 Obviously, you're hearing from the community. >> Is there also just some some sort of registry or some way that we collectively are reporting >> signal back to you in some way as opposed to waiting or having to have the outreach be extraordinarily proactive? I I don't know. And maybe you're going to tell me this is already a thing, but I would love to be able to say, "Hey, you know what? Like feedback, >> this was a dangerous output or a flagging I, you know, this alert is insufficient for the gravity of the what the alert is for or some I don't know like I don't know if there's an answer here because there's so many players involved, but it almost feels like a way to prioritize all the different things we have to tackle."

31:19 >> Yeah. Yeah. Absolutely. So let me like I think it is is a huge challenge. I don't want to minimize that, but also like maybe I'll I'll like we could like talk about some like there's a lot that we can build on, right? So the FDA's always had a riskbased framework, right? And so I think like a a big part of this is really being very clear that like hey these are the things that are not regulated and these are the things that are and then of the things that are regulated these are the things that we think it really matters to do this postmarket stuff and this is the stuff that we don't right and I think like because I think a that this takes from a thousand things to maybe like you know a couple dozen things right so like for example ambient you know scribes right um like wellness tools right there's places where I think we've already said these things are not regulate or these are places where we don't need that level of oversight. So that kind of helps cut it down. The other piece is that there are scaffolding to build on.

32:08 So like in the implant world, right? So FDA regulates like 20% of the economy. Part of that includes right like you know knee implants and things like that. There are there's a system called unique device identifiers. And there's a whole system if you're a surgeon and you put in a device and you find that it's faulty, like you can put up the flag that says, "Hey, I just put something in that I think really harmed a patient."

32:31 And there's a whole process for when that recall happens. Like just like, you know, car recalls, there are device recalls. Um, and even though they don't change as frequently as software, they actually change more frequently than you think. And the oversight for the changes is not as much as you think. >> Um, the the canonical example is sort of screws like Orthoscrews. There's a ton of companies that make them. Some of them are kind of small. They iterate on those screws. And so we really, I'm not going to say, but a big part of the signal comes from individual surgeons saying, "Hey, I used this screw and it didn't seem to work the way it intended to." And so part of it is just extending some of those frameworks. Obviously, there's some unique challenges, but it's really building on some of those pieces.

33:13 Well, on that on that optimistic note and again, thank you so much for ju joining us on the conversation again which was the reason kind of we started this which is how do we bring people together at >> you know what I think we all believe is kind of the most transformative time for healthcare period uh and and how do we set this up set this up right totally. Thank you. Thank you so much. >> For anyone out there listening that has made some progress on any of these fronts, like you know, go take a look at the um the request um for information that's out there and like definitely patch it. are looking to collaborate and learn from the baston.