0:15 Welcome to our first conversation our water cooler conversations in healthcare AI uh we're hosted we're doing this from uh the perspective of two Stanford faculty members Physicians who are trying to find our way for what's happening next in healthcare and how AI is going to going to change the field uh Matt's the chief scientific officer for Microsoft and Healthcare in addition to his Stanford faculty role and I'm Justin nordon uh CEO and founder of a company

0:45 called qualified Health um and also we teach together a course on generative Ai and medicine at Stanford the the purpose of this discussion here is these are conversations we've had for the past years trying to figure out what's coming next and especially with the pace of development what's happening in the field the news and articles and research is changing by the week and uh when we were asked by uh Stanford to put this together a little bit more broadly we

1:14 said okay we'll record ourselves having these conversations and bring some of our you know best friends and and people across the space um and so today to kick off our inaugural episode we'll have Troy tazbaz who uh recently was a leading digital health for the FDA so we're going to cover a handful of current topics we'll pull up some data we'll see where the conversation goes um and as a note these conversations are our personal opinions and do not reflect

1:40 the opinions of Stanford or or any of our companies or organizations that that we're a part of uh so to to kick things off uh we wanted to start with like where are we at now in in the healthcare AI landscape and I want to pull up one chart and then uh Matt I'll turn to you to kind of explain what we're looking at a little bit here on what's coming next on you know what we see right now for

2:04 effects of using these tools since the launch and kind of where people are getting some games yeah absolutely I mean this is this is a great this is a great paper just just uh from from the past week and and really starts to show uh despite those of us who've been tracking this very closely for at least at least a couple years as gener a started to permeate through kind of just the general Workforce we always knew that

2:26 people were kind of using the models I think we saw a lot of us I think people would talk about it but they wouldn't uh maybe they wouldn't even surface that they were using it I think Ethan mullik had that had that great blog talking about the secret cyborg right people were getting some work done but they weren't like asking for more work right so that was kind of a phenomenon uh but this has been this is maybe one of the

2:47 best recent papers kind of looking at okay let's actually break down all the different knowledge work tasks and actually see what the difference is leveraging you know the the best new model to to to complete their task and how much time they're saving and I mean the takeaway here again I think with some of the caveat that this is very uh you know specifically designed to look for this but you know tripling productivity for knowledge work tasks

3:12 across the economy is is a pretty eye popping result to me um and and I think the other thing that I took away from this was how many people were using it uh and and really have only come around to using it in maybe in the last six months and I sort of feel like you know of course everyone's using it right but but I think it's still kind of slowly making its way and then you know of

3:33 course brings me to the healthcare um and and thinking about maybe just the only study I've seen which is the British medical journal study that talked about you know the GPS that they how many are using in theirin practice so I think it raises a lot of interesting questions um but but it's definitely starting to to paint this pattern that as the tools get better um people are adopting them and and absolutely saving time uh no matter what

3:57 they do for a living yeah and this for for everyone the GP uh study from the British medical journal showed I think it was last year 20% of GPS are using these tools on a at least weekly basis and for clinical care not just in general you know writing emails Etc uh which is which is pretty astounding um and I'll pull up one more thing and then uh try I want to see what you hear on this but what's fascinating

4:27 is that's the backdrop of what's Happening generally in AI uh this was a study done by hyms they released end of December just a survey and you know most people or most medical workplaces uh don't have access to these chat Bots or any training related to AI uh many of them are you know restricted from using AI at at work and so it's this weird like moment in time where at least many people are using these tools

4:56 at work uh at least in healthcare many people aren't supposed to be using these tools at work uh or don't have access to these tools and so you know I had a funny conversation with a um head of an Healthcare organization to to main namelist was like Hey do people have access to tools like chat GPT like oh yeah every everyone has their phone uh which which which you know hopefully doesn't scare Troy too much as

5:21 a regulator and worried about Hippa data violations well I'm no longer the regulator so you know yeah what do you make of this Troy like how do we put put these pieces together you know this is an interesting I between the two charts right and and uh in my previous role before going to the government and then now back uh to a similar role we we used to do this um

5:51 study to ensure that you know our development Investments were appropriately uh you know prioritized and one thing that uh we looked at was like it spend as a percentage of the revenue of an industry and one thing that uh was quite evident is that healthc care tends to be on the lower end of that and which means that when it comes to technology technology Investments technology adoptions it it it Trails some of the other industries that go on so frankly I think the uh

6:22 this maybe speaks to that and the question now becomes is it is the the lack of use because there's just not enough uh maybe approved technology available in healthcare or is it uh more of a systematic problem around whether we're teaching this stuff in in U you know medical school as two people who both gone through medical school but also teach at a medical school you know one fundamental um truth exists which is

6:53 it's still new like being a medical professional is such a knowledge worker it's a it's a it's it's all about the knowledge right and so I think that maybe perhaps tradition is playing into also some of this this lack of adoption uh so I think it's always going to be fundamentally probably two prongs which is are the tools really available and are they Healthcare Centric and second of all are we actually ensuring that the Next Generation medical professionals

7:24 are coming up with uh you know you they have an understanding how to use these tools in their day-to-day you know process of delivering care yeah and and I let me let me interject there because I I I I fundamentally agree that I think at the top down approach that we typically take to health enterprise software training like you know uring Roi all those things come into play I feel like we're seeing a bottoms up though don't you agree like

7:55 you know everyone's got a phone right everyone's using it most likely in their day to day and those that are sort of those that have used it and experimented with it maybe even six months ago a year ago may have dismissed it because of a flaw or hallucination whatever the issue was and maybe they haven't been keeping Pace with what's actually changing because it is so fast it's more like a a river than a lake to me and every time

8:18 you look at it it's it's different and and you know those that aren't really keeping up aren't really getting that those those turns at at bat to to to get the experience to be able to leverage it more I I I I I struggle with this right because like I you know there are institutions like Stanford UCSF and others out there that have set up Enterprise you know hippoc compliant instances of these models for their Workforce to allow them to you know to

8:43 start to explore and and in a very Bottoms Up way but but you know at the end of the day like I I struggle with try to how does this fit into the sort of traditional Healthcare sass box U you know for for all the different use cases it's it's powerful for I I don't know like I don't have an answer but it's part of me feels like the pragmatist or the realist or the fatalist of me

9:03 whatever sort of feels like they're just going to find a way to use it if you aren't able to make it available uh to everyone do you know what I mean I I to I mean and the the the reason in my head is like these tools like this is the chat GPT right and actually there's a interesting graph that we can pull up later off like different usage from the same study we just looked at this is the

9:25 fastest growing consumer piece of technology ever right right and this is marketed to Consumers they're pushing there's all these companies competing pushing to pace and it's so good that it's coming in in a you know Bottoms Up way to healthcare and like we saw this before in healthcare we saw this with like social networks and people figuring out like posting Healthcare information and institutions freaked out as people are posting informations and pictures on social media so we've had a few like

9:54 early starts of seeing technology seep in like consumer first into Healthcare Institution and this is just the next iteration of that I think what's potentially different is the speed at which it's happening and theity um at least from that I'm candidly I don't believe all those 3x graphs that's a little bit too bright for me I don't I get improvements yeah all uh but it's useful yeah here's what I'll also say about I think this technology in general

10:24 about the whole Bottoms Up versus tops down type of thing right I mean if you look at what happened with generative AI right I mean particularly once chat gbt 4 came out is that you know they go okay here's this great new capability where could we apply it to that's a very Bottoms Up approach kind of technological adoption strategy right it wasn't like hey here are the problems that we're trying to solve for whatever the industry might be and here are the

10:52 tools available and and generative AI just happens to be yet another tool for that and I think so I think by by just the how the technology was integrated into you know society and or how it's being integrated in society was very Bottoms Up the other thing that I'll also say and you two know this a whole lot more than I do but you know healthc care system especially in the United States is highly fragmented anyway right

11:18 and so which means that any change any uh improvements any progress really gets done at the B from a bottom as well and so you know when I was kind of wearing a regulator's hat you know I always kind of used to work with all the hospital systems and say you know instead of have the technological providers technology providers come to you and say Here's what we'd like you to integrate why are you not all coming together and say

11:45 these are all the problems that we're all trying to address uh that are consistent across whether it's a large hospital system at AMC or or a community center and I think that that type of uh you have to flip the switch a little bit and say that the hospital systems and the healthcare I guess ecosystem has to be essentially providing the demand signal to say hey technology providers these are the issues that we need to address and the priority we need to

12:15 address them in I really like that comment because that that I think is what we're starting to see I think there there I don't know if we have the paper uh Justus or not but but we can maybe uh put it in the in the notes but you know uh some institutions have said now now that we've implemented this sort of hippoc compliant way to leverage the models for whatever use case across the the Enterprise now now we can actually look

12:37 at a bird's ey view and say well what are they using it for and who's using it for what and then that really does start to to your point like get you know boil down the things that it's actually having an impact for in a very organic way and and then allow them to say well do I want to operationalize that as you know or do I want to look for a vendor for that or or do I just want to have

12:59 folks share best practices and their and their prompts and maybe even have like you know sort of prompt workflows already baked in I I mean I really feel like this is still a very early days um and and you know the part that that I struggle with still though is that you know we we kind of make decisions around where the capabilities are with a snapshot in time again but I feel like it's like again taking a

13:22 picture of a of a movie or something like it's like you're going to look again it's be totally different thing going on and it's going to be like does that does that change everything you know and that that makes it so challenging right it's a moving Target you got to talk you got to talk about this then at least on the performance curve that you're talking about well I mean this is just one of many and I'm sure you know folks in the

13:43 audence have seen a ton of these and and I you know there's literally several dozen right so there's benchmarks that um you know folks have for a long time felt are going to be very very difficult uh for models and as soon as you hear someone say I've created a new Benchmark that's going to be years before AI solves it and then like without question three or four months later you know you'll see one of these these

14:06 exponential curve graphs and I think you know if if I'm being honest I think over the course of maybe the last you know four or five months since we really kind of started to understand you know uh test time compute inference scaling on I'm essentially letting the model think quote unquote uh they really doesn't seem to be a lot of uh benchmarks left that we aren't saturating to the point where you know any further you know

14:31 reference to them is almost not not very useful I still feel and this is especially true in medicine that there's a there's a disconnect between some of the Benchmark that at least we use for healthcare broy Med QA is obviously one and and actually the performance on specific real good you know use cases that will actually drive impact but but nonetheless to me it's still is a signal I think and and maybe a call to action

14:56 because what I would like to see and again I love the GP QA I love the SWA bench verified I love all these benchmarks where are benchmarks in healthcare of similar things for tasks that we actually really do think are hard to solve but are are are really important and then I would love to see a bunch of these curves doing that for those too you know what I mean and and that's the one thing I I sort of I'm a

15:17 little jealous if I'm being honest I kind of look at these I me man coders you know they're they're killing it right it's like what do they say now uh Sam recently said what 03 is you know amongst the top whatever 100 50 coders in the world or something like that like I would love to have a model that is you know top position in whatever specialty um with a really good Benchmark to prove that out well well people are trying to talk

15:43 about this with 01 and case management tasks for for things like this where it keeps getting better and better and um you know Jonathan Chen one of our Stanford faculty has been putting out paper after paper of hey actually you know we all thought AI plus you know humans would be better than just AI or humans and actually we're finding at least on the benchmarks currently that we're talking about in testing against AI on its own is doing better without

16:12 human inos interaction and so how do we how do we make sense of that are the benchmarks correct are are they right are you know like I guess the flip side of that is do we even need those benchmarks anymore Matt like the AI are AI is winning it's it's very hard for me to wrap my head around this because like you know I've been doing Health AI right building models for like what over 10 years uh of all different shapes and

16:37 sizes and uh every time I'm on a panel or I'm talking to a class it's always like AI plus human is going to be better than you know human or AI alone I I I it's actually like it's ingrained into me that that mentality and I yet I cannot find a really well done study that that shows that right instead I see studies like the one you're showing here uh which is that these models do really

17:04 well on their own and they do better on their own that Physicians using the model now I don't know I'm sure there's some aspect of human computer interaction there's some aspect of the Benchmark itself like you know and there's probably an aspect of just like learning the familiarity I think all of us know it takes a while to start to understand how to use these models effectively Etc but but it still doesn't change the fact to me here where I'm I'm

17:28 I'm begging the Community to to prove me right otherwise I have to go back and like correct so many statements where you know I was convinced that AI plus plus plus Physicians was going to be better than either one alone I don't know I mean Troy I don't if you have a take on this U and I want to know your your your like regulatory take too or like how were like we've always been in that Paradigm but then we're kind of

17:51 seeing data differently like how how are you thinking about this well I I I think it's really hard to [Music] um determine what is it better at I mean like kind of looking at this uh this particular chart and then having this conversation it's like is it better at just being a physician or is it very very specific tasks that it does actually uh better right so I think without that context it's it's kind of

18:23 hard and I think these are also in my opinion um I I don't want say they're dangerous but it's more of a it's a it's missing the point right I mean the point is that and I've said this before and we've talked about this before it's like if you start kind of pitting AI against let's say some sort of a field and in this case it's it's medical professionals what do you think the adoption curve is going to

18:52 be right right and and and maybe that's why the the previous chart actually says what it says and so I think the question or the dialogue should really be is that where can we apply this that it's going to perform better than human and for various reasons where the human can then focus on the things that AI can't do right or it could um solve for the huge labor shortage problem it's like I I think the The

19:22 Narrative when I see these types of uh studies it it to me it's very academic it's not it's not in my op uh real in a sense that it's not solving the problem right sure it's the analogy that I use uh often where it's like you know you can put a Ferrari on rush hour traffic on 101 and and is it going to perform better than something else no probably you're going to get to from point A to

19:52 point B at the exact same time but you just spend another $20 something thousand dollar more getting there and so I think the narrative needs to change that we we we need to stop comparing the two we need to say that where where is it applicable where can we apply it to that is going to be beneficial for both the uh people who are working in the healthcare field but also uh the patients which is the ultimate you know

20:16 goal here and I think that's why most you know medical professionals got into the profession to begin with you know from a regulation um well I actually would love to hear from your point of you because you know both of you are are Physicians one of you really are practicing and so um exactly and and so the question for you all is like how do you feel about this stuff well I'll give

20:46 you I'll give real physician I give give a comment from a medical real phys class together no which was hey you you just asked this very simple question of like Professor Norton you know historically you know doctors had all this knowledge that were unique in their head and we would have this conversation and build that trusted relationship with the patient where there was a lot of knowledge transfer happening in inter action it's like watching the fields and

21:14 things we're talking about in class our patients are going to come in knowing more because they spent two hours going through every part of their disease and what's happening than we are like what is our role and so like just like those questions as you start to answer at least for the medical knowledge answering tasks like the AI is better today but then it's like what is that role question and that's where I I'll tee you up Matt with with how you want

21:40 to take that now as you're practicing today I mean I so I have I have like I live in kind of two it's like you know yes like I still walk by a fax machine at work I still like sometimes pause and reflect on what I actually do in a clinical day and I think about what I used to do when I started and how different it looks in terms of the number of things that you know every

22:03 couple years there's a new thing or a new form or a new process or whatever and then it's just like we've got this huge multi-layered you know onion that we have to peel back right to to sort of stop the bleeding I think in in the healthcare Workforce and so to Troy's Point yeah the benchmarks are telling one story but the reality on the ground is I will take anything that will help me peel back that onion and get me back

22:25 to taking care of patients the way I want you know the way I was was trained to do the way I the way the reason I went into Healthcare you know um and I think you know we've seen some of that right there's some there's some solutions that are purpose built for a to solve a problem and it doesn't lead with quote AI right they're like the ambient notes I think is a great example there's a bunch of these out there uh

22:45 and they're they're changing the game I think I mean I think they're having a real impact partially and again it sounds like a simple solution but it's a real problem I mean you know how much time folks spend on you know administrative note taking and there's so many more I think that are low hanging fruit on the other hand when I see these kinds of you know to to to Justin's point that pH you know a

23:09 patient can spend a couple hours with the model and you know in theory again uh ignoring some of the challenges there walk away with a pretty good download of the concepts and principles of their disease or whatever their conditions I mean that that to me to take your analogy Troy of a Ferrari on a freeway now I'm looking is it is that actually not a Ferrari but maybe it's a helicopter and and the way that we're

23:32 implementing it is we're trying to take it down the freeway as if it's a car but maybe this just a fundamentally different technology that does have the opportunity to fly to our destination but we don't have we we don't have you know the language the ability the the infrastructure necessarily to to take advantage of it I I just don't know yet I I feel like so I kind of live in both worlds a little bit right I see this

23:53 huge opportunity acceleration and but I also see that we just have real simple and you know burning problems to solve now yeah but here's the thing though you're you're actually now talking about the concept of of completely reimagining this the healthc care delivery right doesn't have to be the way it is right now it's it's very episodic in many ways right I mean you get sick and you you you go in and see you know some sort of

24:17 a medical professional and I've always said this in the past it's like we don't have a Health Care system in the United States we have a we have a sick care system right we really you know Healthcare is really when you're engaging with your your health when you're still healthy and so but so it could do a variety of things right I mean and just effectively um self- agency uh as you when you are navigating through sick care is is definitely one

24:42 of them and you know I have obviously a personal experience in that one and and I used to remember saying that like here I was you know I I kind of dealt with complex problems at that point in corporate world I managed uh you know very high- performing individuals uh and so and I remember going like how does someone who doesn't have my skill set ever navigate through that that system without having any knowledge deep knowledge about this stuff and so maybe

25:11 in that sense it's going to be fantastic right it's going to provide agency to a patient or a or a caregiver um as they're navigating through that system and the problems that come along with you know navigating chronic conditions how and and I know we only have a few more minutes together but how do we G of contrast that Troy and that potential you're talking about

25:41 with how that fits in a regulatory framework and software as a medical device like I know you're you're not not there anymore but like how do we even start wrapping our heads around how we would get that you know helicopter different path with Healthcare with current regulatory paradig I I I think you have to in my opinion look at this end to end thing right I mean I always kind of looked at Healthcare as a as a value chain uh from

26:09 that starts with uh maybe and you can look at different starting points Medical School drug development medical device development and all through that and regulation in my opinion is just one part of a a critical Milestone of bringing you know healthc care to to the market itself and so you know when I went into the government and my perspective of Regulation were really just um I kind of said it's just a a fancy way of saying uh guardrails

26:39 it's a parameters where Innovation needs to happen within and there's not a single other industry that doesn't have those parameters and sometimes those parameters are driven by laws of physics and um other times they're driven by something else right I mean uh just like we have safety parameters when we're driving or Aviation or whatever that may be and so regulation in my opinion should always kind of look at this thing and say okay what are the safety

27:05 parameters that you have to establish in order to make sure that you're taking advantage of of all of this this Innovation that is happening but you're also reducing the risk of that and so regulation shouldn't just be done or be considered by the government the government's angle is about coming up with regulation but it has to work with the industry say what are the common sense kind of guard rails that you have to establish so you know when when I was

27:33 working in the government that is the that's the tech that I took to say that you know anything that we do here has to be effectively implementable by the industry itself because if it does if it can't be implemented it's just a piece of paper at that point and so you know we looked at we focused on things like life cycle management make sure that you know you truly understand how these things are are not just developed but

27:59 integrated into healthc care system how it's being the value is being delivered whether it's a standalone product or you know maybe through something like an EHR and then how do you monitor this thing and so like the biggest thing about uh generative is that it it continuously adapts right it's continuously learning that's the that's the main difference and so that means that you have to have uh the monitoring capabilities that goes in with that ongoing learning process

28:29 I I frankly I come from the Enterprise kind of software world and it was kind of surprising to me that you know monitoring whenever I would talk about monitoring that people would look at me funny I'm going I don't ever remember releasing a piece of software without actually having monitoring capabilities and why is this any different is the question that I would I would pose well welcome to healthcare where no sof has monitoring capabilities well well uh Troy thanks so

28:57 much for for coming on and being our inaugural guest to kicked us off uh you know unfortunately we won't have time to hear all of Matt's uh wonderful comments on deep seek uh and and everything it means for the field but I think there are plenty of comments online that that people can can read about there um but uh thank you guys for for joining us here and uh if if this is we're going to keep doing this for at least a a few

29:22 more times and and if this is interesting uh let us know what we should cover it's always fun to to have these discussions with you too so I always enjoy it good to see you Troy