Transcript
0:05 So today we're pleased to have Tatsu Hashimoto here with us. Tatsu did his PhD at MIT, did a postdoc at Stanford, spent 1 year as a researcher at Microsoft Semantic Machines, and he's joining Stanford as of last month as a fresh assistant professor. So welcome to welcome back to Stanford. He'll actually be teaching 221 in the winter. so if you like his talk, you should go tell all your friends to have them take 221 in the in the winter.
0:37 So Tatsu has worked on a number of areas from computational biology, text generation, and NLP, but he's probably really well known for his work on you know, robustness in machine learning. And I think throughout this course we've emphasized that machine learning is something that's really being deployed in the real world all over right now, and having real impact in the world. Just last week we heard from Tina Queler about this. So I think robustness of machine learning systems is a really really important area, and Tatsu's an expert in this. So I'm really happy to have him tell us what robustness in machine learning is all about and where things are at the moment. So take it away, Tatsu.
1:23 Okay, great. So I want to start with emphasizing sort of what Percy already said, which is that there's been this enormous and rapid progress in machine learning over the last decade or so, in especially in tasks like image recognition. 10 years ago ac- or errors were at the level of like 20, 30%, and human-level performance was, you know, sub 7%. And there was this huge gap in performance, and everyone said it'll take a long time to reach human-level performance. But nowadays really human-level performance is being achieved on all sorts of tasks, image recognition as of say 2015, but also in tasks like natural language processing and much more challenging reasoning-based tasks.
2:08 these systems are now getting really close if not exceeding human performance. And so, machine learning has really achieved these sort of great successes and they're being deployed and we can sort of ask what has machine learning been good at and what is it good at? And it's really good at extracting patterns from training data and applying this on a test distribution to do some prediction. And so, we can think of this as, you know, classic digit prediction task. You have some images of digits and you need to return the, you know, numbers that are associated with them. As long as sort of the source and the target distributions look the same, modern machine learning systems based on large amounts of data and neural nets are going to do exceedingly well on these tasks. But, really the challenge is what if the training data doesn't look very much like the test data.
2:55 in these cases, we're going to have a lot of challenges. So, on the image that I put here, you know, in the source domain, we have these like black and white images and sort of desaturated settings and now at test time you have these yellow cabs in New York and, you know, your predictions might not work so well once you have this what's called distribution shift. And so, once we start to think about going beyond just sort of data that looks like the training data, we see a lot of problems on the horizon and we've discovered a lot of these problems beyond test set accuracy.
3:25 and I'm going to at the beginning of this talk cover sort of three classes of problems that hopefully you'll think about as you sort of continue on your journey in AI and machine learning. The first one is sort of discrimination and performance on minorities. Another one is vulnerability to adversaries in high-stakes secure applications. and then last one, which is a little bit more abstract, but I will get into this in more detail, is that models don't really display a understanding of the task that they're actually performing. And this is going to be a little bit abstract, but because it's the AI-focused class, I think this is an important thing to be discussing and going through.
4:03 And so it's sort of the unifying theme, like these seem like very different problems, right? Like problems that machine learning systems have today, but really they're all sort of connected with a single underlying theme, and that many of these problems can be cast as these problems in robustness. and so when the training distribution and the test distribution are different, these models break down because they're brittle. So to start with, let's talk about sort of discrimination and fairness and minority groups.
4:30 So a really typical thing that happens in a lot of machine learning systems today is that there is sort of a majority group. let's say, you know, Western cultures, English text, or sort of males in many cases. So in this majority group that dominates the training data, you get extremely good superhuman performance in these systems. and often you're going to be deploying this to a wide variety of users, and so you will have minorities using your system. and in these cases, you end up with horrible sort of near-random performance. And you can sort of immediately see how this is a discrimination issue and sort of an equity issue.
5:07 And I'm going to go over a lot of these examples in turn, but these just show up in all sorts of places that you might not initially think about when you think about fairness problems. Like say, a dependency parsing, or video captioning. face recognition is a very common one that people probably already know. but in these sorts of like common, widely deployed ML systems, you start to see these gaps between how these systems perform on majority groups versus minority groups.
5:34 So the first one that I think is probably maybe surprising to many people that there's these kinds of gaps. it's a task called dependency parsing. So the input is just sort of sentences tokenized and sort of split up. So an example here is bills on ports and immigration were submitted by Senator Brownback, Republican of Kansas. and the output is that you're supposed to analyze sort of the syntactic structure of this sentence and create dependencies between what are called head words and their dependents. And so, you end up with what looks like a tree here.
6:04 and so, the sentence above, like the bills on ports and so on, can be parsed into this tree-shaped structure here on the bottom. And so, this is called dependency parsing because there's these explicit dependencies between tokens that show up in your data. And in sort of classical NLP pipelines, such as say, if you want to extract relations between people or entities, you know, who was the person that submitted the bill in the sentence, for example, you might use something like a dependency parser to look at dependencies in your sentence and to extract relations, right? So, this is sort of a first step in terms of getting these kinds of more sophisticated analyses in these sort of classical pipelines. Nowadays, many things are sort of end-to-end and neural, but that's sort of beside the point here.
6:47 And what's sort of surprising, or maybe not surprising if you've thought about these kinds of problems, is that these parsers do much, much worse on data that's not commonly used to train these dependency parsers. So, this is a study from Su Su Lin Blodgett in 2016, where they took a bunch of different dependency parsers and applied them to text from Standard American English as well as African American Vernacular, and that's the column labeled AALAS. and the performance here is measured by what's called label attachment score, so that's how well do you reconstruct the tree?
7:22 And the numbers here, you know, you might not really know how to internalize this, but you see these big gaps, right? So, in terms of Standard American English, you get these 57 sort of F1 score type accuracy, and then 43 in African Americans. And that you get a 14-point gap. And sort of state of the art for this task, say, you know, you're competing over like one-point difference. So, these are enormous gaps once you go from Standard American English to African-American Vernacular.
7:47 And these kinds of things can have huge downstream impact if they're used in things like relation extraction or QA systems, right? Because text from African-Americans are just systematically not going to get extracted into say relations or entities when you build knowledge bases and things like that. And so, you might sort of see how this begins to affect these kinds of minority groups through these kinds of robustness problems. Another example is video captioning. So, many of you have already interacted with systems like this through YouTube's video captioning system where the input is, you know, you have a video with some spoken text audio and the output is text captions that are automatically added to the video.
8:32 and these things are increasingly important because say if you have I know that in medical domains, if you have Medicaid-funded sort of videos that you need to put up on the internet, you need to have captions. And so, in these cases, you have you either run these systems or you have people transcribe the videos. And what's been found is that these kinds of systems work a lot worse for women. so, this is a study by Rachel Tatman in 2017 where she basically showed that if you took male versus female speakers and you random through YouTube's video captioning system, you got systematically higher error rates for women. And you see that sort of the median error rate is essentially the upper quartile error rate for men. And so, that's actually a pretty substantial difference in the word error rate between these two groups.
9:17 and you also see sort of expected differences between dialects, which is, you know, Scottish speakers get substantially worse video captioning accuracy whereas, you know, people speakers from California get really good word error rates. And so, you know, you can sort of see how this sort of manifests, right? YouTube being based in in obviously dog fooded with people with Californian accents. and when tested out of distribution on Scottish speakers, suddenly performs a lot worse. And so, this is the kind of robustness problems that you initially don't think about because you sort of think about, well, is our model performing well on, you know, really a complex input. And so, you might put in some really complex inputs as a California speaker, but really you haven't tested out of distribution on Scottish accents.
10:05 And then we'll come to another example, which many of you hopefully already know, in facial recognition. This has sort of been really widely discussed even in the media. And just to go over what the task is, the input is images, possibly containing a face or not, depending on the task. and you can do many sort of things with these images, and there are many outputs that are associated with face recognition or identification tasks. and so, you might ask, is there a face in this image? And that's sort of face sorry, recognition. you might need to match a given face to a database of faces, and that would be identification.
10:41 or you might need to predict attributes. Is this face a female face or a male face or, you know, happy or sad? You have many sort of attribute prediction tasks that can be built on top of faces. And this is one of the original studies, I think, in terms of highlighting how bad these kinds of systems can be in sort of widespread ways. And so, there's a study from MIT Media Lab, Gender Shades, and Joy Buolamwini Buolamwini in 2018, where she basically took a whole bunch of portraits of legislators from different countries, African and I think Northern European, and ran them through different face attribute prediction systems for whether or not they were male or female.
11:28 And what you can sort of see on this on the top right is that dark female skin results in much worse gender predictions compared to light-skinned males, where you basically have perfect prediction. And these kinds of things are pretty problematic. If you've been testing your systems on light-skinned people, you think your system is near perfect, and so you might be using it for really high-stakes tasks, where you need 100% performance. But then, when applied to these sort of darker-skinned demographic groups, you end up with substantially worse performance. And so, you're not you don't you don't even realize the kinds of harms that you're causing by using these kinds of systems.
12:11 And what's sort of problematic, and sort of you can see, is that they reflect a lot of the benchmark data that's been constructed for this task. And so, on the right on the right bottom here, you see the distribution of sort of skin color and gender for benchmark data sets in this kind of like sort of gender identification from face image tasks. And what you see is that there's sort of a sister systematic underrepresentation of both females and darker-skinned demographics.
12:42 And you might say, you know, this really just reflects the underlying data distribution. And so, maybe all we need to get is, you know, unbiased data. You hear this term a lot from, I think, people who haven't thought too deeply about problems of robustness. But the issue is that there's really no such thing as truly unbiased data, in the sense that there will always be an underrepresented group, if you slice your data fine enough. So, we need to really just go beyond thinking about balancing the data set. And we need to think about how can we make our models work well, even for really small groups, really small demographic groups, and even individuals.
13:16 Another task that's has these kinds of issues is language identification. So, as an input, you might be working at Twitter, and you need to identify the language of a tweet so that you can run a machine translation system and automatically translate a tweet into someone's sort of speaking language. But, in order to do this, you need to first identify what text the tweet is written in, right? And so, you might have a lot of different kinds of inputs.
13:41 And this figure one shows the challenge in this task. So, you might have dialectal text. So, the top one is Nigerian English. the second one is sort of sort of Irish tweets. And the last one you can have code-switching. So, you can have a mix of both Indonesian and English. And so, in language identification, when you're given these kinds of tweets, you need to identify the source language that they were written in. And so, the output of the task is the language.
14:08 What's sort of been identified is that there's systematic biases, once again, in language identification. And sort of one that's immediately a little bit troubling is that African American English often gets identified not as English. So, is there sort of like a implicit normative judgment being made here that African American vernacular is not English? And you see this error rate here as AAE having almost double the error rate of language identification compared to a more standard American English data set.
14:37 And you also see this across languages. and this is a study by Jurgens et al. in 2017, where if you sort sort of the languages by the human development index of the countries, you see that there's this decreasing recall or decreasing accuracy as the countries get less and less developed. And that's because as often these kinds of countries have under-resourced data sets. And so, there isn't as much data with which to train these language identification systems.
15:05 So, you see these systematic biases in terms of how well-developed and how internet-connected these countries are. And this this leads to sort of representational harms, right? Like if you're an African American English speaker, and a system, you know, tells you that you're not speaking English, that's kind of harmful. And there's also utility harms, right? Like if your text doesn't automatically get translated to English, you know, you're going to your tweets won't reach as wide of an audience.
15:29 And so you can think of these as having sort of pretty serious implications for fairness as machine learning becomes more widespread and more useful and more impactful. And so there's these problems of serious sort of active discrimination where this was a story in the New York Times where a face I recognition system identified a person as being a criminal and this was faulty and this was essentially the only reason for arresting this Michigan man. And so if you have a system that's much more error prone on African-Americans, you're basically going to have a much higher error rate when deploying these kinds of algorithms. So you have these active discriminations and harms that are being done.
16:08 But on the other side, right? We think that, you know, as people taking machine learning and studying machine learning, we think that these kinds of technologies are broadly beneficial and useful and increase efficiency. And so, you know, there's a study by Eric Björn Ylvisaker which says, you know, the application of machine translation systems increased, you know, exports on eBay by 17.5% because it's really easy to translate text and so people from other countries can buy your product. But if, for example, language identifiers can't identify your language and so you can't use machine translation systems, then you don't get these benefits, right? So you get unequal access to sort of the fruits of these kinds of AI systems. And so this can lead to sort of harms in both ways. You don't get access to the benefits and you get these kinds of active harms from the errors that these systems make.
16:58 and I'm going to stop here because I think fairness is a topic that many people have feelings and comments about and I'd be happy to sort of just sit around and discuss for the next couple of minutes if anyone has questions about sort of how fairness and these sort of robustness questions interact with each other. Yeah, that's a that's a bunch of questions in the chat. You might take a look. Yes. sorry, I have full screen, so I'm going to give me a moment to pull up the chat.
17:25 Yes, okay. okay, I see it now. okay. So, I think there I'll start with the first question, which is on having unbalanced and biased data is not enough. And so, this is a very subtle point. So, there are data that you can construct that will make you robust to certain kinds of groups, right? Like so, let's let's go back to just this slide. so, if we look at this sort of distribution of data, it's clear that at least for this like top one, Indians, we're probably going to have some sort of bias in terms of light versus dark skin, because dark skin is so underrepresented. but if we balance this data out, right? We might still have sort of unbalanced unbalanced demographics in certain other areas, right? Maybe it's not dark versus light skin, but maybe it's geographic region, it might be income, right? These kinds of problems are innumerable. And so, what you really need and not is not necessarily the search for this un- reachable perfectly balanced data set, but sort of a model that can sort of do well on small data sets or small amounts of data. So, you want to have a model that can take in these kinds of imbalance data and do well both on the male and or sorry, the dark and the light skin. And the important thing in this task is there's no real trade-off, right? Like there's no real reason that you can't do well on both the light and dark skin. And I think that's the sort of crucial structure here. Like if you can do well on both groups, then it's not really about the amount of data or the distribution of data. It's more about the model and sort of how you're learning it.
18:58 And the second question, is there a way to audit models without having access to the model? So, that's a interesting question. I mean, I'm not sure if you meant by this like access to the model's outputs or something else. Like if you have access to the models' outputs, right? You can perform a study like gender shades where you run the model on certain sort of challenge examples and you look at what the error rate is and you say, well, clearly we are doing much worse on dark skin females than light skin males, so there is some sort of sort of bias.
19:30 So you can audit models that way. It becomes much harder to audit models if you can't execute the model on your own data. Then you'll have to do something a little bit tricky and it requires very specialized conditions, I think, to be able to audit those kinds of models. Also, feel free to ask follow-up questions if I didn't answer any of these. So similar to the issue with a person in Michigan, there has been efforts in applying AI to model future problem human behavior. oh, this is a comment. Yes.
20:00 And that's highly problematic. I think in one of the earlier slides or talks, there was a discussion about sort of how amplification of feedback effects are really insidious. And yeah, so predicting the future and actioning on predicted future behavior is even more problematic than the task I described here because acting on the real world will change the outcome, right? So if you predict that crime will happen in a certain area, you assign more police and you find more crime, that's going to lead to a pretty vicious feedback loop.
20:26 So you need to really think about sort of the whole socio technical system rather than just the classification system narrowly narrowly when you're in those settings. The last one, it seems that we can always slice our data into more subpopulations to test for fairness. Are there industry standards for what we should usually start with? that's a great question and also a really important academic one. so there is a sort of easy answer, which is that a lot of research and a lot of industry work has focused on sort of legally protected groups.
21:00 and that's a well-defined set of attributes that you can't discriminate on. And so you can group by those, you can group by intersections of those, and you can say those are the groups I shouldn't discriminate on. but sort of academically this seems unsatisfying, because why should those be the only things we care about? and there's a lot of work on sort of individualized fairness, making sure that you do well on sort of individual people, make sure you treat people that are similar similarly, and things like that. and that's a whole active area of research, and sort of not really something where there's an obvious and clear answer yet.
21:33 Okay. any other final questions before I move on? Okay. so now I'm going to move on to sort of the second point that I talked about before, that machine learning systems aren't really secure and can't really be used in many high-stakes situations. So I'm going to start with one of the most well-known examples of this, called adversarial examples. Where on the left we have an image, this is a panda, and a classification system gets this mostly right. It says panda with 57% confidence. That's great.
22:11 Now what we're going to do is we're going to add a very specially designed and visually imperceptible perturbations. So this middle panel looks like complete noise. We scale it down so that it looks just like zeros, and then we add it to the panda image. And we get the image on the right. Now we run our image classifier, and what we get out is it's almost certainly a given, which is, you know, completely wrong. And so what this tells us is we can find visually imperceptible perturbations that lead to very confident misclassifications. And I'm not going to, you know, show you the results of this adversarial example stuff, but you can do this to almost any system, and you can completely and catastrophically destroy the accuracy of all of these systems. And this also happens in NLP systems and so on. And so this is a really sort of hard to avoid and almost sort of universal behavior.
22:58 and I want to show you how, you know, sort of robust this kind of behavior is. and so it doesn't have to be images on a computer screen. It can happen by putting little black and white patches to a stop sign. And so on the left, our system is going to classify that as a yield instead of a stop sign. the middle one is a fun 3D printed toy where if you try to run an object recognizer, it will say gun from almost any angle.
23:26 And the right one is adversarial sticker where if you stick it anywhere and you take an image, it's going to say that it's a toaster instead of a banana, which is what it should be. And so these are very many different formats, but you have this same and kind of disturbing phenomena where you know, it's obvious to us that something shouldn't be tricking us. Like, you know, black and white patches that that are that small or like weird texture on a turtle shouldn't really be fooling us into changing our predictions, but it really fools these image classifiers. and when you first see this, you think there must be a really simple patch, right?
24:01 Like, maybe you like run it through a JPEG compressor, maybe you add a little bit of extra noise to every image. and so this has led to an enormous number of papers over like a hundred or so over the last five or six years in which people have tried a lot of different things to defend against these kinds of what's called adversarial perturbations. But the problem is that every time someone comes up with a defense, soon after someone's going to someone breaks it by finding you know, a better attack or even somewhat more disturbingly just running the all the attack for longer.
24:35 And so it kind of seems like this is a really persistent and serious phenomena. And I think the recent view of a lot of these adversarial example type problems is not that, you know, there's some really degenerate artifact about the way we train models or the way we optimize things. It's really just the fact that there are a lot of ways to have a high performance prediction system. And many of the ways in which we can predict accurately rely on these what we're going to call non-robust features. and so when we try to say classify a dog or a cat or so on, we as humans rely on these what we're we're going to term robust features, right? We try to identify eyes and snout and these parts.
25:21 and so these kinds of things are pretty robust to pixel level perturbations. But actually, you know, low level textures and really small image patches are also very predictive of classes of dogs versus cats, let's say. And who are we to say, right, that that's like an incorrect way to make the predictions? Because when we train the model, all we're saying is, you know, just classify these dogs and cats well. and so you can think of this as saying, you know, our problem is under-specified and there are many ways to get a good classifier and some of them really rely on the use of these sort of like non-robust features.
25:58 and this has kind of serious security implications, right? if you're trying to make a self-driving car system, you know, the stop sign being classified as a yield is pretty bad, you might run over a pedestrian. And this really prevents the application of machine learning systems in things like self-driving cars or at least we should be very hesitant if we believe that these kinds of problems are inherent, right? Because the world is kind of designed so that they're really easily perceptible to humans and not necessarily designed so that small perturbations, say by putting on stickers, don't change stop signs to yield signs.
26:38 and in other cases, right, vision systems, I think, are being increasingly being used in high-stakes applications. Like we might reasonably imagine, say, right, at a TSA checkpoint, there's a camera that's running and it tries to identify whether or not you have a gun, right? And if you can make these adversarial examples that say make a gun a not gun or a turtle a gun, that seems very problematic, right? We can't use vision systems for those kinds of high-stakes applications that we might want to use them for. and so both of these really pose challenges for the use of machine learning in these like sort of high-stakes life or death kind of settings.
27:13 and I'm going to stop here to take questions about adversarial examples for the next couple minutes. Do I know why the first example was classified as a yield sign? that's a good question. I mean, with all these adversarial examples, the reason why they're being classified as yield is pretty confusing. I think, well, for example, like why is this turtle being classified as a gun? I'm really not sure. It doesn't look anything like a gun and the textures don't look like a gun. the way these things are constructed, right? Is they're constructed by optimization process.
27:50 You're basically looking for perturbations on say a normal turtle texture that lead it to being a gun. And so there's no real interpretable reason why say this looks like a yield sign or this is being classified as a gun from every angle. can I train these train on these examples to correct classification? oh, sorry. I I skipped a question. How do we define the non-robust feature versus yes, okay. So this is a ad hoc definition. So the split just here is just whether or not a feature can be flipped by changing the image slightly in pixel space.
28:28 and so that's really the working definition here of robust versus non-robust. and I think if we were being more precise, I think this should really be split as saying visually imperceptible is non-robust and visually perceptible is possibly robust. And I think that's sort of a pretty reasonable split. Anything that humans really cannot visually tell should not be being used as sort of features as inputs to reliable prediction system. can I train on these examples to correct classification? I'm going to try to interpret this question because I'm not 100% sure. I think what you're describing is then idea called adversarial training. So the idea is to basically instead of training on just the input image. Let let's just go back a little bit. Instead of training on the pandas, right? We try to train our system to basically classify this image, the adversarial image, as being a panda, right? And you might think, "Okay, this is good. we can now make this a panda."
29:29 But now we need to prevent some other sort of adversarially designed noise from making sure that we look like a panda, right? Because there's probably many many many different attacks that will change this into a panda. And so the idea that you're describing here is basically called adversarial training. And I think that's in Yes, that's one of the earliest defense approaches. and it's empirically reasonably effective. but it's you can still attack it by sort of more sophisticated methods. You can find still visually imperceptible attacks after adversarial training that break the system. So this is not really a foolproof way of trying to make models for more robust. It's better than nothing.
30:12 are you saying yet unfound defenses are needed for ML self-driving cars to be secure from nefarious attacks? I think this is a great question. I think I was I'm being a little bit too aggressive in terms of the things that I'm saying, right? sort of it's a open question whether or not these kinds of attacks are really feasible in the real world or whether or not they're things that we should worry about, right? Like in the real world, I can easily cut out cut off a stop sign, you know, using a saw. And that's an adversarial human attack. but we're not too worried about that attack, and so maybe we shouldn't be worried about adversarial attacks on self-driving car systems. But I think there two things that this highlights.
30:53 One thing is that we should be a little bit careful when we deploy these self-driving car systems, right? Like we should have fail-safes that, for example, don't rely just on vision. that seems pretty important. We might want to use like radar, lidar. Radar doesn't work on stop people, but let's say lidar. try to make sure we're not going to run over people when we misdetect the stop sign, right? Like having lots of orthogonal checks becomes increasingly important once you realize that there are ways to fool these vision systems.
31:18 and I think people are working on sort of provably robust machine learning systems. maybe in settings like sort of military applications, those become truly important. and so there is progress on that, but it's just that provably robust systems achieve much worse average accuracy than non-robust systems. So there's this big gap right now that we don't really know how to close. Okay. In my opinion, is research now shifting towards reformulating models to rely on robust features instead of finding ad hoc defenses?
31:48 that's a good question. I think there is still a big gap in terms of provably robust defenses versus these what you might call ad hoc defenses that work well for like, say, one or two target attack types. but I think there's things like randomized smoothing and procedures like that now that sort of get the best of both worlds in some sense. They're getting increasingly they're provably correct in some framework, and they're getting increasingly better. and so I think for high-stakes applications, I think we'll end up in a place where we'll lose some average case accuracy, but not catastrophically so, and we'll still have sort of adversarially robust models.
32:27 That's sort of where I imagine the field will go. it does seem like ad hoc defenses keep getting broken, so that's not really a path towards like truly robust systems, even though they might make for more useful systems overall. If, for example, adversarial training leads to a more interpretable latent features. does producing adversarial attacks require access to the model? If so, isn't this just an issue of info security equivalent of I I can't parse that second sentence, but yes, I agree with this general sentiment as well, right? So, if you need access to the internals of the model, then really at that point, you know, you've rooted the system if you're attacking, say, a medical imaging system, or you've like have access to someone's car and if you're the Mossad, you can probably like mess with their brakes, right? and so, it's true that those attack models are pretty obscure and weird, but there are for what are called white box attacks, which only require to evaluate the model for one. And for two, a lot of systems are shared, right? So, you only need to learn to attack them once. And so, if you're trying to attack Tesla's auto driving system, you need to get a Tesla, you need to figure out, you know, the adversarial sticker that's going to make your system go haywire, and then you need to like paste that everywhere, right? That doesn't require a particularly sophisticated threat model in order to execute.
33:52 So, I think there are models in which these are sort of real and problematic, even though I think there is a lot of, you know, questions and debate about whether or not we should really care about this, like cost-benefit tradeoffs of robust versus non-robust systems and so on. But, it's an important thing to keep in mind, right? Okay. Any other questions for the adversarial examples part? Okay. And so, now I'm going to get to the last part, which, given that this is a AI class, is maybe the most important of the three failures of sort of machine learning now in terms of robustness.
34:33 and I think it's one of understanding and people sort of throw this word around a lot that models don't really understand and it's hard to pin down what understanding is, but it's very easy to show when models don't understand. And so, we can go through some examples here. We'll go through them again in more detail. but this is from an overview of what's people call shortcuts in the citation on the bottom. and for example, let's say we're trying to caption the this image here and so we need to, you know, describe in text what's happening. But really sometimes these systems might just use the background instead of actually recognizing hills and skies and sheep.
35:11 adversarial examples maybe, you know, because we're recognizing textures and not actually recognizing the shape of things like teapots. And if we're doing medical image diagnostics, you know, we might be looking at markings on x-rays like what hospital the x-ray came from instead of actually performing prediction. And so in all of these cases we're making use of these pieces of information that shouldn't really be central to the task. Like they're not the core prediction task that we care about and somehow the model has picked them up and learned to do really well. and I'm just going to group this broadly under this label of shortcut learning.
35:46 and the way to think about this is that when we train models in machine learning, we're training them to do well well directly on the training set. And these days we now expect them to do well on the test set. But really what we would ideally like them to do as sort of systems that reason and understand and so on is that they perform well, you know, on these like challenge sets, like really difficult examples that we've constructed to break the model.
36:10 and so you can think about this as, you know, there's a lot of possible rules that work well on the training set and there's fewer that work well on test sets and there's, you know, very very few that work well on sort of these challenge sets, the intended true reasoning that we would like our models to extract. And so, we can think about machine learning today as you know, we've gone from this like tan colored circle where we were before to this you know blue colored circle where we are now where we do all the test set. But where we really want to be is still further. We want to make sure we learn the right sort of mechanism.
36:45 and you may have heard this classic AI story of tank identification. And I think this is like a really old like you know Cold War kind of story where I'm going to read this out loud. You know, the army trained the program to differentiate American tanks from Russian tanks. It got 100% accuracy on a test set. But later people realized that American tanks were photographed on a sunny day and Russian tanks were photographed on a cloudy day and the computer learned to detect brightness not actually detect tanks.
37:13 And so, you know, this is exactly the kind of problem that I'm talking about, right? where we have this extremely high test accuracy and we are super happy, but then we realize we haven't learned anything about the underlying task. and this has been attributed to a lot of different people. It was I think originally in written form on Dreyfus's textbook. but it turns out it's not an actual real example. this can't really be attributed to any actual experiment run by the army. the citation here is Werne. He has a website where he has gone through all the possible attributions of this urban myth.
37:44 but really this urban myth has been is so popular I think at least in the AI machine learning community because there's a kernel of truth to this sort of story. And we're going to go through several examples of tasks today where there's these kinds of failures. so, one of them that's kind of fun is this vision task apparently of where people have tried to predict gender from iris patterns. and there was apparently some belief that this was a task that you can perform because you can actually get reasonably high test accuracy if you train CNNs on cropped images of irises and you try to predict gender.
38:26 but there's this paper that identified that actually this is not actually because of the iris, it's because female eyes often have mascara and that systematically shifts the brightness of the images. and this sort of histogram tells sort of, you know, a thousand words in one image. And so on the top one you see this distribution of males and this x-axis here is the average brightness of the image. And so the distributions look pretty similar. Like males and females have a similar brightness distribution when when females have no cosmetics on. but if you restrict yourself to females with cosmetics, you know, this red distribution shifts to the left and the image becomes darker. And so you see that there's this, you know, very strong confounding effect of the female eyes having mascara, therefore being darker, and therefore the systems predicting quite well based on this average darkness.
39:20 Even though really apparently wasn't learning anything at all in terms of this actual prediction task. another one from sort of the sort of Guerin website and sort of the investigation into this tank phenomenon, which is interesting, is Kaggle fisheries competition where the task is you're given images of fishes being caught on a fishing boat and the task is to identify whether or not these boats are catching fish illegally. So you're supposed to identify whether or not these fish are part of a set of sort of fish protected category of fish or not supposed to catch.
39:53 and it turns out on the training set you can do extremely well on this task using a very simple heuristic. These images come from a relatively small number of boats. So you first identify each boat and then you identify for each boat whether or not they've been catching illegal fish. And this approach does really well because it turns out only a few fish or a few boats catch these illegal types of fish. And so by first identifying the boat and then by identifying the fish, you can get extremely high accuracy, even though you have learned nothing about actually performing this fish identification task.
40:25 Another one that seems maybe more high stakes and problematic is medical prediction. There is a lot of talk about, you know, tumor identification or chest x-ray, sort of malignancy prediction. And in these cases, right, we it's pretty important to ask whether or not we're doing well because these are high stakes situations and you would like to make sure that you're not being fooled by some sort of feature that makes the task easier than it actually should be.
40:53 And there's often claims now of these systems performing just as well as human doctors in terms of their diagnostic accuracy and so on. And one sort of really interesting, and maybe a little problematic, example is when you have these sort of tumors, sort of skin lesions that you're trying to classify as whether or not they're cancerous or not. doctors will often put surgical markers to highlight tumors that they think are more serious than others. Just so that when someone else is looking at them, they can immediately identify the more problematic ones.
41:28 and the training set apparently for these systems contain a lot of these markings. And so there was a examination into these sort of tumor classification systems where they artificially added markings to these images as well as cropped out markings from already marked images. And they showed that they can basically flip the classification output of these systems. And so in some ways, the high accuracy of these kinds of classification systems are not because they're identifying tumors. They're it's because they're piggybacking on humans who have already, in many cases, classified the tumors as being malignant or not.
42:04 an early problem that someone identified in one of the earlier works in oh, sorry, identification of this Esteva in 2011 is when people are trying to identify whether or not tumors are malignant when in serious cases people would include rulers to show how big the tumor is. And so the existence of a ruler would serve as a sort of spurious correlation or a confounder in terms of whether or not a tumor was malignant.
42:34 And finally, one that I think people are now aware about but initially I think people were sort of not as aware of is that hospital ID often serves as a really reliable indicator of both sort of base risk level as well as the type of procedures being performed at a hospital. and this you can think of as like the analogous to the boat example in the fishing problem where if you identify hospitals that say have a lot of smokers, you're going to much much more likely find cancer in lung chest x-rays from those types of hospitals. And so it's really important try to remove the effect of sort of identifying the hospital and then identifying the base risk.
43:19 a really interesting one I wasn't aware of in image classification until yesterday or so is Pascal VOC is a pretty common object identification data set. And a bias that's been identified is that the horse class for this I guess was taken by a single horse photographer who put in watermarks at the bottom left of the image. So around 20% of the horse images have a watermark and rely a classification system just learned to pick up on the watermark. So you can make cars classified as horses as long as you add the watermark on the bottom right.
43:55 And so this is something where unless you really carefully looked at the data set, you probably won't even realize that this kind of bias exists until you've actually sort of carefully examined and adversarially examined the data sets that you have. So the finally, there is a I've mostly talked about vision examples thus far, but this is a this sort of shortcuts and lack of understanding is a problem that's common to every area. And I'm going to give probably a very well-known example in natural language processing.
44:31 The task here is entailment prediction. So you're given a pair of sentences. One is called the premise and another one is called the hypothesis. So the first one is a sentence like the economy could be better. And the second one is a sentence like the economy has never been better. And the goal here is to say does the hypothesis logically follow from the statement made in the premise? And so if they follow, you say it's entailed. If it's contradicted, you say it's a contradiction. And if it's neither, you say it's neither. And so it's a three-class classification problem. And the way these tasks are or sorry, the way these examples are constructed is through crowdsourcing where you extract a premise sentence from some sort of large internet text or newswire text, I guess.
45:13 And you have a label that you randomly pick. So you say, I have a premise and it's going to be a contradiction. And then you ask crowd workers to write down a contradiction. And so they write something like the economy has never been better, right? And so what happens here is that crowd workers, because they're writing the hypothesis text after seeing the label, have systematic biases. And the bias that's really strong is this negation where they learn that negation is often or sorry, where the bias is that when something's not entailed, they use negation. And so model will often learn to associate the negation or lack thereof with the outcome label. And so instead of actually doing these sort of entailment tasks, they're often pick up these negation biases. And even more problematically, systems have what's called or sorry, what's called a hypothesis-only baseline, where you don't even look at the premise, can do extremely well, right? And there's no way to do well on this task while looking just at the hypothesis, right? Because how can you know that the hypothesis is entailed from the premise while only just looking at one?
46:17 And so this shows like this really strong bias that these crowdworkers put into this dataset. And so this has kind of serious implications for the project of pushing machine learning and getting towards understanding and general AI, right? because thus far all of machine learning has been predicated on benchmark progress. And that's the way in which the field has really grown and done well. You know, ImageNet and MNLI and these sort of well-known tasks, you get everyone together and we push on these numbers and we hope that improvements in these benchmark performances lead to understanding. But it's clear that because of these biases, that may not necessarily be the case, right? And so like we need a different sort of paradigm to link machine learning performance to understanding.
47:05 And the other problem that I hope by sort of going over so many examples I was able to impress upon you is that, you know, there's so many shortcuts, right? And like, you know, in this negation bias from crowdworkers, like you wouldn't know about this unless you sort of looked at the dataset carefully after being told that there was a bias, right? Or the watermarks were horses. Like I don't even know how they found that given that how sort of minor that is.
47:31 So it becomes really hard to say like we're just going to construct a dataset free of shortcuts. Like when you're told about these shortcuts afterwards, it seems really obvious, but like how can you construct a shortcut-free dataset? And so that's sort of the real challenge. Now, if we think that we can't get datasets free of these shortcuts and these biases and these minority groups, you know, we need a new way of sort of trying to make sure that our models really learn the right thing.
47:58 and I'm going to stop here for a moment to sort of talk about shortcuts and understanding. And hopefully people have lots of questions because I think this one is a fun one in terms of thinking about how machine learning relates AI and so on. I had a question. Sure. Sure. Just thinking about like some committee modeling and you know, I went back to that stop sign with the patches that seems as a yield sign. If you had you know, does it make more sense to use one big model trained on every piece of data you can find or does it make sense to you know, train a bunch of models on some partitions of the data and that might overlap in some way and then combine those votes in order to make it less I don't sort of I don't know the right adjective sort of robotically easily fooled in a robotic way.
49:00 Yeah. I think that's a generally good thing to do. Like so I guess there's two answers and like the more general one is to think about the trade-off between model capacity and your ability to fit these like sort of minority groups or these like shortcuts, right? So the idea you're describing, let's say we have you know, 10 or like 100 different models, right? And we fit them to different parts of the data. Then we might have a model that's dedicated to shortcuts, but we might also have a model that like really learns the right thing, right? and so you know, the more flexible our model, the bigger our sort of model class, the more we can say like part of the model might be dedicated to shortcuts, but that's okay because the rest of our model will still learn the right thing, right? But that's sort of still a hope. There's no real guarantee that this will happen. And if the shortcuts are strong enough, that's what the model will learn. So I think it seems really important to have bigger capacity models. Like, that's like sort of given. But, you know, how can we learn, big models well? Like, without overfitting? How can we make sure that they still learn sort of the right thing? Like, if one part of the model fits the shortcuts, how can we make sure the rest of it learns to do the right sort of prediction without shortcuts? That's sort of the open question, I think, in this area. Yeah.
50:16 There was a question. For image segmentation, do we have a way to know which part of the image contribute to the prediction? We could call it prediction traceability. Yes. so, I of course glossed over quite a bit, but the this paper, Lapuschkin 2019, here is exactly about that. Is about trying to identify or to attribute, predictions to parts of the image using, interpretability methods. And that's how they found, I think, this horse, problem, where they attributed predictions to locations, and they found that for horses, they were always localized to the bottom right, and it was because of this watermark. And so, I think a big important use case of interpretability methods is exactly this. it's to identify these kinds of shortcuts by attributing predictions to locations in the image or to subgroups in the data set.
51:11 going along with that comment, are there methods of finding what parts of the image or data example has high weights associated with it? Yes. so, the general approach, I think, that people have I think there's of course many different methods, but, Shapley values are a pretty general framework. You can sort of think about, the analogy is kind of like this. Think about each pixel or each region or sub part of the image as a participating in the prediction. And you can ask, when I remove that part of the image, how much does the prediction accuracy go go right? And so, you can do that after sort of randomly removing other parts of the image. So, you randomly remove the other parts of the image, like drop random pixels, and then you drop the part you're interested in, and you ask on average, how much more accuracy does this part that I'm interested in give me, right? And that's sort of this estimate called the Shapley value. And a Pongwe, I think who's here, has a paper on, you know, approximations to that based on the influence function. And so, there's all sorts of ways in the interpretability community about doing what's called feature attribution.
52:16 another question. For a problem with discrimination, would a reasonable approach be to adopt active learning by default, where the model can train with more emphasis on wrongly categorized examples? Then the hope is that the model could steer itself away from the initial biases over time, or is it not as simple? Yeah, so that's obviously sort of actually one way around it, right? Like when you can collect your own data, and you have the ability to know what you're getting wrong, then, you know, collecting data in the places where you're wrong serves as a negative feedback loop, right? You get more data where you're wrong, your model gets the training signal it needs to correct, and so eventually you'll learn the right thing. It's just that active learning on the scale that you need is very, very challenging, right? Like, can you actively collect ImageNet-size style data? Very challenging. Also very challenging to know what parts of the data you're doing badly on, right?
53:08 you need to know well enough to say, "Oh, the part of the data I'm doing badly on is horses without the watermark, right?" That's sort of a hard thing to be able to say. So, you need to know what you don't know, which is almost equally as challenging as robustness. oh, wow, there's a lot of questions now. physicians are experimenting in end-of-life care based on AI to nudge conversations. Do you have any suggestions for patients and doctors?
53:38 that is, yeah, that's very challenging. I do think one important thing about sort of these high-stakes settings is to think about sort of the alternative and the whole decision systems that they're part of, right? So, say in medical diagnosis, or I'll give another example I'm more familiar with, which is predicting whether or not someone will commit crime again and so should be released for parole, right? These are both really high-stakes prediction tasks. And the way they're performed is there's, you know, a machine learning system and a human, and they sort of jointly make a decision.
54:09 So, you need to think about not only the system, which is sort of what I've talked about here, but also the human part, right? And how they integrate and how their decisions get combined together. I think that's actually the important part. Like, how do humans override the machines? How do they incorporate the suggestions of the machines? Even more so than the predictions, which I think need to always be taken carefully. Next one. How can we combine models' objectives to gain greater understanding of the world and to combine it to create intelligent behavior?
54:36 Yeah, I think this is this is basically the the open question, right? The unfortunate thing that we don't know how to do. And I think that's sort of the thing that we are struggling with in sort of the robustness generalization literature. Like, what is the right approach? And I think there's not really even a consensus. Is it like more data collection? Is it smarter ways of training the model? Or is it better models? It's not clear yet what the right way is yet. Unfortunately, we don't have a great answer beyond that.
55:04 For shortcuts, I think even people use shortcuts to identify something. Do we have some way to understand shortcuts based on training data? Yeah, this is a good point. So, humans will also use shortcuts, but I think the important thing here is that these shortcuts are a lot more crude than the humans ones that we use. And at some level, they don't even pass basic sanity checks, right? Like for entailment, we know that the output should depend on both the both the inputs. But in reality, it's only depending on the hypothesis, which is very problematic. And what people do today is they have these sort of challenge sets, like examine the performance of a model just based on the hypothesis. And these kinds of sort of what are close to the sort of unit tests help catch these kinds of shortcuts in many cases. So I think one way to detect them is things like that. Like our model shouldn't be sensitive to certain perturbations or it should be sensitive to certain perturbations and you go off of those kinds of assertions.
56:01 Is there currently a correlation between model capacity and number of shortcuts employed? Are larger more likely to have been on the correct correlations or are smaller models more likely to use shortcuts? That's a good question. I think the general sense that I get from reading the literature is that smaller models are more likely to use shortcuts in many ways. For example, in this paper about sort of watermark based shortcuts, linear models did a lot worse. Like they would really pick up heavily on these watermarks whereas CNNs did so with less weights and less frequency.
56:37 And I think generally that's true that like large capacity models trained with a lot of data can use some of its capacity just to model the shortcuts and it'll still do well on the data without shortcuts as long as they exist. But really the key thing here is like you need to at least see some data without the shortcut pattern. How much are you Oh, I already answered. Okay, great. Yeah, so I want to get into the breakout now. Actually, especially because someone asked the question, you know, what sort of the solution and you know, are humans robust? So if Woody you could drop us into a breakout session for say five or six minutes.
57:16 It'll be great to talk about these two questions. So the first one is are the brittleness issues inevitable? You know, what do you think are the solutions? Like what are the right approaches that you think of? And the second question is are humans robust? And you know, if you think so, then what's the key ingredient that makes humans robust or more robust than machines. Awesome, yes. I'll create the breakout rooms. If If everyone wants to take a quick screenshot or try to remember these, I'll post the questions in the chat as well, but they won't be in the breakout session.
57:46 Oh. Okay. Great. I think I'm unmuted, right? Let's see. Yes, we can Okay, great. All right, excellent. All right. So, I'll go through the second part a little bit quicker. I'm glad that we got so many good questions on the first part, which is the more important of the two parts of this talk. in the second part is thinking a little bit about how can we do learning? How can we fix these problems and the kinds of research that Percy and I and others in Stanford have been doing. and so, the key problem, right, that I think you should keep in mind with all of this is that the training distribution is very different from the test distribution.
58:27 This is the root of all evil for these robustness problems. and so, we need to think about, you know, is the limitation that we can't generalize from train to test inevitable? Or can we come up with some clever data collection schemes or model training mechanisms that allow us to generalize? and so, to do this, we need to think a little bit about, you know, how distributions can shift. And so, I'll give you a little bit of definitions. The first one is like covariate shift. And this is what you usually think of when people say the distributions are different. So, you might get, you know, these Let's say you're making a face recognizer, you have these really nice well-lit portraits in the training data, and at test time you're using it with CCTVs.
59:05 So, all sorts of different environments, but the underlying task is the same, and there should be a single predictor that does well both on portraits and images cropped from CCTVs. Another example is label shift, where, you know, the input features look similar, but now the output labels distribution has shifted. So, for example, if you're making a face recognizer, and at training time, you know, you need really precise matching. So, you're only going to be calling detections when the images look exactly the same, but at test time, you know, you're making a product for your camera and so, it can be a little bit looser and so, you might deal with blurry images and so on. And so, the limits test for this is, you know, you have the same predictor, but you're changing your threshold. You're just saying it's okay if, you know, I'm a little bit less confident. I'm still going to make the call. And this is, you know, an instance of label shift.
59:50 And the final one that is basically intractable in all cases is concept drift. and so, here, you know, you might have a prediction task where you're trying to initially recognize faces of, you know, the same people, but then at test time, you want to match people across time, like young pictures and old pictures. The task is fundamentally different, right? Like whether or not you're matching the same person or person shifted in time. And so, no one predictor is going to do really well on both of these tasks. And so, there's sort of a fundamental change in the task definition.
60:21 and I'm going to go over two sort of ways to deal with all of these problems. the first one is we're just going to collect more data. Like someone sort of talked about this as a question earlier. And this is sort of the key thing, right? Like if we get more data, we can do a lot more things. And the second part is a little bit more ambitious and it's going to say, you know, let's try to only do with what data we have. So, the first idea is let's just say, you know, we're going to try to generalize to a test set and we're just going to collect more data. So, a classic example of this kind of task is you're recognizing digits. So, you have images of digits.
60:52 The left one is MNIST, which is really old. USPS Yes, even older actually. And SVHN, which is a more modern recognize digits from I think mail numbers that are taken from the wild. And so, in all of these cases, you need to output the number, right? Like this one's a two, this one's a seven, and so on. When we're collecting new data, let's say we trained a model on MNIST and we want to do well on SVHN at test time.
61:15 And so, this is a distribution shift. but we might be able to collect more data, right? It's unrealistic to maybe say we only have this and we have to predict this. And so what we might have is we might be able to collect some unlabeled data from SVHN, right? Like we can't afford someone labeling them by hand, but we might just be able to get the images. And this is called unsupervised domain adaptation. and if we can collect labels, that's all called supervised domain adaptation, and that's even better.
61:43 And so we can ask like when can we do learning, right? Like if we're in covariate shift and we have this like source data, we might be able to do learning because we have a better model that can adapt to these different kinds of distribution shifts that occur. And so the general thing that you should sort of think about is, you know, if we're in covariate shift settings and we have source domain data, so unlabeled data from our target distribution, then we can actually sometimes generalize to our target task, even though we don't have labels.
62:12 and so this is sort of the setting that we're going to be talking about for the rest of the talk, the rest of the couple minutes of this talk, where we're going to ask say, you know, we have this covariate shift problem where the prediction task is fundamentally the same, and how can we generalize? So the easiest thing to do and the most classic thing to do is re-weighting. So let's say we have training data that's 90% frontal images, but 10% images taken from the side, and we want to generalize the test data that's 50/50 front and side, right? So how can we do this?
62:44 Well, let's just re-weight the data set. So each frontal-facing image counts for less, and each side-facing image counts for more, right? So we've artificially rebalanced the data. And this gives us this like assumption-free way of getting estimates of how well our model will perform on this sort of 50/50 test set. And, you know, this applies to all of what I've talked about before. Like if your data's imbalanced and has a minority group, or maybe has shortcuts, maybe we can rebalance it to get rid of all these problems, right? So are we done?
63:12 No, because, you know, even though I talked about it that way, if we reshuffle or like restructure the data set, you know, the training data is 100% men and the test data is 100% women, so there's no overlap. And if we try to reweight this data, we're going to get infinite error, right? Because we need to infinitely upweight the women that we don't have in our training data. And so this is the fundamental problem with all these approaches. Like when you don't have any overlap, your estimates all blow up and go to infinity.
63:38 And so in the real world, everything is non-overlapping, and so usually these kinds of estimates don't work. But intuitively we might think, you know, these kinds of tasks are possible. And the reason why, you know, you and I and many others think this is possible is this intuition, right? Like let's say we have training data that's blue images and test data that's orange images. Like clearly there's no overlap between any of these images, right? They're on different color channels. but if we desaturate the images, we can perform prediction on the desaturated image, and we'll get really good performance, right?
64:12 And so the intuition that if we can't distinguish the two domains because we've desaturated the images, we might do really well. And this is the idea behind, you know, most of modern domain adaptation, where you learn to represent your data in a space that doesn't change when you go from training to test distribution. And so you measure how much your data changes in the sort of a higher-level representation. And if your data's close, then you're going to do really well. And the sort of thing think to keep in mind is, you know, your test performance is going to be your training data's performance plus some sort of distance that measures how different the training and test distributions are.
64:48 And if you can keep this small, you're going to do really well. and so you can think about this sort of very simply as saying, you know, this gap. You know, the test error of a model is the source performance, how well we do on the training data, and the gap between train error and test error. And this idea is that, you know, we're going to look at a domain distance, where no matter what model we pick, we're going to do well because the distributions look so similar, right? Like if a image is a desaturated and they look identical, it doesn't matter what the model is. They do identically.
65:20 and so, if we do well on the source domain and the domain distance is low, we might be able to generalize. And this is really interesting and optimistic, right? Like if these all seem like things that we can measure and think about and they give us conditions under which we might be able to do well on a test distribution. And there's been a lot of work over the last almost two decades now on these kinds of domain distances and bounds and how you can learn from unlabeled data.
65:46 and they give great intuition. They let you think carefully about these kinds of problems, but unfortunately, if you try to actually compute these bounds, you know, what's my guarantee of test error, they're usually vacuous. So, you'll say that things like the accuracy will be greater than zero and the error will be less than one, which is not super helpful. And just to go over how these kinds of things often work in practice, domain distance, the thing I just described, is the basis for a lot of these modern domain adaptation methods. And the key idea is that neural nets are everywhere because neural nets work and you use them as a way to measure the domain distance. and the idea here is that you have one part of your model maximizing performance on the training data and you have another part of the model making sure that you can't distinguish between the training and the data distribution on this sort of what's called like a bottleneck feature space where at this level your training and test data should be indistinguishable and yet useful for actually doing the task.
66:47 And so, you know, we have this hope that, you know, we have this reweighting thing which has no model dependence and this model-based domain distances that, you know, requires us to carefully construct a neural network, but on the other hand, that's the only way we can get the these things to work in the real world, right? Like with reweighting, often these weights are infinity. And so, there's no free lunch. We either need model assumptions or assumptions about overlap on the domain.
67:12 but, if we have one of those and unlabeled data on the test domain, we can actually sometimes do well. The other approach that I'm going to go over with a little quickly because I'm running low on time, I can describe very at a very high level as this idea, right? As I said before, the main issue is that our training distribution and the test distribution are different, right? And if they were the same, we'd be done, but they're not.
67:38 But, what if I told you I give you a list of a hundred possible test distributions, and I say your true test distribution is going to be one amongst these hundred, right? Then we can train a model to do well on all of these, right? We just go through each one of these test sets, and we say our model has to do well on the worst one. And if we can get a model that works on all of them, we know that our model's going to do well on the true test set. So, this is thinking about, you know, a potential set of test distributions and considering the worst case. And this is what's called a minimax optimization problem.
68:09 So, we're going to find the best model, that's the min part, that works well over the worst possible potential test set, and that's the max part. And this idea is going to work whenever the true test set is contained in this uncertainty Q, right? We're saying worst case over this big Q, which is the set of potential test distributions. And this fails whenever Q is too small or too big. If Q doesn't contain the real test distribution, you got no guarantees. If Q is so big that it contains everything, then your model is going to be so pessimistic, right?
68:38 Because it has to be prepared for any possible distribution, that it's just going to predict 50/50 or, you know, something vacuous for all of your inputs. I'm going to skip over bit of examples, and I'm going to go to this slide and say these kinds of ideas can be applied in each one of the settings that I described before. Talking about minority groups and fairness, or adversarial examples, or understanding by carefully choosing the kinds of worst-case groups.
69:07 In the case of minority groups, we basically explicitly list out all of the possible minorities that we care about, and we consider the worst-case performance over all of those. In adversarial examples, we know that these images before and after perturbation are close, so we consider all distributions that are nearby each other in pixel space. And then we maximize over the worst case over those. And then for shortcuts, what we can do is we can explicitly construct groups that don't contain some of these shortcuts, and we enumerate all such groups, and then make sure that these worst-case groups work well. So, for example, if we have a model that relies too much on backgrounds, we construct subgroups of the data that have sort of mismatching backgrounds and objects.
69:50 So, the So, basically wrap up here, right? The limits of this kind of approach is that if we pick too small of a worst-case group, we get no robustness. And if we pick too broad of a worst-case group, we get vacuous models. And there's no simpler or general principle for designing these losses. Even though this approach gives us really nice ways of thinking about and optimizing models for the worst case and getting guarantees. Okay. I'm going to wrap up there. If anyone has questions, I would be happy to answer them. I can stay for a little bit longer and chat if people have questions.
70:25 All right, maybe we should just thank Tatsuo for his really insightful and interesting talk. And then maybe people can run off if they have to go. So, thanks, Tatsuo. Thank you. Yeah, really a lot of interesting things, food for thought. So, I hope that everyone is kind of has their eyes opened with the respect to all the different problems that we're we're seeing, and hopefully motivated to help solve some of these, because I think there's a lot of interesting open research questions here.