Chapter 2.5 - Eigenfaces

0:08 all right we're going to do a kind of a fun little exercise and what's called igen faces it was one of the more interesting you know applications of igen values and igen vectors in the early days of computer vision in sort of the early 1990s about how to extract features from images to do face recognition and so it was really based upon igen value the IG value problem that we've talked about and I'm going to show you this example

0:36 in in context of some image image recognition tasks that are very basic but they kind of highlight some of the main ideas that we're actually interested in so I'm going to call this igen faces because that's exactly what it was called from that early developments in the 1980s and 90s when we were first developing image recognition task for human face recognition so first of all we need to bring in images and you're going to see here a collection of images that I

1:04 bring in we're going to start using some of the specialty tools out of python in fact some of these image packages that we would bring in here and so we can start reading in images like image read fed one. jpeg this is a image of Roger Federer and I'm G to have five of these images I'm going to turn it into a grayscale I'm going to take it make an RGB to a grayscale picture and I'm going

1:28 to resize it to 20 x 80 pixels so all these are commands that we would use to take these images that I actually scraped from the internet and I brought it in and then I resize them all to the same size so I can start doing comparatives right so it's important that you know you don't have some images that are high resolution and some that aren't and you try to compare them you kind of need them all to be the same

1:53 size and roughly the same cropped faces so here are the images that I brought in so we have pictures here of George Clooney Barack Obama Margaret Thatcher Matt Damon okay so these are ones that I could scrape off the internet easily and I SP these date me to you know when I did this was around the time that Barack Obama was President so it's a little older but still these are just five faces you can scrape off

2:24 and notice a couple things first when I scrape them I I cro the pictures roughly the same but you know to have most of their face in there roughly proportioned the same way all these images have 120 by 80 pixels 128 in height 80 pixels in width okay and so what I want to do is start thinking about this pixel space and start thinking about features in this pixel space so I'm going to look for commonality among these things so

2:53 the first thing I can do is look at the average face of George Clooney the average Barack Obama face Matt Damon Margaret Thatcher so again all that's doing is taking these five images averaging them or these five images and averaging and this is what these average faces look like and so you can start seeing each each of these even though you get you know they're not perfectly aligned that's why you get a lot of blurriness you can still kind

3:19 of tell who these people are even with the averages that you have here so here's what I'm going to do I'm going to start looking for correlations among the imag in other words what correlations mean here is essentially dot products among images so for instance I can take one mat Damon picture and Dot product against a Barack Obama picture and if I do that I can actually ask the question all right so when I take that dot product how much

3:50 of Matt Damon is projected in the Barack Obama direction or a George Clooney direction or Margaret Thatcher Direction so inner products have this really important representation here as giving us essentially what's how much is correlated among the images in fact the correlation of these vectors is just the dot product so here's how I'm going to arrange everything I'm going to arrange it all into a big Matrix image one image two they're going to be row vectors

4:19 remember it's 120 by 80 so each row is 120 120 times 80 in length and I'm going to have 20 images five for each of them five for Obama five for Matt Damon five for Clooney five for Thatcher okay so it's a 20 by 120 times 80 okay that's that's my Matrix of data so what I'm going to do now is compute What's called the correlation Matrix which is B transpose

4:50 B so what b transpose B now is doing is taking one matte Damon image and asking what's the inner product with itself which will give you a good score because the same image but now what how does one Matt Damon image correspond to the next Matt Damon image to all five Matt Damon images but then now against how does IT project onto the Clooney images how does IT project onto Obama images and Thatcher images same thing for every

5:16 single image so this correlation Matrix is giving me a score which tells me how each of the 20 images projects onto all the other 20 images okay and a large value means means there's a lot of correlation and in fact a low value means very little correlation in fact a low value means that they're almost orthogonal okay but it turns out with faces the correlations be high because everybody has eyes and nose and ears so there's a lot of commonality and faces

5:48 that is picked up by this inner product projection which is given by this correlation Matrix and then once I have the correlation Matrix what I'm going to do is I'm going to find its ion value and igen vectors and these going to be the IG faces and they kind of represent in some sense the features we can look at for building face recognition algorithms this is how it was thought of in the early days so first of all from numpy in

6:13 linear algebra I'm going to bring in LA and so I'm going to do this Matrix multiply BB transpose which is what I did here and actually this is backwards should be BB transpose and then I'm going to find its IG values okay so I'm going to go ahead and come back with igen vectors and the igen values here and this is what they look like these are the first five igen vectors and this is the IG values that I

6:39 have associated with all 20 so first of all notice what's happening here these I vectors that are this is the most dominant one second most dominant third most dominant they actually represent something like a face what are the largest dominated features among the faces this is what the aen vector is telling you is that this direction here which look what it's capturing eyes and some hair and a face so this is sort of in some sense capturing the dominant

7:09 features among all the faces that's the igon vector one ion Vector two is filling in some features three four five and what we can do is start representing any one of those images in terms of this new coordinate system which is the igen face coordinate system so another way to say it is I could take my face and we'll do this in a minute and project it onto these igen vectors here and ask how do I

7:35 look in this coordinate system which is the IG Vector coordinate system in other words I have these IG vectors which are these natural coordinates and I want to project any face into here including the ones that I'm using here with Margaret Thatcher and Obama and Clooney and Matt Damon so for instance I can take one of these images and ask how does IT project onto these 20 igen Vector directions so I can take the average Clooney face and

8:06 here's how the average Clooney face these are what these bars represent is the strength of the projection into these 20 different directions you can see Clooney is sort of spread out a bit here whereas Obama has very strong features in the first two directions and also this fifth ion Vector Direction in other words you can look what they here's what they look like these are the what the ion vectors look like here and so this is telling you how much of each

8:31 of the IG vectors it has Thatcher is here very strong projection onto the second and third igon Vector directions which are these two here are really dominant for Thatcher and then here is Matt Damon which has the first third and this fifth are very strong projections so you can start doing in other words look at the features that are happening here if I'm looking in the igen vector space Clooney and Thatcher and Obama and Damon all look very different and this

9:04 is the basis of doing of some of the early facial recognition techniques these weren't nearly as good as they are today but this was the start point in the 80s and 90s of how we started thinking about doing facial recognition we didn't do facial recognition by looking at pixel space we started doing it in this feature space which are the igen vector IG face space so in other words you would take an image and you would say look Clooney has a certain

9:34 representation in the igen face coordinate system and so does Obama and so I'd start to take a new picture and I would project it into this igen face space and actually look at what does it look like in igen face Direction and then from there can I tell if it's a thatcher picture or an Obama picture or Clooney or D okay so that's the idea these are the features you're using to do an identification of whether

10:01 this is George CL a new picture is a picture of George Clooney or not so let me just show you this for recognition I can take now a new picture of Margaret Thatcher one that wasn't on the IG and I can project this new picture of Thatcher directly onto those 20 I vectors and here's what that reconstruction looks like in a 20 dimensional space so this picture here in a 20-dimensional space is giving me this representation and I can also ask

10:30 how does IT project onto the five images that I actually had what's the dot product between five other images I had a fer and here's what they are I can also take someone like Merill stre Merl stre actually played Margaret Thatcher in a movie and so the idea is here's her what she was supposed to be looking like in the movie and I can take this and project onto the igen face directions and this is the Reconstruction of what

10:55 it looks like in that Igan faces Direction but here's the projection of the inner product it has against all those five pictures of Margaret Thatcher it's pretty high correlation especially this this one here this picture here this is the fourth picture looks very much has a very high correspondence to what Market Thatcher looks like take another politician Hillary Clinton here and project and again here's the projection of 20 mode decomposition here's what she looks like in the 20

11:24 mode decomposition here's her projection onto Margaret Thatcher modes so see the real you know look right here this is telling you that Merl Street looks a lot more like Margaret Thatcher there in that movie than than Hillary Clinton does in fact you know she can even look more like Margaret Thatcher than Margaret Thatcher does in some of these projections so that's just a little fun fun little fact so I decided to do the last thing on experiment by the way all

11:51 this code's available if you just download if you have the GitHub link and you have all the data and all the code to run this and play around with it which is kind kind of fun and I took my picture here I am there I am look good that's younger Nathan and n and I'm going to go ahead and project onto my igen face directions and here's my projection to I 20 20-dimensional igen face space and

12:14 here's my reconstruction of me in that 20-dimensional space so remember when I do this projection into this low dimensional space 20 directions you know it's an approximation and here's my approximation to this so you know get blurred out a bit here but then I asked the question who do I look more most like right so I was hoping to look a lot like Clooney but it turns out my projection onto the Clooney pictures isn't that high here's Obama a little

12:42 bit more in Obama but apparently I have a much bigger projection on a thatcher and Matt Damon so I look most like this one Thatcher picture but I'm gonna go with Matt Damon like in a born identity and so like I can totally you know I can totally Crush you I think that's the moral of the story is I have a really good projection on to mat Damon and so that means I'm awesome Matt Damon and like you know

13:09 he's like a Terminator in Born identity so there we are this is just fun in games for image recognition and so forth and and projections so this this kind of highlights a couple things one is this idea of projection two is idea of a coordinate space and the coordinate space we're using here is based on the igon vectors and those igen faces have real they they're intuitive like you can look at them and start seeing that

13:35 there's structure there that you can exploit and that's exactly what we want to do in all of this is exploit structures so that we can U start using them effectively for doing different operations and it's all based upon this very simple linear algebra and normally when we think about I values and I vectors it comes in many different Frameworks but here's one of the early Frameworks in computer vision that we're using this concept of igen values and I

14:01 vectors in a really new way and and has progressed from there now to very mature technologies that are more based on neural networks now than than igen values and I vectors but this is a great start to start starting to understand how to progress that technology