transcribe

Stanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Stereo Vision

Stanford Online · 49m · transcribed Oct 2026
More from Stanford Online Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Transcript

Speaker 1

0:05 So today we're going to be talking about stereo vision, which is something that Marco alluded to last time. But basically the point is that in general, it's not possible to recover the structure of the 3D world from just one image. And this is due to the intrinsic ambiguity of the 3D perspective transformation. So in this lecture we'll discuss different ways for recovering 3D structure. And here you can see some of the sections for where this material is taken from.

Speaker 1

0:35 So, stereo vision involves two processes. The first is the fusion of features observed by two or more eyes, and then the second is the reconstruction of the three dimensional properties. So the key problem here is actually a correspondence problem. So, given observations P and P prime in, in camera one and two, respectively, how do we know that such observations correspond to the same 3D point capital P, or equivalently, that P and P prime are actually in correspondence, that they actually do belong to the same 3D point.

Speaker 1

1:08 So, to help us, there are several constraints that can be leveraged to simplify step one here. And so, for example, there's a similarity constraint, a continuity constraint, and most importantly, an epipolar constraint. And we're going to talk about that right now. So, a priori, if you think about how when we have two cameras set up looking at the same scene, it makes intuitive sense that any pixel that you see in one camera, you should probably be able to see in the other camera, especially if they're looking at the same scene.

Speaker 1

1:43 Indeed, matching pairs of pixels that are restricted to lying on the corresponding epipolar line is one of the main ways that we will be able to generate 3D information from 2D images. So let's actually break down what we're looking at here, because this is the fundamental image that we're going to have going forward through this lecture. And what you guys should keep in mind when you're thinking about 3D vision or stereo vision. So these two planes here on the left and right, PI K, capital PI and PI prime are the image planes point O and O prime, that's like the pinhole camera center that's behind them, some focal length behind them.

Speaker 1

2:24 And then point capital p is some 3D point that's out in front of the cameras that they've both captured in the images. So in PI, this 3D point capital P corresponds to lowercase p. And then on the right in PI prime, it corresponds to lowercase P prime. So these are the image, These are the 3D world coordinates point in the image, and these are just the image points.

Speaker 1

2:56 And specifically, something that's very important that we're going to be talking about when it comes to epipols, or let's even disambiguate what that is. So when you have two images from two cameras looking at the same scene, so the baseline, which is the line from one camera center to the other camera center, will go through some part of the image, and even if it doesn't, it will go through some part of this plane, unless they're parallel.

Speaker 1

3:22 But we'll talk about that later. So, and this is the general case. So the point at which this baseline intersects the image plane, those are called epipoles. So there's going to be an epipole for the image on the left here, and that's E, and then an epipole for the image on the right, which is E prime. And so now for any 3D point, which then has a corresponding image point, if you draw a line from the epipole to that image point, we call this line an epipolar line.

Speaker 1

3:54 Okay, so this is just some terminology here. And then the plane that's defined by these three O, O prime and, and capital P, this is called the epipolar plane. So when I was alluding to this previous epipolar constraint for determining which point in one image maps to a point in another image, which then both map to the same point in 3D, what happens is, as you can see, if we have some image point, capital P, which corresponds to this image point, P prime, in the left image, this point must lie somewhere on the epipolar line.

Speaker 1

4:34 Why is that? Because it lies on the epipolar plane, and the epipolar plane intersects the image plane along this epipolar line. So this is the fundamental constraint that we're going to be working with for developing 3D information from 2D images. Let's see if I want to say anything else about that. No, the only thing I will say is that although this is like a fundamental constraint, it actually is really helpful because it means that when we want to search for correspondences between two images, rather than searching over the entire image, which is maybe like what a naive method might do, we only have to search along this line.

Speaker 1

5:09 So it reduces our search space from one of a 2D search space to just a 1D, which is significantly faster. Yep. Yeah, this is. This is the general case.

Speaker 2

5:22 Yeah.

Speaker 1

5:23 So there's. There's no. Like, the baseline doesn't have to be. Well, it's all a matter of semantics. Right. If even if I have one image way behind the other, the Baseline is still, you know, between those two and, you know, they can be rotated whichever way. Yep. Any other questions about this setup?

Speaker 2

5:45 That IP photo line, if you don't know where P prime is?

Speaker 1

5:48 Well, the one thing that you know. Well, you do know where. You know where P and P prime both are, because you have both images. Like, you know, we, we are the ones who have these two images at our disposal here. So we know both P and P prime and we're trying to determine capital P. That's kind of the whole goal of stereo vision. Yeah. So we know, because this is our, you know, camera setup, we know the baseline and specifically there's two things we know.

Speaker 1

6:17 We know the baseline in terms of the translation between the two cameras. We'll talk about this later. But we also know the rotation between the two because remember, we're the ones who set up the camera so we can measure these things. Well, this helps us get there because we know that we just have to search on a line. But in terms of how we exactly determine that, we'll actually come back to that later in this. In this lecture.

Speaker 1

6:43 So. Yes. Yeah. So just to. Yeah, this might be a little bit confusing, but if you think about it, a match for P. So we're looking for a point in PI prime that matches P. When we say matches P, we mean it corresponds to that same 3D point. So we're looking here for P prime, which must lie on L prime.

Speaker 2

7:13 So there's another line, not visualized, that goes from 0 prime to P and that intersects L prime.

Speaker 1

7:22 Not necessarily.

Speaker 2

7:26 Why? No, I guess what is a potential match for P mean?

Speaker 1

7:32 Yeah, so potential match. It means that we're looking for a point in the other image that corresponds to the same 3D point that P does. And so that's why you see how we draw those two lines from P into this image and then P into the other image. So in that image, P prime happens to be where capital P lies in that image. But for us here, it's P for the left image. So when we say we're looking for a potential match for P, we're basically saying that we're looking for that point P prime.

Speaker 1

8:06 All right, so to. To visualize what this might look like. So kind of towards your question, how do we actually find. So if we know.

Speaker 2

8:15 Oops.

Speaker 1

8:15 If we know what the epipolar line is, again, we will talk about. We will dive into this more deeply.

Speaker 2

8:21 You don't have P5 yet. Somebody is.

Speaker 1

8:26 We will. So, OK. Yeah, well, no, we'll actually get to this, right now we're just giving a high level overview, but we'll dive into it. We will get to this. So keeping with the high level theme in mind, so if we do have L and L prime, and if we do have P, but we want to find P prime, then what we can do is, as I said before, you can just search along this line, which is much simpler than searching along the entire image.

Speaker 1

8:53 But again, we'll talk about it in more detail. Keep that in mind as we go forward. All right, so let's actually derive how we get this constraint, and this is now from just like kind of this geometry here. So we know, as I said before, that O, O prime and then capital P, which anyways they form this plane that we're interested in, this epipolar plane.

Speaker 1

9:25 And specifically because P and P prime lie along this plane, we know that each of these three points, which is these three lines, the baseline op prime and op or O prime, P prime and op must be coplanar. They must all lie on this epipolar plane. And the way that you can write this mathematically is that you can say that the vector formed by O and P must be perpendicular to the plane's normal vector, which you can define via a cross product as such, which is a cross product from the other two vectors that you have.

Speaker 1

10:04 So one thing to keep in mind is when we need to find the cross product between two vectors, you can actually write this out as you can actually write out a matrix form of the cross product for one vector as follows. This is more of just like a, let's say, like a mathematical trick that makes writing out some of the matrix algebra later easier. But all this is, is just saying that you can write the cross product as a matrix vector product as well.

Speaker 1

10:36 So just keep that in mind and you'll have these slides for reference anyways, or they're already up anyways. All right, so how can we actually derive. Yeah, so continuing the derivation of this epipolar constraint. So if we assume that the world reference system is located with camera, camera one, which means you put your initial frame of reference and everything there, after a bit of algebra, we can find that the epipolar constraint that we're talking about comes out in this form of P transpose with some matrix F, which we'll talk about, and P prime must be zero.

Speaker 1

11:15 So this is directly writing this constraint. Here we are directly writing this constraint, but we are defining a few of the terms differently. So here lowercase P just corresponds to the vector, because remember we're going from. We're saying that the world origin is here at O. And so kind of everything is with respect to the left image here. So P is just the vector from O to P. And then P prime is the vector from O to P prime, not from O prime to P prime.

Speaker 1

11:48 Remember, we're considering everything from this perspective here. And so now there is this kind of interesting matrix F that comes up which is kind of similar to. If you take the cross product matrix of this vector as well as a few other rotations and translations. Because you have to take into, you have to take into account the rotations and translations from camera one to camera two. So specifically, if you were to expand it out, F looks like.

Speaker 1

12:21 So here's where that translation and rotation comes in as well as the camera intrinsics, which you know, because these are your cameras which you calibrated. So some key facts about this matrix F. F is referred to as the fundamental matrix. The reason it's called fundamentals, because with this matrix you can basically find out everything given, you know, for two images of the same scene. And so specifically something that was of interest is specifically something of interest is how can we define epipolar lines?

Speaker 1

12:57 Well, if we have this fundamental matrix. We're going to talk about how you get the fundamental fundamental matrix later. But if you have this fundamental matrix, you can simply say that the epipolar line is this fundamental matrix times the image point that it corresponds to. So remember, for an epipolar line, you need, well, the epipole and then some other point in the image. And so for L, which is defined, keep in mind, this is defined in the left image.

Speaker 1

13:25 We're using an image point that's defined in the other image. The reason this makes sense is because, and that's why actually these three red lines are drawn here. If you pick some other image point, P, I don't know, double prime in this right image, and you move it just somewhere along that epipolar line, you know, maybe along some of these other three red lines, then it's gonna, it's basically gonna correspond to different points on the left image.

Speaker 1

13:54 Actually, if you move it on the epipolar line, it's gonna correspond to the same point. It just might be a little bit bigger in scale.

Speaker 2

14:00 Sorry.

Speaker 1

14:03 Yes. Oh, yeah, it's either the transpose of the inverse or the inverse of the transpose. Yeah, that's equivalent. Okay, so this is how we can find the epipolar line in the left image from a point in the right image. And you can Also do this the other way around. So if you want to find the epipolar line in the right image from a point in the left image, well, you can do that just by using the transpose of the fundamental matrix.

Speaker 1

14:30 So something else that's very important to keep in mind is that F is actually singular. It's not invertible. And why do we know that kind of right off the bat? Well, the reason is because if you actually try and find the epipolar line corresponding to the epipolar, the epipole itself, because I mean, you can put any image point in there, if you try and find what the epipolar line is corresponding to the epipole itself, you get nothing.

Speaker 1

14:58 Well, because it collapses to a single point, you don't get a line anymore because if you're trying to map this to the other image, it's just a single point at the epipol. So if you use the fundamental matrix for this, you get zero. So if you get zero, that means that it has a null space. If it has a null space, then you know that it's not invertible. Now the other thing to know about F is that it has, so F is a three by three matrix, but specifically it has nine elements in there.

Speaker 1

15:30 But we only really need seven to find seven because there are some other constraints that then help us find the other two. And again, we will talk about this in more detail in a few minutes. So some, I mean, as we talked about before, some of the obvious usefulness of the fundamental matrix is that we can find these epipolar lines just like that. Given we have some point on the left. And so towards one of your guys questions, if we have a point in the left image, we don't have the point in the right image, but we want to find the point in the right image.

Speaker 1

16:04 What do we do? We start from this point, we multiply it by the fundamental matrix that gives us a line in the right image and then we literally search along this line until we find something that matches this. We're going to talk about how you do this as well in a couple minutes. But there's, there was a question that came above that came earlier which was related to how that works. So hopefully that answers that.

Speaker 1

16:36 Yeah, great question. So there's actually no assumption of presence or absence of noise because I mean, there's many things you can do in computer vision rather than like a direct pixel wise comparison. And actually even if you had a perfect image, a direct pixel wise comparison wouldn't even work because If I'm looking at, you know, for example, this statue and I one camera is a little bit rotated so it sees, let's say, the right side of the nose, whereas this other camera sees the left side, then even a direct pixel match is not going to work at all.

Speaker 1

17:08 So actually, even in the first place, we have to do something a little bit more interesting than that. But yeah, so a little bit of noise shouldn't be that much of a problem. Too much. It'll be harder. All right, so the other cool thing in mind. So remember that the, the whole point of this procedure was, you know, we're using this epipolar plane, which is actually defined by this 3D point. But the really great thing about the fundamental matrix is there's no mention of the 3D point here.

Speaker 1

17:38 We don't actually need to find the 3D point in order to find the epipolar line in the other image. So this is very important, right, because then there's no chicken and egg problem, because we can find the epipolar line first without any need for capital P. Then we can find this other P prime in this image, in the right image. Now we have P prime and P, then we can do triangulation and then we can find capital P.

Speaker 1

18:03 So there's no chicken and egg problem of, oh, you need to find, to find the 3D point, you need to first, well, find the 3D point to get this F and then, you know, do whatever you want. So how do we actually compute this fundamental matrix? So this is like another kind of core question. Yes, this matrix is amazing, but how do we actually find it? So the first question is, is it even possible to estimate F if you just have two images of the same scene and without knowing really the intrinsic and extrinsic parameters.

Speaker 1

18:37 Because remember, I mean, I said that, yeah, we can measure R and T, but let's say we can't even measure R and T. Let's say I just took these two images from Google Maps or Google Images of, you know, the Rome Coliseum or something, and they're just from two different viewpoints. So can we even do this? Yes, it's actually possible as long as we have a sufficient number of point correspondences between the two images. And so while there are several algorithms for computing F, we're going to discuss here a pretty simple one.

Speaker 1

19:06 It's called the eight point algorithm, the naming of which will be pretty clear. So what we're going to do is we're going to use the fact that the fundamental matrix satisfies this epipolar constraint, which we've talked about all this time, for all of the pairs of corresponding points. So specifically, if we let P and P prime be defined in this homogeneous form, where U and V are the pixel coordinates in the left image, and then U, U prime V prime are the image coordinates in the right image, we can actually write out that exact same p transpose F P equals zero constraint here on the right, where we kind of leave the elements of F as well undefined because we don't really know them now, so we'll just leave them as symbolic variables.

Speaker 1

19:55 And then what we can do is if you actually write out the form of what actually you get, if you carry out this quadratic form you're going to get, you can actually rewrite that form linearly as such, because, well, you can kind of look at it if you take U times the first element, F times U prime plus all the other terms, you, you can see that, okay, well if we just factor out those first two terms, then we can, we can write this linear form.

Speaker 1

20:28 So now if we have this linear form, this is great because I mean, this is just for one set of correspond one pair of correspondences. If you now have eight of these correspondences, then you can actually fully determine the the matrix F. And if you have even more, then it becomes a least squares problem. And that's great because we know how to solve least squares problem very easily. So W is basically how we write these matrix of known correspondences.

Speaker 1

21:00 And the one thing to keep in mind is that even if we have all of these different terms, there is still some ambiguity. And so we actually use. So what we do is we m end up solving this least squares problem, but we do have a constraint that the norm of the learned F values is one. So this resolves our ambiguity with respect to that ninth point.

Speaker 1

21:31 I mean, this is called the eight point algorithm, but there's nine values in F. And so we really, we only need to find eight divided by the last one. And it's all good because the fundamental matrix is, you can use it up to scale. So it's completely fine if you have differently scaled values. But that's why we normalize such that the largest is one or the ninth one is one. So while this F that we just learned will satisfy the epipolar constraint, we never talked about that singularity of F, which is also very important to enforce because, well, we need the epipols to map to zero when you multiply them in the fundamental matrix.

Speaker 1

22:10 So specifically we're going to enforce this rank constraint via svd, it's not very important that you understand exactly the internals of this, but just know that the way that you can formulate and solve this problem is if you, it's again a least squares problem over the Frobenius norm of the matrix where you have a subject or you have a constraint which is that it's not, not invertible. So in total, in sum, the eight point algorithm is to use the normal least squares that we're used to to compute an approximate fundamental matrix, F, and then to fix up the fact that we're missing this singularity property with svd.

Speaker 1

22:59 So now if we have the fundamental matrix and we can go between these two images in terms of finding the epipolar lines in both of them, something that we might be interested in doing to make our search over these lines easier. Well, the problem that you have is, I mean, I'm trying to think in terms of like, is there a massive difference?

Speaker 1

23:42 It really depends. It really depends on the scene that you have in front of you and like how good your correspondences are. But really that the key thing is that you have to make it non invertible because if that doesn't hold, then your epipolar, your epipoles then don't map to each other, which means that you kind of have two baseline lines. And that's very weird. That's like a very strange. So even if the, like, even if numerically the matrices are not very different conceptually, it is very important to actually fix this condition.

Speaker 1

24:16 All right, so does anyone have any other question? Yeah,

Speaker 2

24:23 using the UV prime of D prime and of course the same point, but how did you know they were.

Speaker 1

24:30 Yeah, the core question. Yeah, so the way you actually know this. So unfortunately this is just kind of the answer here is just that we look at the two images and we just, you know, either a human can do this or someone else, someone who's kind of trying to solve this problem here for us. You pick points and you say, I know that this point corresponds to this point. And there's like software where you just kind of click in one image and you click in the other and you say these two points belong to each other.

Speaker 1

24:56 So you kind of, you actually have to provide these initial correspondences. And then, I mean, to be honest, that's one method of doing it. Another method of doing it is just by starting out using your naive 2D image search, right? Where if you have a point in one image and you're like, okay, well I want to find this point in the other. Well, I can just scan across the entire second image, find it, and then. Okay, great.

Speaker 1

25:20 I know that these two correspond to each other via some similarity metric, some visual similarity metric, and then just do it again for other points. But there's like. Again, this can be noisy. Yeah.

Speaker 2

25:40 So.

Speaker 1

25:40 Yeah. What? What? That's a great question. So we saw before here that F has a couple terms in there. So if your cameras, like physically the cameras are the same, then your k's are the same. So that's great. But it's exactly as you pointed out. Yeah. They have to be set up the same way because if you move them, I. E. Your translation changes or your rotation changes, then F will change. So F is only as good.

Speaker 1

26:01 F is only good for the specific scene setup that you have at that point in time.

Speaker 2

26:13 Yep, yep. Can you explain one more time why has to be singular?

Speaker 1

26:19 Oh yeah. F has to be singular because specifically the epipol must map to zero. You cannot have a line that comes out of. If you try and find the corresponding epipolar line when you pass in the other image's epipolar, you have to get zero. How can you get a line from that? Right. And so now what does that mean? That means that this metric has a non empty null space. You might remember this from linear algebra, you might not.

Speaker 1

26:48 But this kind of important property of the matrix means that it is not invertible. If you have a non empty null space, your matrix is not invertible.

Speaker 2

27:01 In a specific way such that the sensitivity of L on P is like weaker depending on the noise of that, the sensitivity.

Speaker 1

27:14 Could you.

Speaker 2

27:15 Could you explain your F is a. Is a. Is a noisy measurement, but it depends on your. On the frame of reference between the two. So do people choose that in a specific way or is this. I guess I imagine that if the cameras were super far away, that would cause a problem. If they were really close, that would also cause a problem.

Speaker 1

27:32 We're going to talk about this actually. Yes. Yeah, we are going to talk about this. About, you know, baseline distance, for example, and like extreme rotation stuff. We're going to talk about it. Hold on to that question in your mind. Any other questions about fundamental matrix and estimating it? Great. All right. So something that's so okay. If we have this kind of general setup where we have epipolar lines that are. Well, they can really be anything.

Speaker 1

28:01 I mean, they can just be general lines in the other image. Something that's kind of annoying is searching along a diagonal line is kind of annoying in image space because if you think about it, what is an image? It's a 2D matrix, which means it's discrete. I mean, especially in a computer, it's discrete. You have specific pixels. So if you now have to search along this diagonal line, that might cut through some pixels but not others, and cut through them, I don't know, 40% of one pixel and 40% of the other.

Speaker 1

28:28 That's very annoying to work with and quite slow to iterate through, especially in computer vision. And so what we do is it's. Well, ideally, we would actually just work with parallel images where specifically there's no rotation between the two of them, but there might be a translation. And so this is fantastic, right? Because first of all, your epipolar lines are horizontal. I mean, that's okay. That's cool. What does that mean? That means that we can just search along one specific coordinate.

Speaker 1

28:58 We know that if you have a point here, you know that all you have to do is search along one specific row of the image and the other of the other image. And this is now much faster computationally to deal with. And it basically simplifies your correspondence problem, which is one of the two fundamental problems that we were talking about earlier. So the real question is, is it possible, if we do have two images which are kind of generally rotated and translated, is it possible that we can somehow rectify them such that they end up having parallel or horizontal epipolar lines?

Speaker 1

29:37 So, of course, the answer is yes, because we've engineered it. So. But basically what you can do is now, if you know what the rotation and translation is between two cameras, in between the two images that they produce, you can actually do something called homography. I think that's. It's called homography in the general case, but it doesn't matter. You can do. You can perform a specific transformation to one image or to both images in order to rectify them such that after this transformation, they are both parallel views of the same scene.

Speaker 1

30:13 So you can see, of course, that this will warp the image, but that's just something that we have to accept as part of the territory. You absolutely can. I mean, we might see this later on in the slides, but basically, most robotic sensors, I mean, they just put the cameras side by side because then you don't have to deal with this. But we're just saying, in general, from the theoretical perspective, this is one way by which, from two weirdly rotated cameras, you can obtain, computationally nice and easy to work with, rectified images.

Speaker 1

30:51 But, yeah, that would Be like, that's actually what's done in practice. All right, so if we come back to a high level overview of what we're doing and what we're solving. So we have these two problems, which is this fusion of features that we observe by two or more cameras. So this was called correspondence. This is finding one point that matches another point. And then we have to actually using these matched points, we then have to actually reconstruct the scene that we took an image of in 3D.

Speaker 1

31:26 And so to specifically now solve to address this correspondence problem, we need to find observations that match. Like we have to find observations P prime that match P. So this is another question that came up earlier. And there's two main methods, methods that we can use to do this. The first is area based. So these algorithms, they usually consider like a small patch kind of illustrated here in one image.

Speaker 1

31:56 And then you look for the most similar patch in another image. And you can do this by means of some visual similarity metric or some visual correlation metric. It's up to you as the designer. But the nice thing is that if you have this rectified image, which we talked about how you can get, then if I have some image patch in one image, I can just very easily scan along that exact same row and see if I can find some image patch that looks similar.

Speaker 1

32:28 Now, the problem or the benefit of using an area based correspondence method is that it's quite dense. What that means is you can basically cover your entire image in points that you want to match. And then because you're not just looking at a single pixel that'll be very hard to match, you have a single pixel as well as some context around it. So you can actually do a pretty good job of matching up one patch in one image with a patch in another image.

Speaker 1

32:59 The problem is that in regions where your image is quite uniform, like for example, if you're going to take a picture of this whiteboard and you're going to try and match two images of this whiteboard, which is very textureless and featureless, area based methods can fail. And so all that nice information about dense about this dense correspondence that you want is kind of out the window. So another popular method are feature based methods.

Speaker 1

33:29 And so what these are is basically these algorithms first try and extract some salient features, features in the images and then match between those. And when I say what salient features means is these are basically some robust, these are some points in the image for which this algorithm has determined that they are robust with respect to, let's Say lighting changes, rotation changes, scale changes, stuff like this. And there's many popular feature, there's many popular algorithms for this.

Speaker 1

34:00 You might have heard of some of them, like sift, maybe you've heard this term before, Surf, orb. These are just different kinds of, these are like different names for different algorithms that find features that are robust with respect to scale, shift, orient, rotation, stuff like this. The problem with feature based, or the benefit of feature based methods is, well, they're more robust. So even if you have textureless images that you're trying to match together, you might actually have an easier time finding correspondences.

Speaker 1

34:32 The problem is that usually you don't have such a good dense covering of the image. So although you might have a good matching a good correspondence, you're going to find less points. So you're probably going to have a more sparse, like a more sparse depth map that comes out as a result of that. And then you need to interpolate that and you might have errors along the way. So if we have two parallel images and we have now points which are matched between them using either of these algorithms, then we can finally try and solve this problem of finding capital P in the real world, given what we've seen about it in these two images.

Speaker 1

35:18 So the way we can do this is via triangulation. This is something that you guys have seen before where you use similar triangles in order to find how far away something is from the camera, at least theoretically. And so the way we can do this is by defining something called disparity. So disparity is the difference between how far one point in one image is from the other point in the other image.

Speaker 1

35:49 And then it's actually from this it's easy to show. So we show you the kind of the resulting formula on the right. But from this it's easy to show by comparing similar triangles that the disparity which is here, Pu minus Pu prime, is inversely proportional to the distance z of the point capital P from the image plane and it's directly proportional to the baseline B. And so this is actually, we'll talk a bit more about this, but this actually is why having differently spaced cameras either far apart or close together has different properties when you're dealing with 3D reconstruction.

Speaker 1

36:30 So thus this equation here can be used to compute the depth from this baseline distance, focal length, which we know, and then as well as this disparity which we got from corresponding these points in the different images. So yeah, as I was alluding to before, so some properties of having different sized baselines so if you have a very large baseline, you can resolve quite far, like pretty well because you'll have very large. You'll do really well to, to resolve farther away points.

Speaker 1

37:06 The problem is, if they are, if they are quite far separated, then points that you see in one image you might not see in the other image. So you kind of lose a portion of your image that you can't see because it's too far apart. Now the other thing is that if you have points that, if you have cameras that are very close to each other, you now don't have this correspondence problem because almost all the points in one image will be in the other.

Speaker 1

37:30 The problem is you're going to have a lot of depth error because what's it called, your disparities are going to be very small and might even be sub pixel for points which are sufficiently far away.

Speaker 2

37:43 When you rectify two images so their

Speaker 1

37:47 image planes are parallel to each other,

Speaker 2

37:49 what happens to the focal length? So that becomes different as modeling.

Speaker 1

37:57 No, so when you do this transformation, you're keeping. What you're doing is really just fixing the rotation such that they're parallel because you're still keeping that origin point behind the image plane and you're just kind of rotating this image plane such that it is now parallel to the other. Or in this case, in this case you rotated both.

Speaker 2

38:22 But.

Speaker 1

38:23 Yeah, so. Yeah, does that answer your question? But I will say this. The other thing is, even if it did change your focal length, actually. No, it does. Yeah. No, it's not a concern simply because it keeps all the other properties about the camera the same. It really, all it does is really just change the R with which they're rotated with each other.

Speaker 1

38:56 Where were we? Okay, we were here. So the output from doing this triangulation process for many corresponding points is something called the disparity map. So I mean aptly named because you're using disparity compute to compute this, but basically you get an image which looks like the right. Where you might color it such that lighter points are closer to the camera and darker points are farther away. But basically, given that you have some dense correspondences from one image to another, you can compute the disparity, which is that distance, the difference in position in the image of one point and another, and you can compute the depth.

Speaker 1

39:37 And the one question that's interesting is if you look at these two images, it seems like we can kind of, maybe it would be hard for us because the images look pretty similar, but it seems like you'd be able to come up with a pretty robust 3D reconstruction of the scene. But if we look on the right here, I mean, yes, it looks like we came up with a good 3D reconstruction, but we have these black regions and that's a bit weird.

Speaker 1

40:01 What are these black regions? I mean, why do we have some, some points for which we can't resolve any information? And so, well, the first answer, especially this is something we talked about earlier, especially for this left side, is that we have some points in one image which we cannot correspond to the other because, well, the camera can't see them. And then the other ones are, well, occlusion. And this is like another, this is another very fundamental problem in computer vision is if objects are blocked by objects in front of them and so they kind of, they almost form these like shadow, like appearances.

Speaker 1

40:35 But I mean, from a computer vision perspective, occupancy kind of looks like that anyways. You have one object blocking another behind it. So now we have this kind of solution for if we have two cameras and one scene, even if we have general rotations and all that, what if we have multiple cameras and one scene? Can we do even better? Or conversely, what if we have one camera but that one camera moves through the scene?

Speaker 1

41:11 Because now, I mean, yes, having two cameras is great. What if it's too expensive? What if you only want to have the one camera and then just kind of move it around and pan it around some object that you want to scan? If you think about it, if the object's not moving, maybe if you take some snapshots every one second, then if you move this one camera, it's kind of as if you have, I don't know, three or four cameras that are just placed differently throughout the scene.

Speaker 1

41:35 So in Structure from Stereo, we address the problem of how to recover the structure from the environment from two images of the scene. So here, structure from motion, we're going to talk about recovering the structure when one, the camera relative pose is unknown. I mean, if I am moving a camera between, I might have no idea what the rotation and translation are. I mean, especially because I'm a human and I'm just moving it. So suppose we have m cameras with camera transformations MK between them.

Speaker 1

42:07 So this encodes both the intrinsic and extrinsic parameters of the camera. So remember, intrinsic are like the different focal skew, whatever you guys learned about last time. And then extrinsic are rotation and translation. Then the different capital PJ are different 3D points in the scene that we want to reconstruct our Aim here is to recover both the structure of the scene, so these 3D points, as well as the motion of the cameras, specifically these projection matrices mk, which we also want to find.

Speaker 1

42:48 So the first thing that's important to realize is that in structure for motion, it's actually impossible to recover the scale of the scene. So the problem that we have here is that let's say here we have some large cube, and then I take one camera and I move it to the other side and I take an image that's going to look exactly the same as if I have a smaller cube and I put the camera kind of closer to it and then did the exact same, well, not the exact same transformation, but a similar transformation.

Speaker 1

43:21 So the problem is it's actually hard to recover the absolute scale. And intuitively this corresponds to a well known property of the perspective projection, which is anyways how we see the world. So the apparent size of objects depends on their distance from the camera. But the problem is if you have an object that's twice as large, twice as far away, then well, it's going to look the same as something that's well, the same size and two times closer or smaller and two times closer.

Speaker 1

43:51 So the reconstruction that we're going to obtain is going to be correct up to a similarity transform. Similarity transform means rotation, translation and scaling. Because who knows how an absolute sense this scene was reconstructed, was constructed. So at a high level we're not going to go super deep into like how you actually solve this problem, but at a high level structure for motion, there is an algebraic approach which actually just continues exactly what we were doing before, where you leverage this concept of a fundamental matrix between two images to solve this SFM problem.

Speaker 1

44:31 So what you could do is you first compute this fundamental matrix F via the eight point algorithm that we had from before. And then you use this to estimate the projective camera matrices. And then you use these matrices to determine the world coordinates of the points. And now so, and, and then basically you can, you can use these projection camera matrices to get the triangulation. So you can do exactly what we did, but you kind of work with pairs of images and then it basically becomes a stereo vision problem.

Speaker 1

45:04 You're just kind of ignoring all the other cameras. Well, you're not ignoring them, it's just that you're kind of, you're using the two images to get the 3D points, then you use the next two images, get the 3D points, next two images get the 3D points, etc. Now there is another, and I Will say significantly more popular method called bundle adjustment. So bundle adjustment is a nonlinear method for solving the SFM problem. So computationally it's more difficult, but the results you get are significantly better.

Speaker 1

45:37 And this is actually what a lot of methods will use for structure for motion, for solving structure from motion. So the way the bundle adjustment works is that internally it's an optimization problem. And specifically the optimization problem is to minimize something called reprojection error. Reprojection error is basically, if I believe that a 3D point is somewhere out here, and I then try and project this back into my cameras, how far away is that projection from the actual observed point in the image plane?

Speaker 1

46:08 So this is a way that, and the reason it's called bundle adjustment is that if you draw lots of these projections from lots of 3D points to the, to the camera image plane, it just, it kind of looks like bundles of lines. So you're adjusting these bundles of lines. so fundamentally this, this bundle adjustment optimization method, it's actually pretty similar to the triangulation method that we talked about. It's just that it's a bit more involved mathematically, but it does give you better per better performance.

Speaker 1

46:41 So a specific application of structure for motion that is actually very useful in a robotic sense is something called visual odometry. So visual odometry is the problem of estimating the motion of a robot or that of a vehicle by using just visual input. And specifically, the basic principle behind visual odometry is a simple iteration of this two view structure for motion, where you kind of take two camera, two, two image views, do the whole SFM approach, do it again, and you can recover this rnt, and then you use this RNT to inform your own odometry.

Speaker 1

47:22 So specifically, visual odometry, it doesn't really care about all the other reconstructing 3D. It really only cares about localizing the robot and localizing its motion through a scene. But it's not uncommon at all to see results also showing the 3D map of the environment that is output by such methods and actually what we're seeing here. So if anyone can take a guess as to where this image and where this visual odometry algorithm was used, does anyone recognize it's on Mars?

Speaker 1

47:56 So the rover, the Mars rovers actually use visual odometry and they actually use computer vision in order to inform their own localization. And what's kind of interesting about that is that if you think about, well, I guess who here is aeroastro, Eh, good enough, good enough amount of people I'm going to tell you right now, Mars looks kind of like the desert. And deserts are kind of featureless, which is annoying, especially if you're trying to do computer vision.

Speaker 1

48:28 But the great thing is that every time you have a couple rocks or something, which happens frequently enough, then you get a pretty dense amount of information. And actually what we're specifically showing you here is an image from the camera from the Mars rover's camera and these lines. So the blue dots are where those points were in a previous frame and the red dots are where they are now. So what you're seeing is how the image, or at least how the rover moved forward between one image to the other and how each of these pixels were pulled backward.

Speaker 1

49:02 So maybe I flipped that. Maybe red is where they were. Yeah, red is where they were before and blue is where they are now. So that's how the kind of pixels were moved backward. All right, so that's kind of all I have to actually say about 3D reconstruction. But next time, what you guys are going to look into is a bit more in depth computer vision methods, specifically ones which deal with, well, altering different image properties and applying different operations to images.

Speaker 1

49:30 Specifically something like how could you blur an image? And also how could you do something like detect edges from images? So that's all I have for today.

© transcribe · For agents Built with care and craft by Gokul Rajaram