Transcript
0:05 So this is mostly just an introduction to deep learning. We're going to talk about deep learning, or the application of deep learning in other areas, in future weeks, as well, like in diffusion models, in large language models, so on and so forth. So these two lectures are mostly just setting up the basics of deep learning, especially in the context of supervised learning. We are going to introduce just a general framework for supervised learning for nonlinear models, which is pretty much the same as what you have seen in the lectures by Chris again about linear models.
0:38 You just extend it in a pretty simple way to nonlinear models. Then we're going to define neural networks, which is the most popular nonlinear models. If it's not the only nonlinear models people are using. Of course, there are many, many variants of neural networks. We are going to define some of them and some of the basics. And with this lecture, we're going to talk about back propagation, which is about how do you compute the gradient of the loss function with respect to these models so that you can train these models by computing the gradient and update based on gradients.
1:10 And I'm going to use the whiteboard. And the reason mostly just because I found that when I use the slides, there's only one slide there, so there's not enough memory there. So once I switch to the next slide, sometimes people don't remember the previous slide. And also this is a regulating effect for me to slow down a little bit, because after teaching this for so many years, I tend to speak fast and go very fast. And this, at least, you see six bars as memory.
1:41 So you can check them if you forget, and I can check them if I forget about something. And it's still not that slow in some sense. And feel free to ask me questions, because, as I said, once I teach this kind of lecture so many times, I tend to move a little bit faster than I should do. And also, we did update the lectures over time. So some of these new materials are new, and we drop some of the other materials that are out of date.
2:11 But generally, still this lecture is still just a very basic stuff for today, you know. So I think Chris has talked about a linear model, which is, you have a model, which is parameterized by theta, and the input is x. Let's just recall the notation, so that we're on the same page about notation. I think Chris is using almost exactly the same notation. It cannot be exactly the same, because sometimes it's a nonlinear model. But for linear model, recall that if you have a linear model, then the model is kind of like this where you have theta-- H theta x, which is equal to theta 1 x1 plus theta 2 x2 up to theta D xD where x is the input, which is a D-dimensional vector.
2:57 And theta is the parameters, which are also a D-dimensional vector. And once you have nonlinear models, what does that mean? Generally, nonlinear just means that as long as this form is nonlinear then it's nonlinear. And here we are mostly talking about nonlinear in the parameters, because this nonlinear in the data is not nonlinear enough in some sense. So suppose you have something like theta 1 x1 squared plus theta 2 x2 squared. So on and so forth.
3:26 So the D xD squared. I think we didn't talk about this kind of models, because we skipped them, in some sense, in the new version. But this is not sufficient nonlinear to be interesting, because you can just define new data, which is the square of the old data. So suppose you say I'm going to replace this by Z1 and replace this by Z2, and then it still becomes a linear model. Another way you are not learning-- you are not trying to find out-- you're trying to learn the theta, theta i's.
3:54 You have to figure out the parameters of theta i. You don't really have to figure out-- there's no learning on Zi's. So whether it's X or Xi square, it doesn't really matter. So that's why this is not exactly what we're talking about. Even though some people will say this is still a nonlinear model, but this is not exactly what we are talking about. We're talking about cases, like you have theta 1 squared, for example, X1 plus theta 2 maybe.
4:16 I'm just giving an example, something like this. Whatever nonlinear model you can write. As long as it's nonlinear, it's called nonlinear model. So of course, the real model we are talking about is not like this. This is just one example. So the model we are going to talk about is neural networks, which is more structured and more specialized way. But before talking about neural networks, let's talk about how do you learn nonlinear models and what's the framework?
4:45 This is, in some sense, mostly just recalling what Chris was talking about with a little twist about the model. So basically the model can be a general function as opposed to a linear function. So let's just quickly recall some of this. So let's say you have a data set. I think Chris probably mentions the concept of data set, which is a file of data points with pairs of inputs and outputs. So here, let's talk about supervised learning.
5:12 By the way, in other cases, we have data sets of different format. But here, if you have a supervised learning data set, you have a sequence of input and output pairs. And let's say we have n of this. And let's say we assume Xi is in D. And just recall that in this lecture, in this whole course, we are trying to use this notation where the superscript is for the indices of the examples, and the subscript is for the dimensions, the coordinates.
5:41 So when I write x sub I, this is the first coordinate of the vector Xi. And when I have superscript, it's indexing one particular example. Go ahead. nonlinear just x, or is it? And what is the difference between the square root expression and the one variable? Yeah, that's a great. So if you are only nonlinear in x but not nonlinear in theta, then it means that you can replace the x by something else.
6:16 You can replace Xi-- for example, in this example, you can replace x1 square by Z1. And then you just say Z is my input now, and then I have a linear model in both Z and theta. So fundamental is the same as linear model from a learning perspective. But here, you cannot do this. There is no way you can easily transform it into a linear model. Actually, there's no way just fundamentally. So that's why it requires different kind of-- slightly different kind of techniques.
6:44 So some of the linear models techniques cannot work anymore. And that's just one example of a potential nonlinear. Yeah, yeah. You can write-- yeah, yeah. This is just the one. You can even add more things. You can take more square roots. Whatever. You can take squares. Yeah. You'll see some real examples. Yeah. can we just add another data and x1, so not like an x1 squared with that, so it can benefit the problem. In which cases?
7:11 In this case. Yeah. If we just had no square. OK So you would add on theta, like D plus 1 x1, etc., and so forth. I didn't get what you mean, so you add on something here. Yeah, x1-- you take x1 and split it. So you add? Yeah. But then you change the model. Suppose my model is really-- suppose my modeling assumption is just literally this. What my argument was that you can replace x1 squared by Z1 so that you maintain the model's functionality, but you can reapply the same techniques that is useful for linear models.
7:56 OK, sounds good, so anyway. So how do we-- so now let's just assume you have a general-- OK. So let's talk about the framework. And let's say we have a regression problem. I guess Chris lecture talk about regression versus classification. And for regression problem, let's say-- one of the distinction between regression and classification is that we have regression problem in the y(i). The labels are a real number.
8:28 These are real numbers. And then your model, let's say, is h parametrized by theta, which is a function that maps the input, which is the Rd to a real number R. And you can have a cost function, or maybe loss function. I think these days people use the word loss function more often. In the lecture notes, we still use cost function, but most researchers understand what you are talking about. So this is a most likely-- of course, there are also many different type of mean square error functions or loss functions you can define.
9:07 By the way, error and loss and cost is the same. Let's just use loss for all the cases. And the loss function could be something like this with j for the ith example. The loss function is a function of the parameter, because the data is not something you optimize for. It's not a variable. It's given. So loss function for the jth example is y(i) minus H theta Xi. You apply the data-- you apply the model to the data, you get some prediction, and you compare that with the real label, and you take the square.
9:44 Something like this. And then you can also have the average loss, which is something like 1 over n times the sum of all the losses. And then you try to minimize this function over the parameter theta. So that's for regression. And let me give another example. I think this is mostly just the repeating some of the stuff we have discussed. The only thing here is just that this is a nonlinear model as opposed to linear model.
10:15 So in this lecture, I think this is just theta 1 x1 plus theta x2, so on and so forth. And I guess for multiclass classification, we can also talk about single-class classification, which is actually in the lecture notes. I think I just-- given an interest of time, let's just only talk about multiclass classification. Basically, you just have to define the loss function the same way as we have done for linear models, except that you replace the models by nonlinear functions.
10:45 By the way, I think in most cases, you don't have to think about the superscript i because this is just indexing examples. So if you remove-- if you say I only have one example x and y, then everything without a superscript still works. Maybe I should just write in that notation to simplify it a little bit. So let's say you have-- so in this case, the y-- I can write y superscript i for the i-th label, or I can just use y to indicate one label.
11:13 Let's say suppose you have one label-- in multiclass classification, you have a discrete choice of labels, 1 to k. And here, I guess you can say you have a model, let's use h bar, which is mapping the input x. So the x is still in Rd. Think of this as an image. Think of this as some kind of labels for the image, which is mapping the input to k numbers.
11:44 And these k numbers are often called logits. This is the same logit in the linear case. And then you say, with these logits, the predicted probability of each class, for this example, x, let's say class j, is this kind of exponential form.
12:16 You look at the j's-- this is the j's output right. Index j means that this is the j-th output of this k dimensional vector. And then divided by the sum of the exponential of all the-- this is the so-called softmax. So this is the so-called softmax. Sorry.
12:48 So this is the so-called softmax function of this h. So basically, you say-- yeah. So basically, your model is like this. So this is your model. So P of y is equal to j given x is this. This is my parameterized model for the prediction, parameterized by theta. And then I need to compute the log likelihood. So I'm going to use negative log likelihood, which is also what we have discussed before.
13:23 By the way, here, this is also log likelihood model if you assume there's some Gaussian probability distribution. I didn't really define go through that calculation. So if you think of that your probabilistic model is something you either Gaussian to the h theta, and then you compute the log likelihood under the Gaussian, then it's the same mean squared error. Here, I'm just going through more details. So I define the probability distribution. I compute the log likelihood on the data.
13:47 So this is the minus log P of y given x and my parameterized model. Then this is a minus log of this thing. Oh, sorry. I think one subtlety here is that this is the-- so this model specifies what's the probability for every choice of j. But now I have a particular choice of j which is equals to my label y.
14:18 So that's why I'm plugging my y. So this is exponential h theta bar x. This is for the particular label I'm given the y-th index-- the y-th cardinal of that logit vector divided by the-- the denominator is the same, just the sum of the exponentials. Oh, I think I have some typo here.
14:54 And this is just called the entropy loss. I think if you look at the multiclass classification case, this is the so-called cross entropy loss. You can have many different form of it, but this is the cross entropy loss, where the-- you can just write this h theta bar x, y. So this is the logit. This is the label. And you can plug in the cross entropy loss if you use PyTorch to implement it, I think you can find a module, which is just called cross entropy loss.
15:29 And you give it to the inputs, and you can compute the loss. But, of course, the underlying math is this. This all sounds somewhat familiar to some degree? Good. Any questions? Can you explain why it's called cross entropy loss? Is there an entropy? It's cross entropy-- oh, so what's the second part of the question? Where can we see entropy with the equations? Yeah, I think this is the cross entropy between the probability distribution defined by h theta x.
16:11 So I think-- here's the thing. So the question is why this is called cross entropy loss? I think if you-- how do I write it in some sense? I think if you have this distribution-- let's call it maybe P theta. And I think this cross entropy loss is actually equals to the cross entropy between the true-- I guess you have to take-- if you take the expectation of this l cross entropy loss of this h bar and y and expectation over y from the true distribution, from the true label distribution, whatever you call it, and then this is equals to the cross entropy between this distribution P theta and the true distribution P star, which is something like you-- I guess the cross entropy is just-- it's actually the probability of one distribution times the log of the probability of another distribution.
17:33 Yeah, I think the derivation is probably a bit long, especially if you're reading this language, a little bit cumbersome, but this is fundamental. I think actually, probably I have it in the lecture notes I forgot exactly. Yeah. Any other questions? What would be a one-hot? What is the one-hot vector? Just, like-- In my case, in my notation, y is just the discrete variable. So that's why you can see I have h theta bar theta x indexed by y.
18:03 y is like 2 or 3 or 4 or k. But you can also encode write y as a one-hot vector, which is the same thing. So then your math needs to be a little bit different. Then this is going to be the inner product between this vector times the one-hot vector. But here, I'm just using y is really a discrete choice between 1 to k. Can you explain again the regression as for why it should replace the Gaussian distribution?
18:29 So, can you repeat the question? Sorry. You mentioned with regression, if something is Gaussian distributed, you have some-- it simplifies to-- So the question is y for the regression case, it also is not log likelihood. So I think the-- let me try to see what I can do it on the fly. I think it should be easy. So I think the modeling-- so if you make the modeling assumption that if you assume, suppose you assume that your modeling assumption is that y is generated from say, h theta bar theta x plus some Gaussian noise with epsilon is from N 0, maybe sigma squared, something like this-- Then the likelihood of y given x is, I think, 1 over-- I guess this is challenging my memory of this, something like this, exponential y minus h theta x squared over 2 sigma squared, I guess, something like this.
19:39 And then you take the negative log likelihood. So then this is a log of P y given x is equal to some constant-- you take the log. You get a constant, which doesn't really matter. And you got the log of this, which is 1 over 2 sigma squared y minus h theta x squared. So then it's kind of like this. The scaling doesn't really matter because it doesn't change the optimization function fundamentally. Sounds good.
20:14 Any other questions? Cool. And in this case, here, I only have one example, eventually on average over many examples. So I guess maybe just-- let me right here. So if you continue here, so eventually, your real loss is something like the final loss is the average over all the examples.
20:44 You take the cross entropy loss of each of the examples xi and yi. And each of these function is a function of theta. So basically, your total loss is also a function of theta. And now, the next task is to optimize to find theta that makes the loss smaller. The same, I think, we also introduced a little bit about gradient descent like Newton method.
21:15 So these are all ways to optimize things. So here, you cannot use Newton method because the function is not convex. So that's why Newton's method is not guaranteed to work. Actually, no methods can guarantee to work for all nonconvex functions, but stochastic gradient sense seems to be a good idea. So just to quickly recall, you can use an optimizer, so optimization. I think one of the option is you take gradient descent.
21:46 So let's say, suppose you have a loss function. Let's just summarize all the cases as this. Let's be a little abstract. So let's say each of the examples have the loss ji of theta. And you have an average of these loss functions. Within this sum means that you just say I'm going to ignore the structure here. I just treat this as a single-loss function. I take the update. Theta is updated to-- I think we use this notation to denote update-- something like eta times gradient of j of theta.
22:17 So this is gradient descent. And how do you compute this gradient? You compute the gradient of each of the loss, the gradient-- sorry. You compute the gradient of each of the loss with respect to parameters, and you add them up. So that's gradient descent. And for stochastic gradient descent, so what you do is you say, your first sample, say, B-- Let's see. So you sample-- I guess there are-- maybe let's say-- the most vanilla version of it is you sample one item.
22:54 You sample j from 1 to n uniformly. And then you run this. So, of course, there's a for loop. And you sample, and then you say theta is updated to theta minus eta nabla the cost of loss of example j theta.
23:26 And kind of the justification here is that if you think about what this is-- so the difference here is that here, you compute the gradient of the average loss, and here you compute the gradient of one single example, the loss on one single example. And the justification here is that you say, even though this is one example, it's not actually the loss of the whole gradient. But actually, if you look at the expectation of the per example gradient, and your randomness come from j, which is sampled from uniformly from 1 to n, this is equal to the gradient of the average gradient, the gradient of the loss function.
24:11 So, in some sense, this quantity is an approximation of this quantity. And the approximation is in the sense that this quantity in average is equal to this. It's just that sometimes it's biased or-- it's never approximate. Let's say suppose this quantity is in this direction, and this quantity is just-- it can be any directions. But if you average out all the possibilities, you get the original direction you want to go. So in some sense, you have a much larger noise.
24:37 So if you do gradient descent, it's-- I guess, maybe I use two different colors. So suppose you use gradient descent. This is the gradient of the average loss. And stochastic gradient descent is stochastic. You may go here. You may go here, or you may go here. Maybe sometimes you may even go in the reverse direction. But in average, all of these blue ones will average out in the same direction as the black one. And then if you have a trajectory-- so suppose the black one goes like this.
25:15 The blue one, because it's stochastic, you have noise, so it will go look like this, maybe something like this, maybe like even-- but in average, it's roughly tracking the gradient descent. Of course, how much the two things tracks over time is a very complex matter. It depends on how large the step size is. It depends on many subtle things. But conceptually, you should expect that the blue one-- some would track the black one.
25:45 The blue one, in average, is exactly equal to the black one in one step. But the error sometimes can be amplified. Sometimes the error can be shrinked. So it depends on the situation. Ideally, you want them to track to some degree. Generally, you want to use stochastic gradient descent on a nonconvex? I think you can use gradient descent as well. But I think you guys have a question. This is a great-- I think it sounds like the question is really about why you use stochastic descent?
26:14 In what cases, you want to use stochastic descent? And that's a great question. So why you even consider stochastic gradient descent? The reason is that computing this gradient is very costly each time because you have so many examples. n could be-- in 2015, it's one-minute examples. In 2026 now, I think it's like 1 trillion tokens. There's no chance you can compute all the gradients. Even computing one gradient is very costly, let alone computing all the gradients.
26:41 So that's why you have to find an efficient way. So that's why you want to have a sampler which is less costly. And maybe just one-- and also another reason you can do this is that computing the full gradient sometimes is actually wasteful, because maybe it's the case that all of these individual gradients are actually they are correlated. They are highly correlated. So there's no reason to compute each of them to understand which direction should go.
27:12 So you just only need a sub-- it's like every example tells you to go that way. Why you have to compute all of them? You just need a subset of them. And another thing is that even this is not entirely true, in many cases, you don't really have to be very precise in this dynamics. It's kind of like, even you have some mistakes, it will be corrected later. So maybe another way to think about it is that for some cases, for example, if you have a convex function, so in some sense, one way to do it is you do this.
27:46 But sometimes, if you overshoot or maybe you go in the wrong direction, still eventually, you're going to go down. How do I say this? Basically, even you have some perturbations-- and this is not exactly the right example. But even you have some perturbations in your dynamics, then over time because you are-- every time you have to share our target, eventually, those kind of perturbations got kind of washed out by the shared force in some sense. So I just wanted to ask on a nonconvex function, if I get stuck at a local point, and I believe that this is going to use stochastic gradient descent?
28:24 Are we still able to use gradient descent, or would it not? Yeah, I think that's a great question. So one of the theory for why stochastic descent is good is that it can work better on nonconvex functions. And let me first expand what you said. But actually, it's not entirely right. So basically, suppose you have a nonconvex function in 1D, let's say, like this. So if you start here, you keep going down. This green just means you go downhill in some sense.
28:54 So you go to the directions which is going downhill in this-- Sometimes it's called steepest descent. It's basically you're finding the direction, which is going downhill the fastest in some sense. So you just go here, and you get stuck here because you can never leave this place right here. The green is 0. The curvature is 0. And I think what you're trying to say is that if you do stochastic gradient descent, because there's some noise, maybe even though you want to go this way, maybe the first step you just-- because of the noise, you go the other way, and you jump to another local minimum.
29:27 I think this is plausible, but generally speaking-- of course, the theory here is very limited, but I think the community don't believe that this is the case. This is why stochastic descent is helping. I don't think people believe that stochastic descent can help you to jump out of a local minimum. However, on the other hand, we don't even believe that there are many bad local minimum. So in 1D, it's actually very easy to have local minimum.
29:53 But in high-dimensional cases, it's not that trivial to have that many local minimum. So I think, at least, one-- actually, I was involved in developing some of the theory. Of course, nobody really knows exactly whether they are right, but the common, let's say, hypothesis or hypothesized consensus in some sense is that they are very limited number of bad local minimums in this high-dimensional function you are optimizing. Maybe there's even no bad local minimum. All the local minimum are actually global minimum.
30:23 It's like, you think about it, you are skiing in a mountain. I think if you just randomly try to go skiing down hills in 2D, you're probably going to go back to one of the parking lot, which is one of the global minimum. But in 2D, I think it's a lot easier to have a local minimum. You probably go backcountry and you got stuck somewhere. That's possible, but you always find some river going down-- you always find some valley going down eventually.
30:51 I'm just going to give an analogy. It's not exactly sure in 2D. But in high dimensional, it's more likely to always find a way to go downhill in some sense. Could you give some allusions to the mathematics that actually shows why, which I find to be counterintuitive, you don't see that many local minima in high-dimensional spaces? I just find that kind of surprising. Yeah, I don't think there's a very simple way to say it. I think it's not universally true.
31:26 So in some sense, I think maybe-- if I can only use one or two sentences, I think it's like, just in high dimension, there's always some ways to-- a local minimum means your Hessian at that point is just in all directions positive semi-definite. That's a very hard thing to satisfy. So you find a local-- you find a stationary point where the gradient is 0 and you get a Hessian, the Hessian is kind of always have some direction, which is negative.
31:56 And I was thinking about how to formalize this. So for example, if you think about it, the gradient 0 is already a pretty strong constraint. Suppose you have a D-dimensional function and you add d constraints, that each of the dimension of the gradient is 0. That's already restrict yourself a lot. And if you add another thing, which is the Hessian is positive semi-definite, I think that just makes the number of points much smaller. And in some cases, you can compute.
32:23 In one of my papers, I think we try to compute the number of local minimums by using this-- I think, in differential geometry, there's something called categorize formula. And actually, not only my papers, but many other people's papers. So in some certain cases, you can compute. You can count. And you find out that, oh, all the local minima are actually just those-- number of local minima is the same as the number of global minima. But generally, it's very hard to count.
32:48 Any other questions? Sounds great. So going back to here, so that's the algorithm people do. Of course, practically speaking, you don't also just only sample one example each time, because sampling one example-- using all the examples is too inefficient because you have to compute all the gradients. Doing one example, the downside is that it's too noisy. So sometimes you want to have a trade-off. You want to have a few examples so that you can be a little bit less noisy.
33:21 And another thing is that, which probably we won't go in detail here, but another thing is that when you have-- one examples, your GPU utilization is too low. It's not always the case, but in some cases, using too few of examples make your GPU utilization too low. For example, if you have 10,000 GPUs and you just only compute one example, one gradient each time, then most of the GPUs are actually idle. So that's why you need a so-called batch gradient descent.
33:47 Oh, sorry. I think it's also called stochastic gradient with some kind of small batches. I think SGD just means both the version with batch size 1 or a bigger batch. So when you have a bigger batch, you can imagine what it is. Basically, you just say, I'm going to-- with batches-- I guess let's call it SGD. So I think you just change the algorithm to be you sample say B examples from 1 to n.
34:23 And then you say theta is equal to theta minus eta gradient. Sounds good. So this is just a general introduction to the framework for the supervised learning.
34:54 And actually, in some of the unsupervised learning, it's the same thing. We define a loss function for each examples, and you average them. The only question is, when you have unsupervised learning later that we will cover, you don't have the y's. But still, the framework is the same. You define the loss functions, but the loss functions that involve y. And then you have average loss functions. And you run the optimizers, or you can choose different type of optimizers.
35:17 And then pretty much there are two questions we're going to address. So the two remaining questions-- question 1 is that, what is this function h theta? So this whole framework works for every function h theta and-- h theta bar. I think I use bar just because it's a little bit different. With all bar, I just want to distinguish it from the regression case, but they mean the same thing.
35:48 So question 1 is, how do you define h theta? And question 2 is, how do we compute-- how to compute the gradient j on each of the example? And once you know how to compute the gradient for each of the example, you can compute the gradient for all the examples or batched examples by averaging the gradient for every examples. So this is called backpropagation. The algorithm for this is called backpropagation or auto differentiation, which is the topic of next lecture.
36:18 And in the rest of this lecture, I'm going to define a bunch of h theta that people are using, mostly for a more classical cases like vision, cases like supervised learning. And of course, for language case, you can have even more complicated nonlinear functions like transformers. So that's the plan. I'm going to start with the very basics, like neural networks. So we are still using this kind of housing examples.
36:55 I know that this example is probably too old. It's 20 years old. So let's say suppose you have to motivate the introduction of nonlinear models. Suppose you have size, and this is the price. That's housing price prediction. So if you look at the real data, most likely, it's something like this. Maybe some-- I don't know exactly what it will look like. But at the beginning, it doesn't increase that much as the size goes up, and then eventually, it becomes linear.
37:29 Let's say this is the case we will see in the data. So if you use a linear model, you're going to fit something here. Then you're going to not be accurate enough. So that's why you probably have to have a nonlinear model, which is something like this. So you want your model to be able to output a curve like this. Of course, you don't really know which line is this. It could be like-- probably without seeing the data, you can have multiple curve like this.
37:52 But at least, you should allow the model to generate a curve like this, as opposed to only generating a linear function. And what this function is? So this function is actually of this form. So this blue function is actually of the form is a max of, say, wx plus b, which is a linear function, and 0.
38:22 I guess, especially if you-- suppose this is 0, then you just-- maybe let's say this is max wx b and this 0 plus C. So max of wx plus b is kind of like a function like this. And then if you-- sorry. This one is like this. And then if you-- plus c, you can shift it above or below a little bit. So basically, you just cut off everything that is below 0, which makes sense because for housing price, it cannot be negative.
38:56 It has to be positive. So actually, this introduced a primitive that is used very often in deep learning these days, which is this function, which is max of t and 0. So I think in mathematics, people write this something like this. But for us, we just call this ReLU.
39:28 I think this is the standard in machine learning. So this function is called rectified linear units, which is literally just this function where you have linear functions on the positive plane and then 0 on the other side, so 0 and then linear. And linear with slope 1. So this is a very important primitive. And then you can write this function as this function can be written as ReLU of t maybe plus c, something like this.
39:58 Oh, sorry, ReLU of-- let's say this one will be written as ReLU of wx plus b plus c. So basically, you introduce-- so if you don't have the ReLU, then it's basically linear function. But if you have the ReLU, then you somehow change the shape a little bit. And also, the ReLU is often called activation function. I think the reason for the word "activation" is because there is some kind of connections to biological neural networks.
40:30 I think in biology, in our brain, the neurons-- how the neurons are fired up is a nonlinear function of the stimulus it receives from other neurons. And it's not exactly this, but it's something like this. So when it's passing some threshold, the neurons got fired up and. And below that threshold, the neurons just doesn't do anything. So that's why it's called activation if it's fired up. But I think in biology-- I'm not an expert there, but based on my knowledge of-- the function is never exactly like this.
41:09 Sometimes it's like sigmoid. And that's why sigmoid was used at the very beginning of deep learning. But then later on, people realized that this ReLU is actually a bad idea for networks for various computational reasons, which we'll probably discuss in a bit. Anyway, so for now, I think that's just to remember this concept. So, so actually we have already defined a one-dimensional neural network.
41:39 So basically this is an artwork with one-dimensional input and a few parameters, so that's already a new artwork. So then for the rest of the 15 minutes, we are just going to generalize this to multiple higher dimensions. And you apply this multiple times. And that will make it a more complex artworks. And then we're going to even build more on top of that. So next, what do we do? So this is only one-dimensional input right.
42:10 So now let's try to do a higher-dimensional case. Let's say suppose you have a data with dimension d. And then how do you imitate this and make it nonlinear? So the way to do it is that you say I have x in d, and then I have a parameter w, which is also in dimension d. And often this is called the weight vector. And actually, eventually, it will be a weight matrices. So that's why weight and matrix are associated together.
42:39 The weight vector is a rare cases. And then you also introduce another number b. This is called bias. And then you define your model like this, say, ReLU of w transpose x plus b. This is the inner product between two vectors, and you get the scalar out of it. And then you add another bias, which is still a scalar. So the output of this is a scalar. So then basically, we have high-dimensional inputs and one-dimensional outputs.
43:12 This is actually called a one-layer network. So basically you apply this ReLU once. And now let's try to stack multiple-- so this is one neuron, actually. Because you apply one activation in some sense, it's one neuron. And then you can have multiple neurons. So I think one of the typical examples I used to give-- maybe this is not exactly the best example these days. For example, let's say suppose you think about the housing price. So it probably depends on a lot of intermediate features.
43:44 Suppose housing price depends on, for example, maybe-- maybe let me give you a full example, just slow down a little bit. Let's say suppose you have this cases where x1 is the size, x2 is the number of bedrooms, and x3 is, let's say, the zip code, and maybe x4 is-- let's see what's-- maybe the-- I don't know what x4 is in my notes means.
44:17 But anyway, so this is your raw inputs right. So maybe you can think of there are some intermediate things that actually matters more than this raw inputs. Let's say you have intermediate quantities that you quantities. For example, you say walkability, how easy to walk around the house. Maybe there's another one which is the school quality.
44:51 So these are, in some sense, variables that are not present in your original data, but they matters a lot to the price of the house. And each of this-- maybe let's call this a1. This is a2. Each of these is actually a function of x1 up to x4, which can be computed as some nonlinear function. So you can imagine a1 is probably some kind of a nonlinear function of maybe ReLU of some of these linear combinations of this thing.
45:24 And you can imagine maybe a2 is also some nonlinear function of this, maybe, let's say, theta. I'm using notations in a pretty sloppy way just to give you an example. Finally, we're going to define things more formally. And then you can say, oh, maybe my final y should be a linear combination of these guys. Maybe I have another parameter, theta 9 a1 plus theta 10 a2, something like this. So then this way, you can stack two layers on top.
45:53 You have two neurons, and you stack another linear functions on top of it. So in some sense, you are applying this recursively. So that's like the fundamental idea of how do you compose. So basically, you apply this nonlinear functions. You apply a linear function. You apply some nonlinear functions. And then you get some outputs. And then you apply another linear version. And you can probably apply another nonlinear function here. And then you keep compose. So that's the rough idea.
46:21 But let's try to make it a little bit more formal so that it's probably more obvious. Let's see what's the right way for me to erase this stuff. I think we can probably start from the left.
46:52 So one of the ways that people used to draw this kind of things, at least 10 years ago, was that they draw on this inputs like this. So suppose you have x1 up to xd, and then you say I have some intermediate variables a1 up to ad, maybe am. And then each of the intermediate variable depend on all of this. And they have some dependencies here. So let's say am depends on all of the input variables.
47:19 And you just draw this kind of edges. And then you can say I have another layer of intermediate variables. And they connect to each other. So these drawings are-- once you know what they are talking about, the drawings are very useful. But when you don't really know what they are talking about, it's kind of confusing. So what I really mean here is that the way you do it is that you say, first-- let's do only one layer, maybe just formerly.
47:44 So we have x1 up to xd. So then let's have a1, the first intermediate variable, is defined as maybe ReLU of w, say, 1 x1 plus b1, where w1 is a vector, even though I have an index 1, but it's the first vector, w1 transpose. So x is here. x is a d-dimensional vector. W1 is a vector. So the inner part together, you get a scalar.
48:16 And you either scalar b1 here. That's how you get a1, which is kind of what I wrote there like a1. So I use w just because I think people use w and b to distinguish the weight matrices and the biases. And then you have a2, which is a ReLU of w2 transpose x plus b2, where w2 is Rd and b2 is R. And then you have a bunch of this.
48:46 Let's say you have up to am. Let me see where m is my notation here. It's very easy to mess up notation in this kind of cases. Yes, am is ReLU of wm transpose x plus bm. And that's how you define a1 up to am. And each of the ai depends on all of the xi's in this particular way. And then you can say-- I'm going to apply another-- my output will just be a linear combination of all of these ai's.
49:16 Let's say the output is some linear combination of the a's plus another biases. So here is the thing. How do I write a linear combination of the ai's? I can write something like theta 1 a1, so on and so forth. But if I always expand this, it's too complicated. So what I'm going to do is I'm going to write a vector. Let's say w transpose a plus another b. But you realize that when I write this, this w needs to be distinguished from other w's.
49:47 And that's why I think I'm going to use this 2 to say that this is the w on the second layer. In some sense, these are the w's on the-- so there is another-- so all of these ai's are combined to h theta x. And that's why people write this is the w on the second layer, and these are the w's on the first layer. So I'm going to write another bracket w a1, something like this, to distinguish the first layer and the second layer.
50:23 And this w2 superscript bracket 2 is a vector of the same dimension as a, not of the same dimension as x anymore, because you have to compile. You have to inner product with a. So this is in dimension Rm because m entries here. And b is always a scalar. So that's how you define a two-layer network. So you have a lot of images. And you can see this notation is kind of messy. So you probably shouldn't-- we have too many superscript, subscripts.
50:58 At some point, it's too messy. So that's why for most of the cases-- some cases, you have to live with this kind of notations. But in most cases, you can simplify even further. You say-- actually, if I write everything as matrices and vectors, I don't have to really think about each of these details. So basically, I just write a as a vector in-- this is a vector in Rn. You can write it as ReLU of a matrix W times x plus b.
51:29 And here, w and b are a little bit different. So w-- I guess I need to write w1 and b1. So this is a matrix W1. If you want to understand the correspondence between this notation and this notation, so W1 is this matrix of dimension n by d. And each row is w1 1 transpose up to wm1 transpose.
52:06 So you put all of this row vectors because once you transpose-- by the way, all the vectors are supposed to be column vectors if you don't do any transpose. If you really want to understand this, you should try to understand, at least, when-- depending on the situations. But in many cases, it's good to keep in mind whether these vectors is a row vector and column vector. So all of these are column vectors. And then you can transpose this into row vector.
52:31 So this is a row vector. And you put all of them, stack them into a matrix. And then this b1 is just the vector of all the scalars. But in many cases, you actually don't have to necessarily remember this correspondence. You just remember this. It's fine. You have a vector. You multiply a matrix. You get a vector. And then you add a vector bias, and you take ReLU entrywise. That's probably important. So ReLU applied to a vector-- t is the same as you applied entrywise.
53:10 So in many cases, you just say, oh, one matrix application-- one layer is just multiplying a matrix and bias vector and apply ReLU. And you get a vector out of it. So you don't have to necessarily understand all the details. Sometimes you do, but in most cases these days, you don't have to. And if you count how many parameters are there, this is n by d. This is the dimension. So n by d, because the input dimension is d, the output dimension is n.
53:40 And in a matrix, that turns the d-dimensional input to m-dimensional outputs. That's why you need m times d parameters. And then this guy has m parameters. So the total parameters parameter count is md plus m. So d is the input dimension. m is the output dimension. And this is output dimension. Is b to the one, as in the first layer?
54:11 This is b1, yes. And this b1 is this b1. So any questions? And I think eventually, actually, many of these details don't even matter if you don't care. So for example, you can even just write this.
54:45 For example, you can just say, oh-- I even simplify notation here. I think you can say, let's just say-- this is a matrix multiplication. So let's say you have a matrix multiplication with w and b. So w is the weight matrices, which is this w. And b is the bias. And you apply z.
55:16 And this is just equal to z plus b. And then you can say I'm going to apply a sigma. So let's say sigma is-- I guess maybe I didn't-- probably I go a little bit too fast. Let's see. So maybe just before I talk about that-- I think I already alluded allude to it. So this ReLU here can be used to also replace to other nonlinear activations.
55:56 So you can replace the ReLU with some other nonlinear activations. And all the activations are functions that maps R to R, so it's always a scalar function. But you can apply this to vectors by applying it entrywise. And other activations could be like sigmoid.
56:28 Sigmoid look like this. Let me just save some time to now define the equations. You can read the lecture notes to see the exact equations. There are also tan h, which looks like going through 0 and like this. So all of these activations are generally increasing functions, but not always the case. I think there's also leaky ReLU. There's a bunch of different activations functions. For example, leaky ReLU looks like-- none of this is actually used that much.
57:02 If you want to see what's used in the latest network, I think you have to check the lecture notes. I just added a few, which I don't have it on my notes. But there's also functions like this. Anyway, generally is almost increasing functions. And often on the left-hand side of the 0, it's kind of close to 0, but not exactly. So for example, the one that I just draw, this is called GELU. Let me now write-- I think the definition of it is-- the GELU's definition is this, just in case you care about the exact equation.
57:39 So it's not exactly nonlinear. It dips a little bit. I think it doesn't go above 0. It's like this, strictly speaking. So it dips a little bit, and then it goes up. So exactly why use any of this is like a little like a magic. I think you don't-- I think people seem to find out that using some kind of somewhat smooth functions is actually useful. So ReLU is actually not necessarily great because it's not smooth enough.
58:08 But other than that, I don't really know exactly. Sometimes it's a stability thing. Sometimes it's just a trial and error. So that's the activations. So now I'm trying to have more and more abstract versions so that you don't have to deal with all these notations. So then once you have activations, you can define this. This is just the sigma of Wz plus b.
58:40 This is the abstract version of that. And then you can compose many times. You can say I'm going to apply another one, so my model can be like-- say, I apply a matrix application on top of this sigma, so w, b, z, and then apply another thing, apply another thing, and so on and so forth. And each of this is one layer. And then if you apply another matrix application sigma is another layer.
59:11 And if you do another one, it's another layer. The number of layers is defined pretty much as the number of matrix application. But here, one thing I'm going to describe to you about is that I didn't really tell you what's the parameters here in this matrix application. So technically, if you want to write this formula, you have to say, oh, this is w and b. And this is what? It is still using w and b or some other b?
59:33 So oftentimes, you use a different w and different b. The shape is similar, but the variables are different variables. So then you have to introduce the bracket again. So let's say this is the bracket 1, b1. And then here, the parameters are W bracket 2 d2, so on and so forth. And then you index each of them to fully specify the network. So It's not just an plus 1 matrix bias vector transpose.
60:04 Which matrix? So this matrix? Yeah. Yeah, so what these matrix are-- so the only requirement is that you need to make sure the dimensionality is compatible. So your input-- wait, did I write z here? Oh, yeah. So let's say suppose your input z is, say, dimension d. Then you can only apply a matrix with-- so to make Wz make sense, you have to make sure W is of dimension something by d.
60:44 And then something is what you can choose, almost freely. Of course, not all the options are the same, eventually in terms of the results. But for the compatibility issue perspective, you can choose them arbitrarily. And let's say, suppose the first layer you choose-- let's say suppose you choose matrix multiplication application, say, W1, b1, z plus. And suppose this W1 is m1 by d.
61:20 So that means that you map the d-dimensional vector to m1-dimensional vector. So let's say-- so this output is going to be m1-dimensional vector. And the next time-- and if you apply sigma, it's still m1-dimensional vector. And next time you apply the matrix multiplication, you have to make sure that the input dimension of that matrix to be compatible with m1. So basically, if you do another one, say, W2, b2, then it's required that W2 is to be something times m1 dimension.
61:54 And something-- the output dimension is still what you can choose freely, but the input dimension is not. And the next-- and you have to do this again and again. So once you have multiple dimensions then-- so basically-- maybe the easiest way to think about it is that you have a matrix times, say, z plus b. So this dimension is what I call the input dimension because this dimension got multiplied with the dimension of z. And this dimension output dimension.
62:20 But the output dimension becomes the input dimension for the next round. So that's why you need to make sure all the dimensionalities to be compatible. That make sense? And in many cases, I think the original input dimension is something you cannot change. So that's fixed. That's based on our data. And then afterwards, you don't change dimensions that often. You typically use the same dimensions for quite a while, or maybe you change it once or twice, something like that.
62:58 And just in case you read some of the old papers, which-- I think even these days still the architecture papers sometimes draw like this. They just draw-- people go more and more abstract. They say, I have an input x, and then apply a matrix modification, and then apply another, maybe, sigma, and apply another matrix multiplication, so on and so forth. That's how people actually draw these networks without expecting all the details. And you can even merge these two together to call it MLP.
63:30 So sometimes people just draw this. You have x. And MLP is called multilayer perception operator encapsulate one or two layers of this-- a few of these components. When people say MLP, I think sometimes it's one matrix multiplication, one sigma. Sometimes it's two matrix multiplication, so on and so forth. So usually, the vectors like this become smaller in size? Or do they usually-- is there a common strategy there or-- I think in most cases, you get smaller dimensions.
64:08 The intermediate dimensions become smaller. And that's often because the input dimension is often very big. For example, for language, it's like, your initial-- the vocab is like the token-- I guess I'm jumping ahead a little bit. But your initial dimension could be like 250k or something like that. And then you shrink it to probably 2k and then keep using 2k, something like that. And for vision, I think the same thing because initially you have RGB-encoded image.
64:35 You have three channels, three RGB, three numbers. And each is like 256 by 256 or maybe-- depending on the resolution. And you have to shrink it a little bit. What else? I think one thing is called ResNet. I think that's still useful, so probably I should tell you about it. So let me erase something here. But eventually, you can define many components like this. Most of these components are matrix multiplications, activations. I'm going to tell you a few other components that are still largely being used in different cases.
65:11 And some of the very modern cases use in more modern components, like attention using language, I'm going to talk about those in a large language model lectures. So these are some-- so one of the component is called residual network, or just residual connections if you think about the component. So the residual connection is like this.
65:42 So instead of keep using this kind of things, the residual connection is like-- let's call it residual z-- is this interesting thing where you have a two applications of this alternating between matrix multiplication and activations. And then you add your input again back to it. So one thing that is-- at least, by definition, it means that you don't change the dimension here, because to make this compatible, your input dimension, output dimension are exactly the same.
66:24 And of course, these two matrix multiplication are not using the same weight matrices. I'm not just write exactly what they are, but they are not using the same matrices. And this is called a residual block. And this was introduced in 2015 in this very famous residual network paper I think the rationale was that if-- let's say, suppose-- by the way, this z doesn't have to be our original raw input.
66:58 This could be one intermediate layer, and then you apply this z-- you apply this residual block. And you can actually apply many, many residual blocks. So eventually, the residual-- this is called residual block. And eventually, the residual network, the residual net, looks like with some approximation is like you apply a lot of residual net many, many times. And at the beginning, you need to do something maybe to change the dimension because you cannot work with the original dimension.
67:30 So you first change the dimension to some dimension of the intermediate dimensions, and then you apply many, many residual network. And maybe you have to apply one more at the end to change dimension again. But largely speaking, it's application of many, many residual blocks. And that's called residual network. And the original motivation was that, why you apply this? That's because, suppose you believe that at some point, you are already getting close to your labels. So suppose at some point, z is already approximately equal to y, let's say. y is our target, and z is our intermediate variables in some intermediate layers.
68:07 Then you probably should model the differences explicitly because you are already close. You just probably should use networks to model only differences. So basically, you should probably just model z minus y as some residual, or maybe y minus z, sorry-- say y minus z as some residual of z. And if you believe in that-- sorry, not residual z-- y minus z to be some kind of neural network, say, something like this, plus, like this.
68:41 So if this is your belief, then if you move z to the other side, then you have y is equal to sigma mm z plus. And you can see that this is almost the same as here. The only difference is that they found out, actually, probably shouldn't use a one layer to model the difference. You should use two layers to model the difference. So they say, oh, let's say you do two of this. And then the same thing, you just have this.
69:08 So this is just a story. It's not like there's any formal proof there. And actually, people have tried to, 10 years ago, maybe seven years ago, to see how many blocks here you should do. Can you do two layers, three layers, l layers? And eventually, I think two or three layers are the best also for reasons we don't fundamentally understand necessarily. Yeah. So that's the residual network. Yeah.
69:38 I thought like that the term on the right, that's our hypothesis. I'm little bit concerned by this. So suppose this is the modeling assumption. You model only the differences then-- so this implies this. They are the same. So which one is the hypothesis depends on what you believe in. But you somehow believe that it's more beneficial to model directly the residual, then you do this. And then you rearrange, you get this.
70:10 That's the only thing I'm saying, nothing profound. but again, a little bit better than it seems to help. And there are other kind of explanations for why this works. For example, some of the theories that they try to analyze the optimization perspective of different modeling. So basically, if you don't add z here versus you add z, in some cases, you can argue that adding z will make optimization easier, make your optimization more well-conditioned, including some of my papers.
70:42 I think there are some degree of truth there as well. It's just that it's very hard to fundamentally-- you have many explanations for it. It's very hard to know exactly which one is the right explanation. Maybe-- let's see. I have seven minutes. Any question? Can you specify the intuition for why the differences can be modeled like this? There's no-- I got a question, what's the intuition?
71:13 But maybe, let's say-- so the intuition could be that, oh, maybe this is already small, then maybe I should directly model that, as opposed to model the original label. So it's a simpler task. Predicting original label is very hard. But if you predict the difference, probably it's easier. So that's why you model easier task with a two-layer networks. And that's pretty much it. There's nothing more formal than that. Yeah. Let's see. I have seven minutes.
71:43 I think-- let's see. Should I attempt to talk about-- two things. So maybe let me quickly talk about layer norm. And I can see whether I have time to talk about the convolutional neural network. So layer norm is also something that people still use in large language models, of course, in slightly different form these days. So the original layer norm is the following.
72:14 So basically, it's a component. So it's a block. So you have some input. Let's call it z. And this component is trying to normalize z in some way. And the way to normalize it is that you say you first turn z into some normalized version of z, which is z minus mu 1 over sigma 1 comma z minus mu m sigma m. Let's say z is m-dimensional z-- sorry, z1. My bad. I should read the note.
72:40 I should follow the notes more closely. I think this is one of the same-- yeah. Maybe I should even use-- So let's say-- I don't remember why I use hat, but let's use hat. So this is how you normalize, where mu hat is the empirical mean, which is the average of all the zi's.
73:17 And sigma hat is the empirical-- sigma hat squared is empirical average of the standard deviation. So this will empirically normalize it to norm 1. So you can see it's pretty standard. And then, it's not done yet. And then you turn this into-- sometimes people found out, oh, actually, mean 0 and norm 1 is not that great, for whatever reason.
73:50 So they say, oh, let's be flexible. Let's also add a flexible tunable parameters beta and gamma. So the next step is turn this into beta plus gamma z1 minus mu hat sigma hat, where beta is a tunable parameter-- beta is a variable that you can tune, and gamma is a variable you can tune.
74:25 So basically, this makes a mean beta and standard deviation gamma. So that's the original layer norm. And then later, people realized actually, you probably don't have to do this. So this is the layer norm. And now I think there's something called RMS norm, which is even simpler. You don't subtract the mean anymore. You just divide by sigma. So you just say-- you do z. It's turned into z1 minus sigma hat, say, z minus sigma hat.
75:02 And sigma hat is also different. Sigma hat is only the average-- is only-- you're only averaging over the-- you don't subtract here. So you just try to make the-- so sigma hat is something like 1 over m zi squared.
75:34 And then you still do the beta thing. Then you turn this into beta plus z1 over sigma hat. Why this is more better or not? I think it's mostly empirical. So basically here, you don't really want to make it-- let me think about this. Wait, sorry. I think you still need gamma. Sorry. Did I keep gamma or keep the beta? I keep the gamma. Sorry, my bad.
76:05 You do this. And gamma is trainable parameter. So basically, this is saying, I don't have to care about the shifting mean. I don't really make a mean 0. But I still care about normalizing it in some way. But I still leave some degree of flexibility gamma, which is not really normalizing. I still allow it to be normalized to something that I will decide later in my training. So this is like the magic here. It's not like you try to force it to normalize to 1 standard deviation.
76:30 You actually say I first normalize to 1, and then I leave a gamma parameter there so that in the training, I can normalize to anything. And I think-- let me see. Yeah, I only have one minute. So one benefit of all of this is that you make it a scaling invariant in terms for both the parameters and the data. So what does that mean is that if you multiply-- so maybe let's say you call this whole thing called, let's say, z-- let's just say z.
77:05 So let's say, suppose this whole operation is called the layer norm. So you apply z, and you get a layer norm of z. And you can find out that there's a scaling invariant, which means that if you apply a scalar-- let's say you multiply a scalar, say, alpha to z, and you get an alpha z. But an alpha z is actually equals to LNz. So basically, no matter how you multiply modify the input, you don't really change it.
77:35 So that's why, no matter how you multiply your matrices or you multiply your input data, you never change the output, unless you change some of the parameters like beta and gamma, which is-- your test time, you're not supposed to change them. So basically, this makes the model more scaling invariant. And in some cases, it helps stabilize the model, for example, at least intuitively, with some degree of approximation. At least, in the past when people tuned all of these models, it happens that sometimes you apply these layers many times, and then the output becomes bigger and bigger.
78:14 And it becomes like really big-- it's like 10 to the 20 or something like that. And then everything just blows up. So when you normalize every time, then you never got into this issue. And also, it makes the initialization a little more of easier so that you don't have to-- before, if you don't do something right, then suddenly, everything just blows up. And now the scaling doesn't really matter that much. But there's also subtlety here because when you have-- this might be a too advanced comment.
78:41 One side effect of this is that when the forward pass is scaling invariant but actually the gradient is not scaling invariant still, so some of these issues got pushed into the optimization, which is a much more complex matter. Anyway, so I guess-- let's see. So I think I'm done. The only thing I didn't cover is the convolutional network. I think it's in the lecture notes. So if I spend one minute on it, I think it just means that instead of multiplying a full matrix, you actually multiply a topless matrix.
79:15 I'm not sure you heard of this word. So basically, it's a matrix with some structures where-- so instead of having this matrix, convolutional network, which basically means you have a matrix that's like-- look like this. The nonzeros are only like this. And it's actually the same set of nonzeros. You shift this vector here, and you put it here, and you put it here, and you put it here. It's actually called a filter in some sense. So this matrix-- all the other parts of the matrix is 0.
79:44 And only this, what I drew here is kind of nonzero. And also they share the parameter. So basically, this vector is the same as this vector. So you apply a shifting invariant filter to each to the data. And it's called convolution. Of course, the real thing is a little bit more complicated because when you have 2D and so on and so forth. It becomes complicated. But in the lecture notes, we have that. But convolution neural network is not used that much, except for very hard-- for vision cases.
80:12 Even for vision cases, people use language models like transformers a lot. So that's why probably it's fine to skip them. Sounds good.