MIT 6.S191: Reinforcement Learning

0:05 [music] >> Okay. Hi everyone. Welcome back to lecture five on deep reinforcement learning. Today we're going to be talking about a yet another paradigm of machine learning. So in the past few days we've talked a lot about supervised learning. Then yesterday we talked a lot about unsupervised learning. Today reinforcement learning is yet another paradigm that will will learn about. And specifically we'll focus on the the subset of reinforcement learning which focuses on deep reinforcement learning.

0:42 Meaning that we train reinforcement learning models using deep neural networks. So I think actually one of the reasons why this field as a whole is so exciting is because it moves away from really everything that we've seen in this course so far. And like I mentioned so far we've really been dealing in this world where we have pre-collected data sets. Either we have labels with those data sets or we have no labels in the unsupervised case. But in the reinforcement learning world we actually have neither. And actually we're often times looking at situations where we are actively collecting data while the model is learning. Which means that we're really trying to learn in very dynamic environments. Which if you think about it is much more similar and much more aligned with how we as humans also learn as well. We learn by interacting with our world and learning from those interactions.

1:33 Now this obviously has very huge implications in many different fields that touch the real world, right? And you can imagine that for example in robotics reinforcement learning plays a massive role in training robots how to also interact in our world just like we learn in the same world as well. But a lot of the actual original advances and breakthroughs in reinforcement learning, especially deep reinforcement learning, have come in the field of game play. And the reason for this is because in the world of games games are perfectly defined worlds, right? In many ways, right? They're much less uncertain than our own world and much more predictable than our own world. So a lot of the huge breakthroughs that we'll actually use as case studies in the class today will actually focus on games and strategies as a as a case study. And then we'll see actually how this has extended into the more recent advances with language modeling. Because again in language modeling things are a lot more predictable than they are in the real world. And we'll see how these deep reinforcement learning algorithms can also extend to massively improve the preferences and the alignment of language models to interact with humans as well.

2:40 So to start let's actually summarize and set in place where reinforcement learning lives with respect to all of the other types of learning paradigms and learning problems that we've seen over the first few days of this course, right? We'll start first with supervised learning. The first three lectures that we had were all focused on supervised learning. And the paradigm here is that you have two things. You have a data set that has X's and Y's. The X's are your inputs, the Y's are your outputs. And importantly this is a paired data set.

3:10 Meaning that for every X you have a paired Y. For every input you have a paired label that is associated to that input. So for example uh let's say you show a picture of an apple. That's your X, your input. The Y, the label could be that a classification that this is an apple, right? Now in the last lecture, lecture four, we saw an extension of this to unsupervised learning. Which all that means is that now we only have X's without associated Y's.

3:42 So we only have a bunch of examples of things that look like apples and things uh other things in the world as well. And unsupervised learning algorithms should just learn from that unstructured data not that this thing is an apple, but this thing is probably like this other thing that it sees. And it learns relationships between the distributions of the data that it that it's shown. Now to reinforcement learning. How does this compare to reinforcement learning?

4:08 So with reinforcement learning we have data in the form of triplets. Now we have states, actions, and rewards. So every piece of data, every sample of data, has these three components associated to it. And the objective function, the goal of any reinforcement learning algorithm is to take those triplets and to take your your actor in the environment and to be able to maximize these rewards, which is one of the pieces of your triplet and to maximize that reward over time.

4:42 Right? So naturally reinforcement learning is a temporal process, right? You're learning something over a sequence always and you're trying to maximize rewards over the course of that horizon of the sequence. So for example the same apple example that I gave earlier, the analogy would be that uh you don't necessarily need labels that this is an apple or that apples are good to eat, but just by observing and eating some apples and eating some other, let's say not healthy food, you could observe that apples make you live longer. So apples are probably a good thing that you should eat in the future, right? This would be the connection to kind of place mentally and intuitively between these three different paradigms.

5:21 So today we'll focus on the third paradigm. We'll only focus on RL and we'll focus on how we can build new algorithms to train RL models from scratch. So before we go any further, let's set some foundations uh with terminology because a lot of the terminology in RL is slightly different than the terminology that we've seen before. But there's actually a lot of overlap as well. And we'll see that. And I tried to even show that hint at some of that in the first slide that we just walked through.

5:52 So starting first with an agent, right? The agent is the thing that is taking actions in your world, right? So for example if you're trying to train a drone to make deliveries, the drone is the agent. If you're trying to train a Super Mario character in an arcade game, Super Mario is the agent, right? In the real world each of you are the agents acting in the world, right? Now for every agent there's also an associated environment in which that agent lives and interacts in, right? So this is where the agent exists and where it moves.

6:31 Between these two paradigms, between the agent and the environment, the agent can always send actions to the environment, right? This is things that it can execute on or it can act on. Let's say for example movements in the environment. Right? And these are typically defined within, let's say for simplicity, a finite set of actions. So you could have a vocabulary of different actions that the agent could act upon within any given environment. And it can do this at any time step. So for example in this very simplified example, let's say you're in a two-dimensional world and your actions could be moving forwards, moving backwards, left, right.

7:07 Let's say there's only four actions in this very simplified environment that this agent could take. So the action space, or what's called capital A here on the bottom, is simply just the set of possible actions that this agent can take. So here the set of capital A is just covering four different actions. Now going back in the other direction the environment also responds. Every time the agent takes an action, the environment responds in kind with a new observation of what that agent should see. So for example if the agent moves forward, takes a step forward, its world, what it sees around it, changes as a result of that. Right? So for every action there is a corresponding new observation that comes back in the form of what's called a state, right? So state is another key component of that triplet if you remember. There's actions and states.

8:02 That actions are being sent from the agent to the environment. States are getting sent back as a result of those actions. And every time you make an action, let's say at time T, let's say action T, so A of T the environment will respond back with a state of T plus one. Okay? Along with the state you'll also get a reward associated to the action that you just took. Okay. So for simplicity again, let's say that you're collecting coins in this two-dimensional world. Every time you move in one direction, you may collect one coin or you may collect no coins, right? The coin you can think of here as roughly as a reward. But often times you may have very sparse rewards, what's called. So you may not get any reward on any step. So for example you're playing a game of chess, you may got get no reward until you get to the last step in the game and then you get all the reward or you get none of the reward. So those are called very delayed time horizons for rewards. And a big thing with rewards is that you don't need to receive them in a uh in a dense fashion. They can be extremely sparse and that's one of the big challenges.

9:11 Now the total reward or what's also called the return of the agent is simply going to be defined as the sum of all rewards that this agent gets starting from the current time T. So capital R of T you can think of is just the sum of everything that you collect from this time point going forward. But in in general actually we can actually expand this out, write it out like this. But there's a key point here is that often times it's actually useful to not just consider the sum of all rewards that you take from this point on, but also to discount some of the future things that you could get in the future.

9:49 Let me give an example. So if I could offer you $5 today or $5 in a year from now the reward is the same. It's still $5. But obviously, you would rather the $5 today because you discount the future reward. So, what we say here is that capital R is not just a function of the sum of all future rewards, but you actually discount the future rewards by some gamma, and that gamma compounds over time. So, as rewards go forward into time, you actually discount them, and that is actually your total return from this point on.

10:24 Okay, any questions? Yes. Can the reward be negative? Absolutely. So, simplistic example would be let's say again going back to the game of chess, you win or you lose, you could say positive reward plus one if you win, negative reward negative one if you lose. And oftentimes it's actually quite helpful to define like a normalization of rewards. So, even to even if let's say like in the natural framing of a problem, let's say collecting coins for example, it could be you could say this is a reward of zero or one if you collect a coin in the world or not.

10:56 Sometimes it's actually helpful to make it a negative reward just to encourage a penalty. It's more of like a penalty in that framing to the algorithm as well. Yes. In the game of chess, you get the reward at the end. So, how do you measure reward with the discount factor? Okay, great. So, the question is in the game of chess, you get the reward all the way at the end, and you get nothing in between the current time point and the future. So, the question is how do you measure that, right? And the And the answer is that it's still measured in this. You'll just get it all the way at the end. So, one of these time points somewhere far on the on the right-hand side will be a positive one or a negative one.

11:32 And just you discount that back all the way into the future, and that turns into your positive your your total return. But we'll see actually how the agent has to learn from that. So, the the tricky part is that even though it's coming from a very far time horizon in the future, you still have to learn what to do today. Right? And that's the hard part. Well, so we'll see more about that later through the lecture. Yes. How do you Let's say you're at time T, how do you How do you even factor in reward that's going to arrive many time steps in the future? We'll get to all of these questions. So, right now just definitions. Are all the definitions clear in terms of how what each of these terminologies mean for now?

12:10 Okay, good. Yeah, we'll get to all of like the the how questions in a second. What's the difference between state and reward? Okay, great question. Yes, so difference between state and reward. So, state is Think of this as just a snapshot of your environment, right? If you're playing a game of chess, it's where all the pieces are. Reward is telling you what the what the result what the um the penalty or the reward, I guess you could say, on that action is. So, both happen at every time step.

12:38 Some may like rewards may be zero, of course. You may have sparse rewards, but each one you'll get it every time step. Think of the state as a snapshot of your environment, though. So, state importantly does not depend on the goodness of your action, right? You will see a state regardless of what action you take unless you lose the game entirely, of course. Right? But rewards will directly depend on the goodness of your action. Okay, excellent. So, let's continue a little bit with some of these definitions, and then we'll go into more of the learning side in a second. There's this very important function in reinforcement learning called the Q function.

13:20 Right? And actually almost at least half of today's lecture will be focused on this function, this Q function. So, we already learned about the the R of T, capital R of T, which is the return from this point T onto the future. And that's the total return discounted by future rewards. Q, or the Q function, is going to be a function of two things. It's a function of the state and the action. And importantly, this is a function of the state at time T, the current state, and the action that you take at time T when you see that state.

13:52 And all the Q function tells you is that if you're in this state and you take this action, what is your expected total return that you should receive as a result of that action in that state? Okay, so obviously, if you take a very good action in that state, the Q function should should demonstrate that the that this was a good action. It would be a higher total return. If it's a bad action, then you would also see that reflected as well.

14:22 Okay. So, now let me pose this as a question, which is let's assume that I give you this magical Q function. I haven't told you how to learn it yet. We'll get to that in a second. Let's assume I just give you this function. So, you can plug in any state and any action, and I give this to you. It's a black box. And the question for all of you is, if you find yourself in some state and in in some state, you observe that state, is there an algorithm that you can think of? Is there a step-by-step process that you could think of in order to determine what the optimal action should be if you're in that state? Any ideas?

15:03 Yes, you can maximize the Q function. Yes, exactly. So, the answer is you can maximize the Q function. All that means is, right, you would plug in all possible actions, right? Let's say you have a list of possible actions, you plug them all in, and you see which one maximizes the Q function, you pick the action that does that, and you continue. Excellent. Okay. So, that's exactly right. Ultimately, though, what we really need for an agent is what's called a policy. It's doing exactly what was just mentioned. A policy is nothing more than now a function off of just the state. So, policy is what's here called pi. Pi of S is going to be resulting in the action that you should take for any given input state, right?

15:47 So, if you're given a state, what is the corresponding action that you should take at that state? Yes. How does Q make sense without the future actions? future So, Q actually So, if you look at the output here, right? So, your expected, right? So, like if this is your expected return if you take this action in this current state. So, it's like a distribution of actions too. Exactly. So, like some actions, even though let's say again go back to the game of chess, there are some actions that very concretely, if you take this action, you're in a bad result of the game, right? And like there's always going to be this expected return that even a good action or a bad action will take. Some actions are in the middle, they keep you in the in the spread of the distribution, but importantly, this does not capture the spread of the distribution. It's the expected value of the returns that you'll get for all future times.

16:42 Okay, so now maybe going back to the the strategy that was mentioned earlier, right? If you wanted to define this policy, and that's just simply saying for any given action, how do you pick the right Excuse me, for any given state, how do you pick the right action? That's your policy function, the strategy, as as uh was mentioned, uh should just choose the action that maximizes your future reward, right? You do this by simply maximizing, doing a search over all possible actions, plug in all possible actions, find the maximum reward that comes out, and pick the action that yielded that maximum reward.

17:23 Okay. So, this is all Yep, go ahead. What's the same question again? The expected value is on on what? Like precisely, if you were to write that as an equation. Mhm. So, the expected value It's the mean of the of the reward future rewards that you would get if you take this action from this state. So, it's it's effectively measuring how what It's effectively measuring the goodness of this action from your current state in time.

17:57 So, the randomness is coming from the fact that your policy is random. Also, the environment that generates states and actions is also random. The randomness is coming from a few things. Number one is that you could take future bad actions in the future, right? So, there's a full distribution of things that you could take after this time as well that will spread the distribution. But some things, even at this one time, will impact that distribution and cut off certain parts of it for you already, right?

18:25 It's just a distribution of future actions. That's that's what you're looking at. Correct. Yes, but given the current action and given the current state that you're in right now. Yes. Yes. Correct. Correct. Okay, last question, and then we'll push forward. >> for that policy, though, you're not doing like a a search in the entire space and then coming back and assuming you're you have a policy that's like good enough. Right. So, right now exactly. So, right now I've told you nothing about how to actually compute uh R of T. I said I just give you this magical Q function. We're going to get to actually learning this function in the next part, and that's actually the whole part of the lecture actually. It's like how to learn this function. Cuz of course, if you had this Q function, it's trivial, right? It's like you just literally plug in every action, and you pick the one that's the best result.

19:14 Okay, so that's exactly what we'll focus on for the rest of the class. We'll focus on two different ways to learn uh that policy function. One does this by learning the Q function, and then basically plug in every action exactly as we saw before and picking the action that results in the highest total return. And then the second way is going to be something called policy learning algorithms, which try to directly learn the policy and skip over the Q function, right? So, there's two different ways of doing this. We'll see both ways over the course of the class, but let's first focus on what's called value learning, which basically what means that you want to learn the value of the the function, and then we'll use that in the way that we previously saw.

19:55 Okay, so let's start by just building up a bit more intuition on the Q function because I can see already there's some questions on that part. First, I'll introduce this Atari game. Does anybody know this game? Have you Yeah, okay, good. So, I won't explain too much on it. The actor is this little paddle on the bottom. It moves left or right. These are the actions. Literally two actions in the action space, left or right. And you have this ball coming towards the paddle or away from the paddle.

20:22 And the goal is for the paddle to move in the position that would bounce the ball back up into the top, and the objective is to basically break out everything on the top, all of the colored points on the top. You want to get rid of them. You get rid of them by bouncing the ball up and hitting a point on the top. Okay, so now let me pose this problem. Uh and the motivation of this problem is actually to demonstrate that forecasting a Q function is actually really hard.

20:48 It's very intuitive It It's intuitively uh contradictory for a lot of humans. So, um I'm going to show two different state action pairs, right? First state action pair is A on the left, which is the ball coming straight down towards the paddle, and the action that the paddle takes is no action. So, actually there's three actions. I should have corrected myself. You can go left, right, or stay stationary. The action here is stay stationary as the ball is coming down.

21:15 Very conservative action. The other The other state action pair is a very adverse uh situation. The ball is coming from the side. The The paddle is trying to catch up and move to the right to to hit it. Okay, so out of these two options to take, maybe we can do a poll. How many people think A will have the higher Q value, or B will have the higher Q value? Let's start with A. Okay. Okay, probably 20% 10% I'd say. Okay, B.

21:47 Okay, a lot of more, but probably 40%. So, probably 50% of the audience doesn't vote. Uh okay, how about someone that raised their hand for A? Let's hear. Okay. Doesn't the state have to include much more than that? Doesn't the state have to include what bricks are up there? Yes, it's true. But let's assume it's the same on the top. You're correct. Yeah, the state would be the entire board. Uh but for purpose of demonstration, let's say it's the total. It's the same.

22:14 Sorry. Okay, so someone that raised their hand for A. Yes, go ahead. So, if I remember correctly, if you have a perpendicular hit, it would just keep bouncing up and down, and it could go through the entire uh row of uh six rows that needs to break through, and uh do nothing. Um and so But this was in the '70s that I played it. And that's exactly it. Okay, so are you saying for A or for B?

22:43 For A. For A. Okay, because you would just keep hitting up and you keep breaking up the bar. Yep. Makes sense. Okay. And then for B. Yep. The same reason, which is once you would you would just sit there forever, and you would just bounce, and then eventually not get any. >> [clears throat and cough] >> I see. So, it's a it's a reason against A. It's less than a reason for B. Okay, how about a reason for B? I'm curious.

23:08 Yes. Uh because you Same answer. You'd be stuck in that Okay. Okay. Okay. Sure. Sure. Okay. Okay, yes. I think you ultimately want to get all the bricks, not just Yeah, sure. Okay. You You want to basically get all the bricks on each layer. Correct. Okay, so let's let's watch actually let's let's see actually both of these policies in action because we can actually train a policy that optimizes something like A and something like B. Let's see actually what the result would be in both of these cases. So, A is a very conservative policy. Let's watch it in action. So, it's it's definitely doing a good job. It's hitting the center of the bricks more as everybody guessed.

23:45 Um but it's Yeah, it's doing a good job. I wouldn't criticize it so much. If you saw this in isolation, it wouldn't be a bad policy, for sure. Yes. Okay, so now let's watch policy B, what will happen. And the trick here is that policy B learns a a hack in the system, which is that it can break off the corner, and it will unlock a bunch of balls on the top for free. And this is like a a hack in the system, right? It's like uncovered this that there's like a huge gain that can be had if it can break through and then get like a bunch of free balls off the or blocks off the top. Right? And the point here is that it's probably really hard to to forecast this for us as humans. We as humans may or may not play the game like this, right? But our RL algorithms on the other hand play like millions of times, and they observe a lot of different possible scenarios, and they're able to think in ways that we as humans don't necessarily develop our own policies. So, B is the correct answer, but not necessarily for the reason that a lot of people think.

24:53 Okay, great. So, now we can see that if we know the Q function, again, this problem is pretty easy. But let's actually see how we can train a neural network to learn the Q function, right? So, remember again, Q function is a function of two things, a state and an action. So, that means we have a neural network that takes as input a state and an action, and we want to predict a value, which is the Q value at that input, right? So, given a state, given an action, predict a number, which is the Q value for that state action pair.

25:28 Now, we could also reformulate this a little bit just for efficiency reasons. Instead of inputting a state and an action, we could input only a state, and we could output the Q value for every possible action. Why would we do this? This is just a bit more efficient because now you don't need to run your network for every possible action. You run it once for the state that you're in, and you get all the Q values all at once, and you just pick the action that you actually are going to execute.

25:54 Right? So, it's same thing, just uh you know, reformulating the model a little bit. Oftentimes, you'll you'll probably see that number two on the right-hand side is the more efficient way to do things. So, it's usually the way that's implemented in practice. So, how would you actually train a deep Q network? Think about what happens in the best-case scenario, right? So, in the best case, this is how the agent would perform ideally. What would happen if we took all of the best actions? And this would mean that we Well, number one, this would mean that our target return, capital R of T, is going to be maximized, right?

26:32 This can serve as the ground truth for our agent, right? This is a number. It's a continuous value, and it serves as our ground truth, right? So, let's just uh let's write this out now. So, this is uh the formulation that we're going to be working with. It's our current return, or excuse me, our current reward at this time, R, but it's also going to be uh the action that maximizes the expected return for all future states, and of course applying that discounting factor in front as well that you see here, right? This is going to capture both the current time step as well as the discounted future time as well, and this will basically be our uh target for training our network. Now, the question is, okay, what does our network predict? And the answer is that it's directly those those outputs, right? It's predicting the Q value for any state action pair.

27:25 But these two sides are directly forming now our target and our prediction, which means that we can create a loss function, which is subtracts them and takes their absolute value. Right? So, we have a ground truth target, which is a numerical number, and we have a predicted number from our network. We train our network to match the target. And that's how we train deep Q networks, right? So, let me just summarize this whole process and just go end to end, and we can pause for questions before we go to policy networks.

27:56 So, our deep Q network is going to see now an input state, right? It will predict every possible Q value for every possible action that you can take in this given state. So, here for example, given the state of on the left, there are three possible actions, right? You can move left, you can stay still, or you can go to the right. Those are three numbers that the model outputs. Right? So, for example here, let's say the model predicts 23 or zero given this state that we're in.

28:29 Right? We can now produce a policy on how to act in this current state by looking across those three numbers, picking the one that's maximized. Right? So, 20 in this case is the maximum. That's our best action that we should take. So, that's the action that we execute upon. Right? And of course, when we train this, we send the action back to the environment. We get the new state that comes back, and we train this with backprop like we saw before, given the target values that we have from an entire rollout applying this game.

29:04 Okay, any questions? Yes. For this game, we exactly know the state action pairs and the maximum likelihood of a good return. For certain games like chess, after a while, it's very hard to know what is the reward at that point. I think you are 10 steps into the game. It's very hard to know. So, um how do you know that? Okay, so the question, let me repeat for the audience. The question is, in some games, maybe it's easy to know the reward at any moment in time. In some games, you have to wait long into the game to get the reward.

29:42 The answer is that in neither of these cases, they're actually not different because in both cases, you always play the game to the end. You collect all rewards that you gather until the end of the game. And that's when you actually curate, let me go back to here. This is how you get the target value. So, given a particular action that you execute on right now at this moment, the outcome of the game is already decided.

30:07 But you you you can rewind the game and you see what you did at this moment. The reward is a deterministic parameter as a function of the game. So, the the the the network does not learn the reward, the network observes the reward from the game. Right? So, the game, let's say you win or lose the game, you rewind the game now and you say, "Okay, what was the action that I took at this point? And what was the outcome of the game? Did I win the game or did I lose the game?" Of course, that that individual action may or may not have contributed to you winning or losing the game.

30:43 But this is the noise. Like this is noise to the model and it's a that's why it's a it's a hard learning problem. The point is that you did take some actions and you eventually won or you eventually lost the game. And you rewind it and you go step-by-step accordingly. So, basically what's happening in the breakout that you're putting 20 reward there, that's not a label done by any human, right? No, there's no human labeling in this task, but there is a game that has pre-definitions of what rewards are coming at what points in time.

31:17 Awesome. Okay. Now, exactly what you've just seen in this problem, these deep Q networks and learning these, this approach is has been scaled actually and this was one of the first big breakthroughs in deep reinforcement learning, happened in 2015 by DeepMind with the Atari game actually that you we just used as an example. And this was using convolutional neural networks. So, it took what we learned in the image processing lecture, literally took the pixels on a screen of this game and predicted a continuous value. That continuous value is the Q function and then learned this using the exact algorithm that we just saw. The result was actually really impressive for such a simple algorithm that you can implement in probably a few dozen lines of code, you can win over 50% of the games and surpass human-level performance on over 50% of the Atari games.

32:16 Right? You're below human-level performance on these right-hand games, but considering the simplicity of this algorithm, this was a massive breakthrough to the field, right? No labels, no prior information about the games or how they're played. Just in a completely unstructured environment, you let the algorithm free, you let it play a bunch of games, collect its rewards, and optimize its own actions as a result of those rewards. So, considering the simplicity and the the cleanliness of this idea, it's amazing to me that this this has scaled and and worked at all. But there are some very clear downsides to these approaches and that I will focus on for the the later part of the lecture today.

32:55 Uh probably some of you have already picked up on some of these downsides. So, number one, it's a it's a very limited complexity type of model. So, number one is that the model can really only handle scenarios where the action space is discrete in the way that we put it. So, the vocabulary of possible actions that the agent can take is from a predefined set of actions, right? And that's obviously having some limitations uh in the real world, right? Real actions probably are continuous as well.

33:27 Um and flexibility from a flexibility point of view, of course, a lot of games are not deterministically computed. They have probabilistic components to them. Some are stochastic as well. And using this policy, because you're always picking picking the maximum uh return using the maximum as the action that you you go after, you're you're not really optimizing for stochastic games. Right? That's because you always pick the action that results in the maximum reward.

33:57 So, to address this, we'll consider a new class of RL algorithms called policy gradient methods. And actually, we'll see a lot of parallels between what we just saw and these algorithms. Yes. So, just if you go back to that loss function, is it correct that the the labels are also relying on the network itself? In the loss function, the the target was this this time step's reward plus discount factor times Q. And that Q factor that Q value comes from the network. Can you not just use the actual rewards that you got in the rollout? No, it is the actual reward. So, probably it's a misnomer in the notation. So, the Q is then defined as like the true Q that's actually observed as the function of the rollout. That's why it's like a true true ground truth value for your predictor.

34:49 Excellent. Okay. So, we've just covered Q-learning. We'll cover the other side where we're not going to learn a Q function at all anymore. We're directly going to learn this policy function. The policy function is only a function of your state. It will directly learn what action you should take before any given state. But importantly, it's not going to be a deterministic function. This will be a probability distribution where we sample from our policy pi to compute a sample of our action that we should take for that given state. Right? And again, this is one subtle difference that we'll see between Q-learning and policy learning.

35:30 Okay. So, let's draw a direct parallel now for Q models or Q-learning models, right? We saw this already from the previous slide. We compute the optimal action A as a function of the argmax of all possible actions. So, again, we do this by plugging in all possible actions and picking the action that resulted in the maximum reward to come back. Now, the difference with policy learning is that instead of predicting the Q values, we're doing going to directly optimize the pi function. Pi is the policy.

36:03 So, this is the policy distribution over our state or excuse me, over our action space for a particular state. So, the outputs here are going to be again over the action space, just like we had before. But now, these are not going to learn the value of our excuse me, the value of our reward, but rather they're going to output the probability that any given action is going to result in the highest likelihood return to the agent. So, these are now probabilities.

36:38 Right? And importantly, we can see that if we predict these probabilities, we can aggregate them into a full distribution. Because they're probabilities, they will sum to one. Right? And we can now not just pick the argmax of these probability distributions like we were doing before, we actually sample from this distribution itself. Right? So, we encourage now stochasticity. We encourage the fact that we want the model to sometimes, let's say, pick this left action 90% of the time, but 10% of the time, it should actually also be encouraged to stay still.

37:15 That's something that would never happen in a Q-learning approach because you always pick the argmax. Okay. So, immediately, I think one advantage that comes from this is like what we just saw, the stochasticity, the ability and the flexibility for this model to extend beyond very deterministic uh types of environments. But beyond this, actually also a more subtle point is that this already has allowed us to go beyond discrete action spaces. It now allows us to also have continuity in our action spaces as well.

37:49 Let me show an example. So, one example between this is that we can now handle these discrete versus continuous spaces. Discrete action spaces look like this. Right? There's a finite number of possible actions that we could take. Let's say, left, right, or stay still. Every possible action will have an associated density on the Y axis. Right? Instead of this, of course, we could have an X axis that is not binned into three discrete bins. We could just have, let's say, instead of left, right, or stay still, we could have a speed at which we move to the left or a speed at which we move to the right. So, moving farther to the right means that we're going faster to the right. Moving farther to the left means we're going faster to the left. Right? And this allows us to now plot uh you know, not just a density point density for each of the values of our actions, but a full distribution across continuously across this full space.

38:49 And now we're looking at these densities, you know, on a continuity and like a infinity number of bins across the X axis. Now, let's take a look at how we can actually model these continuous distributions and probably uh a lot of this will look familiar from the last lecture that we saw with VAEs that I have a gave as well. Because now, instead of predicting just point estimates for every action, we're going to learn a distribution, right? We'll learn a mean and a mean and a standard deviation that defines that distribution that you saw on the previous slide. So, for example, let's say again, given the state on the left-hand side, this image of the screen with this ball coming towards the left and the paddle is on the right, the mean that you may learn here is -1 and the variance is, let's say, 0.5.

39:41 This distribution has a shape. It has a density to it. And you can sample from that distribution to infer one possible action that you should take given this possible this current state that you're in. Right? Now again, remember you're not taking the maximum peak of the distribution. So I did not say that you should go with -1 uh meters per second for example, if that's the yeah, meters per second. That's not the the optimal action. That's just the mean of your distribution.

40:12 But you sample from this probabilistically to compute one possible action that you take at this point in time. And of course, again, the same trend holds because since this is a probability distribution, we still have the the fact that this sums to one, right? This density function here sums to one by definition because we're outputting a mean and a variance and we're following a very structured form here. Yes. Why do we assume it's a normal distribution? Why do we assume it's a normal distribution? Uh it's an arbitrary choice. I think in this case it makes sense because it's a unimodal distribution. Often times you'll see other cases where it's multimodal. You could have multiple possible uh possible actions that you may want to take and it's not just one peak for example.

41:01 Uh often times you pick a parameterization of your distribution that makes sense for your problem. Right? In this example, we pick normal. Normal is a very common one because it's very easy to learn. It requires two numbers to learn this distribution and it's a good approximator for most tasks. But it is flexible. This is completely arbitrary. Instead of outputting two numbers, you could output an arbitrary number of numbers that parameterize any other distribution as well. You defined previously the P A given S.

41:34 What is exactly What does it mean? Is it probably of the action existing or probably A given the best action or Great question. Uh it is the probability that any given action results in the maximum return on future investment for that given state. Awesome. Okay. So let's tie this all together with a concrete example. Let's revisit this schematic of the reinforcement learning learning loop one more time and let's think about for a case study here, let's think about self-driving uh because I think all of the variables are quite intuitive, right? So let's say the agent is the vehicle that's driving.

42:18 The state is going to be let's say what your camera, what your lidar, what your sensing systems observe. For simplicity, let's just say it's the position of the car on a lane. Right? And let's think of the actions as again a very simplified framing here. This is the steering wheel angle, right? It's just one continuous value of how the wheel should turn, the angle at which the wheel should turn for any given state that the model's in.

42:42 And a very simplistic reward function would be how long the car can drive, the distance traveled before it crashes, right? So you want to maximize the distance traveled. You don't want it to crash. You don't want human intervention. So that is going to be the thing that you optimize for and you want to extend that as long as possible. So let's look at an example here. So training a policy gradient network in the context of self-driving cars would look like this. We are going to initialize our agent and all that means is that we have a random network, randomly initialized parameters, and the parameters take as input a state.

43:23 They output this distribution over possible actions that would maximize your your reward, but remember because this is randomly initialized, that distribution will also be random. So we run a policy until termination. Because we're starting from a random policy, this will crash very quickly, right? But importantly, at every point along this trajectory, we have recorded the triplet of state, action, and reward pairs that we execute at each point in time. Right? Because we have those rewards and the recordings of each of those prior states, what we can do is we can take all of those observations and take the ones that happen closest to the crash, let's say the 50th percentile, the things on the first the last 50%, we'll take those actions and we will decrease the probability of ever doing those actions again when we're in those states.

44:18 And we will also take the first 50% of actions because those happened earlier and they were not as correlated to the actual crashing event and we will increase their probability of happening again in those future states. Right? So very basic assumption. All we've done here is we're saying let's decrease the probability of ever doing anything that we did close to an accident. We'll increase the probability of doing things that we did away from the accident. Then you repeat this process again, right? You train this using policy grade You train this using gradient descent.

44:50 And now the second time you run it, the car is able to travel farther, but importantly, you do the exact same process, right? You decrease the probability of those actions closer to the crash. You increase the probability of these actions uh farther from the crash until eventually you keep repeating this until you get convergence and eventually you do observe that this car is going to uh stably be following the lane, right? Now the only remaining question I think to tie this all together, this algorithm together is how we can actually execute four and five, right? What is the loss function that will decrease the probability of things closer to bad rewards and increase the probability of of actions happening far from bad rewards or close to good rewards, you could also say.

45:39 So let's walk through uh exactly what this loss function looks like in this case. So the loss here consists of two terms, right? The first term is this log likelihood of a particular action that you're taking. So this is the probability, like we saw before, probability of a given action resulting in a uh maximum total return that the agent can take. And then you also have this total discounted reward RFT.

46:10 Right? So let's assume for example that we get a lot of reward for a current action that we take right now, a lot of rewards right now. That means that uh the action that we took probably had a high log likelihood because it was sampled, right? We sampled an action that had high likelihood and we got a big reward for that. That would be two big numbers multiplied by each other. You take their negative, that would be a very negative loss. It's a very good loss, right? You always want the loss to be low.

46:40 Let's take the converse case. If you take a action that was very probable but had a very low reward, you'll get the opposite, right? And you don't want that to happen. Right? So you want to discourage those cases. One more case I'll give you is that if you take a very unlikely action that resulted in a very high reward, you want to actually encourage that from happen encourage that to happen again in the future. And again, this loss function will encourage that because of the negative.

47:09 Right? And when you plug this into this uh this SGD, stochastic gradient descent equation, you plug in that loss function right here, the reason why this is called policy gradients is because you're actually taking the gradient of this policy. It's a very uh simple naming convention that was had and then this is where the name came from. Yes. So for actions that are extremely unlikely and that never get executed by your training procedure, but may have higher reward, you never end up seeing them, right? Your training will never be able to utilize that data.

47:42 Yes, great question. So the question is for very unlikely actions, you may never see them and then you'll never learn them effectively even if they have super high reward. The answer is yes, it's true and that's a downside of the approach. That's why it's very important to actually encourage random initialization in the beginning so that you do actually see a lot of actions. Even as you start to train, often times what you'll do is you'll still encourage more randomness into the model just to make sure that it gets exposure to explore other possible states. And that's this exploration versus exploitation trade-off. Even though you think you're in a good path, you don't just want to exploit that. You always want to have some exploration around as well.

48:20 Is there some version of this approach where you intentionally include non-sample trajectories in your training? Absolutely. Yeah, so there's a lot of flexibility actually in this approach because I actually the a lot of the nuance comes in running the policy, right? And sampling the forward policy. The step four and five, you follow the loss function and it's this this simple loss function encourages things that that were good rewards basically. Uh running the policy can actually give you a lot of diverse data if you sample in different ways, right? And often times you may not always want to sample the uh the exploitative strategy, right? To your point.

49:00 Okay. I want to talk a little bit about how to extend this type of problem of RL into real life as well because as I gave this previous example, hopefully there was a pretty obvious uh downside to the example that I just gave. So across these five steps here, can you think of one step in particular which is a really problematic step of why we don't really do reinforcement learning in the real world? Yes. Step three because it would be very expensive.

49:30 Step three? Uh if you run let's say a car on the road, you're anyways collecting a lot of data and recording data. So it's not too expensive to just record what you're already seeing. Yeah, any other ideas? Yeah. Okay. You don't want to crash all the cars. Yeah, exactly. So, yeah, you don't want to crash the car. So, step two is is definitely the the problematic step. So, yeah, doing this doing this algorithm that I mentioned, it explicitly defines that you have to crash the car in order to learn or you have to run to a terminal state. Let's say like this, you have to run to a terminal state in order to be able to backtrack, right?

50:14 Recording the states is okay, but actually getting to the terminal state, you don't want to do that just to learn, right? Um so, of course, simulation has really turned into a very big field for RL and that's why gameplay is actually also a very big component of RL because it's very safe to die in games, right? It's okay. Uh actually, I want to highlight one really cool result here at MIT, which was developing a brand new photorealistic simulator that's entirely data-driven. So, one big problem with simulation in general is that they're model-based. And if you look at a lot of simulators, they look very game-like.

50:48 They don't look very natural or realistic and it's very hard to actually understand what you know, how how you could translate a policy that was learned in a game-like simulator to reality. So, what we want to do is actually build what's called data-driven simulation, which means that directly from real data if you drive down a road, you should be able to simulate an autonomous car also driving down that road taking different actions. So, actually all of the roads that you see here are actually real roads, but being driven in simulation not using a a real car.

51:19 Um but, these are actually real roads with other real cars next to it under real times of day. Um in fact, we actually did exactly this. We trained in that simulator and then we deployed it on that same autonomous car that I mentioned earlier that's actually just downstairs one floor below here. And at this point in time, this was in 2020 a few years ago, but this was actually the first time any autonomous vehicle was ever trained end-to-end in simulation was successfully deployed in the real world.

51:48 So, it was a very cool first instantiation that we had. Um and I want to cover a few more applications of these techniques because now we have a lot of RL permeating into big algorithms like LLMs, for example. Uh but, a lot of this did start with game and with game playing. So, Go was actually another huge example where RL agents were put to the test against human champions and achieved what was at the time an extremely exciting result.

52:17 Uh so, for those of you who are not familiar with Go, this is a board game played on a 19 by 19 grid and you have two players, black and white. And the objective here is to hold more space on the board than your opponent. Uh this is a massive combinatorial problem. There are more possible board states than atoms in the universe, things like this. It's like classically hard problem for machine learning algorithms to learn on.

52:42 And the goal is to train an AI to master the game of Go, not just only beating the existing AIs that existed, but actually outperforming the human Go champions, right? Uh Google DeepMind did this very successfully uh a few years ago. They developed a deep reinforcement learning paradigm that actually relied both on supervised learning to start. So, they started first by first collecting a bunch of human data and just training not using reinforcement learning, but using supervised learning given a state to predict the action that the the model should take and just copy, imitate basically, the human champions.

53:22 That was step one. Step two was then to take that bootstrapped model that was learned by imitating humans and then play them against each other. And then every time you play them against each other, you do get a reward signal that comes back. And now you can train using reinforcement learning because this is now effectively your simulator, right? You're simulating against yourself or another agent or another AI that was trained using imitation learning. And then you repeat this, right? You cycle on this approach and eventually, you also have to build some intuition about what is a good board state. So, in training this model that's self-playing against itself, you're actually teaching the model to build up some intuition about given any state was this a good state that you want to reach because it's very important for a model to understand this because then it can avoid other states in the future that may lead you into a case like this.

54:14 You can I won't talk about it so much today because we'll talk about it later in the class is the extensions of RL also in LLMs because LLMs have a huge preference alignment problem where they're first trained to imitate a bunch of text exactly like this. We train LLMs by showing them a bunch of text from the internet and you imitate the process of next word prediction. But, actually what we're seeing here is that you eventually want to also go beyond imitation and be able to optimize for preferences of human likes versus dislikes in their answer as well. And those likes versus dislikes, those thumbs ups or thumbs down that you give when talking to a chatbot are exactly training a reward model for reinforcement learning algorithm to maximize the thumbs up and minimize the thumb down that you talk to on your LLMs. Yes.

55:06 In [clears throat] between like two and three, was there also a step of like take that great one that you built and then use it to train just from scratch without ever using human stuff? Great question. I actually wasn't going to show this slide, but yes, there was an extension of RL from scratch uh where you start from a random network. And the question was actually, is it first of all possible to to train from scratch where you actually know nothing about the game? It's was very nice actually to imitate humans first because it it made it accelerated to to learn quicker, right? On the self-play part. But, it turns out that yes, you can actually do this. The green is some of these baseline models. You can see that even training from scratch, you really do get to their performance very quickly and it translates across games, right? So, what I showed previously was on Go. Um you can see it across many other games as well. Chess, Stockfish, and so on. Yes, go ahead.

56:01 Perhaps I might have missed something. The board sizes of all of these games are different. Would that not necessarily mean the input matrix is a different size? Uh is AlphaZero using some sort of machine learning? No. Some sort of machine vision thing and using a picture of the board instead of a matrix representing the board? Great question. Uh so, all of these use a matrix representing the board, but the important thing that you are correct is that they do have different board sizes.

56:36 So, each one is playing a different game. Like you have each model is optimized for its game that it plays. Okay. Yeah, so so, yes and no, I would say is the answer to your question, right? So, yes, they're different, but also no because they're trained separately as well. You could probably train them jointly as well. Like, for example, you could bring them all up to the resolution of the biggest board. For example, right? So, chess is I believe 8 by 8 I think, right? And uh Go is 19 by 19. So, you could bring them all up into the same you know, common denominator resolution as well.

57:17 If I understood I see two ways to play that the trained policy network can just play the game. Or you can use the value network to choose the next play. So, is there a performance gain or loss from the last step? Oh, I see I see the question. So, the question is between two and three, are these actually just two different ways to pick an action? Uh the It itself when they're giving you a policy on how to play.

57:44 >> [clears throat] >> Yeah, good great question. So, three is informing the policy itself, right? So, the the value network that you learn is also going to go back and actually help inform the different actions that you should be taking to get to good value states as well because you're getting some intuition about what are good states to reach versus what are good what are board positions that you don't want to be in, for example. Excellent. Okay, so we covered a lot in today's lecture all the way from the beginnings of RL from the definitions and terminology to Q-learning and policy learning. I hope a lot of this was clear. We'll answer a lot more questions today during the labs uh if there are any questions.

58:26 Uh you will get a lot of experience with some of the algorithms talked about today in the labs uh to actually fine-tune some of your models and there will be actually opportunity to use RL as part of that process. Um we'll switch over now to the the new generation, next generation, and limitations type lecture. It's a very fun lecture that we do every year to look back on a lot of the different works that have been coming up in the field over the past few years and highlight the next-gen models as well as remaining limitations that the field uh presents. And we will also be showing t-shirts coming tomorrow as well as well as some exciting guest lectures. So, um I'll do a quick transition to the next lecture and we'll be back in a few minutes. Thank you.