MIT 6.S191: Deep Generative Modeling

0:05 [music] >> All right, so in this lecture for the second half of our lecture portion, we're going to be talking about generative modeling. And it's perhaps no secret now that we're in this tremendous age of generative models and generative AI as you may have heard. And so today we're going to learn the foundations of generative modeling where we're going to talk about how we can build neural network systems that not only learn patterns in data like we've seen in the past three lectures, but actually learn to model distributions of data so that we can sample to generate brand new data instances.

0:47 This is a very, very powerful and complex data idea. How can we learn distributions of data so that we can model that distribution and also sample from it to produce new instances. So to demonstrate how powerful these algorithms have gotten, I think the best way to do this is by showing you. Traditionally, we've done a similar type of poll where I ask you to consider these three faces. Think for a moment and we're going to do a poll in a second of which face out of these three you think is real.

1:26 Raise your hand if you think it's A. Okay, we have like four people. B. Quite a lot. C. Also a fair amount. I think perhaps equal between B and C. And in past years, I've done this as a trick where all three of the images are actually fake and generated by AI, But, that's not the case. This year, last minute, I decided to switch it up because the models have gotten so powerful that I thought for the first time it was truly indistinguishable between what is fake and generated by a generative model and what is real. So, the right answer is that picture C is the real image.

2:22 And so, these these A and B were generated by Google's latest Gemini model, which if you've played with it is really awesome in terms of its capabilities in image generation. And I think this goes to show how far the field has come over the past, you know, 5 years or so, where now we are getting this these advances that truly, truly are remarkable in terms of the quality of uh generative modeling that we're able to achieve.

2:53 So, let's get more concrete with this motivation in hand. So far in this course, across the past three lectures, we've been looking at a broad class of problems, whether they be through feed-forward models or recurrent models or CNNs, that generally fall into this bigger picture paradigm of what we call supervised learning. Meaning, we are given data where X are the data and each data instance has some sort of a label, Y.

3:24 And the goal of our neural network modeling is to learn a functional mapping between data and label, X to Y. And these types of functional mappings can be classification, regression, object detection, so on and so forth. But, there's a larger class of problems in in machine learning that falls into this new categorization of unsupervised learning, where we're given only data, not necessarily labels, and the goal is to learn a model that captures this distribution of data, so that we can learn the underlying structure of the data, that is the distribution itself.

4:06 And if you're familiar with approaches that you may have used in data analysis, like clustering or other techniques for dimensionality reduction, these are also methods for unsupervised learning. But generative modeling gives us a powerful way to tackle these sorts of problems. Where our fundamental goal is to take as input training samples from some distribution of data, and learn a neural network model that represents that distribution, that probability distribution.

4:39 And it really amazes me that even at very advanced levels, people often don't realize that this is really the core fundamental principle of generative modeling. Capturing and learning probability distributions over data. And so we can use this in a couple of main ways. The first is this idea of density estimation, which is precisely what I referred to, where given some samples that fall along a distribution, we want to learn a model of that distribution. Basically describing the underlying probability distribution from which these data samples generated.

5:22 The second use case is very closely related to this first. Where we can think of now sample generation, generating brand new data instances. That is, if we have effectively learned a probability distribution over our data based on some training samples, we can then sample from that probability distribution to generate brand new instances of new samples that did not exist in the true training data.

5:53 And in both these cases again, the fundamental question in principle is this idea of how can we learn a probability distribution over the data modeled by our model that is as close as possible to the true probability distribution over the data itself. So, with this in hand, why do we actually care about generative modeling? It's not just about generating hyperrealistic images and things like that. The use cases are far beyond that.

6:24 One powerful way that you'll actually get hands-on experience with in today's lab is to learn the underlying features in a data set and use this to improve fairness and representation of the model itself or perform what is known as debiasing. And we know that in many, many cases and examples in the real world, there are fundamental biases in the data sets that have been collected and as a result, models trained on those data sets will capture those same biases. So, using techniques from generative modeling, we can actually leverage the models learning of that data data distribution to now debias the model itself to learn more fair and representative data sets and distributions.

7:16 Another great example is in outlier detection, right? Remember again, generative models are capturing this probability distribution over data. In cases like self-driving or also in medicine, it's those really, really rare edge cases that are going to present problems versus, you know, 90% of the distribution. And so, thinking about how can we use a generative model to detect when there's an event that's a rare event or an outlier and basically mitigate that and and correct for that behavior.

7:51 And finally, perhaps the use case which you may have had the most experience with is in actual sample generation. So because again of this principle that generative models learn probability distributions that means that we can sample from that distribution to generate new data instances. And this is the backbone of the generative AI advances that we are seeing behind natural language models like GPT to image generation if you've played with, you know, nano banana or copilot designer or um Dolly or all these other types of models, these are generative models that are doing sample generation over image space.

8:36 And it goes beyond that, right? To even biology and chemistry and we'll hear more about some of these ideas in applications such as drug discovery. So hopefully that sets the stage of how powerful of a modeling paradigm this idea of generative modeling is. And as with all our lectures in this class, we we like to go back to the foundations of where we came from, how we got there and build up to where we are today.

9:06 And so today we're going to focus on two classes of very foundational generative models that sort of kick this whole thing off which are autoencoders and variational autoencoders and what was previously the dominant paradigm for sample generation, generative adversarial networks or GANs. And in particular autoencoders and variational autoencoders are very powerful class of what we call latent variable models And they're so very widely used today.

9:40 And when we think about them and when we talk about this notion of data distribution modeling, it I think it's very helpful to contextualize what we actually mean by these underlying representations of data and latent variables. So, the example I love to use to describe this is this little story from Plato's Republic from back in you know, ancient Greece, which is known as the myth of the cave. And in this myth, there's a group of prisoners who, as part of their imprisonment and I guess torture in some way, are constrained to face a wall and just sit there and not do anything else.

10:23 And the only things that they see on this wall are shadows reflected by objects that are passing behind them and luckily they have a fire to keep themselves warm. And so, to the prisoners, right, they these shadows that they're observing are their truth, right? Their observations of their reality. They don't know that what exactly is going back behind their heads of these objects that are casting the shadows on the wall. And while they cannot directly observe or measure those objects, they seem to have a sense that they exist because the shadows are being cast on the wall.

11:01 And so, with latent variable modeling and with encoding approaches in general, the goal is very similar and kind of inspired by this, where data may fall across some distribution and it that distribution is what we observe, but in truth, there are some underlying features or variables that are governing the core sort of axes of variability in that distribution. And so, with latent variable modeling, our goal is to try to numerically capture this sort of behavior. Use neural networks to encode this information in a finite set of latent variables that can capture the big axes of variability in the data.

11:46 And so, the the dominant approach for doing this is this general principle of encoding. And it started off with a modeling framework known as autoencoders. And these are very elegant in their idea. And as the name describes and we'll see, this idea of self-encoding the input data. And the principle behind this is that we take in raw data, let's say it's of images. And we train a neural network, pass this data through many neural network layers, and try to arrive at a low-dimensional latent space or set of latent variables, Z.

12:27 Which you can think of as a representation of the core features in this data. And the idea behind encoding is we're trying to learn a mapping from the original data distribution to this lower-dimensional space, Z. Now, the core question here is, remember we're in this world of unsupervised learning. We don't have class labels or scores for our data, but we automatic we want to inherently learn the structure in the data. How could we actually achieve this and why would we even care about doing this in the first place and learning this lower-dimensional feature?

13:10 Well, in a way, right, this is a form of compression and information compression, right? Your data may be very very very complex, but if you distill those axes of variability into something lower-dimensional, you can capture the key patterns in the data. The next question is, how do we even go about this? What is our objective or task to be able to do this sort of encoding? Well, the idea is quite clever because of the fact that we have the data itself. So, perhaps we can task the model to actually reconstruct or regenerate the data itself and use this as a signal to train the model.

13:56 And this is the idea that autoencoders employ. We go from an encoding into this lower dimensional space Z and then go back up from that encoding back to the original input space and task the model with trying to reconstruct the individual data instances. And we'll we'll denote the reconstructed output as X hat, right? Notice there are no Ys here, there are no labels. The prediction itself is over the data space and the model is trying to go from input to reconstructed output.

14:35 The way we can actually give a loss function to the model to learn this is by simply comparing the two, the true ground truth input X and the predicted reconstructed output X hat and task the model to minimize the difference between the two. And if you'll notice, right? Here in this example with images, this loss function L of X and X hat is looking at sort of the pixel-wise difference between the input and the reconstruction.

15:11 And what you'll hopefully appreciate is that there's no dependence on Y, any sort of label here. We can train this model using a loss only dependent on the input and this reconstructed output. So, this is gets This is a very powerful idea, right? And in the remainder of this talk, like we've been doing um in the prior lectures, we'll abstract these individual layers away using these symbols for the encoders and decoders here.

15:44 And so, this is really a powerful idea because it gets at this notion of compression and compressing information from very complex, high-dimensional inputs into something lower dimensional, like the latent space Z. And because of this notion of compression, what that means is that the dimensionality of the latent space is going to give a bottleneck on the amount of information that we're able to capture in those latent variables.

16:15 And so, something that's smaller will force a larger compressive bottleneck, whereas something that's larger will enforce less of a compressive bottleneck. But, we don't want it to be too large because then what's the point of learning a small set of latent variables in the first place. And so, this gets us the idea of autoencoding and using autoencoders as a form of what we call representation learning from these learned latent variables Z.

16:46 And the core idea is is that we have this bottleneck layer, this latent variable set Z, that forces this learning of a compressed representation, and we can train the model in this unsupervised way by just reconstructing the input and taking the loss between the two. So, this gets at this idea of encoding data or self-encoding data that autoencoders use. So, in practice, right, this is very elegant, but it's not so useful for generating new samples because of the fact that this traditional approach to autoencoding is deterministic. Meaning, once we train the autoencoder, if we have an input X, every time we pass that input in and make a call to the model, the reconstruction is going to be exactly the same.

17:45 And so, it's not useful for sample generation because there's no notion of variability or really of a probability distribution here. And the way that we can get around this, and the way that was proposed to innovate on this to actually get towards sample generation and also better representation learning is this idea of variational autoencoding or VAEs. And the core principle here is can we introduce a probabilistic twist on this idea of self-encoding?

18:21 With standard autoencoders, right, as I mentioned, it's deterministic. The model is trained, you take an input in, you get a reconstructed output out. In VAEs, instead of taking that deterministic layer Z, we can replace that with a sampling operation. And in in particular, instead of learning the latent variables directly, we now learn a mean and a variance for each latent variable. And the idea here is that this mean and variance can parameterize a probability distribution for that latent variable that will then enable sampling from these variables to produce new data instances.

19:07 And concretely, this means that we go from learning set of variable Z to learning a vector of means and a vector of standard deviations or variances for each of these latent variables. So, as you can tell, right, the architecture on both sides looks very, very similar to the traditional autoencoder. The core part is in the middle here, where now we have this parameterization of a probability distribution over each of the latent variables.

19:41 Now, we can formalize this a little further and think about our sets of weights. Instead of deterministic mapping, now the encoder is capturing a probability distribution over the latent variables given the input data X, and the decoder is computing a probability distribution over the data space X given the set of latent variables Z. And importantly, like all the principles that we introduce in this class, we parameterize these with weights shown here, phi and theta, that define these encoders and decoders respectively.

20:23 And these weights are what is actually learned by the neural network over the course of its training. So, how do we actually do that? That's kind of the final piece of the puzzle. Again, think about inputs, think about outputs, think about and finally think about the loss or the objective that we use to actually train a model going from input to output. With VAEs, we have that same term of the reconstruction loss as before. And now we are going to intro- introduce just in a little bit a new term called the regularization term, which may also be termed a distributional loss, and that helps us enforce learning of good probability distributions in the slate and space.

21:11 And we'll break this down step by step. But importantly I'll note that this loss is dependent on these two sets of weights phi and theta. And keep in mind in learning we're trying to optimize this loss with respect to these two sets of weights from the encoder and decoder and have these two terms that dictate the function by which we perform gradient descent. So breaking it down first the reconstruction term. This is very similar to before. We're going to have some way to compare inputs to predicted or reconstructed outputs.

21:50 The second term, the regularization term is where now things get a bit more interesting. If we go into this a little bit in more detail, right? Remember our goal here is to learn mu sigma parameterizing distributions of these latent variables. Similar to how in reconstruction loss we compute this distance between reconstruct input and reconstructed output. With this regularization term, we're going to compute also a sort of distance, but now it's a distributional distance.

22:26 What I mean by that is this, right? The encoder is inferring this latent distribution Q of Z given X. And we're going to place some prior or regularization on the form of what that distribution should look like. Which is captured in this term P of Z. And in regularization we compare the distributional distance, meaning how similar is the shape of our learned distribution to the shape of the prior distribution using this term D.

23:03 The reason for doing this is if we do not impose any prior, right? The network would be very unregularized. There's no sort of constraint or sort of you know, enforcing on what shape this distribution could take. And we're going to break down why this is actually useful in training and in learning high-quality representations and high-quality latent spaces. To understand that, we have to consider what is often chosen as this choice of a prior.

23:37 In practice with VAEs, no matter how large your encoder, how large your decoder are, it can be very typical to use a normal or standard Gaussian distribution as the choice of prior. And in particular, this distance term can be thought of as a divergence between the learned distribution and that normal prior. To get at why we choose the normal prior and what this regularization achieves, I think it's very, very helpful to think about the intuition behind this in terms of why we even want to learn these latent spaces and ultimately what sort of features or properties they should have.

24:23 First, right, to get at that question. What is the quality of a good latent space? It should be continuous. Meaning that points that are close in the latent space should be close in meaning or content or the form of the data itself, especially after we decode and reconstruct them back out. The second core property is this notion of completeness. Meaning that there's some sort of semantic structure that's learned and captured in this latent space.

24:56 Such that when we sample from the latent space and do decoding, we get meaningful, consistent things back out. Without any regularization, the issue becomes that you lose these two desirable properties of continuity and completeness. You could have instances that where points may be close into the latent space, but they're not decoded similarly, right? And this example is showing that in intuition with a simple example of shapes. Where let's say these green and red points are very close in the latent space, but one corresponds to a square and one corresponds to a triangle, which are totally different shapes.

25:38 You could also have instances of incompleteness, meaning there are parts of the latent space that are not meaningful. And if you have a point there, it doesn't yield um consistent content when you decode back out. In contrast, our goal with regularizing is to counteract these issues, right? Such that when we have points that are close, they can be decoded back to uh meaningful things, meaning we have a good representation of the data, and it's complete.

26:13 So, that gives us an intuition of what are properties we want to achieve. Now, how does the choice of the prior figure into this? Could we choose any arbitrary distribution and be guaranteed to have these two properties of continuity and completeness, or is there something special about normal distributions that allows for this? It's not always guaranteed that any choice of a distributional prior will achieve these two properties.

26:45 Because, for example, we may have a case where the Here the shading represents the variance, so like kind of the spread or the width of the distribution. Where if the variances are not large enough, the distributions could be very pointed. And so as soon as you get out of that point, right, maybe the content is the same, but you're not going to decode meaningfully. And further, if the means are different, if they're not, you know, zero centered, these different latent variables could end up very, very far apart such that there could be discontinuities in the latent space, where now you sample from some part and you don't get anything meaningful back out.

27:33 The intuition behind the normal prior is that it helps promote this notion of continuity and completeness because we center the we have this prior of centering the means and regularizing the variances. And so this is the core intuition behind why we regularize, we have this distributional similarity, and we choose the normal prior in particular. Now, does this mean that all the latent variables are going to end up 0 1? No, right, because we have this other term in the loss of the reconstruction term.

28:09 And remember that the regularization is just a choice of prior and we're still keeping this distributional distance between the inferred latent distributions and the prior. So we still should be good with proper weighting of the reconstruction term and this regularization term. So the final piece to the puzzle is backprop, right? We've defined our loss, we know our inputs, we know our outputs, we know the structure of our architecture overall.

28:45 Now it gets time to train. How do we actually pass gradients through this thing? The problem is this sampling operation. Because we introduce mu, sigma, and we want them to capture variability and uh stochasticity in the data, we can't back-propagate gradients through something that is inherently probabilistic. And so, the way that VAEs correct for this and get around this is to say, "Okay, latent variables, in principle, yes, we think of them as sampling from this normal distribution, but in practice, to actually make this trainable, we will consider these latent vectors as the sum of a fixed mu vector, fixed vector of means, and a fixed vector of sigmas or standard deviations, but we scale these sigmas by random constants that are drawn from some prior distribution.

29:46 And so, by doing this, we enable a structure where we still have a probability distribution parameterized, but that variability, that stochasticity, is basically offset to this random variable epsilon that we're not going to learn directly and therefore not back-propagate through. You can kind of appreciate this more visually by looking at this diagram where in the original form, Z is kind of the bottleneck layer here, and if it's a sampling operation, we can't back-propagate through it.

30:25 But instead, if now we keep Z as a deterministic node in this computation graph, and offset the sampling and the variability to these random samples epsilon, this allows for back-propagation and update of the weights with respect to these variables. Yeah, question. >> Would it not be possible to sample on the on the output? So, you don't sample at the latent space latent space, keep it as a distribution.

30:56 You compute some distribution in the output. You sample from that, and then you compare that to your [clears throat] input. >> Yeah, so the question is can we think about this paradigm from the perspective of just going to output directly and doing sampling there and then using this in some comparison to input? Absolutely. This is another paradigm for generative modeling where now in an instance where you don't care so much about having some like fixed feature representation, your goal is really in sample generation, we can think about ways to define noise or variability in the transformation of input directly to reconstructed output and use the signal there to basically train the model. And this is closely related to principles of, for example, diffusion models that have been introduced more recently, and we'll talk about that in this tomorrow, I guess. Yeah, tomorrow's lecture six. Yeah.

31:58 Great question. Okay, so yeah, the the the core of that question is is really again coming back to this principle of how does your model inform your task and vice versa, right? What is your end goal? What are you actually trying to do with the neural network you build, and how does that inform the architecture choice? With VAEs, we're saying we care about this latent space, right? We want to learn this compressed set of variables so that we have that as a representation and also perhaps be able to sample from it.

32:33 The cool thing that VAEs enable us to do is because we learn these latent variables, this gives some degree of interpretability or control when it comes to sampling. So, it turns out that in a high-quality, well-trained VAE, you can think of these latent variables as capturing individual features, such that if we perturbed the value of the latent variable slightly, increasing it or decreasing it, and decoded back out, we could yield subtle differences in the reconstructed outputs.

33:11 And the core idea here is that in a well-trained VAE, the different dimensions of the latent space would encode different interpretable features. And so, coming back full circle to the motivation that we started with, right? In today's lab, focusing on facial detection detection systems and debiasing, you'll actually put these two pieces together of CNNs and VAEs to train an approach for debiasing facial detection systems and using the latent distribution to achieve this.

33:47 So, to summarize, right? This This is this core idea of compressing the representation of our data to something that's uh we can use for learning or feature encoding. And reconstruction allows us for this unsupervised approach. We can use the reparameterization strategy to train these models end-to-end and make things differentiable. And we can look in and interpret some of the values of these latent variables using techniques like perturbation, and ultimately be able to generate new examples.

34:25 Now, coming back to the the question, maybe sometimes our goals are different. We don't care so explicitly about a set of interpretable latent features, but just want to generate new samples. And this is the principle behind a variety of different generative modeling approaches, but the one that kicked things sort of off was this approach called generative adversarial networks or GANs. And so the the core idea here is let's say we don't explicitly compress the data distribution into a set of latent variables, we just care about sample generation and generating new instances.

35:07 The challenge still is our data distribution may be very very complex. So it's hard to do this directly. And the idea of behind GANs was can we learn a neural network that now maps something very simple like a distribution of noise to something more complex like a distribution of true data. And the core idea here is behind GANs is we're going to turn train a generator that learns this mapping from random noise to a set or representation that captures the data distribution.

35:49 And the goal is for that generations from this network to be as close to the real deal as possible. And with the structure of GANs, the idea was very clever. The idea was can we make this generative model by pitting two separate neural networks, an encoder that is a discriminative encoder, to and a generator against each other and having them sort of compete in a loss function that captures this notion of competition.

36:24 And the two networks, the first is a generator, which is trained to turn noise into a uh imitation of the data that's as close to the real distribution as possible. And a discriminator, which takes as input some examples, which may be real or may be fake, and outputs a classification of probability score of whether that instance is real or whether that instance is fake. I mean, you all were discriminators in the very start of this lecture when I showed the three faces and asked, you know, which one is real.

37:02 The The task of the discriminator in the GAN is similar in spirit. And so, this is this notion of competition. Can the generator produce better and better fake instances to try to fool the discriminator? And so, one of I think, hopefully, a a very um clear and intuitive illustration for this idea is looking at just something simple, points along a line, right?

37:34 The generator starts from random noise, and it has, you know, a random initialization before training. It produces some samples that are going to lie somewhere on this line. The discriminator can see these points and also see real examples. And now, we task the discriminator with classification. Are these real? Are these fake? And let's say, you know, initially, it produces some prediction.

38:04 Maybe its predictions are okay, maybe they're not if it's not trained. But as we train it, the predictions are going to improve such that it increases the probability for real instances being real and decreases the probability that the fake instances are real. Over time, this improves and eventually we get perfect separation. Now, we come back to the generator. The generator sees these examples of where the real data lie and in its training, over the course of its training, it it gets better and better at generating new instances, moving its generations closer and closer to the real data.

38:46 So, now we go back to the discriminator. It looks at these points and it estimates the probability that each point is real. And because the fake instances have moved closer to the real instances, they're going to be assigned a higher probability of being real. But still, over training it learns to adjust and do better at discrimination, continuing to adjust, and then we go back and repeat again one last time. Generator starts moving its samples closer and closer to the real data, such that now it's going to be very, very hard for the discriminator to distinguish because our generated instances are so close in distribution to the real the real data.

39:33 So, this is really this core intuition. And again, coming back to the very beginning of the talk, the core idea of capturing distributions of data and sampling new instances that are as close to that distribution as possible. We can concretize this intuition more formally, right? When we look at the objective for training GANs, where G, the generator, is producing fake instances to fool the discriminator D, which is trying to identify these from truth.

40:06 And so, in training, what this amounts to is these sort of adversarial objectives between network D and network G. And the best possible optimum that we could arrive at is that G reproduces the true data distribution exactly, right? That's the ideal case. How do we actually do this in practice? In practice, this means that the objective in the loss has to be adversarial, right? Generator and discriminator are at odds with each other.

40:38 And so if we break this down a little further, it looks perhaps a little scary at first, but we are going to go step by step. With the perspective of the discriminator, its goal is to maximize the probability fake data is fake, real data is real. And here G of Z is generations from the generator. And so this first term is looking at the probability that the discriminator correctly assigns a fake instance as fake.

41:13 And the second term is looking at the probability that the discriminator correctly assigns a real instance as real. And together, we want to maximize this with respect to the discriminator. On the other On the other side, back to the generator. While the generator can't directly affect the objective of the discriminator, what it can do is um produce instances that are as close to real as possible.

41:48 And so its loss is equivalent to minimizing the probability that the generated data is identified as fake. So putting these together now, we can summarize this as follows where the generator is trying to minimize this likelihood, the discriminator is trying to maximize this likelihood, and this is the principle of adversarial in again. And at the end of the day, right, we train our network uh using this objective, and our goal is to now use the generator to produce new instances.

42:27 And a quick note that in practice, right, perhaps as you can imagine, these adversarial objectives can actually be quite tricky to get this model to train stably. And so over the years, there have been many advances to improve the training stability of GANs and make them more scalable. And varying success. In some domains, GANs are still used, but there are other approaches that have moved beyond, you know, the this principle in terms of just having one model that can generate new instances, as we'll see tomorrow.

43:05 But coming back to GANs, and again, thinking about this this core principle of data distributions and how we model that. The principle, right, in doing this transformation is to go from noise to this learned target distribution. And it seems really crazy at first. How can you turn pure random noise into something meaningful? But because of the fact that we have these two networks that are adversarial with each other, we can update and use the power of of gradient descent and back propagation to actually learn weights that do this functional transformation.

43:47 And so because of this, what this means is that one a sample of noise from one part of this random distribution produces a particular instance. And a sample from another part of the noise distribution produces a different instance. And as a result, you can and about, and people have shown that you can actually traverse the space of a Gaussian distribution and use that to create instances that are semantically consistent in terms of how the the data is actually shifting itself.

44:22 And so this is this core idea of distribution transformation, right? A standard GAN goes from Gaussian noise to some target data distribution or landscape. There are also variants of GANs that rather than going from noise to one data landscape, transform one data landscape into another data landscape. And one common approach to do this is an architecture called a CycleGAN where there are two separate um uh generators, two separate discriminators that basically do this unpaired image-to-image translation where the model takes as input instances in one domain and can produce instances in the other domain without necessarily having a paired instance.

45:18 And so the goal here is that generations in the target domain follow the distribution of the target given the instance in the first domain as sort of a starting point. And the way that this is achieved by is by um a cyclic loss that connects these two domains. And in fact you know, these principles don't just apply to visual images, but we can think about instances across other domains and modalities as well in transformation of speech from one style of waveform to another where you can you could feasibly think about training in the audio space directly or converting the audio waveform into an image representation and employing a cycle GAN approach.

46:10 And so, audio generation has come a very long way, but when we started playing with these types of models, we used a cycle GAN approach to actually do the um audio to audio synthesis for taking Alexander's voice and producing the Obama style voice um that we had done a few years back. >> Hi everybody, and welcome to MIT 6.S191.

46:43 Hi everybody, and welcome >> I will say it was many a few different takes in a conference room that we did now now some years ago to actually get good samples to to um to do this and probably that was the least fun part of that. >> [laughter] >> Um Anyway, so this summarizes this lecture and um again, we've we've talked in depth about these two core foundational generative modeling approaches. One with GANs in which we our goal is primarily about new instance generation.

47:24 And secondly, with or rather firstly, with autoencoding based approaches where we're thinking about feature encoding and learning this representation in latent variables to capture the key axes of variability in data. And so, again, with with all the principles and methods and architectures we introduced in this class, right? Our goal is to think about how these foundations set the stage for the advances that have uh, come since then.

47:57 And now we're at a place where with the recent advances in generative models, as you saw at the beginning of the lecture with the faces, the quality of these generations is very, very, very high fidelity. And it's not just for images, it's also for videos or you text or even other modalities. And we'll see more in lecture six tomorrow about some of the modeling frameworks that have been introduced that take some of these same principles of data distribution modeling using noise as a technique to do reconstruction and have unlocked a new series of capabilities and quality of generation in the recent years.

48:40 So with that, I'll close and remind you all and encourage you all to dive into the software labs. This is really a very strong highlight of the course and it comes with the perk of entry into the competitions for the software labs. And so please, uh, have your hand at this. We'll be here, our TAs and ourselves to answer questions over the next hour or so. And finally, um, we do have a, uh, we will prelude that we hope to see you tomorrow with the possibility of receiving course t-shirts at the end of class.

49:22 Um, and so stay tuned. We like to keep you on the edge of the seat uh, your seats over the next 3 days. Um, and thank you so much for your attention. >> [applause]