SHRED 3 SINDY and Koopman

0:07 we're going to continue now talking about shred but now we're going to use something that's a nice trick for people which is to start exploiting the latent space that shred is learning between the lhm and the decoder Network itself so what we're going to do now is think about imposing structure in the latent space the specific structures we're going to go after are things that make a lot of sense for dynamical systems one is to impose something like ascendi

0:33 architecture which is the sparse identification in the line of Dynamics or something like a coupan architecture in there which is enforce a linear Dynamics inside of that latent space this can be done very nicely within the shred architecture and really what it involves is essentially accessing the latent space of that lstm before it goes to the decoder and imposing a loss function consistent with models that do this so if you remember this is the

1:04 basic architecture we're creating a map from trajectory information on your sensors through a a Time sequence model lstm grw RNN Transformer whatever you happen to use here to a decoder back to the state space so it's a time space separation of variables but now what we're going to do is in this latent space representation this gener erated by the sequence model we're going to impose a regularization or a loss

1:36 function that goes along with that so that's going to be the idea it's regularize the latent space to something that's advantageous for us to work with which is something like a Cindy model or a cman model or a DMD model so that's the basic idea here just taking shred ramping it up and what you're going to see here is we can do some really fantastic things with that so first here's the mathematical framework we're really talking about so this is a very

2:03 generic model Discovery framework normally we're taking measurements trying to do discover Dynamics but notice what we're doing here we're going to take measurements of the system these are the yftk and we're trying to map it from that measurement space to a state space variable X through a measurement Model H so once we get into the state space the idea is to impose some kind of structure around the Dynamics F itself so this is an illposed problem unless we provide

2:30 regularization and the regularizations that we're going to Pro provide here is that we either want a linear model for f so learn coordinate system in which the Dynamics is linear or learn something that's parsimonious which is something that the Cindy model architecture captures in either case this model now becomes very nicely interpretable for us to use either for future forecasts State PR St space representation but also just learning what the underlying Dynamics is of potentially a very high dimensional

2:59 system system and how it collapses down in fact in reduced order modeling then this is our reduced order modeling if we pinch down that latent space as far as we can so let's first talk about what would it mean to constrain to a linear model now linear models are very nice because we can actually write down their Solutions right so you know we might have a nonlinear Dynamics itself but the idea is to approximate it with some best

3:25 fit linear Dynamics and the reason we like this often is because we can just write down the solution explicitly do forecast have a representation of the modal structures inside of this so this is a very powerful technique it's typically what we do in our courses in dynamical systems Odes and pdes we always usually are trying to Sol typically what we can do close form is solve linear models and that's the solution to the linear model now how

3:51 would we do this in a data driven way there is What's called the dynamic mode decomposition and the dynamic mode decomposition is a regression directly to that solution form now there's different ways to set up the dynamic mode decomposition what I'm showing here is the optimized Dynamic mode decomposition which is taking some data snapshots we're actually going to in fact do what's called bagging optimized Dynamic mode decomposition so this is work from DIA and Travis here that what

4:18 you're going to do is sample time stamps of the data and fit it to this exponential solution which we know as a solution to a linear model and by sampling over and over again we have an ensemble or a bag of models that we can actually look at their mean and their variance and and we can have now in some sense a linear model that is very robust because the the bagging itself robusti vises procedures the

4:43 modern version of DMD also allows you to constrain the I values to the left half plane or on the imaginary Axis or even come in complex conjugate pairs all this DMD technology is now in the pi DMD package that's here so in some sense that's what we're going for so we go to do a constraint like this in the latent space of the shred this is what we're constraining to find this kind of model that we can put there in the latent

5:11 space and you can see once we have this model it's very easy to characterize the behavior and know it exactly in some sense so again these are approximations but they're approximations that make sense for us to try to enforce so that's a linear model if we choose to do that there's also nonlinear models and the way we want enforce a nonlinear model is through parsimony so we're going to take here here's the Dynamics maybe the system and I don't

5:37 know what f is but what I'm going to assume is that f is parsimonious in other words there's only a few terms in the right hand side of this dynamical system that allow me to represent the Dynamics most physics models are quite parsimonious if you look at what we do in fluid dynamics quantum mechanics electromagnetism there's just a few terms in our governing equations because that's the dominant Balan physics of the system and so parsimony is an excellent

6:03 thing to enforce in regularizing the model that you want out of the system the dynamical model you want out of the system okay so how do we do it we do this through the sparse identification of nonlinear Dynamics so this is a a fairly popular technique it's a basic simple ax equal Tob regression which promotes sparsity here's how it works we take a system for instance here is L Rens and I give you time series data now

6:31 this comes from x dot equals F ofx you don't know what f ofx is but I give you the time series X which means you can produce x dot so you have x dot in the language of ax equal to B you have B now the Matrix a is this Matrix here which is essentially a library of candidate right hand side terms you don't know what f ofx is but you can propose a large set of candidate models that might

7:00 be part of the right hand side and so you put them in this Library here so this is we call fet X or in the par of ax equal to B that's the Matrix a so we have a and b all we have to solve for is X and for in this case what it is is this library with loadings or in other words the weights of those liary elements is equal to the derivative and the fundamental assumption we're going

7:21 to do here is that only a few of these Library elements matter so we're going to promote sparcity here so we want to solve this ax equal to B with a penalization of something like the L1 Norm or l0o Norm okay and when you do that these dots represent the nonzero elements that are extracted out of this okay so that's the idea behind Cindy and it's a very simple architecture for learning dynamical systems you can also

7:49 learn pdes this is work with Sam Rudy here's a canonical set of PD that you might do where I just give you the data and you discover the PD Evolution from them so this is a powerful architecture which has been used broadly in the community now and it's all integrated in this package here pindy which is now in a second edition which is much more powerful than the first version of the Cindy package you can pip install

8:14 it and there's a lot of examples there to work from but this Cindy regression is what we're going to impose in the latent space of the shred model so this gets us to really being close to being able to do GoPro physics GoPro physics is just this idea that I can film physics maybe with your phone maybe with some sensors and directly from what you film extract out not only the governing equations but the right coordinate

8:43 system that is there so that's kind of the bed idea it's kind of in some sense the dream of modern physics which is I want to just take measurements of the system and I have a machine learning agent that automatically extracts from me the right variables and coordinates as as well as is the right dynamical governing equations associated with it directly from the data stream itself so that's the idea behind GoPro physics and we're going to do it here in several

9:08 examples so what you're going to see here is this architecture which is the shred architecture applied to some data that we've collected real data none of this is synthetic it's all real data and for instance the first example here that shows the architecture this is work by Mars go so Mars built this whole architecture up and what you're looking at here is snapshots of let's say sea surface sea surface temperature data so this is a data downloaded from Noah and

9:35 what we do is we have three measurements so we're going to start using the shred architecture there's three measurement locations and this is the time series of what the sensors are doing so in the shred architecture what we're doing is we're going to say I would like to reconstruct the full State space in other words the full entire temperature field of the earth of the sea surface temperatures from three measurements randomly placed and so those are where those three

10:01 measurements are placed but what I have access to is not just the measurements but to their time history so that's what the shred architector does right it's a separation of variables method that is actually leveraging toin embedding theorem okay so we're going to run it through our time pipe here so this is now going to be a GRU network not an lstm and then we're going to decode from the latent space back out to the full

10:25 State space but now there's a big imposition or regularization of the Laten space z it's learning in particular in the Z Laten space we are going to impose a Cindy loss function now there what I had shown you with the Cindy architecture which is do a sparse regression onto candidate variables that's exactly what we're going to pose in the loss function as we train this neural network overall so we have different aspects of the loss function

10:53 and this one here is just saying I'm going to impose losses on the on that latent space now when you do that for this specific data this is what you get you in fact you can pinch the lat space way down this is also something you want to do which is bring the lat space down impose the Cindy loss and we can pinch it down all the way in the lat space to three dimensions and here's the Dynamics

11:14 in three dimensions and here is the model you find so all the C- surface temperature data is basically the following model there's three modes so if I take and if I pull out what the decoder is learning here's mode one two and three let's call it associated with Z1 Z2 Z3 and here's the dynamical evolution of Z1 Z2 Z3 it's a linear model linear coupled Odes that's it so I can write down the solution for this

11:44 explicitly and then I can actually give you the solution in the future as an analytic solution I'm not solving anything anymore on the computer because it's actually a linear solve it's a linear model I have the exact solution for and what's interesting is I do have Z1 and Z2 Z3 so other words I have linear superposition in the Laten space but on the decoding side it is a nonlinear mix of these three modes that allow me to essentially reconstruct and

12:12 predict the se- surface temperature data in the future so let me just show you a roll out of this so what you're looking at here is the top row is real data so this is the truth rolling out 100 weeks right so almost almost two years into the future and what you're looking at at the bottom is just this is our analytic solution to this in the latent space that we can roll out and then just decode whenever we want to

12:37 produce what the solution looks like back in the original space so this is sort of this GoPro physics Discovery idea directly applied here where we're actually just saying look just learn the right coordinate system or embedding and it's a decoding only strategy previously Kathleen Champion Steve Brunton and myself had worked on encoding and then building a Cindy model and then decoding but this decoding strategy decoding only strategy is much more robust

13:08 requires almost very little hyperparameter tuning and and actually can work even on more more robustly on some of these data streams that are just directly from video so let me give you one more exam a couple more examples here so what you're looking at here is a video of flow around a cylinder so we pulled this off of YouTube so we wanted to go directly from video and of course many of you who've done data driven

13:32 modeling maybe have looked around flow around the cylinder typically it is a simulation so it's perfect data so this is a video and so it's the real the real physics playing out here and we went right from this physics to build a model and so what we did is we take some points Downstream as our sensor locations and map their time history to the full State space and so in doing that what we have here is for instance

13:58 here's the original video on the top here's what the real data is now and sort of in some sense in a black and white color frame and if we do 180 frames roll out into the future we you can see the difference between the real data and this Cindy shred architecture and what the Cindy shred architecture is learning is this model right here a 4x4 nonlinear system of Odes which has a limit cycle behavior and this is looking

14:24 at the Z1 Z2 plane versus Z3 Z4 plane and this is also just looking at the time series of these variables so essentially what this thing finds for you is a model like this that has a stable limit cycle which is exactly what this here is and we of course discover this straight from the video and it does an amazing job in future State predictions of the of the video itself okay one other thing to note is

14:52 obviously you want to compare it to other methodologies that have been developed for basically predicting video frame sequences and so what I'm showing you here is the original data the real data the P Cindy shred package along with some other competing methods and not only does our method give better error the training time is a fraction of many of these other routines it's a very simple lightweight neural network architecture and the regularization with Cindy is

15:23 just in some sense the right thing to do because it's capturing the physics and it works much more r rapidly and this here is actually this is basically done on video data here which is a video of a pendulum so what you're looking at here is a video of a pendulum and so this is the latent space Dynamics this is the model that's discovered you can see here quite an interesting Dynamic so is Theta dot is negative sin Theta

15:49 dominates and the dominant diffusion term or dissipation term is Theta dot squared not Theta dot which is typically what we teach undergrads to Model A damped pendulum with but you also get a little cont contribution from a Theta cubed and also you get a small contribution from sine of theta dot so this is a kind of an interesting model for that's coming directly from this real data right so we typically make approximations for these physics but here you're actually getting out

16:20 exactly what the physics is exact the Dynamics that's actually producing so these are three examples flow around the SAR see surface temperature data pendulum we have more examples and more things that we've tested on the archive and in our GitHub link but it's really a a really nice methodology because now instead of shred shred which just works really well mapping sensors to Output very robust and stable is now we're getting in the latent space interpretable models that we can use for

16:52 a variety of purposes especially for long range forecasts and rollouts this is you can create these models that are stable by Design whereas if you try to roll out an lstm or a Gro any of these other sequence models often times what happens you roll them out far enough they will blow up go to zero or go to a fixed point but this stable Dynamics long time into the future with kind of these interpretable measures so again

17:20 we're leveraging all the power of The Shred architecture but now adding this one extra feature and able to go directly to real data sets with video data and get to GoPro physics which is we can do cman shad and Cindy shred all in sort of very economical Fashions you could train this on laptops and be working on models very easily