Deep Learning of Hierarchical Multiscale Differential Equation Time Steppers

0:00 hi everyone welcome my name is ying liu today i'm gonna present you one of our recent works which is called hierarchical deep learning of multi-scale differential equation time staffers and this is a joint work with my advisors nathan curse and steve brunton so after this video if you are interested in this work you are more than welcome to check out our archive and also my github page for more details of this project okay so before entering the main context

0:32 of this paper i'm gonna i'm gonna to briefly talk about two different regimes of dynamics modeling okay so traditionally when we try to understand the dynamics what we usually do is basically say we want to derive a set of differential equations first while some first principles for example for this pendulum we're going to use like newton's law which will give us like a set of equations that

1:04 that describe the dynamics here and after we have a set of differential equations we are interested in the solutions of such equations but since most of the interesting dynamics are non-linear in nature and these equations do not admit closed form analytical solutions and that's why we often use the computers to run simulations on them and there are a bunch of timestamping schemes for us like prepared in the field of

1:36 numerical analysis to help us run simulations on these systems for example one of the most popular time stepping scheme is the fourth order rangakata scheme which has a local truncation error of o delta t to the fifth order the reason why it's called the fourth order rungcata is because like the global error it's o delta t to the fourth however here i want to point it out that for this simulations we conduct

2:09 each step we're gonna have some small error although the error would be small if you choose this delta t to be very small however these errors will inevitably accumulate as you do the simulations so this problem makes the simulation of these chaotic systems very challenging why for example this lorenz system these chaotic systems are very sensitive to the initial conditions any

2:40 discrepancies in the initial time frame will be exponentially expanded and cascaded to the future that's the problem okay now let's talk about the second regime the second regime is about like when we try to model more complex dynamics for example the human brain nowadays it's really really not easy to derive a set of governing equations by some first principles right so what we're going to do with

3:11 this system is that we're going to collect the experimental data from it we record the brain activities and the data typical typically looks like the following okay we have a bunch of recordings indexed by the superscript i here and you can see like in this data set we have n recordings and for each recording we have p snapshot or p plus one snapshot in it right x t x t plus delta t all the way down to x t

3:42 plus p delta t so these are time series our goal is to convert this scientific measurement data into some sort of model maybe not in the form of differential equations but still we hope this model could be predictive so that we can run for example some time stepping scheme on it to do the predictions so that's the setup of the second region of like the modeling the dynamics this is more like a data-driven modeling

4:14 regime so in our work there are a bunch of like methodologies for doing data-driven modeling for dynamical systems for example like the methods based on sparse regression or symbolic regression etc and etc but for this work we're going to focus on using the artificial neural network to model the dynamics okay there are a bunch of advantages and disadvantages of using neural network

4:45 against the others i would name for example the advantage of using neural network is that personally i think you can deal with the experimental data directly without computing any numerical time derivatives so it's quite robust like with respect to the data right okay so now let's talk about neural network so what we're going to do is to use the neural network to model what we call the flow map

5:16 but what is the neural network the neural network is very simple it's basically a function parametrized by some parameters theta there okay and you have the x as the input of the neural network and once you feed x to the neural network it will give you some output wise so it's that simple so what's special about this neural network function is that it is compositional in nature it is a like a sequence of operations

5:49 and for each block operation it's basically a linear operator followed by a nonlinear activation function so yeah this is what the neural network is and to use the neural network to model the measurement data or the sequence data what we're going to do is basically we take the data here the d here and we assume it comes from some autonomous system in this form which is an initial value problem and our goal is to learn

6:20 the delta t lag flow map with this neural network okay so you can see that finally once this neural network is trained the goal is that if you fit this neural network with xt the state variable at time point t is going to give you the prediction at the time point t plus delta t so that's why i put a like a hat here you can see this because this is a prediction not the real value okay okay

6:51 once these neural network are trained okay how we're gonna use it so we're gonna feed the initial snapshot x0 to the neural network and it's going to generate some predictions x delta t right and this x dot d prediction will be the input of the next run which will be fitted back to the neural network again and then it gives you the prediction at time point two delta t and then so on and so forth

7:24 you get like x2 data t here and also x3 delta t so on so forth so this seems to be very similar to what we have with the physics-based simulation models where you fit the algorithm with the initial value then with some sort of timestamping scheme you can go away like all the way to the future but the question here is that is that is that the best we can do are

7:56 there any problems with it okay so the problem is that very similar to what we have in the classical numerical simulation settings we have the error accumulations at each step okay you might say okay this is not a big problem we had that like in the numerical simulation settings why should we worry about that it's because we're now thinking about doing data driven

8:26 modeling which is completely different and we have a way to eliminate this error accumulation somehow okay and this brings us like to the key methodology of our work so this picture basically shows you like the big picture of our methodology what we're gonna do is to say instead of training one neural network to do timestamping as i previously showed

8:56 how about we you know like trim multiple neural network with different size of delta t's and then somehow combine them to do the time stepping simultaneously okay so so that's the basic idea so in this example here we have illustrated an example with three neural networks and you can see the red neural network is associated with like this big delta t1 here the time step is very large

9:26 and the yellow neural network here it's has some sort of like inter like an intermediate value of delta t2 right and this blue network there has some like small delta t here you can see that right this is very small they are associated with different time stepping size now consider once these neural networks are trained how do we do the predictions

9:58 of the like the state variable at time point t which is shown in green there how do we do the predictions there once we feed our algorithm with the initial state x0 here here okay so what we're gonna do is basically to do timestamping for four times you can see we can do one step one big step with this red network and then a step with the yellow network

10:30 and then two steps here with the blue networks and then finally we got to xt so meaning like we only do four time stepping to get there if you don't have this rad network and this yellow map network if you only have this blue network what you're going to do is basically starting from this x0 you do the timestamping again and again and again very slowly with some like very small time steps you're gonna imagine you can

11:01 imagine that if you do this the error will accumulate along the way so that once you get to xt actually your predictions you know may not be valid at all so that's the key intuition of our method we trim multiple networks and we use them simultaneously so here the red neural network can march forward very fast and this is the key to prevent the error accumulation when you do time stepping with these models

11:31 and the blue neural network is responsible for more accurate local predictions what do i mean by that so like let's consider the case where we only have the red network but we don't have the yellow and blue ones if you want to go from x0 just a little bit more like to the future maybe to here if you only have the red network the best you can do is basically to take the x0 here and also

12:01 the first step stays here and you do an interpolation to estimate what's the state variable here you basically achieve that with the you know the interpolation and that's not going to be accurate because nothing's like basically guarantee you the accuracy there so that's where the blue network gonna help the blue network gonna complement this because the blue network can beco because the blue network can just shoot you just a little bit away from

12:32 where you were at and then you can get information there so that's how we couple these two different scale networks okay so by coupling different scale networks we actually got a very like a bonus point in addition to that timestamping accuracy the bonus point is that we somehow circumvent the explode exploding vanishing grading problem when

13:03 we do the training of such networks why is that because like typically when people try to do time stepping using the neural networks they go for the recurrent neural networks and when you train the recurrent neural networks you will have an objective function like something looks like this okay this objective function it's a mean square error of your prediction and the ground truth okay and you suck you basically you need

13:35 to run over all the samples or all the recordings that's what this i from one to n is and also you want to run through all the steps j from one to p okay you can see like the j basically index like the future steps you're gonna you're gonna make if you want your single scale neural network to predict very far from the future you'd better choose a large p to trend this right for example p might

14:03 be a hundred or a thousand but this gonna give you the exploding and vanishing grading problem when you do the training because you basically what you do is to say we apply this neural network again and again and again the superscript j here means like you apply the neural network for j times when you apply this for a lot of times then when you try to use the back propagation to compute the gradient these gradients will typically vanish or

14:35 explode depending on the eigenvalues of that system okay so that that's the problem but if you are using our proposed methodology here you don't have this problem why because when you train each individual neural network you actually don't need a very large p there okay think about a problem where we want to predict from like the from t equals to zero to t equals to eight

15:06 if you only have the blue network you probably need to train a blue network with p equals to eight here so that like when you do the prediction the prediction at the a-step may still be accurate but with our network okay with this three network for example this blue one is associated with one time step the yellow one with two time step and the red one with four time steps what you need to do is basically

15:35 training these neural networks with p equals to two because this red neural network will handle like the longer scale prediction you only need to go forward for two steps to get to t equals to 8 with this red one and like the blue one only needs to focus on its own range of interest to go two steps forward that's all we need to do so yeah that's how we circumvent the exploding and vanishing gradient problem

16:06 with this architecture now let's talk about some results from this what i have just told you about are just some intuitions of that like proposed architecture and how that can be achieved these are the actual accuracy we have obtained in our paper there are a lot of information clearly in this class and let's see it like column by column so the first three columns are very simple the first column are the toy non-linear differential

16:37 equation examples that we used in our experiment and the second columns are the corresponding phase space of these systems the third column in the third column basically we visualize some specific trajectories of these systems with like blue and red curves okay and on top of that the the black dots basically represent the time stepping or or the predictions that we generated using our multi-scale

17:09 hierarchical deep learning scheme you can see basically the agree pretty well except for the lorenz system so yeah for this plot what i really want you to focus on is actually the last column so the last columns are the plots for the error curves the horizontal axis represents the simulation time and the vertical axis actually represents that the error in the logarithmic scale and different colors represent

17:39 like the different error curves for different methods for the colored ones you can refer to the top panel here for example that purple one that represents the error curve when we only use the neural network with delta t equals to one dt so what the error looks like that's the blue curve in each example and so on and so forth and most importantly the black curve is basically the coupling results

18:11 when you use all the neural network you couple them together what is the error curve you achieve by using this so you can see the black curve is well below all the single scaled error curves which means by coupling different scale neural networks you can gain like better accuracy by doing like for predicting the future another thing i want to mention is that you might ask why this error curve looks so strange you can see there's like jungling curves

18:41 there a lot of like for example this a lot of local minimum here why it the arrow curve looks this way in general it is because like when you train a neural network for example with delta t it only gives you very accurate predictions over the time point which are the interp the multiples of this delta t's okay within for example zero and delta t what we do is basically do a linear

19:12 interpolation to achieve those predicted values that's why like within zero and delta t or delta t to two delta t you can see there's like a bomb of the error curves that's why it looks this way anyways the key takeaway here is that this hierarchical scheme of using different scale networks can basically bring us the accuracy for doing the time stepping so that's pretty much for the accuracy

19:42 let's now talk about the efficiency okay so are there any benefits in terms of the computational speed by using our method it turns out yes why is that so before when we think about like doing the numerical time step stepping what we usually do is basically we start from the initial time frame here and then we simulated step by step you know a little bit and then all the way down to the future

20:14 but right now you can do the following we take the initial state variable and we fit it to the largest scale network which is represented by this red one and it will output two state variables at here and here okay so it can go very far away to the future and once you get the state variables on that two red bars you can collect them and

20:45 on the second level then you can basically take this initial condition the white one again and also the c variable at this first red bar which is in the middle and feed them to the yellow network this will separately generate like this four yellow bars here you can see that right okay so what i want to tell you that actually when you do the second level time stepping you can actually

21:17 send these two states to different computers if you are working with for example a distributed memory system right so you can do parallelization in this computation right now like the time stepping schemes are not serialized in nature anymore we can paralyze this computation naturally so that's the key takeaway and also when you finish with the second level you get those yellow bars you can gather them with the

21:47 red bar and the white bar again and send it to the like the smallest scale network which represented by this blue one and they will give you the simulation values or the time stepping values at those blue bars so after this step three you will get all the state variables at different bars with different colors here and in the end what you're going to do is basically take all the state variables and rearrange them so that they're in a

22:15 chronic order so that's how we do the simulation with this kind of data driven models and it can be paralyzed in nature what's more interesting is that the similar idea can be used to benefit the classical numerical timestamping schemes as well how we going to do that think about a scenario where we want to simulate a super large differential equation

22:46 online okay it's of high dimensional very hard to simulate but we want to do it online what i'm gonna suggest is basically saying okay first you're gonna do it a bunch of simulations offline with different initial conditions then you collect those data and use those data to train a large-scale neural network and then when you go online you want to do some real-time simulations what you're going to do is to first

23:16 feed that initial condition to the neural network first not just like run the simulations directly you feed the neural network first and this neural network gonna gives you two state variables by doing the time stepping it will give you the two type like the state variables represented by that two red bars there right which corresponds to here and there okay so these two so this step is very cheap you just use

23:48 the neural network and do two-step step like time stepping there but what you have achieved is that you break this computation into different different parts and different parts can be conducted like the simulation of these parts can be conducted independently on different computers so you can parallelize these computations using some for example the runge-kutta 45 time stepper in this example all right you maybe

24:18 you can send this white bar state variables to a computer and this red bar in the middle to another then you run like the typical simulations you you you typically run on a single computer but these computations can be paralyzed in the end you will also gonna gather these blue bars and then rearrange them so that they are in a quantico order so that's what we call the hybrid timestampers it can be used to accelerate classic numerical

24:48 simulations okay now let's see some results in terms of the efficiency okay and in this plot actually there are two plots here there are a bunch of dimensions a lot of information in this illustration as well so first we want to pay attention to the color so in both of these plots different colors represent the different non-linear system the toy examples that we have

25:20 done the experiment on for example let's focus on this hyperbolic system which represents by this like blue dots and blue stars and blue curves there okay and then let's take a look at the left plot so what the left plot tells you is that we compare all the single scale neural network time steppers against the multi-scale neural network hierarchical time stepper that we just talked about previously okay because like the circle

25:52 represents the single scale neural networks and the star represents the multi-scale neural networks you can basically see like in this plot the horizontal and the vertical axis are basically the simulation time and error they're all in logarithmic scale and in general these stars are well below these curves okay meaning our hierarchical scheme can be both accurate and efficient okay

26:23 so so that's the takeaway from this if you are like compared with all the single scale neural network the hierarchical multi-scale neural network scheme is both efficient and accurate so what about the second plot so for the second plot what we are comparing are the single time scale rangakata time steppers we run a bunch of simulations with different delta t's with the rangakata 45 that timestamping scheme

26:53 and those are represented by this like the triangles in this plot and we also use the scheme that we presented in the previous slides which is the hybrid time stepper which is basically built a neural network time stepper on top of the same runge-kutta timesteppers we want to see are there any efficiency gain from it what we see is that in general these hybrid timestampers

27:24 are on the top left you see it's on the top left over this single scale longer quad timestampers meaning they are definitely more efficient yeah we have this efficiency gain but we sacrificed a little bit on accuracy so that's the key takeaway but i want to point out that actually it's really not a fair comparison right because like for philosophical reasons we consider our differential equations to be the ground truth and in our hybrid time step probably

27:57 first train a neural network that mimic the behavior of that you know the differential equations and we then we use that neural network to generate the predictions and compare that against the ground truth which you know are the ones that produced from the differential equation so it's not surprising there's a like a deficiency in the accuracy here yeah so that's pretty much for the efficiency

28:27 and also to explore like like a broader impact of this work we also compared our proposed architecture against some state-of-the-art architectures nowadays for example the lstm network the echo state network which is more like a reservoir computing and also the clockwork rnn and we compare them against the sequence generation task so the sequence generation task is

28:58 basically we feed the data to different architectures of the network we train them and then once it is trained we feed that network with the initial snapshot of some sort of sequence and we want that network to regenerate the whole sequence that's what we mean by sequence generation and here what i'm going to show you are basically like four videos so we have a video sequence of flower blooming and we want to see like

29:30 for different architectures how they're gonna reproduce this sequence and this hist basically represents the hierarchical time stepper which is the proposed architecture in this work and you can see for this hierarchical timestampers it can almost perfectly perfectly reproduce like the sequence the video sequence but for the lstm it does like a poor job and also the same happens for the echo state network

30:00 and the clock work on the end but the key here is not really about like this competition because we really adapt like different philosophy when we try to do this kind of sequence generation in the hierarchical time stepper we're basically coupling explicitly coupling different scale neural networks but for all these others they are basically recurrent neural net network they relies on some non-trivial dynamics which might give you some sort of memory using the gating mechanism like to you

30:31 know to alleviate like the exploding vanishing gradient design pro the gradient problem so yeah this is one of the examples that we've shown in this paper and there are also a bunch of other examples including the fluid dynamics the chaotic chaos equation and also some audio sequences which you can basically see them in this like the the below this link this is like a youtube link which gives you other examples there okay so to summarize in this work

31:03 we have proposed a multi-scale hierarchical time stepper it has several advantages it is highly accurate because we reduce the error accumulation problem and it is highly efficient it can be paralyzed in nature and it is highly flexible because it can be jointly used with the numerical time steppers to promote the efficiency of classical numerical algorithms and in the end it's also easy to train when you train each individual neural network you only need

31:34 to focus on its own range of interest you don't need to trend very far from the future this will circumvent like the gradient exploding and vanishing problem yeah okay i think that's pretty much it thanks for listening