Chapter 22.3 - Foundation Models & Fine Tuning

0:08 >> Okay, I want to just talk briefly about foundation models and fine-tuning. So, what I covered last time was this transformer architecture with attention mechanisms buried inside where we would train a large language model on as much data as we possibly could could get a hold of. And often these large language models are trained on language. However, when you switch to science and engineering, you don't have language, you have numbers, right? We are quantitative in science and engineering you switch from a language-based representation often to, you know, I'm taking measurements of the system, of the pressure or the velocity, things of this nature. And of course, you can jointly train those with words as well, but a lot of the quantitative diagnostics or data we might collect would actually be numbers from an experiment. And so, the question is how do we use these? Well, what people are doing now is within this same architecture of the transformer with attention heads, they're training what are called foundation models. And what foundation models are is essentially taking as much data in an application domain as possible and starting to train that. So, in science and engineering foundation models, you just use the same architecture I showed here, but for instance, suppose I actually had massive amounts of fluid flow data in a channel. I'm just making that up.

1:37 If I had a big enough resource base to look at all the data in that, both computational real data, then what I could do is I could take the foundation I could make a foundation model by either taking a pre-trained LLM and refining it for the purpose of channel flow or taking all the channel flow data and just training it directly on the foundation model we showed you, which is learn put it through the transformer architecture with the attention mechanism inside. And this is happening across scientific domains.

2:12 People with enough resources, but it's not just resources, people with specialty data. If you're the world's experts in widgets of a certain type and you have tons of that widget data, you could train a foundation model for that widget. Replace widget with whatever you might do, right? So, that's the idea is that if I have enough data, I could use this to train a foundation model. And what we're seeing now is a foundation-alization of science and engineering. People with enough data just do this and that is how we might proceed. Of course, this has limitations.

2:49 and really what it requires is refinement tuning for the application you might have in mind and to sort of make it work better. So, I I have a picture here what people are thinking about. And this is also goes toward what are called world models, which is we have all kinds of data streams from words to videos or graphics to image and video generalization. So, this is the world of the data we might collect. But to make it really work well is this last step in LLM training and it's very relevant for scientific and engineering applications, which is what's called refinement trim tuning.

3:27 There's a lot of different ways to do refinement. Some of these they're they're listed here, right? So, you could do selective refinement. You could take of all these attention heads, it's very expensive to update the model that you say randomly select some of the attention heads and you would take with new data specific to your application, I would refine those weights in those specific to do well on the application I have. You can also do here what's called additive. So, you take your large language model and what you do is your output is concatenation of some extra model to go with your LLM.

4:08 So, you're just using this to modulate the and update the values as you go forward. And that also is a That's probably the simplest way to do this, right? You just have an additional layer at the end that modifies what you have to get closer to the solutions you want. Here's some other architectures that you might have. I'm going to just talk about this Laura one here for a moment, which is you start to merge a new model instead of concatenating, you merge this new model with your existing LLM. And so, you can update your model in in this way, okay? So, these are different architectures that people are playing with. There's so many different innovations around this that are happening in the community all the time right now because everybody is dialing in on like how do I refine refine refining and tuning my model to really make it work well? And really what it's about is this.

5:05 no matter how you do this, in some sense you're saying, "Look, I have the values, the keys, and the query matrices, and what I really need to do in refinement tuning is update them." What is the update I need in order to take the weights I have and update them? And again, here's all these kind of different strategies available. You could do masking, differential pruning, fish, auto freeze layers, child tuning. So many different options that are out there for you to improve the foundation model. And this become important for science and engineering because you would take this foundation model and then you or existing foundation model, and you could try to essentially fine-tune it for your specific application, whatever that might be.

5:51 And so again, just a chart, all kinds of ways that people are doing to update this. This is sort of in some sense current state-of-the-art research work, and there's a lot at stake here, so everybody's in this game trying to find the best refinement and tuning strategies for these LLMs. And in fact, in many ways, I would say that this is in fact with the where the real magic has been in the last couple years of taking an LLM, which is sort of a very generic, fairly works pretty effectively to now the performance levels that they've achieved in the last year or two is really coming down to this refinement tuning strategies that you would have to make this thing look it's both to reason, to give back better answers, and to be this incredibly effective agent.

6:44 So, refinement tuning and foundation models are something that everybody's really working on quite a bit these days, and it's really just essentially updating the kind of structure you have, and there's so many different ways to do it. So, you should just be aware that if you have a large language model, that is probably not enough. What you really need to do is take that next step and figure out how to do the refinement tuning because that is where you can now start making foundation models really valuable in specific domain applications, but even in just generic large language model settings, this is really what's happening to take their performance from where they were to the current modern state-of-the-art.

7:26 >>