Weight Folding, CUDA Streams, and the Bug That Made My Model Speak Backwards — Filip Makraduli

0:12 Hello everyone. Thank you for coming and I'll start the talk now. So This talk is around a paper that I did which is very simple. The proposition is very clear. It's basically two lines of algebra that make the RMS norm layer in transformers cheaper, quicker, and kind of improve improve it as like a layer in the transformer architecture.

0:53 Similar to how layer norm once used to be the standard and then it was substituted by RMS norm. This follows along this way of thinking. And I got the chance to kind of meet some people from the open source world and I co-authored this paper together with Niels Graf who was the kind of the creator of this. And the work follows from there.

1:24 So this is presented on archive. You can have a look, read it, test it out. There is a repo as well. And the concept the let's say the idea and the way of thinking it's easiest to explain with maybe flash attention. So in a similar way of how flash attention kind of weights until there's a multiplication and tries to limit this communications between memory so that the whole process is faster. This is kind of a similar thought along those lines and it does certain improvements that make the RMS norm process much quicker and in in effect improve the whole transformer.

2:16 And one question is okay, why RMS norm since that layer does almost none of the math? And that's true. So the share of the kind of math portion if you look at it is quite small. However, the clock time or wall time as they say is quite big and for example in one decode step. So right when like inference is performed the RMS norm can be started like 33 times. Of course, it depends on the model and so on. In the paper, you have the specific models and how this was tested.

2:55 and the question is how this can be improved and how this wait for the matrix multiplication can be kind of avoided. And the reason why this is slow is because the GPUs are not slow or bad at math, but they're bad at everything else around the actual math. So that means starting the work, the actual work. So for example, starting the process as it happens in some of the experiments 33 times that takes a long time and for example, fusing each normalization into the matrix multiplication can help avoid this. Also doing weight folding can help in kind of moving data between memory and that's a process that's also slow for GPUs.

3:55 And also waiting. so, for example, deferring the division that's done in the RMS norm layer is also a way to avoid this waiting step. So, basically, what this paper does is it improves all these three aspects by doing a few algebraic tricks in the way RMS norm is computed. That's it. And math-wise, these are the tricks.

4:25 it's mainly around the first two propositions. One is weightless normalization. you can see that here. and deferred normalization, so that's the second one. And now in more newer architectures, there is a situation where RMS can be kind of can appear twice. for example, in Gemma 4, this happens. So, canceling the pre-normalization also works. and all of this is algebraically proven in the paper.

4:58 And the first proposition is this where kind of the the gain and the weight fold folded to one matrix W, you can see here with an asterisk. And that is computed offline, similar to how maybe in flash attention, you compute some stuff on the side so that there is no communication between memory all the time. So, this is one step that's kind of done, this weight folding. And the other step is deferring the scalar the scalar divide of the matmul so that they can be done in parallel. So, in a normal case, you would have to compute once, then wait, and compute again. In this case, the idea is to kind of split this so that it can be parallelized.

5:48 And the third one, which is kind of a version of this is that there is kind of if there are two because this is scale invariant, one of them can be dropped and this still works. And this is applicable to newer models that can have this architecture and implementation. So, in order to make this happen in real life, especially this proposition number two, so for this one, for example, it's easy.

6:20 There is a repo called Transformer Tricks. You can just apply this to any model and it works. But in order to do this, there is some kernel work. So, it's not as straightforward to do. So, in order for me to do that, I was implementing this and I came out with this experiment once. So, it looks okay in general, where it's like, "Okay, the prompt is the Transformer architecture revolutionally revolutionized NLP because and then there is some kind of expected output."

6:54 But in the output I got, I saw this repetition and one-step lag, as you can see here, the word because appears again. And there was something happening with the GPU streams and I was trying to figure out what was happening. And I was getting this one-step lag and kind of outputs that were from the past in a way. and in debugging all of this, I realized that in the process of building something like this, so as I explained the proposition two or deferring these two operations, in CUDA, you can do two things. You can do like tensor cores that do one part of the matrix multiplication and you can do CUDA cores that kind of run stuff like element-wise operations, reductions, square roots, and so on. So, the idea was to do this in parallel and get the benefit of what I was explaining in the paper to actually test out this concept.

7:58 So, this is how it was supposed to look like. So, there is if you do things sequentially, there is this idle waiting time when you when the vector unit computes the RMS and scaling, and then there is a matrix multiplication. So, the idea was okay, with flash norm, which is the technique in the paper, you're supposed to do those both in parallel. So, the matrix unit computes the matmul and the vector unit computes the RMS. So, in that way you save time. However, you cannot just do this in Python, you have to go a bit lower.

8:31 And I did that with CUDA code like this. And this looked in general okay at my at that time. However, I realized that I did something slightly wrong. And that thing was that the join in the end, where you're supposed to join the two streams, was implicit in my case. And when I tested this out, the unit tests worked, the quality seemed similar, like perplexity testing, and so on, because it's just like similar generation, but over long generation, I was able to see this problem. So, I had no idea what this was.

9:15 And the reason was that when I was doing this implicit join, basically, one of the streams hadn't finished the work, so I got race conditions that kind of read the past from the unfinished matrix multiplication. So, the idea that I had to fix this was around the fact that I had to be explicit about the join and wait until one of the operations is finished so that I'm certain that when I join I'm not reading from the past.

9:48 So, that was the realization in this exploration of CUDA streams. And this is how I had things done. So, the join was implicit. So, the post scale read like an old buffer value. And how this is fixed is with this where basically you need to mark the end of the matrix multiplication, then mark the end of the RMS, and then post scale wait for the first stream and then wait for the second stream.

10:24 And that fixed the bug and made kind of the paper work and the model speak forwards instead of backwards. And that was the cool maybe academic perspective, but I also wanted to try things, right? Deploy this, test it out, see how I can make it work in maybe a more production setting. And you can also read the paper and see all the tests. some are done most are done around llama models, but like this works for other architectures as well.

10:57 so, what you can do for this specific paper is for example, the weight folding that I explained the pre-position one, you can just do it with some code in the repo that's like flash you say flashify and it does that. However, with this second thing that I mentioned, you need to do a bit of kernel work if you want to do that like I explained in my example. And these are some results that are based on llama models and there are different kind of details that you can have a look at as well as well. Like what happens if you do only the third normalization, what happens if you do a full fused kernel. so there are a lot of experiments of going lower here to test all the propositions, and this have been our results in different, let's say, levels of scrutiny and detail.

11:48 But even the simple one with like weight folding shows some improvement. And this also works with like the day-to-day tools that you use in the models. It's not like you have to reinvent the wheel or, you know, do things from scratch. So it works with torch compile because the it's kind of like a new checkpoint, and that's it. Flash attention does similar tricks at a different layer, and also it works with quantized models. So it's totally cool to actually apply this, and you can get a model that has this cool new normalization layer.

12:27 And where you can get this details and code to actually run this is this transformer tricks repo. So it has different algebraic tricks like I explained, as well as this paper that I mentioned. And also there is the GitHub not the GitHub, but the Hugging Face model repo where I've done this with some models, and you can have a Hugging Face link to that model and test it out. and what you also can do with this Hugging Face models is to deploy them in production. And so when I was thinking about doing this, I realized that, okay, now that, let's say, the science is done and there is a link to a Hugging Face model, Superlinked's inference engine was a cool way to actually deploy any Hugging Face model, and we've done this at hackathons where people would bring like a custom Hugging Face model or checkpoint that they have with their fine-tuned stuff, and you can test out like even if you have some version of this algebraic tricks that you want to improve a model and test on test your own research ideas, you can actually try that out and have a deployed version on a cluster of this model and not have to worry about this glue code around deploying models.

13:49 So, that's pretty cool. And the the point is that if you have the full cluster open source and the model inference open source, you can actually test out this kind of maybe more novel research ideas where if you want to do kernel manipulation or flash norm and things like that, it's much more difficult to do that do this at a rented endpoint where you don't own the inference. It's you want something that's portable and flexible to actually allow you to do this stuff, but it's also production ready enough so that you can test things out at scale.

14:29 And you can, for example, use site to combine this with other models like, as you can see in the top left, there is you can have this flashified models with different other models to do agentic tasks if you want and kind of do that end-to-end bigger use case. And the way site works is this production cluster helps you deploy the models, so you can have a look at site's repo as well for more details on this.

14:58 and also there is a smarter queuing mechanism that helps you, especially if you work with smaller models cuz when doing the flash norm stuff, I worked with like smaller llama models and also with small agents from hugging face. So, having a way to deploy smaller models that can also work on like the same GPU so that you don't have to spend your money on GPU cost, but actually kind of switch models around, especially smaller models. It was quite useful.

15:30 And you can also control the model configs through an API as well as the cluster, which is also pretty convenient without having like an infra guy supporting you in your open source research. So, that's cool as well. and you own your cloud, which is useful if you want open weights, open models, open source. And there's also like a catalog that Sci has of different models, not just the ones I mentioned, but you can have a look. There's also re-ranking embedding models if you're building something along those lines.

16:04 And with that I'm kind of finishing this story of my research journey where I co-authored this paper, around the technique that improves the transformer, but also found a way kind of to bring this to, let's say, production and test it out and find a way to play around with this open source models. And feel free to contact me on LinkedIn, maybe if you have any questions or contributions. A lot of this stuff that I've mentioned, like some of them are PRs on like vLLM or on Hugging Face. You might find them all around. You can also see the check out the paper. That's the archive link that you have there. and you also have the Sci repo and my LinkedIn.

16:47 so thank you very much for attending. >> >> And you can catch me for questions. We'll be here, close by. >>