Why Inference Matters 12x More Than Training | YC Paper Club

0:09 >> How's everyone doing today? Woo! All right, it's inference night. Come on, you got to get excited. Okay, I didn't really know how to to represent an inference in a pictorial, so I did Halloween. So, I made it spookier, as my 3-year-old would say, pookie. this is my wife over here. Okay, so why inference? Who cares about inference? well, I think we do a lot of inference investments at YC. We look at this a ton.

0:40 I used to run an AI company where I would train and we would do a lot of inference, and inference was always the bottleneck. It's so hard to do inference well at scale and without losing your shirt on cost of goods sold, a term called COGS, as some people in software are unaware what that word means. and so far, in just the last 10 years, it's produced 220 billion dollars of market cap. and we have two really cool companies Sorry, we have about three four really cool companies talking tonight.

1:13 And number eight here, actually, together, we have the head of research, Dan Foo. and so, why is inference important? Why does this really matter? the only the this compa- I like to think about it from a comparison of train flops versus inference flops, and the only company that actually publishes all their data is DeepSeek. And so, just to think about the the ratio of how much we should care about train flops versus inference flops, DeepSeek reports that it took about 2.8 million H1 H800 hours to train DeepSeek V3, and they do about 43,000 H800 hours per day, and that's like old data, right?

1:55 This is not recent data. So, it's probably much more than that. But even those numbers is about 6x more flops on inference than train for that weight file per year. And if you believe the useful life of a weight file, I don't know, say 2 years, then it should be 12x more important. And as more users of the that weight file come online as AI is proliferating, then it actually gets more and more important. And so, I think about this a lot when I'm writing papers and things like that, but the the train flops that when you're doing a selection across different models in your Nerf paper, you should be looking to minimize train flops, yes, but 12 times more you should care about inference flops.

2:40 But typically, this is like the Chinchilla paper, for example, in the x-axis for most Nerf papers, you're going to see just train flops to be ISO train flops. We should really start caring about ISO inference flops. and then, yeah, for the people that have done SAS, they don't know what COGS means, but it really matters if you are a model server. And if you just these are some numbers that came out yesterday, some people kind of were shocked that that their margins could be this good, but it implies that Deep Seek's margins are 80% and it's because they have innovated so much on the inference side.

3:19 well, first of all, why is it so complicated? It's just like model.forward. This is it's not that hard. it's actually really, really hard. And so, there's some for the non-LLM people in the in in the crowd, basically what happens is you get served some prompt, some set of of tokens, token IDs, and then you have to call prefill, and you'll run that on the the the initial prompt. If you already have some of that pre-cached, you should, you shouldn't run prefill again if you've already done it. And and and even better, if it's if it's hot in in DRAM or if it's in some Redis cache, some some SSD somewhere.

4:02 And then so make sure make make so that you do some LRU cache kind of thing like that. So this is already a big opportunity for savings. And then after that you'll have this decode loop. And if you think about we'll define these terms in a second, but like model flop utilization, like how much of the GPU you're really using for a train and prefill. And in train time you're almost always doing prefill is essentially the same thing. Your MFU might be 50%. So if Jensen Huang is guaranteeing you 2 trillion FP8 flops, you're not getting that. It's just not not even close. Maybe you get half of that.

4:37 That'd be great if you can get half of that. And in decode it's even worse. It might be like 5% or something like that. and so it's really really important that you're you're careful with these things. And so the terms of art, the things that that KPIs that Dan Fu and others are are constantly trying to optimize is time to first token. So this is how long it takes takes to get through prefill. And then time per output token or time per decode step to produce the next token. And then throughput, you want really good throughput for a GPU hour, how many tokens are you actually producing?

5:10 and then good put is like how many you're producing at some service level. And just to put example numbers, these are the the the the DC numbers that just came out I think yesterday. the throughput per GPU excluding cache hits is 60,000 tokens tokens per second on prefill. Including cache hits, they're at 96% input cache hit rate, which is really really good. that will change based on your traffic. If you're doing agents, you're going to have really good input cache, but if you have like my mom is constantly starting brand new chats with ChatGPT that have nothing to do with the past and like they're asking, "What's the weather? What's this? What's that?"

5:49 There is no no KV cache. You constantly have to run the prefill again and again and again. and so that will be much worse. But throughput per GPU, when there is cache hits, is 1.5 million tokens a second, which is really really good. and then time to first token about 400 milliseconds, time to to first token P99 about 1 second. And then throughput per GPU on decode is 14,000 tokens per second. And you can see the throughput for decode versus prefill 1.5 million to 14,000. So you're going to see decode is really really the bottleneck and that's where lots of chip innovations are coming out to try to speed that up.

6:30 and then this is actually the the most interesting one that people clued in on, which is the KV cache retention policy. And so they have petabytes of of users that are using this data for the their KV caches and they retain it for 72 hours, which is how they're achieving such high cache hit rates. some other terms of art here you'll you'll see, which is arithmetic intensity. So as I go from DRAM to SRAM, if I if I transfer, you know, 4 bytes, how much how many how much flops or how much compute can I am I doing on those 4 bytes? If I'm doing an RMN or if I'm doing diffusion, these are really high cuz I transfer an image and then I have to or even noise and then I have to keep calling the same weight file on on a same network again and again and again. I don't have to go back to DRAM unless it's a really big image and I have to tile or something like that. And so but for LLMs, the the arithmetic intensity is quite poor.

7:31 And this is the argument that Volo gave when he came up here for diffusion LLMs. and and there's a next concept model that Herman out of my lab just published as well, which is also has much higher arithmetic intensity. And the two other ones model bandwidth utilization, model flop utilization. So, when you buy an H100, they say that you get 2 trillion FP8 flops, and what are you actually going to get at test time? And it's it's usually going to be much less than that, and that's the MFU is how many you actually get. And then the and model bandwidth is how much what is peak memory bandwidth?

8:09 You have 1 terabyte of bandwidth or whatever it is, how much you're actually getting at test time on average. And then there's this idea of like of roofline analysis, which is like how good could you possibly get? So, it's going to be the min of your chips peak flops per second versus the memory bytes per second times flops. And so, there's lots of innovations people have come up with. I bucket it into these three buckets. Some people might, you know, bicker about which one would where buckets should live, but the way I think about it is there's model innovations where you do this thing called MOE where you if you have K experts, then you can amortize reduce the amount of flops at test time over all of the experts by 1 over K essentially.

8:58 There's SSMs where you don't need full attention on every single layer. Maybe you only need it every fifth layer or every 10th layer, and so you can really reduce the amount of inference needed at test time. And then multi-headed latent attention where I'm actually compressing in the sequence domain to reduce the amount of flops. And then there's I have this quantization. I think we've all seen these like even as low as 2 bytes models, and then distillation into smaller models. So, once we get some big beefy model, and we we we don't want to necessarily serve that, we can distill it into a smaller model if that's all we need.

9:37 So, those are model innovations. Then there's kernel and runtime innovations like radix attention, flash attention, page attention. These are They're kind of a lot of these are are just taking from classic OS and database theory. If you have taken intro databases, where how do you make a production database really fast? And so, there's, you know, are you cache? You can do prefix trees. You can do radix trees. That's kind of the idea of radix attention. TriDao did this GPU tiling idea called flash attention, which is huge innovation. And then page attention, which is mutually exclusive from radix attention, but is basically this idea of a virtual memory paging and buffer pool that you can that you can page into into the DRAM.

10:24 And so, and the last one is the harness innovations. So, of course, the easiest one is routing. We did this at Focal. And so, not everything should go to the big gory model. We can do much more clever things like that. Maybe you can waterfall to the or tool call to the bigger model to the bigger model to the bigger model. And that will save a lot of flops. Speculative decoding, we have a draft model that is validated. And then, my favorite one is is the recursive language model, where never put the full context into into the the KB cache. Just let the LLM decide what it needs and when. And so, you can really reduce the amount of KB cache with with those ideas. And then last one, compaction, which is a mandatory part of every every harness. And there's a really cool innovations that Sabri and Dan Simran in my lab came up with called cartridges.

11:13 >> >> if you want to learn more, I think the the two best lectures I've seen on this, Percy gives an amazing lecture on on his lecture 10 on CS 336 LLMs from scratch. And then the Ryner Pope and Dwarkesh I thought were just super super cool and super good. enough for me. You have the absolute experts in the world on this. first Dan Fu, who runs research at Together AI. and came out of my lab at Hazy.

11:43 Jeffrey Morgan, who is found founder and CEO of Alama. Joe Fiaty, who's founder and CEO of Luminol. And then Lang Shang, I think he's coming. He says he's late. He's hopefully shows up. is one of the core developers and second author of the SG Lang paper. Well, let me or please help me welcome Dan Fu. >> >> Great.

12:15 All right. thanks everyone. Thanks so much for having me. Thanks Francois for that that great you know, overview of inference. so today I'm going to talk about that whole life cycle of a token, life cycle of an of an inference request in a little bit more detail. Kind of walk you through when you send the request in and it gets to the API, makes it to a data center, all the way down to the to the root of the kernels and how to actually executes on a GPU and then how you get tokens back out. and this was a talk that I cut down from before.

12:48 So it says, you know, there's six things that we can do to make it faster. I crossed out half of them and we'll talk about three of them. some of which Francois mentioned a little bit. So, okay. so why do we care about inference right now? So we are in a really exciting moment in in inference. So open source AI has never been bigger. this this figure is a bit old, but a few months ago at some point we're doing 400 trillion tokens a month at Together.

13:18 what was interesting about that is around the same time that we put out that number, Google said, "Hey, this is how much traffic Gemini is doing." And we scrappy little startup we're doing 1/8 of total traffic that Google is doing which is pretty cool. By by by earlier this year more than half the tokens through open router were through open models and the marketing team wants me to remind you that at some point a couple weeks ago on for one day we had more tokens than open AI as well together.

13:49 And these open models are are also one of the drivers behind them is that they're a lot cheaper. So you can get nearly the same quality or in in many cases actually the same or better quality but up to 10 times cheaper per year. So there we're in this really interesting area where the open source models are really improving and there's a lot of a lot more happening every day. So in today's talk I'll talk a little bit about how they're served and then how do you get more tokens? So at together we spend a lot of time thinking about how can we create more tokens in the world and and serve serve more tokens for everyone who wants them.

14:28 A bit about together. We are an AI native cloud so the same way that AWS you know you can rent CPUs and and get some compute but you can also set up SQL databases and storage and rest APIs and all the other stuff that you need to run a web service. At together we're interested in saying hey what are all the things that you need to do to build a AI native company and so that's of course there's pre-training there's fine-tuning there's inference which we'll talk about. And then you know sandboxes for your for your agents to to try out all sorts of things and at together we're building the the cloud to do it all and and especially focused on inference.

15:12 At together so we we have a pretty correct team we're about 80 researchers bunch of PhDs we did a bunch of stuff in the past, and now we're we're focused on inference and making GPUs more performant. Okay, Francois talked about this a little bit, but one of the things that we think about is what do the workloads look like? What do you different people's workloads look like? And so we'll spend a lot of time thinking about if you're running a coding workload, what is your cache hit rate? What is the time between your turn? So when you go ask your Claude code or your open code or your cursor to say, you know, build me this thing, how long do you wait before you go and talk back to it?

15:55 How responsive does that need to be? If you're generating a lot of code, how fast does that code need to happen? And subject to certain SLAs, so like how fast you need your response, how quickly you need the whole thing to be finished, we'll go and and try to optimize these pieces. So what happens when a request actually comes in? So imagine you send in a request to some REST API, it goes through the normal parts of the internet through Cloudflare and DNS and all that stuff. Eventually it hits a series of data centers. So let's say you have thousands of GPUs, you have this thing called the KV cache that that Francois was talking about.

16:34 And then at the first thing that you do is this basic request level routing piece. So you're going to I'll go over all these momentarily, so you're going to you need to figure out which GPU replica to send your request to, then you have to take this high-level human text, turns it turn it to ones and zeros for your model to actually process, then you're going to start the big batch processing loop where there's this prefill thing and this decode thing, and then at the end you get tokens out, which you then translate back into into human text that you send back to the user.

17:06 So that first bit, request and routing. So the way to think about this is so you've got thousands of GPUs, you have all these requests coming in from from all over the world, And really it's a mix of both load balancing, but also trying to think about things like the KV cache. So, if you're familiar with the with the old world of of databases or production web services, you would just say, "Okay, which of my computers is serving too much traffic?

17:35 I'll just round robin and send it somewhere else." In in this in this language modeling era, we have something else that we need to account for, which is if you send the same request to the same GPU over and over again, you actually get better performance characteristics because you already processed let's say in the code in the code piece, that first request you send in your whole code base, it processes it all, figures everything out. But then in the subsequent requests, you've already done all that work, so you don't need to redo that. So, this routing actually has a little bit of has a little bit of an art to it, which is you have to figure out not only how do you balance the load across your data centers, across your GPUs, but also where can you send the request so that the KV cache will already be resident.

18:19 Next, there's the stage of tokenization. So, this part you probably imagine should be fairly straightforward, but there's actually a lot of stuff that happens here, too. And that's because your requests, when you use these models, come with a lot of things that we call tool calls. So, these tool calls are basically like, "Hey, I'm a I want to ask my model to do this. And by the way, you can search, you can use grep to search through my file system, or you can run this command to build the app and test everything, or you can search the internet." And the way that that gets passed in, so the model is just some abstract, you know, combination of like mathematical things. It's like a bunch of matmuls and some attention modules.

18:57 So, it actually it has no ability to search the internet or do anything. So, the way that that happens is when you send in your request, you say, "Hey, here's a list of 10 things that you can do." And then the model needs to go and say, "Hey, I want to call the internet right now, or I want to search through this. and there's actually a lot of correctness things that you need to ensure when you get there. So, I think Jeff is going to talk about this in a little bit how you manage that from the user side, but there's quite a bit of complexity just at this tokenization setting of how to of how to get this stuff to be correct and and work well.

19:33 next you have what you can think about as the core scheduler loop of your model. So, you have all these requests streaming in. They're going to wait to batch together and either because either wait for a free GP or some free cycles or wait for the previous request to finish. The choices that you make here on the scheduler will decide will determine what's your total throughput. So, are you trying to optimize for more users on on the same GPUs or are you trying to optimize for very snappy response. So, is this like if you ever use the Google search and see the Google Gemini, the very first response is always really really fast.

20:11 Then if you try to chat with it, it's kind of slow afterwards. So, that's probably very different scheduling requests, possibly even different models on the back end. Now we get to these two stages called prefill and decode. So, you can think of these these things as prefill is sort of like prompt processing. So, when you pass in your whole code base, you need to figure out exactly what's in your code base, exactly what's being asked and and understand all those pieces.

20:38 That's a very compute bound operation. So, it's going to be very heavy on your GPU and it's going to have some of these these words that Francois mentioned MFU. Your MFU is going to be very high. The arithmetic intensity is going to be very high. On the other hand, you have the second stage where the model is just generating one or a few tokens at a time to get back to you. So, one way to think about this is every time you generate one of those words, you have to load the entire model weights from GPU memory to the compute unit. So, that that's this decode stage. It's a very memory-bound operation and it's a very different type of operation. So, we spend a lot of time thinking about how can we serve these two different workloads in different ways? How do we separate them?

21:21 Disaggregate them? Are there different parallelisms or kernels or or anything that that we can use to to make this part more efficient? Finally, so now you get a bunch of ones and zeros, these token IDs that then get translated back into text and get streamed back to you. This part is is is going to be relatively simple. There's of course complexities around when do you free up your KV caches? When are you How long do you keep those around? how how long do you try to keep those tokens resident?

21:51 All right. So, and together we think about there's roughly six or so things that you can use to to to make things things faster. So, in in in some in no particular order, you have kernels. Kernels is just a fancy word for a GPU program. So, ultimately at the end of the day, when you're running the model, all these things have to actually run on the GPU, so you can spend a lot of time optimizing them, writing different attention kernels, MOE kernels, mega kernels to make this all faster. there's disaggregation choices, so the prefill and the decode are very different workloads, so you want to put them on different GPUs. How you make those choices can affect a lot of things about user experience, total throughput of your GPUs.

22:33 there's going to be parallelism, so Chimney K3, for example, is a 3 trillion parameter model in FP4 at half a byte per parameter, that's going to be 1 and 1/2 terabytes. your best B200 B your best B300 only holds 280 GB of memory, so that means, okay, you have this big model, you have to split it across GPUs. How do you split it across those GPUs? That's a parallelism question. there's routing in the sense of once your request enters the data center, which which pieces do you send it to? A lot of choices to make there.

23:04 And then the last two almost more model layer optimization. So speculative decoding. The basic idea is you have this really big model. It doesn't you don't really need it to answer you know, what is one plus one. So you always have a little model running at the same time that's trying to preempt the big model. And if you get it right, you can just get those tokens for free. And then lastly quantization which I'll talk about is so you have these big models. How many per token do you actually need to store? How many bits do you need to store these these weights?

23:38 For sake of time, I'm going to talk about kernels, parallelism, and quantization today. And go into these in a little bit more detail. So first kernel. So the place where kernels matter is at this core at these core prefill and decode parts of the computation. So this is where the the math is actually running the matmuls, the attention, and all those other pieces. So what are some things that you think about when optimizing kernels? So when you think about the model, there's some pieces that are very relatively easy to optimize. And then there's a lot of other pieces that are harder to optimize. So like all the random layer norms and RMS norms that are that are in your model, random soft maxes here and there.

24:23 Each of these individual operations has very low arithmetic intensity. So if you write a kernel for it by itself, you get a very low flop utilization on on those kernels. So that's historically one of the reasons why you can only get 50% end-to-end flop utilization during training, 5% during decode. And the way that that you would do this that you do this that we have been doing for years is you go look at your model forward pass, try to figure out every place where you can squeeze, have a team of really talented kernel engineers go and optimize this.

24:58 and for a long time this was this was the modus operandi for everyone in the industry, which is you just have a bunch of really smart people work on this really hard. at Together recently we've been thinking about how to use agents to actually optimize automatically optimize a lot of this piece. so this is an internal project called Together Compiler. We're going to be talking about it more in the next coming weeks. I think probably by the time we release we'll name it something slightly different because I had a I had a shocking realization the other day, which is that Together Compiler is not actually a compiler. but yeah, so we'll we'll have to rename it. But the basic idea is you have a target model, some target hardware, a particular amount of traffic, and you're going to use agents to go optimize this inner loop. so a couple key ideas here is that you really want this loop to be driven by real traffic because different traffic or coding workload versus chat workload is actually going to activate kernels in a very different different way. and we found it's been really important to do the full end-to-end research loop. So actually look at what is running in production to go iterate on that. So we what we started to build here is almost the beginnings of an RSI loop, recursive self-improvement loop for inference.

26:12 Okay, so so that was kernels. the the next the next thing that I'll talk about is how to optimize your parallelism scheme. So the basic idea here is you have all these weights. They're split they're too big to fit onto a single GPU, and if you can split up your model across multiple GPUs you get more compute, more bandwidth, more of all the things that we love to to to do these things. a couple basic parallelism schemes that have been around. So there's this thing called tensor parallelism, which is you just take the matrix multiply and you split it up and then you do a basic map reduce style reduction at the end. pipeline parallelism is this idea where you take your many layers and you cut it layer by layer. So, you put different layers, full layers onto GPUs. Data parallelism is another idea where you just send different sequences to different GPUs, but you keep you keep whatever parts of your model there are the same.

27:07 the the game here has always been how much memory do you have on your GPUs? How fast can you write your kernels to support a particular parallelism scheme? And and what things are available? One thing that's become quite popular recently that we've been investing a lot in and one of the core drivers behind that DeepSeek $38 per hour number that Francois threw out earlier is leveraging much larger scale up domains. So, what this means is that if you're familiar, if you try to go out and buy GPUs recently, you've heard about these Blackwell chips and these GB chips.

27:42 And the biggest difference between them is that these Blackwell chips and also all the Nvidia chips before them, you could only have eight of these these GPUs really talk to each other in a high bandwidth way. So, that basically meant that you take your model and split it up eight ways. And if you try to split it up more, if you try to split it up 16 ways, you'd go over a much slower bandwidth path and then that would make your serving just unbearably slow.

28:06 recently Nvidia has started selling building these these racks that can connect 72 GPUs in much higher in with much higher bandwidth pieces. So, what one thing that you can start to do is you can take all the experts in your model and put the experts individually onto onto different onto different GPUs. So, what this means is that you can end up starting to serve it means that you can serve a lot higher volumes on the same GPU. And the basic reason also comes back to these ideas about arithmetic intensity. Basically, if you only have one expert on your GPU serving a ton of tokens, now that looks like a big matrix multiply operation that GPUs are really good at serving again.

28:49 a couple pieces here. So, I think, you know, people have known about these ideas for a couple years now. You can go back 2 3 years and and find these demos of expert parallelism. Recently, we've figured out how to actually make them more reliable. these these GB racks are really awesome, but they do tend to have a node or GPU go out at different times. And if you have a single model replica that depends on 72 GPUs and one of them goes out, if you don't have the right remediation or and reliability pieces behind it, then what that means is a single GPU can going out can take out 72 GPUs worth of of worth of compute, which is not something that you want. So, we've been spending a lot of time figuring out how to make this reliable, resilient to failures, gracefully gracefully keep going even through these through these hardware failures.

29:41 And the last bit that I'll talk about again is is just quantization. And and the this one is going to be interesting because in some sense it's it's quite simple, you know, there are quantization algorithms that you go, there's probably hundreds or thousands of papers on it. So, you just go pick your favorite one and use it. in another sense, it's it has actually ends up being quite complicated because it it really touches the quality and the quality of experience of of your of your end user at the end of the day.

30:09 So, the the basic idea here is that your models will be trained in something like BF16. That's 16 bits per parameter, 2 bytes per parameter. 3 trillion parameter model's going to take 6 terabytes. and but the nice thing about machine learning is that the entire computation is kind of fuzzy, right? It's not exact. It's not you don't necessarily need all those bits of precision. You can still get the right answer at the end of the day.

30:38 so, for a while we've known that there are these algorithms that I can use to take those 16 bits, that be of 16, shrink it down to eight bits or four bits and still have still have decent quality. And and there's a lot of technical detail there that I'll gloss over, but the the main thing that we care about is how do we ensure the right quality and the right user experience for for for the user. So, we're we're actually we're pretty obsessed with this at together.

31:03 we So, we we know both from the system side what is the right quantization for the hardware, what is the thing that's going to get you the most throughput, the most tokens per GPU. But at the same time we know that people are using these increasingly to build companies and and run high you know, very very important things. So, what we've built at together is a sort of a a measurement quality red teaming loop. So, before we send out any model to any user, we are internally dog fooding that model for quite a bit internally. So, this does a number of things. So, it's going to catch all those tool call things that I talked about earlier cuz if suddenly your your Claude code harness can't can't search the internet, you're going to notice that as a user. but you're also going to notice if the model is not quite as smart as it should be. If it if it doesn't quite get the right scores on the benchmarks or on end-to-end evaluation. So, one of the things that we have running all the time is actually a little Flappy Bird loop where we have all our models trying to trying to code up Flappy Bird in in a rapid loop. So, if if something went wrong with the quantization, suddenly the Flappy Bird won't flap anymore and then we have to go and and figure out what went wrong and fix it.

32:16 but basically the name of the game here is of course there there's a lot of algorithmic optimization you can do, a lot of algorithmic innovation, but really what it comes down to is really rigorous evaluation loop. So, benchmarking, agentic evaluations, a lot of in-house dog fooding, etc. one success story here. So, we were pretty lucky to work with a lot of the open source model providers pretty deeply. one of and we worked with MiniMax to serve their M3 model.

32:47 and also their upcoming M3.1 model. not sure if I was supposed to say that out loud, but now now you all know. and one of the one of the interesting bits here was that MiniMax was actually having some trouble quantizing their model to NVFP4. but we were quite interested in it because we know from the kernel perspective, the serving perspective, and also general experience like this is kind of the right quantization you want to use on Nvidia GPUs. so on the left there that what color is that? That that lilac purple, that's the original model in in FP8. In the middle in this hot pink. Yeah, in the in the hot pink in the middle of that was the the the MiniMax attempt at at quantizing with their own open with their own recipe. and but with our recipes in the orange on the right here, we were actually able to match the original model performance and then we were able to serve that, get MiniMax a lot more tokens, and help them kind of close the loop from training the model to serving it profitably and and and help them, you know, recognize the revenue, get the money so that they go and go train their next model that that we'll all know and love.

33:56 so yeah, that this this was a a fun little success story where we were able to use our quantization expertise to to also help help the people who who are doing the the core model development. All right, so that that's the that's most of it for for the presentation. So just to recap, I talked a little bit about how the request flows through the system. we didn't go into too many detail details about most of it, but there's routing, you know, which replica do you send it to, which data center.

34:24 there's the tokenization, how do you make sure that the that the tool calling and all the pieces that you expect to be correct are are the core queuing and batching, scheduling work, prefill and decode often on different GPUs with different kernels, different parallelism schemes, and then finally, once you go through all that, and make the GPU really, you know, prove its worth, you get the tokens back, to do your coding and your chat, and and all the other things that that we know and love.

34:53 so with that, if, you know, just to conclude, if you found this stuff interesting, if you want to go use open models, if your CFO is beating breathing down your neck and saying, "Hey, we're spending too much money on cloud code, and maybe we should try some of these open models." one of these is is a sales QR code, and then if you if you find this stuff really exciting, you want to come work with us, the other one goes to our careers page. so yeah, thanks again so much for listening in for having me.

35:21 >> >> And I think we have a few minutes for questions. Is that right? All right. >> Hey Dan, thank you for the talk. two questions. So in the NVFP 4, I think quantization that you did, can you talk about how did you kind of like measure the quality? Was it GSM 8K or it was it like perplexity, MMLU, with respect to the MXFP FP8? And the other question is with respect to the, I think, the interconnect technology that enabled basically, Grace Blackwell class of chips to connect beyond like like a in a mesh form beyond eight eight GPUs.

36:06 >> Yeah. yeah, so, let's see. These three are kind of three, open benchmarks that are out there. I think, some of them are quite saturated. Like Amy is at 90 plus percent. really, I think the thing there is if you see one point or more of movement, then you know that something's gone wrong. Like the model's actually going to struggle with some, with some real world activity. So, we have these kind of stock standard benchmarks that we release and we talk about that you know, we also we will look at like KL divergence of the individual layers. So, how how much did the activations actually change?

36:45 And then of course we'll look at all the end-to-end agentic pipeline. So, we've built up like I think at this point hundreds of different test cases where So, every time anybody said, you know, anything went wrong with any model ever, we put that into the test case, make sure make sure that it's okay. And then we also have these agentic evaluation loops like Flappy Bird. We actually put it put it back into the together to together compiler project so that the model is optimizing its own kernels. And if we see that the model for some reason can't optimize its kernels anymore, then we're like, "Hmm, okay, we we should take a look at that."

37:17 The second question was a little bit about Oh, yeah, the the the NVLink. So, I'm personally not a hardware person, so I've never built or designed hardware. The closest I've come is I bought one of those Hugging Face micro ducts. So, that that's my closest you know, you know, play play with hardware recently. Basically, if you look at one of these racks, it's really crazy. So, each tray of four GPUs has these really thick cables that go to a really big switch.

37:46 And then all the trays are connected that way. The you know, the bits go through that really thick cable, go through the switch, and then you can do your all reduces and and all the pieces there. I I I'm not super familiar with you know, what was the jump from the 8 GPU technology to the 72. But I'm you know, I love it and I using it. >> So, if I would like you know, batteries they be versus one. So, you have your phone and you don't know what kind of interview that kind of like you know, what they test against and kind of really like with the this possible that I'm on.

38:18 >> Yeah, yeah. So, yeah, we we do we we do the public benchmarks, we do our own agent loops and and everything in between. Yeah. Yeah. Sure if there's a microphone for Yep. >> Okay. Thank you. >> my question is about this graph where you found a recipe to do better quantization. So, I'm curious what was the difference? What did you were you training from scratch or were you running multiple pre-training epochs after introducing quantization? Or was it that there was no training done on the hot pink one and just like straight up just, you know, transforming the numbers without doing any training fine-tuning on it?

39:09 >> Yeah, so so yeah, so for context there there's a lot of quantization recipes that out there that require you to train the model after it's done. we don't like using them because we find that unless you have the entire original training data set. So, unless you're the model provider yourself. >> You don't have the >> Yeah, we Yeah, we we don't do that because we find that when you do that you can actually like the model will forget things. So, this is entirely a training-free approach. it's it's a combination of stuff that is, I think, publicly out there. So, you know, there's some calibration data involved.

39:46 there's this paper called scale search that that we put out a few months ago that that's used pretty pretty deeply, but it's a it's a mixture of being very careful with your with your recipes and then also trying to and and just being very careful and and putting together all the different pieces. So, like there's a hundred training-free quantization methods out there. It's being very careful about which ones you use and and being very sedulous about how you evaluate them.

40:14 >> Thank you. >> Yeah. >> All right, I think we got to move on here. >> last question for you actually would be >> Yes. >> Are there any model types that you're excited about? I know there's lots to talk about SSMs, lots to talk about DLLMs, there's all these other types of architectures that might dramatically make your life a lot easier specifically on the decode side. Is there any ones that you're still holding out hope for?

40:37 >> yeah, I think the funny thing about all the all the new architectures is that so during my PhD I worked on some of these like SSMs and things and and the thing that we always got was like nobody's ever going to use this thing. It's it's a dead end. Transformers are going to take over the world. It transform AGI all that stuff. a few years later and all those technologies are used in in all of the big models. So Kimmy K3 has this thing called KDA which is basically an SSM layer. it might be slightly controversial for me to say that, but it's an SSM linear attention style layer. so it's being used. It's it's in the one of the best open models.

41:14 there's sparse attention being used which for the longest time nobody was using. we use the diffusion language models the DLLMs in the in all the speculators. That's the best thing to do now to to do it. So what's interesting is that all these things are being used. my students have recently been very interested in looped models. So the idea here is you know, in this in this basic computational graph you send the token back through the same layers a bunch of times and then the idea is you can ramp up the training flops without increasing your your weights. and we thought, "Oh, this is a weird idea. Nobody's going to use it."

41:48 and then it turns out they at least that OpenAI and Anthropic were both using it already. so yeah, it's the it's it's a really exciting time where like if you if you put out a good idea right now about architectures and and go through the leg work of showing that it works, it'll show up in GPT-6 or GPT-7. and it the the the time to feedback and and time to impact is is so so quick now.

42:11 >> Thank you, Dan. >> Cool. >> >> Hi everybody. I'm Jeff. I'm the co-founder and CEO of Ollama and zooming out a little bit, I wanted to talk about how all open model tokens, not all open model tokens are the same. And how to think about in a world where there can be, you know, I think we saw a list on François's deck of just how many inference companies just the top set are exist today. in the foreseeable future, there'll be hundreds if not thousands of potential inference providers. And how to think about measuring quality across these inference providers if you're a builder and you're trying to like build an application on top of open models.

42:57 a little bit of background. I have Ollama here just finished dinner with all of us. Ollama's an open source project that originally released in 2023 for running open models locally. Back in 2023, the most popular open models like Llama, for example, were generally smaller models that you could run on either like a single H100 or on a laptop or like a small server. and the core goal of the project has always been to give an OpenAI and Anthropic level developer experience but for open models. since then it's grown a lot and because the models got bigger and bigger and bigger, really culminating with the start of this year when Kimmy and Jill M and MiniMax all launched these massive 500 billion to 1 trillion parameter models, we launched Ollama's Cloud to really provide the same experience that that developers got locally but in the cloud and that's really exploded over the course of this year and it's, you know, pretty much becoming one of the largest if not the largest independent cloud service for accessing open models across the US and Europe.

44:03 and so all of this is important because we spend a lot of time thinking about how to bring a great developer experience across many different models and over many different providers. And so, if you think about the architecture of our cloud service, unlike kind of a traditional inference provider, we really focus on a few things, like getting the latest models, making sure they're really fast, making sure you get a reliable experience. A lot of these inner workings A lot of this is in that router box that Dan was just talking about. and we do this for a bunch of these reasons. I think, you know, reliability and having, you know, early access to models is probably the most important because different providers excel at serving different models. But in the end, we have this architecture which is very similar to a lot of customers that use Ollama. Ollama's deployed by a huge chunk of the Fortune 500. And our cloud service and what From talking to teams that are moving from closed models to open models due to cost or control or other reasons, is that they've all kind of have the similar problem. And so, I wanted to spend some time talking about about this tonight. but the main reasons to route are really that you kind of get in the In the ideal case, the best of all the worlds by combining a bunch of these providers together. And the definition of of an inference provider is kind of loose, like this is even a problem that you might have if you are an inference provider and you have different deployments on different hardware in different regions from different teams that are using different inference engines, for example.

45:32 and so, when you adopt an architecture like this, when you're thinking about, you know, pooling a bunch of inference providers together, what are some of the challenges? Well, the biggest one in summary is that not everyone serves models the same way. In fact, that's a good thing because different teams have excel at different performance optimizations. Some folks run on specific kinds of hardware. I think there's like a new inference provider that just runs on like Mac minis all over the world.

46:00 and so, you know, if you think about it, ultimately customers care about a few different dimensions, right? Cost, intelligence, speed, and then ultimately like region and privacy. and the challenge with routing across a bunch of providers is obviously that, one bad apple can ruin the bunch. And that if you have a single provider that's underperforming on one of these four, criteria, that you, you know, you might have a customer experience that's kind of the least common denominator. and this can be for a variety of reasons. one of them could be the model precision. You know, you may have a provider that has over quantized a model or is running it on faulty hardware or they're using a different version of the weights.

46:45 tool calling is a big area of concern like Dan was talking about. Doing tool calling effectively is a really hard. while it seems simple because it's like parsing JSON and strings, there's actually a lot that can go wrong from recovering from tool calling failures, and you know, enforcing specific schemas for different tool, tool, parameters and names, and also just making sure that the parsing's actually happening correctly at the tokenizer level. different providers may have different sampling techniques, and then inference engines have such depth to them and to what you can edit.

47:20 So, whether it's custom kernels or KB cast precision, a lot of these, you know, changes can really be only seen in like really bespoke situations as well, like when you read a really long context. and there's even silly things that can happen like, you know, models being served with the wrong name. you think you're accessing one model, it's actually a different model, but it's hard to know. they're all a friendly assistant trying to help you. And so, a lot of the things can go wrong in the serving stack and and it can really affect, you know, if you're trying to serve a large customer base across different regions and providers, it can really all it can be tricky to track that down. And so, I spent some time ahead of this talk trying to think through what we do and also what have what has the industry done for this?

47:59 and it ends up it's not a new problem, and there's been a lot of research done about well, how big of a problem is this? And the crazy thing is that, you know, we all talk about the open model to closed model gap, you know, how far behind are the open models? It ends up that for open model serving, just open models, that the gap between the worst provider for a model and the best provider is actually bigger than the gap between closed and open models. And so it's a really big problem, and you know, I the experience you can have with a great provider, you know, like Together, for example, who are good friends and partners with, can be a lot different than an endpoint served by, you know, a a different provider. And it's really hard to know this right up front because, you know, we come from a world of where Open AI and Anthropic really kind of kickstarted a lot of the kind of you know, ultimately the demand for tokens. And in those worlds you you kind of expect a consistent experience no matter how you're accessing those models. but for open models that's definitely not the case. In fact, it can even be a bigger problem than the difference in performance you'd get between the closed and open models.

49:10 and the good news is it's not a new problem. There's actually been a lot of great work done around this. there's a paper in 2024 that tested, you know, 31 different Llama endpoints, and very similarly they found that you know, some models behaved were even closer to like the performance of GPT 3.5 than another Llama model of the same that were supposed to be exactly the same way. It's and so I had a good time digging through this and thinking through well, if you're going to use open models, how do you really make sure you're getting a good quality experience?

49:40 the last thing I'll say before jumping into like some of the techniques and ideas is that this problem gets significantly worse with coding agent and agent workloads because they're such long context or long horizon workloads. they generally involve multiple calls back-to-back-to-back-to-back. So, if you have a model that's serving an inaccurate call an inaccurate response, and then you're compounding that a pretty like you know, modest coding task, for example, that might take 15 steps could be up to, you know, 79% wrong if like 10% if you have a 10% chance deviation from kind of what the model was output was designed to be. which can obviously ruin your experience if you're ultimately getting it a an error rate of 80%. So, it's a big problem that gets worse as our agent workloads get harder and harder.

50:30 so, how do you then go about monitoring and introspecting and looking at inference endpoints? you know, obviously, you can't read through all the tokens like Alamas attempting to do here. That'd be really difficult, especially when you're doing, you know, what is it? 1.6 million tokens per GPU and and preprocessing. and well, the first thing you can obviously do is run evals, which is kind of like the brute force way and probably the most expensive way to do it. I mean, artificial analysis actually does this on a regular basis where they'll just run benchmarks a few times over the different providers and kind of rank them. I think here this is a small picture, but you can see here that by and large for the Kimik 3, most providers get it pretty much right.

51:13 There's some on the right that aren't quite there. there's one that's not doing too well, but generally, they're all in in good shape. The challenge with this is that ultimately, you know, it'll catch like bad quantization, the challenge and like a variety of issues cuz you're ultimately testing how the model was benchmarked in the first place. The hard part is that it's really expensive and it takes a long time to run these, so you can't do it on an ongoing basis.

51:39 the other thing we saw rise at the end of last year was, you know, the folks at Moonshot at one point were kind of ultimately fed up about how different providers were doing tool calling with the Gemini models, which are incredible family models. And so they released the Gemini K2 tool verifier, which was and a scoreboard for which providers did best. And this was awesome because it caused everyone to really kind of enter a race to like make sure that the tool calls were accurate. Gemini models has since done something similar. So if you're building a model, this is a great way to make sure that people actually run it correctly. Give them the tooling to test it. Another more interesting way that these are some of the papers that I've read about and we're thinking about integrating into how we do Olama's routing and model aggregation is a distribution test where you basically compare the outputs of a reference weight and a provider API and then really just look at the distribution between the two and understand you know how much of a gap there is. This is great because it's generally pretty cheap unlike the the benchmarks which could be very expensive. It's maybe like a dollar per test.

52:47 The hard part is that you know while it it might only be may not catch some of the more intricate problems such as like poor quantization. It doesn't really tell you why something's going wrong whereas if you have a more comprehensive eval suite you can like kind of pick out a specific test and then you really do need the reference model running. So which can be an extra expense you have to you have to go under. And take under.

53:13 Another one is to use the same called log probs which is a way to see kind of ultimately the probability distribution of the different token IDs that for future tokens. And this is great because it's generally a pretty cheap call. You're just getting like a single you know the next token log probs and comparing it with a previous baseline. So you're kind of measuring the drift from that provider which then can tell you if things have changed. For example, did they deploy it somewhere else? Did they upgrade their inference engine?

53:45 this is really cheap. Again, it's potentially even much cheaper than the previous test. it's only like a single token request basically. and so, you know, the example in the paper found a ton of changes across different endpoints. And, you know, they were able to kind of identify these silent changes to the serving stack, which is really interesting if you think about it. Everyone's serving stack is evolving, it's getting patched, they're deploying new versions of popular inference engines like VLM or SG Lang.

54:12 so, that can be a good way to monitor it over time to make sure your customers are getting a consistent experience. Log Probs isn't on a lot of the APIs these days, so that's one downside. And you're really only detecting drift from the past versus, you know, and you could potentially compare it to another reference model as well. But in this approach, you're really only getting drift from the past. So, if you're starting from a place that wasn't serving it correctly, this isn't super helpful.

54:38 another thing a lot of popular routers do, and this is what we're we're kind of at this stage right now, is looking through just, you know, without any kind of data retention, can you just introspect how many tool calls are passing, how many of them are incorrect, are there tool calling errors that are occurring, how's the shape of the request changing over time? different providers may serve different input and output sorry, different output lengths, and that could really give you an indication of whether there's, you know, a common issue. I've seen from like the DLM or DeepSeek serving have run on problems where the model gets into an infinite loop. and so, that's where you can really catch these problems ahead of time just by looking at the shape of the request without even having to look at the data.

55:19 and then you can keep a running score, and then ultimately, you know, keep an eye on that and monitor it request to request. Open Router does something similar to this, like for example with like the exacto model approach that they're taking. and so, zooming out, you know, one thing you know, we're always working backwards from how do you get to an experience with open models like you get with open AI and Anthropic? You think about those, you know, they run on different hardware like TPUs. I think there was a a bug last year at one point with Anthropic where like a tiny bug in a TPU kernel caused like potentially like a whole retrospective of an outage just in terms of quality. And so they have such a strong care for the serving quality. How do we get there but in the open model ecosystem? And so I think one way is that the labs can share a much bigger stack for how to run their model.

56:07 Kind of most commonly people will put their weights on the internet on Hugging Face for example or upload them to a llama and then they'll say here's some example code. But it's actually not what they're serving in their own inference stack. It's generally like another applied AI team that generates that so that the community can use it at large. DeepSeek yesterday launched DeepInfer which is their own inference stack. So now you can take the DeepSeek weights and run it hopefully at $38 a GPU hour on your own GPUs. And what's great about that is you can use a consistent basic basically a consistent stack. So if you're just a new upstart inference provider you can set that up and have some confidence that you're serving it with high quality.

56:45 A lot of labs are quantizing the model. and they're working with you know pros like you know Dan his team to quantize the model before launch so that the model that they're serving on their API is the same as what an open model that you would take and serve are. This is very different than what happened originally which is quantization tend to be done by post you know post model release by folks who don't have the training data or we're ultimately doing it to you know fit it onto like a MacBook or something like that.

57:15 This is much more increasingly common that the labs will actually publish their own quantized versions of the model. you can access models through lower level APIs. You know one thing we realized is that some of the tool calling you can actually pull up into the harness and get a lot of accuracy by doing that consistently across providers versus, you know, delegating that to the providers. And same with reasoning, extraction and thinking. I think there's a big opportunity for startups to start monitoring LLM quality. This is something that's super important if you think about LLMs being part of a critical loop of a application. I don't know if this exists yet, but it's a really interesting idea.

57:53 And then one of my favorites was this paper that talked about how, you know, every modern Nvidia GPU actually comes with a trusted execution environment that can attest to the weights that are running and the software that's serving the model. And this can be a way to get a fingerprint telling you that it's being served in a correct way. I think this is pretty advanced, but I thought it was a really interesting idea. yeah, that's a little bit about how to think about, you know, once there's inference everywhere and it's open models are being served everywhere, how to really measure quality and like make sure that, you're getting the best, tokens out of these awesome open models that are coming out. thanks for having me.

58:31 >> >> Hey Jeff, thanks for the talk. My question is, regarding the pre-fill admission, you know, interval criteria for quality. How do you like decide between tool calls and like large context, running context, special especially specifically for multi-turn agent like workloads that span like long sessions. like which would be better? I I am assuming it's the longer context pre-fill admission, interval.

59:02 >> Yeah, I'm not sure exactly, if I have the best background to answer that. You're asking about the admission. >> Yeah, yeah, yeah. The the interleave admission between like, let's say certain decode after like after say say 48 or 64 decode steps, you kind of take every API endpoint tool call like admission versus, from the running context you take the the pre-fill admission. So, there was there was a there was a slide in your in your deck which kind of like, you know, went over that.

59:34 >> Oh, yeah. This is just saying, you know, you can basically send a single token prompt and then measure the log probs that are coming out the other side and then really measure that with like a a baseline. So, you can do that as the over time. And so, it's not really specific to like a specific context length. I think one thing that you could do though is do this with a consistently long like a longer and longer prompt and then start to measure what is the deviation in log probs as the prompt gets longer.

60:03 Cuz what we've seen with a lot of providers, I mean, first off, some providers will just cap the context window for a model. So, that's like one silly functional thing to look at. But as you get especially past the 200, 000 token range in context, be able to test that the next token accuracy is consistent with what the other with, you know, the reference model providers or what it was previously returning. I think it's the general approach here.

60:27 I'm not super familiar with each of these papers, but that was I thought this was a pretty elegant approach to do it that doesn't require running an entire eval, for example. >> Okay, thanks. >> What do you think about quantizers then just hill climbing the evals? So, if there's an increasing number of like open source tests to ensure model quality that they just train on that and we're not really preserving the model quality. Like, how do we get around that?

60:52 >> Yeah, it's a great question. I've heard a lot about this how, you know, increasingly in the in in, you know, when there's a large marketplace especially of inference providers fighting over who can be the cheapest token, you're incentivized to match some traffic pattern, some use case. I think what we've done a lot with customers is that they, you know, they come to us and they say, "Hey, we've got this this specific use case and we can design with them a you know, a set of tests that you can run consistently or a set of evals." And that's really hard for a a single provider to game because it's not public. It's specific to that customer's data or their traffic. And so, that can be a way where just because they don't have access to it or it's more bespoke, that you can kind of catch a provider for specific customer not delivering, you know, at at the performance that it should. and that's good because generally that's where we hear issues on quality is because some a customer would say something. So, it kind of completes the loop.

61:43 >> Thank you. >> I have a quick question. Over here, Jeff. >> Oh, hey. >> you know, like the when I first tried a llama, it was a local model. And then we have like gotten way off of local models. And then as of today with Flint and Richard Socher's recursive launch, we're back to local models. What do you kind of see for the future of a llama? Do you think it's going to be Yeah, how how do you think this impacts a llama?

62:10 >> Yeah, you know, with the launch of a lot of these local models, there's a lot of ways to run models locally as well. Different inference engines, you know, there's the MLX framework that powers a llama now on Apple silicon. >> Calibri, too. I don't know if you have seen Calibri. It does Qwen 3.5 MoE locally. And it's like serves it from disk. It's pretty cool. >> Yeah, there's so many of these approaches. And I think all the problems we talked about here are also problems for local inference model serving.

62:37 and that can be desktop, it can be like desk side, like, you know, the DJX Spark. they're you know, if you have the new M5, we just got ours today, M5 Ultra. or, you know, data center serving for a lot of businesses who do have GPUs. I think it's a problem, too. It's not just a problem with the provider, but it's also what's the technology stack you're using to serve the model. And you need to be able to test that reliably. and so, I think two things. One, with local I mean, local this new model is incredible.

63:04 We're super excited to be launch partners with recursive. the Flint model. but I do think, you know, it is a challenge whether it's a cloud provider serving a serving you tokens or it's an inference engine you're running yourself serving tokens that you're updating every week. This is a common problem. The reality with open models is it's really the wild west, which is an incredible thing cuz these models can be customized and changed and quantized. I think there's like 20 different quantizations online for almost every model that you can just pick up, you know, off of a llama or hugging face and download. The challenge really is how do you benchmark these? But then, if you're serving it to customers live, you need to wait to do you know, you need a lighter test than benchmarks. So, I think it's a very equal equally important problem locally.

63:47 >> Awesome. Thank you, Jeff. Future of Ollama, I think is probably people running a mix of local and cloud models. We've seen this new model is really good at coding locally. And so, I think, you know, the number of tasks you can now run you know, frankly, a lot of the best coding problems were only solvable or hardest coding tasks were only solvable by cloud models that really grew popular this year. Now, with these ultra-capable local models that can run on, you know, this MacBook, we're going to see, you know, this routing problem go from just being from a bunch of providers on the cloud to like local and cloud routing. So, anyhow, awesome.

64:24 Well, thank you guys so much. >> Thank you. >> >> All right. Yeah, my name is Joe. I'm one of the co-founders and CEO of a company called Luminal here. And we are a compiler company, which is kind of unusual, but throughout this talk, I think you'll see why. we really see compilers as being a very critical part of the stack and kind of defining your conversion function between GPU flops or GPU hours and tokens. And how good of a compiler you have greatly determines whether you lose money, whether you make money, or whether you make a lot of money. And so, we make a compiler that's based on a very wacky technology called equality saturation. It's a fairly new technique.

65:07 We are the largest, I think, industrial application of equality saturation, and we think it's the correct approach for efficient compilation of models to a bunch of different types of hardware. And so, we really think, you know, in a large part of why Nvidia is worth $4 trillion comes from the fact that CUDA is a very well-supported software ecosystem. it's very it's a very difficult problem to map models down to complex hardware. It's a very, you know, I think fairly unsolved problem at this point. And our solution is to leverage the better lesson. So, we do a lot of compile time search. this lets us basically not maintain a lot of different complex rewrite rules or complex heuristics that ordinary compilers maintain. and it lets us not maintain a big kernel library that we need to constantly modify every time a new model comes out.

66:01 And so, we're able to take a very, very small shared compiler core, build backends for different types of inference ASICs in the data center, and then run a whole lot of inference on this across both GPUs from Nvidia and AMD, but also TPUs and Trainiums, and a whole host of different inference ASICs as well. And so, you know, at any given point, we're running, I believe, something like 80 or 90 trillion tokens a month. we're not together, but we're also eight people. So, we'll get there someday.

66:34 So, this talk, yeah, it's going to be largely about how we think about compilers, kind of a unique approach versus your typical MLIR-based compiler stack, and why this tends to work quite well for ML applications for large language models and diffusion models and bio models and all the ones that at least make money today. specifically, how we use the egg log engine, which powers our equality saturation, what what equality saturation really even means to begin with. and then how we structure our IRs in order to target a bunch of different hardware and fully utilize all of that hardware.

67:11 and then how we scale up search. And so, you know, really what we do is we we kind of take a look at all of the neural networks out there, all the deep learning applications, and we realized they're not that complex. they are effectively dense linear algebra and not even that many different possible operations of dense linear algebra. And so, we take models and we actually boil them down to something like 14 or 15 primitive ops. So, those operations are like add, multiply, sum, things like that. And it turns out you can decompose the whole wide world of PyTorch ops into this very simple op set and not lose any sort of information.

67:54 And so, you know, we're not Turing complete, but I like to say we're PyTorch complete, meaning if you can represent your model in PyTorch, we can ingest it and compile it to all of our supported backends. So, we take a matmul, for instance. We don't even have a matmul operation. we take a matmul and we decompose that into a broadcasted multiply and then a sum reduction. so, yeah, this this really helps us to bottleneck the complexity of new models in PyTorch, of which, you know, PyTorch has something like 2,000 operations, and and, you know, new models are very complex, and hardware implements tons of different operations as well. And so, doing this complex mapping, you don't really want to go all-to-all. You want to have some sort of a bottleneck in between with which you can map all your PyTorch operations down to and then map back down into hardware operations. And so, this RISC-style primitive op set is that bottleneck for us.

68:50 So, there's a kind of term in the compiler world called super optimization. it's from, you know, the '90s. it it didn't really ever take off except for in compiler research. but basically the concept is that you should try to search through every possible form of a given program and use some cost function to determine which is best. And that way you don't need to, as the compiler engineer, know how to lower some given program onto a hard on a piece of hardware ahead of time.

69:20 Because there is an exponentially, you know, large number of possible ways you can do that lowering. And so, what you often see from these companies is they hire compiler teams and their compiler team head count scales exponentially with how many different applications they want to target. So, you end up with a compiler team of 500 or 1,000 people. and even then, a very imperfect compiler. And so, once you try to scale up the number of models or types of model architectures you want to support and the types of back ends and the types of data types you want to support, you get this big multiplicative explosion.

69:54 So, what you really want to do is you want to write these very small atomic rewrite rules that kind of specialize or that kind of specify different equivalencies. And then you want to let the search go and figure out what to apply when and and not not tell it to apply these these different rewrite rules. So, what equality saturation lets us do is do exactly that. Write these very small equality rewrite rules, build up the search space, and then go and walk through that search space.

70:22 So, we have a compute graph as our IR structure. So, it's a DAG. You can picture just like, you know, your your add operations, your multiplies, and you get your dependencies between those ops. And we use this this concept called an E-graph. So, it takes a graph and it applies these non-destructive rewrite rules, which means that it an ordinary rewrite rule does a pattern match onto a graph and then replaces what it matched with with something else on the right-hand side of the rule. And so, that effectively destroys the information that it matched on. And so, when you apply that rule, you better be sure that that was the right rule to apply because there's really no going back. And what we're allowed to do is we're allowed to say, "Well, we can apply a rule and we're just going to add the right-hand side information to this e-graph, but this e-graph kind of maintains the both sides of the equation, what came before and what comes after."

71:13 And so, we basically apply all of our rewrite rules in every possible order until they can add no more information, and that effectively has grown the e-graph as much as it can go. We call that saturated. And once that saturated, then we actually go and we extract a whole bunch of different concrete graphs, which represent programs. We compile those all down to executables on the hardware, and then we actually go and run those on the hardware, and we basically time how long it takes to run on the hardware, and we go and try to minimize that time. It's very straight-forward and simple process.

71:47 But this basically is a kind of visual representation of what an e-graph actually looks like. So, you kind of have if you look on the very left-hand side there, you have a graph that represents the equation A * 2 / 2. It's kind of difficult to to imagine, but basically the top node there is the root of your expression, and then you kind of go down, and one of the arrows coming out of the root is the one side of the division, and the other is the other side. And so, you have A * 2 / 2. And what we're doing here is we're actually applying different rules, and these applications grow the graph.

72:22 So, they specify that, "Okay, well, actually A * 2 is equivalent to A bit shifted over by 1." and then you can keep applying these different rules, and eventually what you get is the fully saturated graph on the right-hand side of the picture there. And what that does actually is it has expanded the dotted line that contains both that division, which we started out as our root. That that box there contains A as well. And so, what we've actually worked out is that a * 2 / 2 is equivalent to just a.

72:56 So, it's kind of a way that we're able to simplify down this expression without having known, you know, do a do a heuristic-based rewrite rule that has looked for that pattern ahead of time. And so, this is kind of what our rewrite rules look like. They're very, very straightforward and simple. They're written in a Lisp syntax. The reason for that is because the people who made the engine that we use inside our compiler really like Lisp. And so, we have to write all our rules in Lisp.

73:24 But, they they kind of look like that. So, you kind of see the first rule there is is basic commutativity. So, you know, x + y = y + x. Notably, that is not true with floating-points. But, that, you know, the associativity and commutativity are kind of tricky when you get to IEEE floating-points, but that's another topic. and then, you know, what you actually have in the bottom there is a technique called constant folding where if you have like a + b and at compile time you know what a + b are, you can actually just go and add them and then remove that entire addition from your program. You don't need to run that at runtime because you can do that at compile time, so to speak.

74:03 So, you know, we've kind of talked about the our core IR set. And that core IR, that that very simplified primitive IR set is shared across all of our back-ends. But, we want to actually go and compile to a bunch of different back-ends. So, we want to be able to compile to TPUs and training ins and GPUs. And so, how do we actually go and specify each chip's implemented operations? So, what a hardware does is it kind of brings a set of implemented operations. And these are generally quite low-level. So, they could be in a kind of varying levels, but they could be high-level, they could be low-level. But, on a GPU, you can imagine that, you know, a tensor core MMA is one of the operations that you can specify. And that kind of reads inputs from SRAM and writes outputs to SRAM.

74:51 and so you we go in and we can well characterize a given new hardware backend that we want to target and then we go and specify, "Okay, what do each of these operations actually mean in terms of equating them to operations that our compiler already understands." So, we can write those little rewrite rules to do that equality and that kind of propagates out to the rest of the equalities that we've already specified in the compiler. So, bringing in a new hardware backend is a fairly low-effort thing even if it looks very different from the hardware backends we already support.

75:24 So, yeah, this all gets around the the typical need for a very massive compiler team and a lot of complex rewrites. So, and you know, hardware fundamentally is hierarchical, meaning you have a GPU and that GPU, you know, you're actually possibly programming a single thread, you may be programming a warp, you may be programming a thread block, which is essentially programming a full CUDA core or GPU core and SM. or you may be programming an entire GPU or you may even be programming an entire node with an operation. And so, we want to be able to represent that explicitly. So, our IR is fundamentally multi-level. we can model all of those different levels explicitly and that lets us bring in operations that may come from the community, from vendor libraries that are implemented at these different levels. And so, equality saturation lets us kind of bridge all of these different semantics in this like sane way and just specify what is equal to what and then have the engine swap out and do all this do all this search.

76:25 And so, this is kind of what it looks when you swap out a bunch of different implemented operations. And so, you know, you can have a a cuBLAS host operation that basically is called from the CPU and it does a full matrix multiply. and cuBLAS is a matmul library authored by Nvidia and it implements very good matmuls. you can have a thread lo- thread block level operations like as implemented in CUTLASS, which is another Nvidia library, or CuBLAS DX, which is also an Nvidia library, or ThunderKit, which is from Hazy Research.

77:01 and those are thread block level operations, meaning they actually program one GPU core at a given time. And our compiler, because it can reason about these different levels of operations, we can actually go and take subgraphs of lower-level operations and kind of combine them all into a big mega kernel. And so, we can have at any given point a forward pass that is implemented in both kernels and mega kernels. And so, that's great because as much as you know, mega kernels are neat and cool and everyone likes to talk about them, they don't always they aren't always the optimal solution, especially when you're dealing with very large models.

77:37 And so, you know, the final kind of problem here is once you've defined your search space, how do you actually search it? Meaning, what do you know? How do you say program A is better than program B, so to speak? And this is kind of a problem where you you need a cost model, essentially. You need some fitness function to evaluate a program and say this is more expensive than that. And having a good cost model, especially of complex hardware, you know, take a a modern GPU or something, is extraordinarily hard. This is like many, many, many PhDs worth of work to in order to, you know, take a GPU program and say like how long do you actually take to run without actually running it. And so, we as a little little company here can't possibly do accurate cost models for every type of hardware we want to target. And so, you know, in the spirit of the bitter lesson here, the best cost model is no cost model and we should actually just go and run these things on the real hardware that we're going to run on and see how long they take. And it's, you know, it's kind of a dumb solution, but it's a very very effective solution. Where you know, we can have different different moves in our search space that we may not as programmers think are beneficial, but when we actually go and run them on the GPU, they end up being very beneficial.

78:52 So yeah, we can do we don't actually have to run everything in our search space. There are techniques where you can prune the search space because you know some roofline evaluation of a given subprogram and then you can kind of early prune it because you know that it will be in the best case slower than something that you've already measured that is equivalent to that. So we do we do some early pruning here and there, but you know, largely we try to measure everything on real hardware.

79:18 And oh yeah, this is a little video. This is just what it looks like when it's code generating a kernel. There's not a whole lot of information you can get from this video, but it's just like kind of neat to see. But yeah, and obviously obligatory plug. We have done the very rare feat of solving compiler company monetization. There's a very long list of dead compiler companies in the graveyard because they haven't figured out how to make money from a compiler.

79:43 We've finally closed the loop on profitability. But yeah, like I said, we run a cloud powered by a bunch of different heterogeneous hardware. We're at yeah, something like 80 trillion tokens a month, many many millions of images a month, and one of the benefits of being a compiler like general purpose compilers that we can not just run the most popular models, but we can run all these sorts of wacky models. So we run protein folding models, bio models, image models, all this all this all this stuff.

80:10 We're growing really really rapidly. So we're eight people. We're not staying that way. So we are we are aggressively hiring. And then we use wacky compiler tech to commoditize wacky hardware. And yeah, we think the future of compute compute is commoditized. So come visit us at lumina.com. >> >> Hey, I really like your risk style approach.

80:41 I was wondering how much of you adapted to machine learning workloads because you don't need for instance you mentioned that you also network about associativity in IEEE 754. But ML doesn't care. You can even do different sequences hardware style. How much of that do you bake into your does that enter into the cost model or how you model the language itself? >> So, you know, associativity commutativity they do matter so much as they do affect numerical accuracy.

81:13 Meaning if you reduce your you know if you're doing a sum reduction and you're doing a tree sum reduction that is going to give you a different answer than a purely sequential sum reduction. And you know to the point about provider model model quality if you screw up numerical accuracy that really does impact your your output token quality. So, we do we want to be very very explicit in how we model numerical accuracy. and then in in the reasons why some of those rewrite rules still are advantageous though is that often times you have hardware that implements different functionality for input A versus input B. So, you know, for instance Nvidia tensor cores may only actually consume input A from SRAM and input B from you know team M or something like that some other memory space.

82:00 >> Hey, this is Yeah. So, I didn't quite understand when you say why can't we can just run it on the actual hardware because that seems impossible if you're talking about the entire model. And if you are only doing that for a particular kernel, I don't know how you model contention and stuff like that. so, that's part one and part two you meaningfully expanded search space a much larger search space, right? >> Mhm. >> Yep.

82:31 >> Even like a stupid compiler like a TRT, for example, is extremely slow, even with this heuristic. So, how does this search scale at all given like you have basically expanded it to be an order of magnitude bigger from what I can tell? >> Yeah, so two different questions, but the first answer would be that the is it still fairly quick to run a full model pass like on the order of I guess it depends what you consider to be slow, but we can run a forward pass, you know, and do a profiling iteration on the order of like under 10 milliseconds. So, you can still do, you know, a thousand odd profile runs in 1 second. And so, that's still like reasonably quick.

83:06 for the second question, yes, the search space does get quite large and we use pretty standard search algorithms to walk through that. So, we use like a lot of like genetic algorithms. I think there's a lot of room for innovation here where we can use something like like MuZero to do more like a learned guided search, but so far we haven't had to do that. We end up only searching like 1% of 1% of the space, but and people are I think surprised to hear that, but we never are targeting a global optima. We are targeting a local optima that's like 99% as good as a global optima. And so, that that ends up being much easier to find. And then in terms of compile time, that is an explicit trade-off we've made that customers in inference don't really care about compile time if it gets them a faster run time. And so, because you compile once and then you serve across, you know, millions of GPUs, if it gets you 1% or 5% or 10% better run time performance, you won't you don't care if you compile for a full day sometimes. And so, the way we generally model this is we let the user set a time budget or we set a time budget. So, you can compile for an hour, you can compile for 10 hours.

84:11 Eventually, you'll hit some point of diminishing returns, but that's kind of how we we think about this. And scaling it oftentimes it scales pretty well. Again, you hit some diminishing returns after after quite a while, but it ends it ends up working fairly well. >> Thanks. I think the the kicking this profiling makes makes a lot of sense. Do you guys use actual profiles from inference as well to sort of guide the search or this is just pure on in form search in some sense?

84:36 >> So we try to use as close to real inputs as possible. Unfortunately, we run a cloud so we know the exact kind of shapes that customers care about. So we you know in terms of tensor shapes, we know roughly exactly what they're going to feed us at run time. and you know data distributions and such like that. And so we don't need to check for correctness because we already know that you know unless our rewrite rules are wrong that we're going to get the correct answer out but as long as we try to represent the shapes to be this like industry relevant tensor shapes, we end up with pretty accurate timing.

85:09 >> hi, thank you for sharing and another compiler need to compare the different archi different model architecture to different hardware and I feel like it's pretty hard to do the debugging especially we need to identify the which layer that we're trying to lowering the ops health issue. So I'm curious about how Luminos solve this issue. Thank you. >> Yeah, I mean general debugging is always very difficult. we use a lot of tools that PyTorch provides where they're they're you know really nice about tagging the high level ops and then we kind of track that metadata as it flows through our compiler. we use a lot of like the standard PyTorch profiler to go and and work out you know bottlenecks and issues. And then what a quality saturation does, what egglog specifically does for us quite nicely is it lets us say, "Okay, we got an output program out and this program is wrong.

85:58 Give me a proof that connects this output program back to our input program which basically says like here are the chain of rewrites that were applied to go from the input to the output." And then we can narrow down pretty quickly like which which rule is invalid effectively. >> Yeah, I actually like this solution a lot because usually unsupported operations we just kick them back to the CPU very elegantly. but because there is no free lunch, what kind of latency, because now we're expanding an operation when we usually want to fuse operations at the compiler level, what kind of latency impact do you usually see?

86:42 >> So, we absolutely do fusion. we expand them to get to this like core set of primitive IR, but then when we actually go and do lowerings, the the compiler searches through various forms of lowerings, and what we end up with is, yeah, quite fused ops as well. And so, we generally try as a, you know, roofline heuristic, like you can see, okay, if this kernel is is moving in order of magnitude more data back and forth from DRAM than this other kernel, then like, yeah, probably that's bad. so we end up with quite fused operations. It's just that in the mid-level it kind of expands those out, and then our, you know, kind of search has to go and find those patterns and refuse them back down again.

87:20 >> Okay. >> All right, thank you. >> So, one of the comments I picked up on, during, the the slides was that the language you work with is actually not Turing complete. and that's kind of interesting because that's that says it's not going to be subject to like Rice's theorem and things like this. And so, you can reason about, things like what will the program actually do without without even running it, at least for for some properties. And so, I'm really curious, cuz I don't think you mentioned this by accident, so I'm very curious where in the stack do you exploit this, and how?

88:00 >> So, very good point. It is not subject to Rice's theorem, and this was kind of an early realization we had because this is one of the reasons super optimization failed in the '90s is that they tried to apply this to general-purpose programming languages, meaning like C. And C is Turing complete, and there's a near infinite number of possible compilations. And and so, you know, you face a lot of issues when you try to start reasoning about Turing complete programs. We compile dense linear algebra programs, and that's a very very narrowed space versus like all possible programs. So, you know, again, PyTorch complete but not Turing complete. one of the exploitations we can do is we can know exactly, you know, analytically how much byte movement we do between different memory spaces. And that's how we're able to analytically derive this like roofline of performance without actually running a model. And so, we actually do a lot of early pruning, and we end up not needing to run very many profiling iterations relative to the full size of the search space.

89:02 >> Okay. Thank you. >> >> Yeah. So, the solution space that you're searching for, does it also include optimizations like quantization, speculative decoding, and parallelisms? >> I'm sorry. I didn't I didn't hear you. >> Sorry. so, I guess that the like the solution space that you're searching inside is at a kernel level. Does that include other techniques like speculative speculative decoding the the >> So, generally no. We try to basically just handle the forward pass. So, the you know, you you specify like a forward pass graph, and we compile that. And so, you know, it's end-to-end within the span of a forward pass. And then we live inside of, you know, the great inference engines of vLLM and SGLang.

89:52 And so, the great solutions they already provide for speculative decoding, we're able to, you know, just go and directly piggyback on them. And our, you know, thesis is that the large part of the alpha lives in being able to handle very complex, you know, parallelism topologies and being able to do this, you know, handle the complexity of the forward pass and then leave the rest of the innovation out to, you know, VL and SG length. So, we don't really search that level.

90:20 >> Cool. Thank you so much, Joe. >> Thank you. >> >> hi everyone. I'm Leon Shaw and I am president of SG length and ready start. And I'm very honored to be invited here. I thought this is a like paper reading club. Yeah. it's like a previous talking a lot of technologies rather than a single paper. I actually prepared maybe for a 50% of the contents talking about the SG length's original paper and other 15% talking about our open source and our current optimizations, but I I just added a few more pages, but yeah, I can it's not in the slides. Yeah, I can start. So, So, that's the story begins when the agent workload. Actually, the story begins in the in the late 2023. At that time, we only have the Llama. we all have like for all the open source models. So, at that case, people just a guess what the agent workloads looks like.

91:37 So, at that time, we imagine that there will be some tool calling and we imagine there are some parallel sub-agents for for your like something like the future cloud code to do that. So, imagine when you fork a agent fork a sub-agent in the cloud code, there will be a lot of a context. They're sharing the a big context in in the same KV pool. And when you are the monitor to query each the query result and each the prompt you entered is a very short prompt that it will be get preview into the age into the engine multiple times and get out of of the engine multiple times. So, at that case we notify like in that at that time there is there is no way engine can serve the prefix reusing efficiently. So, at that time we brought the Radix attention which can automatically reuse the share prefix and compute with the new with the new suffix.

92:42 So, the idea is we propose the Radix attention. all the agents' contacts they were managed inside the engine system in the memory system just like a tree. So, we call it like a Radix tree in the Astra inner implementation. so, here is the illustration for how we how we do the memory management. It's just like the figure says. we we share the setup like there is a long system prompt and one do the authentication, one do the review and there two agents they they share the same prompt to do the same with the with the same within the same context and do the different things.

93:27 Then one agent they may want to give some results and because the memory is limited, so why the suffix of one agent that got evicted. So, that is basically how we do the memory management. It's it's it's really a Radix tree in our management system and all our all our core designs including our scheduling are around this topic. So, you can see when when the so, the left side is how we do the do the prefix protection for the agents.

94:09 there is a when So, so we do the eviction. the eviction policy is we we evict the unused leaves when the memory is needed. And there will be a lot of algorithms the scheduling eviction algorithm in our in our scheduling base to to fit your workload and fit your needs. And the right describes why we need this. for example, we have the prompts or the agents. They are sending to the engine interve- in interleavingly.

94:45 So, if you can reorder the prompt, then they can be batched in cache-aware order then to reuse the context A rather than interleave the A and B then there may be the risk of eviction. So, that is the story how Islam becomes the a serving engine. The original paper is an idea to build a red tree inside the memory management system. And the second page is we make this universal and general LM inference engine in the middle 2024.

95:27 so, we at the mid-2024, we published the 0.2 version. and with the with supporting to at that time, the Llama 8B2 Llama 4.405B for both online and offline serving. >> >> So, you may you may notice that for for all the for all the batching operations, for all the memory management eviction matching. you have to order all the waiting request, get the priorities for them.

96:00 so, and we know that the GPU their inference time is getting faster and faster. but but at the meanwhile, the the CPU they didn't their efficiency didn't change. So, for this time we can say we can we can notice that if we use the vanilla scheduling strategy, then there will be a preparation for the batch one, then run the batch one, then get all data, then preparation for the batch two. The so so, when the GPU is quite slow, it's okay. the GPU bubbles is acceptable, normally with within 10%, but when the GPU they are getting faster and faster, the CPU the the bubbles the GPU bubbles by the GPUs bottleneck has been has become noticeable. So, for this case, we we designed our scheduling system with the overlap scheduling, which means we we delay all the device to host the synchronization, like we last it for another round, to make sure that there's no blocking operations in the critical path.

97:17 at the meantime, we do the preparation in advance. because the GPU they are running, there's nothing blocked by the I Once we do the delay operations, then there is no there is nothing there is nothing blocking the next batch launch. So, we just launch the batch two when the batch one is running. So, we still using this, and a lot of new engines in including the yesterday DeepSeek DeepSeek they they published their DeepInfra. They also used a the the zero over the zero over life scheduling to support their internal inference engine.

98:03 in the Hopper in the Hopper days it's only in 1.5 1.1 times of throughput comparing to the vanilla scheduling, but in the Blackwell days it's it's maybe sometimes it can achieve two times in the GB settings. So, the agent and the cache story continues because our GPU memory is always limited.

98:36 when we are doing serving with long contexts and the people there like sending their prompts, sending their next round of response within few minutes or few hours, even few days, you have to find some place to restore the their previous prefix somewhere. so, that is why we bring the high cache which is also pretty standard technology used in many engines. we connect the register I mentioned before with the GPU memory, host memory, and the internal storage we call it L1, L2, and L3 cache. every token they can be exist existing on the L1 cache or L2 cache or L3 cache. all these they can managed by managed by some specific policy to increase the cache hit rate and also the the loading time from the L2 cache to the L1 cache or from the L3 cache to the L2 cache.

99:43 >> >> in our experiment, this can lower the TRFT by 56%. >> >> So, as as as nowadays, like the the traditional transformer doesn't make sense for for long context and the for efficient deployment. So, there will be a more and more things things about the some some some innovative designs about the transformer architectures. For example, there is a sliding window.

100:21 As we all know that a lot of open source and maybe closed source model they use the open window they use the sliding window to to to serve their models. And also there is a current recurrent states like a member state, KDH states. They were they were just putting the information into the fixed shape of state like a token by token. So, at that time you may think that like it's very easy we just need to add different layer at different kind of memory pool to do all the hybrid memory pool, hybrid storage, hybrid computation. but that is not so so so trivial because in that case you have to consider how to build the the cache share strategies.

101:10 for example, if you want to if you want to cache hit, let's see this figure. If you want to cache hit at the position 10, let's starting from one. Like you have to match the full KV tokens for the full attention pool. And also if the in in the example, if the if the sliding window size is four, you have to contain at least the four tokens starting reversibly from 10 to seven. if the if the if the retained token is only reaches eight, then there there will be a no cache hit because it's new it's need a full attention calculation for the sliding window.

101:55 And also, if you want to start share the prefix from the position 10, you have to have the member state KDS state exactly at the position 10 to share the prefix. So, so if we manage all the memory pools separately, then there will be a huge challenge to how to do the efficient cache sharing. Imagine when you do the cache sharing from 95% to 99%, you reduce the the expense by by five times, not by only 5% because it's you use four times less computation for all the inference for all the inference workload. so, so previously, we used different radix tree like a which was implemented separately stand-alone without consideration of these different low-level memory pool.

102:54 And for that case, this will bring a lot of compatibility and flexibility issues. So, also, when you try to try to protect your history, try to lock the prefix node, you also have to do a lot of coordination across the full layers, across the sliding window layers, and across the recurrent layers. each it seems for each component, they have to reach the some some specific protocols to to to coordinate to to satisfy an external policy. So, that is a No. Oh, that is a why we why we brought the unified radix tree.

103:43 in the newly designed unified radix tree, we didn't we didn't treat all the components like as a as a standalone components. We used We used a semantic token ideas tree to which was based on the full token. Because based on full token, that is an interesting because what matter how the architecture changes, there will always be the full tokens. Yeah, at least in nowadays. So, we built the semantic token ideas radix tree for the full tokens and all the other tokens, they were based on the based on the internal maintained component. and the the the the unified radix tree will be a controller and a coordinator for each component to make sure we can achieve the best cache matching efficiency. For this, and also for each component, they were designed to fit the unified radix tree.

104:48 They have their own rules. They have their own matching eviction and the memory rules. yeah. So, this will bring all the all the corner cases together to avoid any memory leakage or memory waste in the in the virus workloads of edge edge analytic inference. Also, there's something I didn't post on the slides, but you may notice that if we have the the three types of heavy posts like the full token, the window token, and the recurrent states like a states. But, you can you can imagine like when the context they varies, they they varies across different users or different workloads or even different time zones.

105:38 for example, a user A, they only use context around 4K. user B, they use context maybe 128. And another user, they use context for 1 million tokens always. So, at that case, they always require the full token pool to feed their context. But, the sliding window, as for some purpose, it only feed the last 128 tokens for their working load. So, that is a ratio difference because because you cannot reserve the token pool by the fixed ratio. If you have to design an engine to fix all the different ratios of diff- all the different to workload context, then it may may you have to use a lot of different types. But, if you only have one engine or limited resources or you want to see you want to serve a lot of different workloads, then you must have you must to have some some unified way to to to to manage all the all the low-level memory pools. So, so besides the unified Redix tree, we also proposed our unified memory pools to serve all the different workloads. before, we just pre-allocated all the workloads at the at the precomputed ratio and and the fixed the size is. And but for now, we only allocate one one one like a like a like a one pile of a physical memory, then we we do not set the limit for each kind of memory pools. Like we didn't we do not set the limit for the 4K. We we did we do not set the limit for the window KW.

107:24 We just treat them as all of the combination of the smallest units like bytes. They were just keep growing from the left side of the physical memory and the from the right side of the physical memory they were like growing like according to our usage. At that time when the when the contest getting like a pretty long we may allocate more memory for the for the full token pool. When the contest is pretty short we just allocate more memory for the recurrence state and the window KW. So but for most of the kernels and back-end implementations all the all the all the operation they need to be continuously in physical memory.

108:08 If we do the memory allocation the memory eviction before then there will be some there will be some holes in the in the in the GPU in the GPU's physical memory. So the holes like the empty the the idle available ones they were they cannot be they cannot be reused because because once you have to like like you may hit the the boundary with two different memory pools. So at that case we introduce one of our operation called compaction. We just reorganize all the memory fragments in the physical memory space. Then we pack them with low latency memory copy kernels. Then we remapping all the virtual token I am virtual token IDs. It's not token ID it's actually the the memory page IDs. Then we just do the remapping and all the virtual page IDs they do not to be changed. We just change the the low level mapping itself.

109:17 >> >> That bring much more flexibility for the whole serving workload. Yeah, and also start from serving, we also introduce the Miles inference inside the training loop. XLA and Miles we we are both contributed mainly contributed by Red Hat Arc. And XLA is responsible for the rollouts and is the perfect back end for the for for for Miles RL training.

109:48 And Miles supported the fully asynchronous execution, keep the generation and the and and the training moving. Yeah, as actually I took more things about the paper. Yeah, we start from the Red Hat casual attention and we grow up, we do the open source inference engine, and we do the Miles, even XLA diffusion. We do a lot of other things to boost the open source AI infra at that time. So So at the last page, like we will be holding the XLA summit in the next months. And there will be a lot of feature speakers. we'll be there. You can scan this QR code to register for this. Yeah, and that is my talk and any Q&As.

110:39 >> >> All right, thank you so much. All right.