What's New in Inference Engineering — Philip Kiely, Baseten

0:12 I am here to talk about what is new in inference engineering. So hi I'm Philip and I'm here because I wrote a book. this is my third year at the AI engineer worlds fair. This is my favorite conference in the entire world. It's the highlight of the calendar every single year. I really got my start as a speaker and as an engineer here in 2024. I came back in 2025 and did a bunch of stuff. I'm here again. I love it here and I'm very thankful to the organizers for always having me. I wrote this book called Inference Engineering. We published it three or four months ago and I've just been overwhelmed by the response. we've done more than I yeah I published it in on February 23rd.

0:56 at this point we've done more than 11,000 paper copies. We're coming up on 30,000 digital copies and 24 million people around the world or 24 million Twitter accounts. So we'll see how many people that actually is have seen something out of inference engineering. And with all of this, you know, great reception, there has been one question that people have been asking me. Why in the world would you do this? Like why would you write a book about something that's changing so fast? Well, you know, I believe that a lot of the principles of inference engineering at this point have been pretty solidified and there's a lot that we can, you know, learn and and kind of repeat over generation and generation of model. But today I'm here to talk about what's new in inference engineering. This is the first public addendum of all new information since the book came out. We are going to review the inference engineering principles a little bit and then we're going to talk about all the stuff that's happened since February 23rd of 2026 in the inference world.

1:59 We're going to talk about what happened to Turboquant. Talk a little bit about KV compaction. We're going to spend a lot of time on deflash and some other new exciting things in speculative decoding. And then I'm going to do a little bit of prognosticating, a little bit of forecasting of what I think is going to happen in influence coming up here and what I'm excited about. You know, hopefully being able to talk about next time you guys see me up here. Cool.

2:25 So, let's get started. So, you know, one thing I've been identifying now out of tons and tons of conversations with people about influence is a handful of of shared principles. And one of the big ones I was on this podcast the other day with so we were talking about inference and you know there's two types of inference engineering that have really emerged. There's local inference where the overwhelming strategy is just get it working on whatever hardware you have by squishing the model with quantization, distillation, pruning, however you can, you know, splitting it across whatever GPUs you happen to have in your house.

3:05 And first you get it working and then you make it less dumb. you take away whatever you know catastrophic issues all of this compression of the model has created and you try and get it back to that baseline intelligence running at a batch size of one and then there's there's my world which is the batch size and data center world where it's get it working you know just day zero get the build of VLM up get it working and then make it less slow do stuff like KVAware routing speculation disagregation and you know with within these two worlds I think that we a lot to learn from each other. I am in this talk going to be focused on advances in data center oriented inference engineering because that's what I know but there's a lot of really cool stuff happening in the local world as well.

3:52 So in the book in inference engineering I generally assume that the weights are a finished product and I do think that the handoff from training to inference is an important one to keep in mind and it's a good way of kind of delimiting the space. However, what I've found more and more recently is that many optimizations for inference come from a dedicated training process. And so the lines between training and inference are getting blurriier and blurriier. And that's an interesting thing to keep in mind. We're seeing this cycle where you get faster inference, which gives you more data, which you use to train a better model, which gives you faster inference, which gives you more data, and you just kind of like keep doing that until you're super rich. so with with training for inference we have a bunch of new techniques to talk about across what I like to call the big three. So we're going to talk about some news in quantization, some news in caching, specifically the the KV cache mechanism, and some news in speculation because these are, you know, there's a lot of other stuff in the world of inference, including some stuff I'm going to talk about at the end. But when it comes to the practical day-to-day of how do I make X model faster usually these are the three techniques that people are reaching for.

5:10 So first thing I publish a book it's February I'm feeling awesome about myself. I'm like wow everything you need to know about influence in one place. And then we we had some news in the quantization world. So just as a quick review, quantization is when we use a smaller, less precise number format in order to save ourselves on bandwidth, save ourselves on compute, make TTFT better, make TPS better. it's usually kind of hardware specific, gives you cost savings, but potentially degrades model quality a little bit.

5:46 And by the way, if you want to hear my whole rant about quantization, I did a talk at AI Engineer Miami last month about how quantization is is not necessarily as evil as it sounds and that there's many things you can do to preserve quality through that process. So I was feeling good about my treatment of quantization and then 20 million people saw Toboquant and in fact it like made the memory stock macro dip for for a minute just because everyone was like oh memory is going to be so much more efficient now like we don't need any more flash memory which which was wrong but anyway it was this new quantization approach that was popularized in March of this year that uses polar coordinates for quantization and allows you to quantize the KV cache down to four bits. And it was like super hot and I was like, "Oh man, like there's this whole thing that that I left out." And like what what is this going to look like?

6:44 And so our team did a bunch of research on this. this is if you know shout out Waterlue intern on Twitter. Ali from our model performance team. I'm not sure if he's still an intern actually. but yeah, he is from Waterlue. anyway, so he he wrote this great piece about the math behind turboquant. And basically the benefit you get out of turboquant is that you get to represent the KV cache with four bits instead of eight bits. You save half the room and half the band and and you get effectively double the bandwidth when you're moving KV cache around in your system memory. But the drawback is pretty big for turboquant. Turns out that you need to do additional computation in the forward pass to account for this during decode and it cuts TPS by more than half and that's just an unacceptable trade-off for a lot of the production use cases. So we we took a good hard look at toocquant but but are not using it for you know any any of these real workloads. We're we're still on the traditional NVFP4 quantization. That said it actually is a great technique for the local inference folks. So if you are running a model especially a long context language model on your local computer on GPUs in your basement you have a very limited amount of memory that's the number one bottleneck and so anything that can free up memory from KV cache and allow you to put those longer sequences on there is going to be very valuable and the additional forward pass computation is going to be like less of a a drawback.

8:15 so still Turboquant is a fantastic research paper, a really great technique that just ended up not being as applicable in the data center inference world as it might have first appeared. you know, instead we're focused on, you know, NVFP4 with a focus on quantizing the weights versus the KV cache. you know, doing our best to find rough edges in the quantized weights. make sure that we're not flattening out of probability distributions for the KV cache itself.

8:45 Focusing instead on KV aware routing, KV offloading, KV sharing using you know nickel and using Nvidia Dynamo and other tools in order to move the KV cache around the system and potentially offload to CPU, ordinary memory, etc. versus trying to use to Turboquant to to compress it. and then we're also focused on quantization across modalities. so thinking about how can we apply the benefits of NVFP4 not only to language models but also to you know image and video models. Ali also wrote a lot of great stuff on Twitter about that which you should check out. So that said the KV cache is still very important.

9:26 Let's talk about it. Let's talk about KV compaction. Again quick review. KV cache. If you put in the same prompt with the same prefix, you get to reuse the tokens that you calculated prefill on last time. That makes your whole system faster and more efficient. Broadly, KV cache is lossless memory. There's only a couple sources of lossless memory. When we think about our inference system, we have the content of the prompt, the context, you have the the KV cache, and that's going to scale linearly with the amount of data you pass in. And now if you're thinking about you know million token sequence lengths that actually gets pretty substantial. So a lot of people are thinking about how do you compress memory? How do you compress context?

10:08 Agent harnesses will compress context. Rag search all these techniques that we've been talking about for years are a sort of compression of a larger context into something that you can give to a model. You can write to files. All of these things scale sublinearly with the amount of data that you have. But what if there was a middle load? What if there was a way where you could get quite a bit of compression in the data that you remembering with near lossless information retention?

10:37 So we you know we have a lot of different ways that we can think of what to keep in the cache. You know recent compaction methods have shown that we can replace the cache with a much shorter one. We've got papers like attention matching and cartridges that have given really promising outcomes here with high compression ratios. But both of these are run at inference time. Again, one of the techniques I want to talk about or one of the themes I want to talk about is training for inference. So in this case I want to introduce something called still by the base 10 research team where the synthesis on top of the cache where we're keeping a loaned representation of the information rather than a the information directly or a sort of deterministic subset of it is amotized via training. So Charlie and Mudith from our post training team did a fantastic chalk talk at Kosa compile recently.

11:30 it's up on YouTube. So I would encourage you to take a look at it if you're interested in learning about KV compaction. I do not unfortunately have the time or the genius to explain everything up here. but the the basic mechanism is that still is a perceivable bottleneck that takes a fixed set of loan query vectors crossends it against the full KV cache and produces a set of compact keys and values in a single forward pass. This creates a fast differentiable compressed memory that the LLM can attend to as if it was real context. So if you're interested in KV compaction, definitely check out Charlie and Mudith's work. it's been a really fantastic thing to learn about.

12:14 So that's two of the techniques. We've talked about quantization. We've talked about caching. The final one is speculation. And there's been a lot of change here. as a review, speculative decoding, we're going to use draft tokens. We're going to verify them during the forward pass and we're going to use that to generate more than one token per forward pass. It helps a lot with tokens per second and it is a fully lossless optimization which is great because we don't have to worry about quality at all. Now in in the sort of history of speculation we started with speculative decoding. All of these are in the book. you have spec where you use a small model from the same family to generate draft tokens. Turns out small models are like not great draft token generators. They're great small models. So we invented as an industry a bunch of new methods like Medusa where maybe just you add decoder heads to the model and then eventually Eagle 3 which was hey what if instead of taking a tiny model from the same family, we actually train a billion parameter model on the hidden states of the target model to generate draft tokens. And that actually worked really well. And so, you know, as of of maybe February last year of of February of this year, Eagle 3 was the best method in speculation.

13:30 Now, we got Dlash. Dlash is even better. So, it's diffusion for speculation. Dlash creates a sequence of draft tokens instead of a single token. So the model is a diffusion language model which means it creates a whole sequence of tokens in the same way that a video or image generation creates a sequence of frames or a sequence of pixels and iterates over it rather than doing a auto reggressive token generation. Dlash models might be two or four times slower to run, but they're going to predict eight or 16 tokens at once in that in that window while Eagle is only doing one at a time. So a single D flash forward pass is faster than the entire eagle draft phrase and predicts more tokens. These tokens are able to cross attend to each other and generally create a higher acceptance rate because in speculation acceptance rate is everything. So in the wild we're seeing a more than 3x improvement from deep flash. this is measured with a single B200 quen 38B. and we can see it versus Eagle. It's a substantial improvement in the tokens both the token acceptance rate and the tokens per second. These deflash models are trained with a attention mask for birectional drafting. so the target model is going to provide the context. and within each block we're going to have a subset of clean tokens that are sampled and the attention mask is going to enforce causal consistency. but it is still going to allow for birectional attention. where in you know a traditional auto reggressive model you're only looking at the tokens in a single direction. So that's why we're able to you know take advantage of this diffusionbased architecture and then you know I thought I was done and then a couple days ago DSpark came out. Now Dlash we do have up and running in production. DSpark is is new research. so this one I can basically only say like hey it exists.

15:31 It's cool. We're looking at it. the difference versus Dlash, it still has that diffusion model, but it also pairs it with a sequential model. And the idea is that we're going to improve acceptance weights by having these two models work together rather than having just the iterative speculator just the diffusion speculator or just the auto reggressive speculator. So DSpark very exciting but we don't have any production results with it yet to share. hopefully you know we'll have those for next time.

16:03 what we do have production results though on is continuous speculator retraining. So this is we're back to Dlash here. and this is the idea that you know speculative decoding is very dependent on the actual content of the prompts and responses that you're looking at in your system. And so if you are continuously retraining on those prompts and responses in your live system, you can see a 20% to even 2x improvement in your token acceptance rates. This is actually like really hard to do.

16:35 it takes a lot of storage and you have to make sure that you have permission to use the data that you're processing in this way. it takes a ton of compute and you have to move all of this information around. And if you change the underlying model, you also have to change the speculator model. But when I look forward into the future, I do think that continuous speculation for very very large scale systems is going to be a worthwhile optimization.

17:00 So what is next in inference? the following is like personal opinion and speculation and public information and like if I knew anything that was actually coming out, I wouldn't be able to talk about it. so so this is just like what I think is going to happen. You know I've been through three hardware cycles through the Ampio release, the Hopper release, the Blackwell release. and it always takes time for you know when these chips get shipped to when they get installed in data centers when the entire software stack really is able to take advantage of their capabilities. But some things that I'm excited about are, you know, with with Reuben, it looks like the NVFP4 performance is going to be fantastic. So the more we can like honestly borrow techniques from local inference and get a lot of confidence running models in this NVFP4 data format, the more we're going to be able to take advantage of the awesome performance of of the upcoming Ruben systems. I think that like disagregation and systemwide communication is going to be increasingly important. we're seeing really excellent early gains from PD disagregation and the ability to you know move information like KV cache data around the system is going to be increasingly important and then like I said the theme of training for inference is going to be something that continues to have a big impact in the industry moving forward so thank you all so much for the talk for coming to the talk I am on Twitter I'm on LinkedIn and I'm giving out free books you can download a PDF at the QR code or come down with me to the base 10 booth to get your free copy of Inference Engineering.

18:39 we've got a bunch there, maybe enough for everyone. If not, we will have a CO bring some more from the office. so yeah, I'll be downstairs at the base 10 booth. Thank you all so much and have a great day. >>