Ep. 034 - The Fight for Fast Tokens, TPU v7, Vera Rubin, and Engrams (AI Supply Chain, InferenceX)

0:04 Hello everyone. Welcome back to Semi analysis weekly. I'm joined this week by the inference X team. It's been about two months since we've had them on the podcast to give a little bit of an update. this episode we're going to talk about engrams offloading agent X TPUs and anything else that we think of. Guys have been hard at work. So I got CM Brian and Alec. Guys, welcome to the show. >> What's up? What's up? What's up? Hello.

0:27 Nice to be here. Awesome. Okay, so the latest article that came out, we're going to start with that one, came out September 18th, so almost two weeks ago at this point. it's all about engrams and what happens when people codees the model architecture for a capability of offloading certain parts of the model or the KVs to DRAM or SSD. So how about we just start by talking about what are engrams and give a little bit of an introduction there.

0:59 Yeah. Yeah. I So I think you know engrams basically have been engram models and the the concept of sort of like transferring more things to the embeddings have been around for a while. Brian can touch on more on like the history, but basically when you have a transformer architecture, each token has an embedding, right? And the embedding is some, you know, latent space of like meaning about what that token means. And as you know, you have a sequence of tokens and it you know, makes its way through the layers of the transformer. You can think of each layer as sort of building more information about like grounding the facts of what of what that sequence means, right?

1:45 as the tokens like relate to each other and so on and so forth. So like as it flows through the layers, it more meaning more meaning flows into the the hidden state. So the idea with engram is basically instead of having these layers you can basically during training time you can have the meaning of multiple so two grams and three grams em token embeddings kind of learned at training time. So then these these can be stored in a hash table and then looked up at inference time and this is basically like for free I mean not for completely free but if you go down to the next graph or a couple graphs down you know that deepse ran some experiments right and basically on the right side at 100% right it's the the loss for all mole So no engram layers and then on the left side it's all all engram layers and no no mo layers right so you can see that the loss is actually minimized at like a middle point right so when you have some engram layers and then some some mo layers and it turns out that the the performance is actually quite good so that's like a highle overview but I think Brian maybe you want to talk a little bit more about the motivations for engram and the Yeah. Yeah. So the the paper that came out introducing dig earlier this year actually like talk about the intuition for adding it which was that models had to build up this representation or the meaning of words basically in later layers. So they tested this by like comparing the next token prediction ability of the the hidden state of the tokens at different layers. And there was an earlier paper on this court. And basically with engrams if you calculate the KO divergence between the hidden states of the early earlier layers and the late later layers you notice with engrams the layers are have lower KL divergence with the later layers which kind of show that the model is moving towards a next open prediction earlier in the model instead of later which is good because means that you're not like wasting parameters on memorizing stuff memorizing or trying to build up the meaning of words. Yeah. and the code design of this model which is one thing that dips really specializes in this neck for like considering inferencing while designing the architecture of the model is that the models can be asynchronous asynchronously over overlapped with other parts of the model because the inputs for the engram modules are just the token ids. So in their paper, the original paper, Dipstick also talked about the trade-offs of engrams being put earlier versus later because if you have engrams put engram layers put earlier in the model, they can very well they can help the most and the ablation show this. I believe the engram layers help the most as it can bring in the memory of the meanings earlier on. But the problem with having the engram layers earlier on is that there's really nothing to overlap it right because you have to straight away start waiting for the engram retriever to end before continuing. So this actually shows up in Quen's AAA flash next I think they put the engrams in the second layer because the first layer gives it time for the retriever from DM or SSD to actually come in first so that there's no there's very little overhead but interesting part for V1 V4.1 flash is that they put it at layer one so there was still space for overlap but not as much. Yeah. And this brings me to the results, I guess, comparing we have graphs in our article comparing the DRAM and SSD offload versions of engram. Jordan, could you pull pull up the chart?

6:17 >> Yeah, trying to find it. >> Yeah, no worries. Yeah. Yeah. But while Jordan is finding finding the charts, maybe I'll talk about like this offload, how the offload works. S offload uses UVA which stands for united yeah unified virtual addressing sorry and basically it just moves the engram to CPU pin host memory and oh yes thanks Jordan and yeah we can see that DM we actually tested HBM within HBM engram as well as I don't think we talked the word in the article because because the results aren't very interesting but HBM Take a step back here, Ryan. Like the implication of this is just that there's a significant HBM constraint and people are innovating around that constraint by introducing engrams to make it easier to offload to DRAM or to SSD, right? And >> yes, >> this is just maybe a natural evolution of that pro like progression over time. Is that fair?

7:22 >> Yeah. >> Yeah. I guess I guess one part of like >> I was muted. Brian Brian maybe maybe you have context on this like I I I don't know if necessarily like engram was a specific code for restriction of HBM was it like was that >> definitely partially a lot of yeah a lot of deep stuff addressing like limited HBM like last year even their whole MLA thing was about reducing KV cache for HVM usage and I think like this is along the same lines as Because if you think about it from like first principles, you want something you want a technique that can like be overlapped. So either retriever or so yeah. So just retriever and the stuff you get at the front of the model is basically the token ids. So basically what can you do with token ids with only the token ids that can be added to the hidden states and overlap and I think engram is a very natural consequence that you get to Actually the long cat team from Mtoan claimed to be concurrently working at the same with the same technique. So this actually quite cool that like two labs I'm not sure if they collaborated or what two labs arrived at this conclusion independently that engrams was a good technique. Yeah.

8:49 Anyway going back to Jordan's point about HBM. I think we talked about this in a in our previous four high podcast but yeah HBM is really a big constraint right now and we're trying to offload everything off of HBM right like we heavy offloading firstly I think CM and ALEC are quite experts in this field given that we did for agent X maybe they can explain about it >> yeah I mean a big part of that 12 high 8 high four high Ruben DSP back conversation around HBM was the fact that the American labs are optimizing for HBM bandwidth over capacity or bandwidth per bit. in some ways the Chinese labs have already had that constraint forced upon them by the chips that they have access to. And so yeah, maybe you can talk a little bit about the like Chinese chips like the domestic accelerated ecosystem as well as the sort of already despecked American chips and how if you just don't have the bandwidth already available well you need to do something about it in order to improve serving at high concurrency or like high interactivity. video I guess.

10:14 >> Yeah. Yeah. So the Chinese chips are very interesting. We in our earlier Deep Sig V4 Pro article, the Day Zero article, we did talk about the Sen 950 chip design. And the HBM they use, they don't really call it HBM. It's high high Q. I'm not sure what it stands for, but it's their own version of HBM that that they manufacture in China. And of course the bandwidth is lower than the stuff we get but they're very creative workarounds for their hardware restrictions. And we're actually writing an article on this which will come out very very soon.

10:55 Maybe right after this podcast comes out. I'm not sure when these podcasts will come out but >> yeah basically the trend the trend I mean at a high level the trend forever has been I mean China has been restricted in terms of HBM even more than you know the United States on a chip to chip level. So they've just been trying to make, you know, the KVs smaller and more efficient, compression KV, and then that leads to like having to offload stuff to lower tiers of memory. And I think, you know, engram is just like a natural extension of that because you're you're effectively getting like as Brian was mentioning, you're getting layers for free in a way since you don't have to build up the representation and you just have that offloaded on DRAM and you can as Brian was saying you know you can overlap that with with other computations since you have the token ID at the beginning.

11:52 So yeah, I mean this is like a general trend of just like Chinese labs are just trying to be more efficient because in a way it's a curse that the US labs have, you know, infinite compute and are hoarding HBM and stuff because we we don't have our hands aren't forced in that in that sense. So yeah, it's it's been it's been quite interesting. All the innovations feels like they're coming from China. >> yeah, interesting. Now maybe you guys can talk about the implication on the workload. So obviously the reason to want to store a bunch of KV is driven by many concurrent requests or very large requests. So either long context workloads or high throughput serving both of which are in demand for anybody that's serving their model for coding agents right now like agenic coding.

12:47 so obviously we had this big motivation to try to develop a benchmark that more accurately represents the workload pattern of eenic coding and we've applied this for all of the new models and released both the data set as well as the initial results. It's called agent X. Well, Cam, can you run us through what is Agent X and what's the implication on the change from like traditional AK1K to using Agent X and then how we were actually able to tease out some of these differences in efficiencies from the model architecture and clearly demonstrate them using this benchmark.

13:23 >> Yeah. Yeah. Good question. So, I think before we were doing AK1K, no prefix caching, just testing at a chip level. Now we move to agent X. Basically this is just you know agentic traffic right so as you see here in this image right each the weight you know when you code with an agentic harness what's happening at a low level is you send a message and you know the the output gets appended and maybe some tool gets called and the output of that tool gets appended to your streaming of messages and then this is just keeps getting sent to the model because the model is stateless right but as we know right the KV cache is just growing like at at an insane rate from turn to turn and you know if you know when you have million context length models right you need to store quite a bit of KV cache so the the idea with agent X is we're no longer just testing like the the chip level stuff we're testing the entire system how KD cache moves between chips and pre-filled decode disagregation scenarios and how offloading works right So yeah, actually if if you if you want to pull up inference.com, maybe we can >> take a look at some results.

14:40 >> Yeah, definitely. >> Actually, so in the in the the traces that we replay, right, many people are surprised that the cash hit rate, the theoretical cash hit rate is is very high, like above 98%. Right? So it's just about how much of that 98% can you actually realize in your system. So maybe just click on like I don't know B300 VLM, right? So the points that are have a halo around them, right? This is with offloading enabled. So you see that this is not very important for the higher activity points because really, you know, offloading to a secondear secondary tier of memory is slower, right? And when you're at a higher concurrency and you want higher higher interactivity, you I'm sorry, I I misspoke. When you want a higher interactivity at a lower concurrency, you don't have as much KV to store. So offloading isn't particularly relevant. But when we're at the you know 40 to 60 TPS range with extremely high throughput. So this would be something like offload offline batch inference or even like reinforcement learning rollouts where you have like hundreds of concurrent agentic session. offloading is extremely relevant and you see like maybe like upwards of 20 to 30% cash hit rate just from DRAM instead of HPM.

16:09 So yeah, this is the idea with Agent X and we we really just want to show like what a realistic workload is. and yeah, I don't know, Brian, maybe you want to talk about I think the coolest thing is the the profit calculator. >> Yeah. So the cost of our so our benchmark is really quite cool in the fact that we can kind of like because segment also have TCO models right please buy our TCO model our TCO model actually calculates like the total cost of ownership of our of accelerator chips. In other words, like how much does it cost an hour to run them and basically like breaks down what this cost entails like your data center cost, your electrician cost, the chip cost, etc. And we kind of give a preview for users of infrared sex, the final number which comes down to the one shown right now. And with this cost as well as so this is the cost basis and on open router you can kind of obtain how much the inference providers are charging and from inference x you kind of know how many users in other words the concurrency the throughput and with all these numbers combined we kind of calculate the profit one can make using different chips. So for example right now with the best serving which is GB 200 MBL72 I believe you can make about a 40.70% margin profit margin at those parameters for Kim K3 and yeah with Deep Sig V4.1 flash you kind of see another trend where with certain parameters or certain settings you kind of don't make a profit. So I mean it's really quite interesting seeing this because I mean we do have we we are going to write up an article soon on the methodology of this I believe >> I mean I mean it's quite simple like it's it's I mean go go back to Kimmy Jordan I mean the the greatest takeaway from this is like these are actual feasible profit margins that you can get from running open source inference like these things are just money printers right now until the cost of compute sort of normalizes like to break this down right on the current GB300 like with Dynamo and V I'm assuming a 60% or sorry what are we at put it down to 60% that's probably more so 60% utilization right meaning yeah obviously you don't have users fully saturating the the system at all times and maybe you have like some nodes down or whatever so at 60% utilization with the 30% license fee that you have to give to Moonshot you can literally make a almost 50% profit margin serving this on GV300. It's just a money printer, right? So this is assuming like the cash hit rate that we get on agent X and then looking at the open router data, but this is like very feasible, right? so >> and and sorry Sam Kim let's let's go back a minute to the Asian X you know data set and talk about that a little bit. I think we haven't necessarily introduced that clearly. We we talked about how this approximates the use of agentic coding and how this distribution of requests is interesting because like the P50 request is still almost 100,000 input tokens and you keep talking about cash hit ratio.

19:54 Bunch of people are probably like how do they know that's real? It's real because we're replaying our own data. So can you talk a little bit about how we collect that and why we think it why we're pretty confident that like the replays of the data against these endpoints for the agent X benchmark is actually approximating real world coding. >> Yeah. I mean we we we have a proxy that intercepts all of the semi analysis employees u agentic coding sessions and we just replayed them right. So we we just take them all and then and do some post-processing to get rid of the bad requests and then we replay them against actual servers running open source inference. And yeah, I mean this isn't going to be perfectly representative of all agentic coding traffic, but the fact of the matter is like the workload shapes are quite simple, right? You you all harnesses are kind of the same similar, right? You have some tasks that you want done. You send that, you prompt the model. The model decides to use some tools, goes and gathers a bunch more context, writes that all to cache, and then does this self- aentic loop for however long it takes for the task to be done. And during this time, the bottom line is most of the context is is a cache hit. So with that, you need to be quite smart about how you store your KB cache, how you transfer your KV cache, how you route requests to different prefill servers. And you know, as you can see at the top here, there's there's a lot of parameters you you can set.

21:21 We're using Nixel for the KV transfer engine. We're using VLM for, you know, VLM for KV offloading to CPU. We're using DRAM offloading only. We're using the Dynamo router. And now this is like not necessarily trivial. There there are a lot of ways to do this, but yeah, to answer your question, the data is representative because it's just like straight up just like coding sessions that we captured. so that's quite cool. You can explore all this on inference.com. It's it's pretty neat.

21:53 >> Yeah. So when we're saying like, hey, back to what Brian was saying around the profit estimator per per gigawatt, like if we take our real world traffic, discount it by the utilization rate, and then divide it by real hardware cost numbers, if people are running the latest and greatest chips with the latest and greatest models, it's incredibly profitable right now. >> If you're listening to this podcast and you have like a few million dollars, I would buy like a GB300 rack. No, I'm serious, dude. like you can ROI is just like probably make your money back in like a couple months. you >> and it's probably better as well right for the infense providers because they have their own like private kernels and we're just working with open source stuff.

22:35 >> Yeah. I mean, even if you go back there, I mean, you don't have to go back to page, but even if you put like a 30 or 40% utilization rate, right, which I'm sure some of the providers in open router, no shade, but I'm sure they're getting around there, like pretty utilization, it still can be profitable, right? So, until the price of compute kind of I don't know, I'm not a finance guy, but I'm assuming this will normalize, right? Like, it's insanely profitable.

23:00 >> Well, I think people question this a lot at the beginning, and I want to bring in Alec here to talk a little bit about this. So, let me try and tee them up. two things have been changing. We talked initially at the start about how the models are getting more efficient and higher quality, which generally means people are able to charge a higher price for the new models because they're higher quality, but then make more money because they're it's lower cost to serve them if they're improving in efficiency, especially with all these optimizations for engrams and KB cache.

23:30 But then also the hardware is improving and so there's competition out there and specifically another article that you guys wrote was about introducing the TPU. Probably the most exciting looked forward to announcement here. Here's a nice picture of the TPU racks. A lot of people wonder like you can rent them in GCP. Why are they not on Inference X? Why is everybody talking about Nvidia versus AMD? But Alec, talk us through the high level, man. Like what's what's been up with TPUs? We got we got some data.

24:00 >> Right. Right. So I think so we had we ended up testing out TPUv7 which is Google's first real purchasable accelerator through their own cloud and so we wanted to see like do a quick apples to apples comparison against B200 and B300 you know see how's the performance is it like feasible and so on and as you can see from this paro curve the blue is TPU v7 and it's like at the bottom it's that's the best for it to be it's the cheapest cost per million tokens you know for everything except for like a single point which is fantastic and you know that depends on that depends a lot on the on the TCO for the TPU accelerator I think for this one it's specifically the external TPU or external TCO which is dollar 21 cents I believe I mean that's like super good and like the performance they were able to drag out of it is fantastic.

25:08 >> Yeah. And and and this is like this is early this is this is early days for TPU inference performance. So it'll just get cheaper through software improvements. >> Right. Right. And like this is the open source version too. So like this isn't what the I guess what what they be running internally in big inference providers. >> This is what the public has access to. >> Yeah. Yeah. And talk about the trend maybe a little bit like obviously TPU is a very different chip than B200 B300 both in terms of like the silicon design, the programming model, how many people have access to it, how open source the community is, but there's a lot of like you said apples to apples and then apples to bananas in the article in terms of comparing PPV7. V7 is maybe the current generation, but V8 is coming pretty quick. We're talking about some open source models that maybe aren't Google's highest priority to optimize for. TPU VLM is pretty new.

26:18 What's the trend for the future with TPU? It's probably going to get better from here, right? >> Right. Right. So I think as we saw recently actually we had we saw in or infract and TPU had like the the mega kernels and like they got like a huge performance boost out of nowhere. So I'm I'm sure we can keep looking forward to seeing that kind of massive performance boost from them. >> Okay. One of the things about the TPU for years is that Google would produce a ton of them but then they would use them all for internal workloads or for really big customers where they get the whole support of the TPU engineering team. You guys talked a little bit about VLM plus TPU and how this is in preview.

26:58 Obviously, we're we're specifically reporting results from one model on one workload and comparing it to B200, B300. It's a little bit contrived. What's the broader trend here? I think it seems like we're going to have more TPU over time in the ecosystem. Is that fair? >> Yes. Yeah. So, I'll take this and then hand it off to Brian. You're going to see a couple things. you're going to see mass externalization of TPU as we, you know, see more maturity of V7 and then V8 and so on and so forth. Meaning that Google is going to stop allocating all of this to internal workloads. They'll still allocate a ton, but they'll sell a bunch of it to with that. You'll see massive improvements to the open source ecosystem. So, for instance, right now, there's no native torch backend for for TPU, right? It's basically goes through this whole like different like intermediate representations and like I don't know it's really messed up right so torch TPU is going to come out quite quite soon and this is going to be a native torch back end for for TPU and this is going to make things the programming model way easier right from a software perspective so you'll see immediate improvements in in open source inference and like this is just going to be the trend for for quite a while I Brian, you probably have some things to add.

28:23 >> Yeah, I mean TP in the past may go a bit more deeper into this like he are buon systolic arrays which pises like data movement and one problem with it of course was that the systolic array was of a fixed fixed size. So it's smaller once you have to do padding which is kind of a waste of the compute and this kind of resulted and yeah because TPUs were mainly used for internal stuff a lot of the models were built around this limitation whereas open source did not and this was one limitation or like one thing holding TPUs back for models of inference x so like the full power of the TPU wasn't really used and like a year a few years back maybe just a year back everyone was bullish on Google and deep mind because like they had the most amount of compute out of any lab because of the TPUs. I forgot what the numbers are maybe something like 3 million TPUs being produced. But yeah, everyone was very bull because of the amount of TPUs Google had and the amount of research that they did. I mean if you like went to any of the conferences any of at the ML conferences you'll see a ton of Google deep mind papers and the guys there are really quite correct there's a saying on X that everyone prominent in ML is either in deep mind or has been in deep mind at some point in of their career.

29:57 Yeah. So the TPUs like actually being externalized is quite a big deal I believe and yeah and it's quite some time since you've seen a real third competitor to AMD and Nvidia. It' be interesting to see what the next few years are like. Yeah, I think first thing I'm looking forward to is moving for the the support in inference X from private a private fork to the actual public repo and getting this stuff running on some persistent clusters there. That's really exciting.

30:30 The other thing is we we know that TPUs are being installed in Neoclouds as well. So, for the first time, there's going to be hosting of them outside of GCP, which should open up the ecosystem to have some more customers actually using the TPUs beyond having to consume them through GCP. >> People bring the price down as well. >> People will be happier to not have to go through GCP, man. >> That's like a turnoff. >> Need to make this a Cluster Max call, buddy.

30:57 >> Oh, yeah. I couldn't help myself. >> Alec, talk to us about the GCP console. What's your favorite, Alec? Give me your top three cloud consoles. What do you think? >> My top three? Huh? Well, I can't say I've used that many of them, but I mean I I just kind of leave it to to the command line to be honest. >> Just I find the command line easier >> to navigate through all the crazy UI.

31:23 What do you think Codeex or Claude Code's favorite? What do you think their top three CLI are for the for the clouds? >> GCP, bro. When you when you update GCP on the command line, it takes like 14 prompts that you have to say yes to, dude. It's like installs bunch of >> I think that's free. I think software is free, man. I think can do that whole install for you now. >> Or maybe your open claw.

31:54 >> okay. So, moving on from from TPU, some other new hardware hit the hit the wires on Inference X as well, namely Vera Rubin. let's talk a little bit about the difference between Vera Rubin and Grace Blackwell. So, Jensen sandbagging again, huh? >> Certainly. Yeah, he he's definitely he was sandbagging. I mean, look, it it's it's just an incremental improvement, right? So, I mean, you see in particular at the spot that people would actually So, this is Deep Seek V4 Pro, which I guess no one uses anymore, but in the, you know, in the range of interactivity where providers would actually be serving, it's it's just like 3x better.

32:37 So, I mean, when you think about what that means is you can serve 3x number of users at the same interactivity. So you could charge 3x the amount of users the same price, right? But that's not how you'd price it. But like I mean it's just more money, right? More profit. So if you go to the profit estimator, you you'll see this reflected. But yeah, I mean Brian can speak more to the architectural improvements, but yeah, I mean just more flops, more memory bandwidth. Like it's it's it's just better, man.

33:10 >> Okay. One thing I'll lead Brian in. Something that I've talked about with Vera Rubin on this show before is the fact that it is a little bit more of a minor improvement going from GB300 to VR NVL72 in terms of software support in terms of like the physical deployment of the system because let's say going from H100 or H200 HGX to GB200 GB300 you introduced the ARM CPU you introduced scale up NVL switches for the first time you introduced a new 400 gig network or 800 for the for the GB300's. you know, obviously Black Blackwell SM100 was a whole rev. it had new FP4 data types that the previous one didn't verb seems like more of an incremental change. There's no big change to the scaleup network. There's no new ARM CPU.

34:00 Everybody should have experience with liquid cooling already deploying it physically. So on one one hand it might be easier to adopt but on the other hand you might feel like oh I'm not going to get as big of a performance improvement compared to GB300 and I may be less motivated to go for the new stuff. it seems like that's not necessarily the case initially. So high level like Brian maybe you can talk about the initial software experience of porting to fair Rubin and then yeah back up what Cam said which is like there's definitely going to be some performance benefits or cost benefits on a per token basis when comparing to GB300 right >> yeah yeah I agree I agree with you that the Rubin step up from the previous generation was not as big as black well was from Hopper I mean some interesting half regarding where Rubin was that of course the dispect that we talked about earlier going to eight high the reduction in VRAMm and of course this may be the labs prioritizing memory bandwidth because of the scale network you're able to like spread your model out and do stuff like YP which makes users have more GPU's memory bandwidth so your per GPU memory capacity isn't as important because your aggregate is what matters Right. And another interesting new thing with V Rubin, of course, we can't talk too much about the intricacies of V Rubin is that the new LB quantization that we mentioned in our previous introducing V Rubin. Yeah, it's very interesting. It's the first time we see >> it's quite cool.

35:42 >> Yeah. So for those that did not read our article, L Rubin has a new hardware accelerated path called LD in their internal docs and this was confirmed already like their PTX I forgot which version I already talked about it and there are like open source Triton branches of Fox implementing LB and from a high overview all it does is a lookup table of contization. So use if I'm not wrong three bit keys to look up 8 bit look up table values and this like reduces down to about 3.125 bits if I remember correctly per weight and it's quite interesting that Nvidia gave it a specific hardware accelerated path on beer rubin and right now we don't see any applications of it yet but it'll be interesting to see what comes out of it. Maybe maybe it's some very specialized use case that the labs have that open source does not have yet.

36:46 But very interesting to see yeah what comes of it. Yeah. And thanks Jordan for showing what it is and how it compares to like block scale quantizations like MXFP4 and MBFP4. But it'll be interesting to see what is this is used for specifically and because you can actually emulate this for those ski you can actually emulate this in black well to look at like the efficiency and how well it actually quantiz quantizes higher precision weights and can run like evals on it and to check the perplexity law stuff like that.

37:28 So yeah, a lot of work coming out of this. Yeah, garbin is quite interesting in some regards although not really that much of a difference. Yeah, but but the performance does speak for itself, right? Like we do see crazy increments in performance from infrance X and this is like quite early results some and we are getting like better results probably very soon. I think before this podcast goes out, we'll probably have >> yeah more updated results.

37:59 >> Here's what jumps out to me, right? It's a curve and so obviously it depends which point you pick in the curve to compare to the baseline in order to see what the multiple is if you take some really, you know, contrive point here. It's like infinitely better than just not being able to serve people at 200 tokens per second because now VR and VL72 can do it. at a midpoint 100 tokens per second to get three times better and then on the high throughput cases to not be quite at the three times better be more at 1.4 before. So 40% improvement, you know, makes you question, okay, is this going to be good for high throughput. Interestingly, >> this reminds me actually alo sorry Jordan, but this reminds me of the CPX.

38:36 I mean, the noise has died down quite a lot on CPX, but there Rubin introduces CPX for DSC workloads and splitting up the prefill and this decode. And right now we don't have CPX on our Infantex, but yeah, we look forward to getting CPX results soon maybe. and check how CB CPX improves disagregated work work groups. >> Yeah, it's it's just it motivates people to think about the flops versus bandwidth trade-off at the different ends of the curve and how you can optimize. And so the obvious chart to me is just like the big step up for Reuben as a GPU is the memory bandwidth, which is like as you go from HBM3 to HBM4, you get almost a three times improvement in memory bandwidth. then that roughly seems to correspond with this you know three times improvement on the big middle parts of the curve where inference is memory bandwidth constraint. to your point Brian the fact that this is an early result and these lookup table quantization schemes are not explored and you know some of the other hardware acceleration features in the GPU is not there yet. It just seems like here's us demonstrating that roughly on a per dollar or per watt or per like adjusted dollar watt basis, Bear Rubin is still going to be more efficient at inference than Blackwell.

39:57 It's just a better chip. They're not pricing it super aggressively, you know, aggressively meaning they're not price gouging with this chip because they know people are going to buy it. Like if you're going to get three times the performance and the chip doesn't cost two times more, like that's the that's a trade everybody's looking to make now going into next year. And to go back to the profit estimator calculations from earlier, it just seems kind of obvious that people are going to make this calculation going forward and say, "Yeah, I want the latest and greatest. Nvidia's done it again. Jensen sandbagging performance again. there's parts of the curve where they're really memory bandwidth constrained and you want to be able to serve that. maybe last question for you guys. You do a lot of work with these coding agents. Can you talk about experience using fast tokens? Like a lot of the implications of these performance curves seem to be related to like people are going to pay a premium for 100 plus tokens per second. we have some hot takes on the phone. I I I won't share mine right now. Can you give me your initial thoughts on like SLfast and SL ultraast and if we're going to see a lot more of this going forward for you guys using it?

41:12 >> Oh, Cam has spicy takes on this, I believe. >> Brian, are you Let's start. Let you start. Are you a super are you a a fast Maxi or are you a a fast bear? What do you think? >> N I don't do fast. Like for my personal use cases, I have like multiple agents running at the same time. So I have like multiple VS in my personal use case I use multiple VS code windows and like multiple engines on each VS code terminal doing basically different tasks. So it doesn't matter to me like how fast one agent completes cuz when I come back to that agent is probably already done given how much time I spend on the rest. But I know Cam has some differing views on ultra fast.

41:59 >> Yeah. I mean I really I'm quite quite bearish on it because I I guess my thinking is that as tasks become more agentic you need to I mean really really the the the power in these models is that they can call tools very very well right so they can call tools to bring in more context to accomplish things but the tools take time to run and if you have a model running on like you know Cerebrus or something that can serve like a thousand tokens per second, right? But then the tool use time takes like 30 seconds then that speed up in the generation of tokens just is not is not really realized because you're still bottlenecked by the tool tool use time.

42:48 So I I guess like if you're just like generating like HTML pages all day, like 100,000 lines of HTML, then yeah, I I suppose it would be quite useful. >> wow. That's a great cue for me to talk about my use of these models. >> Yes. >> No. No. Okay. Alec, what's your take, man? What >> No, I I I do have to agree with them. You know, there's times where where I need to do something really fast and or like get an answer real quick or something, gather context, and you know, fast fast mode's going to be nice for that, but for the vast majority of the time, I also have just multiple agents.

43:22 And I mean, I I can just do something else as they're running. I like I don't need the the code immediately. I don't need it to be merged instantaneously and it you know it can happen in an hour or two even. >> So I mean I just don't use it very much anymore. >> I just I mean it's so expensive, right? Like I'm not sure if the ROI is there. So like I mean yeah especially when you're bottlenecked by by the tool use time. It's just like really really especially like the ultra fast stuff like you know I'm not paying $300 per million output tokens for >> okay so let me let me push this counterfactual I I agree with you for my current usage but I also didn't forecast myself using AI exactly the way I use it today a year ago or two years ago. So I find it hard to predict what it's going to look like a year from now and whether fast tokens aren't going to be useful with any sort of certainty. So let me push you guys to two extremes. One is like do you have the experience where a model is really big? In other words, let's not say really big in this case, but like or the network's really slow or the tool use is really slow or the model just slows down because the endpoint's busy and then you get really frustrated because something is just not returning something at all. And so can you imagine a world where the models get 10 times bigger, 10 times less efficient, we just need to speed them up for them to be even useful at all?

44:46 I mean that's that's a very interesting take the the scaling laws do show that the more you scale the better results you'll get. So kind of a interesting approach to just just scale. I mean that's the bitter lesson right just just to scale and let computer take over. But I mean I've been thinking about that quite recently actually like why don't we just scale up and just go very fast to reach a usable token output rate but like seeing stuff at what the industry is actually doing like for example Astra and rumored rumors that 6.1 soul are loop transformers not confirmed usually quite interesting right to show where the industry is going like it doesn't seem to be like just scaling up is the way there's architectural changes very interesting innovations and yeah >> even loop transformers and lots of reasoning is not free in terms of how much computation you need to do right so >> yeah yeah that's true >> actually good point on the reasoning though like if if you're on like ultra max whatever the reasoning that is obviously just purely to output speed like you need to decode very quickly. So that that's a good point. I honestly I think my prediction would be something along the lines of like you have a harness that can like route to a fast mode for like certain tasks and then like it can patch certain tasks that don't really require low latency, right? Because when you think about it, this is basically what we're describing where like if you're writing a you're bootstrapping a RA React app, then it would be really really really useful to have fast tokens. But if you're doing something like Brian's saying, you know, running a performance benchmark and trying to hill climb overnight, I it, you know, the benchmarks take like 50 minutes. So if it takes 50 minutes and then you spend $40 decoding a few thousand tokens and then just wait another 50 minutes, that that's just a waste of money. So that that's my prediction, right? My prediction is like it'll be very specialized, but it'll still be useful for some things.

46:58 >> Yeah. How about the the other extreme? Let's say the models get really really small, really really efficient and still high enough quality. Is there a scenario where you are just bootstrapping a React app or you're using your open claw muse instinct perplexity computer or whatever and you want have you had the experience of getting instantaneous responses and feeling like wow this is really cool. I mean there's there's that's one that's there that's one interesting direction but I don't know personally I don't really think there's a chance of us reaching this smaller model state because like I mean it's been shown in the history of like computing that if you make something more more efficient someone else going to just lump more stuff into there like CPUs RAMs have gotten so much better but then websites have gotten so much more sluggish and so much more inefficient with the libraries. Yeah, I don't think honestly I don't think a smaller model like a world where smaller models are used will will really come true. But it'd be interesting if I'm proven wrong.

48:06 Better for the environment too as well. Like interesting. Okay. Appreciate you guys going on that little journey with me. I have a suspicion that even when you guys use all of these different agents in parallel and you don't really care about the time it takes to return that if it was 10 times slower, you would still find a way to get upset. If you went to bed, woke up in the morning and there still wasn't a response. That wouldn't be good. So, there's a a room for some sort of fast tokens in the future. Maybe not all fast, right?

48:35 >> Also, Jordan, people in America can barely afford Big Macs, right? So, I don't know if they're going to pay like, you know, hundred dollars per million alpha tokens. So I don't know man but yes I I agree. Yes, like speed matters, right? To some extent the intell I always love when Jensen says this, Jensen Jensen says this in every keynote, right? Like faster tokens are smarter tokens, right? Obviously, he's inclined to say that, right? Because he sells GPUs, but yeah, I mean like to some extent that's true, right? I mean, think about if you could, you know, basically have instantaneous output, right? That would be extremely useful for for some cases.

49:15 cuz you're just getting the answer right away. But I don't know, man. We'll see. Time will tell. Who knows? >> We will see, guys. Okay. >> We should talk about Ty RT. >> yeah. Yeah. Yeah. That was a perfect lead in, man. I was about to wrap this up. >> we can wrap it up. It's I don't think there's >> no talk about Tyler RT because a lot of people when they saw the 6.1 Astra or sorry 6.1 Soul ultraast mode get released at OpenAI dev day everybody was like oh well there's a Cerebrus deal with OpenAI that's promising them 750 megawatts over the next few years obviously this is Cerebrous but Tyler RT is super capable and if our take on how these OpenAI models are is correct and they're using some similar optimization to what we've got on screen here showing Tyler RT. Well, 200 tokens per second all the way approaching 500 tokens per second on a decent size model approaching trillion parameters like we got on screen here is totally possible just on Nvidia GPUs, right?

50:22 >> Yeah. Yeah. I mean tilet for those who don't know is basically just a latency optimi a inference engine optimized for ultra low latency right so I think the takeaway here is that you don't necessarily need super like specialized SRAMM architectures whatever to get like high high interactivity so these are achievable to some extent right bound by the laws of physics of course by super optimized inference engines, right? So, this has been shown here. There's there's also an AMD result and I think going forward this will become a trend where inference engines will become more specialized for different use cases.

51:07 We're already kind of seeing this trend. but yeah, Tile RT is super cool because yeah, you don't have to have a new chip. You just make better. >> Can you talk a little bit about how it works? So obviously there's SRAMM on the Nvidia GPUs. There's lots of different ways that you can configure the serving setup as well as write specialized kernels to actually serve the models. Anything, you know, technical we can talk about on that.

51:36 >> I think Brian can talk about like why it's good at higher activity but it's bad at high throughput, right? >> yeah. So basically like TART just you can see TART as one humongous mega kernel for the entire model. So like the problem at low interactivity is data flow. So what companies like Cerebra Sanova have done is they they built data flow machines where the hardware is built for data flow and for GPUs this a bit more difficult and intire you can kind of see it only batch size one is possible right now and the model releases are quite slow this because it takes a lot of software effort to actually get to this mega kernel because well the GPUs are not that friendly for this kind of data flow stuff. But it's really quite interesting that GPUs really are general the fact that they can be used for this and kind of like replace the specialized hardware.

52:40 >> But to what to what extent is this enabled by just having the AI like an AI agent hill climb one giant mega kernel that literally no one can actually read, right? And that it's just like works and it it it's like good accuracy and high interactivity, right? Yeah, but I mean people have been doing this that's for sure. But and of course I personally do believe that like kernel development is a very easy task for agents to do because is is it very easily verifiable and we have see a lot of agents like on the leaderboards of GPU mode and other kernel competitions and I don't know like right now this is a very new space recently I've seen this kind of mega kernel ideas come out MLX X there a repo recently released on MLX as well as other developments for example the TPU mega kernel which uses the same idea of this mega kernel for the entire model which is very interesting. Yeah, >> Jordan.

53:50 >> Sorry Jordan I have a question for you. If you're not to name any names, but if you're, you know, a chip startup and you're basically entire thesis is like fast tokens, are you a little bit like concerned right now with the state of I mean all these improvements with mega kernels like Brian's talking about and Jalapeno for instance like and just the general purpose GPUs just becoming faster at this region of of interactivity. What do you think?

54:20 I'm not concerned because I have a fundamental belief that demand completely outstrips supply and anybody who can produce chips is going to find a customer right now. people just need a lot more chips of all sorts. I think in the long term, all of these chip startups really make or break their business by their ability to deploy things that actually work and to be able to produce a bunch of chips. Like you just if you're going to have a really big business, you need to be able to manage your supply chain and actually produce a bunch of chips. Like you need to be able to get good yield and you need to be able to build servers and just like build a data center and train people and all of that which is you know in some ways a much harder task than actually making something that performs really well. I think the other thing that I've I've been clued into is the fact that high interactivity is high throughput if the interactivity is high enough. In other words, you don't need a bunch of batching dynamics just to get a whole bunch of, you know, throughput, even if it's a 1:1 ratio between throughput and interactivity, because if your interactivity goes to a thousand, your throughput's at a thousand. If it goes to 2,000, your throughput's at 2,000.

55:32 So, I'd be driving interactivity as high as I possibly could with a lot of this custom silicon, >> but it it's going to be like expensive. Like if if you said like batch size one and you have a thousand throughput at 1,000 TPS, then one user is paying for entire utilization. >> You can do batch one, but if it's really really fast batch one, you can just pipeline everything and still get high effective throughput even though there's not a bunch of batching dynamics here.

55:58 So I I don't necessarily feel like fast tokens are by definition inefficient tokens. I think that really depends on the hardware design and if you're as Myron says using because eggs got expensive meaning HBM you're going for caviar which is the SRAMM. >> so I I don't per calorie thing. >> It'll be it'll be interesting. I mean, another axis that I'm skeptical of is like quite a few of the alternative chip startups, not to name names, do kind of rely on general purpose GPUs for prefill. So, like the heterogeneous stuff is cool and all, but if your argument is like, okay, well, people are chip constrained and they need more chips, but like if the chip can't actually run efficient inference, like it's just kind of like you still need Nvidia GPUs. I I don't I don't know.

56:51 Yeah, the setup that makes sense to me is some pool of prefilled decode whereby you oversize one of the pools to be able to serve a non-refill decode disagregated setup for some of the traffic. In other words, if you're fixing your pre-filled decode ratio ahead of time for the entire data center, I think that's totally bunk because it's just going to the workload pattern is just going to change and you're going to have some wasted inefficient amount of silicon, right?

57:19 But if the workload pattern changes, you know, for more decode or less decode and you can move those chips into a, you know, only like a a homogeneous broken up setup, I think that makes a lot more sense in all of these scenarios. If you depend on Nvidia to be one of your GPU partners and you're buying GPUs and stuff, yeah, the business decisions there make a bunch more impact on your success rather than the technical decisions because look, Nvidia acquiring Grock is just obvious like justification for all of these chip startups that they're on to something technically. Otherwise, you know, why waste $20 billion acquiring somebody whose chip doesn't work? It obviously works to do something Nvidia GPUs can't at any price. So, everybody else can do the same thing. But the question is is not technically if this is going to be beneficial to anybody or if Nvidia has the demand signals to want to go and build this. It's business. It's like are you going to be able to build enough of them and deploy enough of them and actually make them work. And I think that's a quite a big question when like Nvidia's controlling who gets shipments of GPUs and backstopping people who want to buy data centers and build them and stuff like that, right?

58:37 that's a hard business. Hard hard to hard to get into. yeah, and I think the other thing is like all of these chip startups have proven at this point, generally speaking, that they've made a chip that they can tape out. I mean and and get like it it works CPU tranium everybody who's building custom silicon you know MTIA even Maya like the hyperscalers have proven you can build general purpose stuff too it doesn't have to be decode specific silicon so why not just build a second chip that's prefill specific if you want to you know go and win build a prefill and a decode chip right >> yeah I mean I think that's kind of what did right or that's what they are claiming at least. So >> I mean yeah like the the design like the RTL is >> not free but people have proven it's a lot less expensive than it used to be and the software mode the quote unquote CUDA software mode is speed to develop these things as opposed to just like people there's something physically setting Nvidia apart where others cannot reach a certain level of performance.

59:48 just like you can if you spend the time and the effort and the money on doing so. And so I expect chip startups to do that. I expect them to build pre-fill silicon and decode silicon. I expect them to realize the benefits of removing the margin of multiple vendors if it's Nvidia or Broadcom or Marvel or whoever they're partnering with. And then I expect them to have a really hard time deploying all of this stuff and producing enough of them for it to actually matter because supply chain and data center operations matters a lot.

60:16 like I already said. So, but I'm excited, man. I'm like, I know it kind of got buried in the middle of this conversation, but TPU1 inference X is to me the most exciting thing here cuz it's just like a one-way street. You know, they don't start selling CPUs externally and then just say, we're going to stop and keep it all for ourselves. >> Yeah. No, no, it's super exciting. I think I mean Alec, you've you've worked with Google. I think you probably can attest to to where you think this is going to go.

60:53 Yeah, it's awesome. I mean, we're going to have Tranium soon as well. We're going to have alternative alternative chip startups. We're going to have, you know, everything that that's the goal, right? So I I think going forward like Inference X is just going to be a consolidated place on the internet where you can literally get free information on the performance of chips and real world inference workloads and compare all of them. So Infertex chat room soon what do you think on the website chat room?

61:27 >> I don't know. We'll see man we'll see. Got to take that out. the boss. >> Dylan, what do you think about being a a Reddit moderator again, except it's on infrancex.com? Alec, you want to be first moderator? What would your username on inference x socials be? >> Yeah, I don't know if I ever want to be a moderator. That sounds like such a pain. >> TPU thing. Aren't you kind of a moderator of the GitHub actions queue for all of the runs that are going on right now? Kind of.

62:04 >> Yes. Yeah. We're like kind of gatekeeping. >> Well, it's your codeex, right? You give me an API key and you say, "Hey, moderate all these runs." >> Well, actually, that's a good idea agent be the moderator. hopefully there's >> Yeah. Yeah. Hopefully, it's more of a Reddit and less of like a >> I wonder if the AI agent will ban Dylan's posting on his own forum or if it'll let it slide. Okay, guys. Nice nice way to end off.

62:34 Anything you think we missed here? All right, little closing sentiment. as we talked about this, it seems like there was six different articles that came out in between podcasts about inference X. We talked about engrams, Agent X, Reuben performance, TPU performance, the DeepSync V 4 0 to1. I guess we already talked about that, so that doesn't count. Five things and dial RT is the fifth. good stuff. Hopefully we can get you guys back on before the next five articles drop and do this a little bit sooner. Appreciate you taking the time today and thanks everybody for listening.

63:11 >> Thanks, Jordan. Yeah, thanks for having us.