Transcript
0:06 Hello everyone. Welcome back to semi analysis weekly. This week I'm joined by Wyao and Kimbo. We're going to talk about everything GLM 5.3. Guys put out a great article recently. the model is about 2 months old and we've got everything from inference performance results to model architecture post training pipeline and just like what people are using it for that we can dig into. spoiler alert, lots of discussions around cyber security capabilities with this model. Kimboa, welcome to the show, guys.
0:31 >> Thanks for having me, Jordan. >> Excited to be here. >> I can hear it in your voice, guys. >> Okay, so the title of the article was how GLM 5.3 sparse attention affects HBM and memory usage. For those who don't know, maybe we can start, Kimbo, with you with just an introduction like what is sparse attention? Why do we care about HBM memory usage when we talk about these new open models especially ones that are coming from Chinese companies like ZAI?
1:02 >> Yeah. Yeah. So sparse attention is a category of attention mechanism where in contrast to linear attention where you try to modify the core core operation of the attention which is usually called a scale dotproduct attention. linear attention changes tries to change the the fundamental algorithm to linear time algorithm whereas sparse attention chooses to change the way you select the number of tokens while keeping the core attention operation. So the idea is to invent some sort of select a token selection algorithm so that regardless of how long the the sequence the context becomes you would be attending to only the most important tokens the most relevant tokens at every at every forward pass and >> yeah so this makes the model a lot more efficient right and a lot of different labs are actually pursuing things like this to make attention a little bit more efficient can you maybe give a little comparison of what Zai is doing with the geo models to some other lab like Deep Seek or Kimmy.
2:08 >> Yeah. So like you said it's almost in some sense inevitable because just physics we we just don't have that enough memory and that don't have that enough compute and GLM uses since GLM 5 they used the deepseek sparse attention the attention mechanism deepseek invented actually for their v3.2 two model and they they they opted that instead. and the the core components are you have a lightning indexer which is the the token selection component and then the the sparse attention. The sparse attention is based off of the more standard multilatent attention multi headl attention invented by deepseek at popularized on deepseek v3. So that's GLM and for deepseek even though they invented DSA a deepseek sparse attention on V3.2 2 they continued innovating they continued changing their attention variants. So for V4 the for the V4 series they invented CSA compressed sparse attention and heavily compressed attention which are most roughly speaking building on top of DSA and further adding more token compression modules and for v4.1 flash they simplified the HSA and CSA into like a slightly different design and Yes.
3:42 >> Yeah. Sorry. I mean, don't mean to interrupt you, but let's say that the the reason they're pursuing this is in the name of efficiency, right? like the I think people speculate as to why the Chinese labs may care about this more than the American labs due to restrictions that they have on hardware, but roughly the outcome from all of these pursuits of sparse attention is that the cost to serve tokens on this chart, a million tokens, goes way, way down. like it's really really cheap to serve if you have the ability to use sparse attention or like additional ways to like h stuff like that, right?
4:22 >> Yeah. Yeah, totally. it's just infeasible to fit that many tokens with a dense like a dense attention anymore given like our usage goes easily above a million token context for like a single session. so so yeah. >> Okay, one let me bring you in here. When it comes to actually serving this model and and trying to analyze the cost to serve a million tokens at 150 tokens per second, this obviously involves us running benchmarks that simulate a workload that you know a real provider like ZAI or somebody else who's downloading the model and serving it like one of the inference endpoint providers would actually be you know incurring. in other words, they want to serve dozens or hundreds or even thousands of concurrent users on a single model endpoint. Can you talk to me about that restriction that Kimbo just mentioned?
5:25 Like if you're going to support all these users with a million token context, it's going to have a lot of pressure on your system like the resources that you have available like memory, right? >> Yeah. Yeah. Absolutely. So I think for for surveying one of the factor that people care the most is like one of the factor is definitely aggregated token output because it represents how many total output it can produce for your users right and there there's actually a trade-off here is that well we only have this fixed amount of memory in the GPU and in order to get more concurrency A to serve more users, we need a more memory. so there there is a technique called high cache which it serve as prefix cache reuse and when there are more concurrencies the concurrency goes up it will offloads the memory from HBM to DRAM and that ends up will make the percentage of host memory goes up as the number of concurrency goes up. So if you see on the B B200 charts that we have on the high cache yeah right here the concurrency of 8 and 16 was very different on the percentage of host memory and this is actually even larger than P300 because P200 it has less memory per chip and yeah it's a really interesting and observation from Mars.
7:09 >> Kimbo, do you remember when highsparse and high cache first got introduced? Like these are clearly optimizations that are meant to save people money when they're serving these models, which kind of implies that the people that are introducing them have a lot of users that are using their models that they need to worry about. It's not a theoretical thing. >> Yeah. Yeah. so highsparse is the the sparse attention version of high cache and the whole idea of high cache is trying to offload KV cache from the the HBM to a lower tier higher capacity but a lower bandwidth memory and only load the most relevant set of tokens during during computation. And I think high sparse is a good example of an explanation to I think an also a misnomer that sparse attention fixes or fixes all the problems of heavy cache consuming too much memory or having high memory requirements. So the concretely sparse attention doesn't mitigate the memory capacity bottleneck. It's a technique that helps a lot with memory bandwidth. But as for capacity, because during the sparse attention computation, you would still need to select the tokens. So you still have to keep all the tokens in in a relevant context and you cannot discard them during computation. So at some point it still has to be loaded into HBM. So the memory capacity consumption will not be significantly reduced and that's where highsparse becomes useful because it can apply the typical caching rules so that you could load the most relevant parts into HPM.
9:01 >> Okay, talk to me about about some of the initial performance results here. I mean like the ability to use high sparse versus high cache. I mean both I guess depending on the definition of terms on this model GLM 5.3 is obviously giving some great performance results especially for particular vendors when you just look like total tokens per $1 of TCO >> 45 million tokens at peak up here like very very efficient.
9:36 It is. It is. And I just want to add up to what Kimble just said. U I think one thing that people that we want to empathize on article is that even though GLM is compute efficient, it doesn't mean that we have efficiency on the memory side and because the model still need to read through the relevant complex history, right? the high spots becomes very very important and it's offloading from HPM to DRM and offloading back is very techniques and we've also introduced in the article and as for the performance we have I think generally Nvidia chips just get ahead of all the other chips at the very beginning and a few weeks later when we about to ship the article very surprisingly at the AMD interactivity at 50 token per second is relatively the same cost as Nvidia's chips on B200.
10:38 >> Yeah, let me let me actually show that animation as we go through it right now. the models released in early October and we've got this nice feature on Inference X where you can go back in time and see how things make progress. So all the way through, sorry, it was released in early August. We're now in October. as you kind of shift through and you start to see these results show up on the dashboard, you know, wow, you can see the red AMD curves overtake the green Nvidia curves, which is quite impressive. And now Nvidia start to come back up. But just explain to me your experience here because you guys are running this stuff every day.
11:23 lots of people have been anti-AMD for a long time, especially the AMD software ecosystem, including us. But this is a relevant model that's not even 2 months old and the best curves for the highest, you know, throughput even at 100 tokens per second per user, which is a real interactivity range that that, you know, the inference end points I can see on open router are not approaching. So, it's it's below that. That's a lot of high performance from the MI355 from AMD. Good for them.
11:58 >> Yeah, they're definitely working hard overnight and trying to do a range of optimizations. I I think it it shows that I guess there are two sides, right? The first side is the hardware where hardware vendors like MVM and they they have the their their their chips ready and for their customers that chips performance was fixed and on the other side is the inf inference engine side where you have to be catch up and continuing doing optimization and iterated it over again to making sure that your performance stay more costefficient and this is a perfect example where We see that AMD engineers working hard on their engine atom and over the time it's cost per million token just catch up to a lot of the Nvidia's chips. Yeah. And we have a the live chart specifically looking at the prefix cache reuse that you guys were talking about earlier. If you if you click this at the bottom, you can see for a given configuration like MI355 on FP4 or flip over to GB300 for the given points on the curve, how much of the host cache like the DRAM is actually being used. Maybe dig into the point you were just making about the implications for DRAM. Like if we go back to the first big open model release from a Chinese lab which was like DeepSeek taking the world by storm almost 2 years ago now, right? But anyway, V3 and R1 memory stock sold off in Nvidia and everybody was really concerned that a more efficient model was mean less use of memory. But maybe the opposite is true where when you guys are describing as you make the models more efficient, you can therefore increase concurrency and serve more users and you have just as many demand as much demand on the memory itself. But could you put a little bit of a finer point on the capacity versus bandwidth thing cuz a lot of our listeners maybe tuned in for the conversation with Myin a few weeks ago where we talked about the dspec of the Reuben GPUs for Reuben Ultra from a terabyte of G of memory per GPU down to 192 matching the R200 that keeps the bandwidth the same but reduces the capacity. Can you explain the trade-off in the context of like inference where HBM bandwidth and capacity versus host DRAM bandwidth and capacity are all used to serve users at a lot of points on this curve? I guess I can first start off and you know on the infant side.
14:38 So if we think about memory offloading and capacity we definitely want to serve more user with more concurrency, right? because that makes the the aggregated output tokens maximum. But on the other side, we still need to think about the trade-off which is the downside of offloading to DRAM which is longer latency especially for longtail latency. So I guess the thing here is that we will customize based on different use cases. So for example, we've introduced another inference engine TL RT in our article where it's extremely focusing on low latency cases.
15:18 Essentially what it does is that it has a mega kernel for decode to reduce the launch overhead and in smaller batch cases such like batch batch size equal to one it it is shipped with blazing fast eco speed which is up to 470 token per second and the trade-off is that it cannot serve with large large batch sizes but we we we've seen that more and or inference engine are trying to serve with this special use cases for general GPU rather than serrus and people who want high interactivity and low latency will definitely love it and I guess the trade-off is really depends on how much of the cost you're willing to pay and what kind of the use cases you're you're you're dealing if you are Jane Street you definitely want your request gets like several seconds to get the output right but if you are just like user Like me, I'm dealing with a normal agent test. I can just wait it. I can just like maximize the output as long as the cost is affordable. So >> you're not using ultra fast mode, one?
16:30 >> No, I'm not using it. It's too expensive. Also, it's hard to it's hard to me to feel the difference somehow. >> Yeah. Kimbo, how about you? Are you using ultra fast on any of the coding agents right now? >> Not really. I I don't think I can't even use fast right now. So, yeah. Yeah. >> Okay. Okay. So, what do you think about Tyler RT and the trade-offs there? Cuz like when Yao said, you got to pay significantly more for that faster token speed.
16:55 >> It's a fun engineering problem, right? But maybe not a thing that is realistic. >> My interpretation of so back to like Jordan, back to you talking about the Reuben Dspec keeping memory capacity keeping memory bandwidth prioritizing that over memory capacity. I think that's a good example of all the design decisions that we we saw as well that memory bandwidth is more important than capacity. So concretely the the way to think about it is like we said sparse attention doesn't help with memory capacity and you still need memory bandwidth to sustain the the the performance that you would like and I guess in in the ultimate speed of light performance which we can see in some sense we can see it in tile RT or the ultra low latency regime is that you're fighting against how fast you can load your model or how fast you can load the relevant data to finish the very small relatively small amount of computation you have. So in in the ultra low latency regime memory bandwidth is the the most important part and the problem with memory capacity is that we we still need a lot of them but the I think dealing with insufficient memory capacity is a very standard problem in I think computer architecture that's why we have cache hierarchy and there are numerous ways of through algorithm to hardware through hardware to overlap the computation with lower tier lower lower bandwidth memory transfers. So in Reuben Dspec's case, I think they're betting keeping the memory bandwidth while trying to connect GPUs through networking and again try to overlap them.
18:44 >> Yeah. Do you think if you had to forecast what ZAI is thinking about right now when they keep the model architecture the same for 5.2 to 5.3 and get such a performance improvement strictly from their post-training pipeline in my view? Do you think they're experimenting with even more efficiencies on future attention architectures or you think they're just going to double down on what TSA and just keep it the same for some future, you know, GLM6 release or whatever comes next?
19:08 >> I think my guess is that they would probably continue my guess is probably continue the same architecture probably pushing further for 5.x or like 5.4 maybe. But based on their release, it seems like they they will continue trying different architecture cuz not too long after 5.3 actually they release 5.3 flash which is not really a flash model. It's in my opinion almost like a Frankenstein model where they do KDA the Kimmy delta attention with DSA as far as I remember. So they're mixing all the efficient alternative attention mechanisms putting into a model and actually trained it and actually the model is pretty good. So I think that the success of that model should be able would probably translate into like a scale up version of a model with even better capabilities and having a similar architecture.
20:10 >> Yeah, let's talk about the postpanding pipeline. I mean maybe as a bit of a backdrop actually before that we can talk about the capabilities but the first time I remember people talking about this was I'll show a tweet from Tor taxes which then got Elon to respond which then got Gia Tang the founder of Zai to respond you know when do you think China is going to get a mythos class model this is back in June when GLM 5.2 2 had been out for a little while. Elon responds probably Q1 and Geotank says won't take that long. I made a comment on I made a comment on one of the episodes we recorded yesterday where I said to Dan to me GLM 5.3 is a frontier cyber security model because I get classified out of using the entropic or the open AI models even though we're in the CVP red teaming program and we're doing stuff with the neo clouds to help them to this because it's not even the classifiers it's the stuff trained into the model that I can't get around and so I fall back to using GM 5.3 and I need to be more creative in prompting if I want to get the frontier models to not refuse me and pollute the whole context window where you have to like rewind if you want to try and do anything again. So anyway, all of this is accomplished on I can show benchmarks and whatever but like it's it's an incredibly capable coding model especially for cyber security tasks and it's open available anybody can download the the weights.
21:35 This is not the biggest model on the market and all of the improvements from 5.2 to 5.3 were done through post training. So I mean I I kind of agree just keep post training going but what what are the unique aspects of their post training pipeline that you think are letting them do this? >> Yeah. So yeah just to comment on the the tweet I think Jay Tang the CEO it's pretty pretty vindicated right cuz it took only like a month I think after that. So can't believe Elon got out Elon by someone. So like out Elon timeline. the only thing Elon is well not the only thing but >> the thing that Elon is most known for for aggressive timeline but but but yeah so >> I think this is mythos class right now but it's or fable class but it's certainly like approaching there it's nipping on their heels pretty getting pretty close I would say right >> yeah true so yeah technically it's definitely not mythos class but I think it's still non-trivial to get anthropic themselves to admit that 5.3 is what do they call it? Fable class. I think fable near fable level.
22:47 >> Let's go find it. Good point. I I'll go find that blog post from them. >> Yeah. So, yeah, it's I think it's a non-trivial feat all within like a couple months of post- training on the same model. And yeah so the in terms of post- training pipeline I would say among all the Chinese model labs I think ZI is the one that invests most heavily on post training and focused really laser focused on post training. as you can see from other model labs where they change the architecture every like every once in a while, JLM has been focused on this same architecture starting from GLM5 despite you see changing already two iterations of this the since DSA Kimmy for the past year keep releasing new versions of their attention and doing scale up experiments. you say Andling doing all kinds of linear attention as well.
23:51 Stefon doing also very cool things. Mimo being even more conservative on that side. JLM has stuck with the architecture mostly while focusing on post training. And the I would say the the techniques in general aren't too far off. The basic techniques of their post training aren't too far off from actually the Deep Seek V3 which is still GRPO based. But I I think GLM the post- training pipeline they have a very clear steps and reasoning to how they adapt their algorithm along with the efficiency they need. So for example the first step they use a typical gpo for general post training. The first step they do it in a synchronous version in a synchronous way so that they could get the most stability out of the training.
24:42 So that will be yes the the first the first part. and then after after that because they would need they would need to further extract the system efficiency out of training and also because agentic workloads they are typically longer in the the sequence length and gpo has always I guess RLVR in general has always had the weak training signal issue. So as your your sequence gets longer the the reward signal the strength of the reward signal gets diluted through the sequence roughly speaking. So they because of that they the second stage they changed to a a more reinforced like slightly different variant of posting algorithm where you could tolerate other issues of the system for example like policy staleness so that you could use older version of the the policy or you could use just the most recent version of the policy instead of having to deal with a lot more racial policy stillness racial related issues during calculation. Yeah.
25:51 >> So, give me a quick summary. This is algorithmic related decisions that you're describing here. These are maybe unique decisions that they've made, but nothing I've heard you say, you know, before this talk or now would lead me to believe that Chel 5.3 is doing something so unique on the algorithm that it's so completely different to any of the other labs that they have a big advantage on in terms of like algorithmic work in the post training pipeline. Is that fair to say?
26:18 Yeah. Yeah. I guess what I've said so far is not too unique to to ZI. Yes, that is correct. >> Okay. Actually, I I want to say what I feel about GLM 5.3 when it first came out like do with the know it's completely based on the post training with the pre previous models. I feel like this is like a wakeup call for me. So, so if you see the benchmark results of GLM 5.3, it's a gentic results on the benchmark was improved it more than 10% by just doing pulse training and you know it's more GPU efficient if you're doing pulse training than pre-trained model from scratch and I think that's a wakeup call for a lot of other company as well that they don't have, you know, enough budget to train a model from scratch but they also want to you know improve their company workflow for some specific area and they happen to have that proprietary and and private data. So they can just like do duples training themselves and achieve a better result than some of the close wars models. If I if I remember correctly that GLF fun 3 was focusing on a genic reasoning and general RL and we see all all the three area has improved quite a lot by looking through the benchmark newslet. So that that's like the the day that I saw the Twitter where GLM 4.3 cames out and the the benchmark improvement I was like no way just download this by post training that's quite quite amazing to me. Yeah.
27:53 >> Yeah. It's okay. >> Mhm. >> Yeah. Go. >> Yeah. I totally agree with on how just how how much juice there's left to squeeze by just post training and yeah 5.3's post training effort is definitely very impressive and also cyber security 5.2 in our article I think we also compared JLM 5.3's cyber capabilities with 5.2 too. And it's like very notable. It's a like a night and day capability difference. And that's all just post training. So, so yeah, the the part unique about 5.3 I would say to be fully honest, GLM hasn't released like a detailed technical report since GLM5. So, working based off of talking with researchers and the the blog announcements, I think there are two things that are most notable. one is the one that I actually published results about or research about the they're doing PO style training. So they're reverting in some sense reverting back to the typical RO training format having like a critic model grading rollouts along the way and co-raining the the critic model along with the actor the the main model. So that's they they call it single rollout asynchronous optimization.
29:17 And the second thing is they in their public speeches and in the the conferences they talk a lot about thank you for putting down the meme by the way. >> They they talk a lot about Thank you. I I don't know. It It's It's probably too Taiwanese too too local. Like even my like Chinese friends actually like the the slime authors and just Chinese community, they don't even understand that unfortunately.
29:50 >> Maybe explain that a bit. Kimbo, it's it's so funny. >> Do they >> Wing, have you watched Swords Arts Online? >> No. No, I haven't. But but I I've saw your explanation on like that was that was fun. Did they >> I I I explained it. I thought I I don't know. Okay. >> Do they understand this meme? Kimbo 10 15 something like that. >> I don't know. no one I haven't asked anyone about that yet, but >> Okay. But I I jumped in earlier and I said something that might get misconstrued here, which is like they didn't have necessarily super unique advancements in algorithms. it and you just said the word slime which I was waiting for like the both the single roll out asynchronous optimization but then also all the infrastructure seems to me like work related to the efficiency of the training that just allows them to run a bunch more experiments potentially much more jobs potentially just like much more compute efficient jobs which is itself an advancement that sets them ahead of anybody else with like equivalent compute or even more compute who just like can't do as much research as them.
30:51 >> Yeah. Yeah, definitely. Slime is highly regarded in the in the community in the open source community. They're probably arguably one of the best if not the best design architecture design of postraining framework. And yeah, the the efficiency I'm I'm sure they have like very good infrastructure to support that. They also JLM the ZI they also released an article I think probably not too long ago two weeks ago or something about how they use their models to co kind of collaborate with models to further help them improve the system efficiency. So that includes bringing up the verifying the GPU kernels for their crazy GLM 5.3 flash model on an unknown hardware. So on the I think it's the I'm not sure what GPU exactly what hardware exactly but not the GPUs. So they have unknown hardware with different different architecture. They they use their models to improve that and they have ways to improve the the loop the loop of improvement and also monitoring system level jobs how to further squeeze out more efficiency out of the whole infrastructure. So, so yeah, I I think the slime is definitely algorithm wise SAO is a clear innovation which is applied in 5.3 and 5.2 2 and 5.3 and then infrastructure based on even just on what they open sourced. It's a very good improvement and definitely would help with >> and let me throw one thing in here that I think a lot of people have missed or is at least something I've been focused on. If we go back to the GLM5 paper, >> I want to read a paragraph from this.
32:46 and remember this is like almost a year old at this point. they say last one more technical contribution lies in the fact that from the first day GLM5 is full stack adapted to Chinese GPU ecosystems. We have successfully completed deep optimization spanning from underlying kernels to upper level inference frameworks across seven mainstream domestic chip platforms including Huawei Ascend, more threads, hygen, meta x and nflame.
33:17 When you're talking about the model helping bring up the infrastructure for serving GLM5, it's not necessarily the case that this is just Nvidia GPUs and kernels and the serving stack that we're talking about for SG lang on the AMD or Nvidia GPUs that we test in inference. They're serving this model at least they claim for many months on seven different domestic Chinese accelerators. What do you guys make of that? First of all, they also publicly claimed that 5.3 Flash the when it's under the name OX Alpha, they're fully served on Chinese hardware. So that's because I I remember when people tested it, they saw ops alpha being very good while also being able to serve I think a 100red trillion or something like a ridiculous amount of of tokens compared to I guess based on the the amount of tokens that they can serve for free. People thought that's it's impossible that it's served on a GPU by a Chinese app because Chinese apps aren't that GPU rich enough. But turns out they're using Chinese hardware. So I think that assuming assuming everyone's honest, I think it's they they're able to serve it like pump just plum through all the the blockers like the the basics of the the kernels, the systems, the plum through everything and while serving those models efficiently.
34:44 >> Yeah. Here's I've got on screen ox alpha was a stealth model and open router. it was only >> Mhm. >> a month and a half ago, August 20th when this happened. It feels like >> yes, >> a while ago now. Much longer. But anyway, things move. >> They did they did bring up I think I think one month or two, I think. So the the 5.3 flashes cuz yeah they released 5.3 on early August and then they released 5.3 flash in late August.
35:20 >> So that means that even if they decided to that the problem is when they start when they start developing the 5.3 flash model if we have to like kind of upper bound lower bound it's probably won't be too long. It's probably like 2 months or 3 months as far as I can tell because betting on KDA is still a big bet before Kimmy K3 comes out because no one really knows whether KDA scales that well and KDA I think Kimmy K3 came out like mid July.
35:52 >> So probably the most aggressive timeline would be they trained 5.3 Flash like within within two months. If that's true, that that's actually pretty crazy. Yeah, >> it is really crazy. >> One one thing I want to add about the the hardware side of things is that I think I also wrote it in the article, we can kind of guess. So, first of all, the arithmetic intensity of the attention operation during decode is typically matched with the arithmetic intensity of the hardware. So, this is shown most notably in Deepseek V3, which Deepseek openly talks about it. So that that being said there is some sort some sort of hardware and software code design in this sense where you try to change the model the configurations of the model to fit the hardware properties of the GPUs that you serve. So Deepseek fitted for H800s but then so it got me curious and then I I tried to calculate the the number for GLM5. One of the most interesting changes they did is they changed the number of heads and also I can't remember the exact configur configurations they changed but because of the changes the arithmetic the arithmetic intensity is very different from the the hardware the the arithmetic intensity for the Nvidia GPUs.
37:12 >> Yeah, I'll bring this up here. Yeah. >> Yes, thank you. So I think the calculation I did was somewhat around 120 for GLM5 or the 5.x series models and so if I and then I did some like bare basic calculations it somewhat fits with the more threads GPUs. So it's somewhere there. that does not prove that they're optimizing it. that doesn't prove that ZI is actually serving models on those hardware. It is doesn't really prove that it's optimized for that, but I think at least it shows. My conspiracy theory is that they're probably optimized for it, but I couldn't get a confirmation. So, Zi, if you're watching this, I would love your your thoughts.
38:02 >> We are high alpha here. >> Yeah, but but at least I think what we can >> Yeah. Yeah. Let's just calculate the earth intensity, right? But at least I think it shows that they're definitely optimizing for something else or they're sacrificing quite a lot of software hardware optimization for other reasons that are not shown in public cuz it obviously wouldn't work well on not the optimal for let's say B200's or H200s even. Yeah, >> it's a sign. It's a sign of what's to come that somebody might make a software level codeesign decision based on hardware that is not Nvidia GPUs. I think that's a quite a big statement.
38:42 >> Yeah. And I guess like adding on that that w with the capability improve of the models we began to see that the R&D cycle has been shortened drastically right and take a kernel side as an example you know now nowadays agent they can with the proper tools like kernel development agent or the cake just released recently with the right harness and the right tools people get to write kernels that performs much better than human themselves. And we also got like leaderboard that agent writes kernels.
39:18 As long as you have the budget, it will write faster kernel than humans. And actually it's like pretty interesting to see how different model company going to you know doing compete with each other with the help of agent. So it's like human plus AI to do the compet do the compete. Yeah. How much how many tokens do you think the AMD tile RT mega kernel for GLM 5.3 cost them to produce? Must be a lot. I think what do you think Kimbo?
39:47 >> I I don't have a good idea honestly. Yeah, I wonder. Yeah, maybe we should ask. >> Yeah, maybe we should up with good ideas here. Okay, guys. Anything you think that we've missed talking about this? I think we went through inference X performance results some high level discussion about the cyber capabilities of this model what we're using it for high sparse high cache you know KB offloading model architecture post training pipeline what else is left rumors and and and vibes on what's to come with GLM 5.4 four. What if what if you want to speculate as we sign off here?
40:21 >> Yeah, I I think we we briefly talked about it. The the 5.3 Flash is definitely a success for them. They're able to mix DSA with KDA. So, while like a very new architecture while still having a very great model, having great capabilities. So I I suspect they can definitely scale up 5.3 flash the the model architecture of 5.3 flash to a 5.3 level size and probably even squeeze out even more performance or model capabilities. Yeah, I think 5.3 flash has that crazy efficiency as well, right? cuz they're linear attention and sparse attention. So, yeah, >> I'm curious about what they're going to do next cuz I think JM they they definitely explore something else by post training and they show us like you can improve model cubably but not only scale up but also squeeze up given your current parameter size even though you're not trillion model size you can still perform as well. And for those company who do inference for for the model and they're heavily sensitive to the cost, I think it will be definitely a good call for them. Actually, we've seen like in in the Bay Area like Harvey had has their own post-trained agent based on Kimi model. We also see periodic lab maybe Kimboy can extend that has do the post training on AI for science. So I'm really excited to see like how this train goes and especially what GLM 5.4 will brings us.
42:00 >> Yeah. One I guess one last thing I think about post training that is very important but not very much discussed and ZI also is not willing to clarify more either is human effort. last time in their public like in like conferences and things they talked about they spent a lot of human effort into supposedly data when training 5.2 2 and 5.3 and the problem is what kind of human effort what kind of well human effort I would say I'm pretty sure it's something related to data labeling but the problem is what kind of data labeling and what kind of effort into data labeling and yeah that's one like I would think it's also pretty important when it comes to post training that they're able to get quality data >> makes sense so what do you think comes next you think we get space bunny alpha Is Miniax and 3.1 Flash? Do you think it's GLM? You think Kimmy comes out with something new? Deep Seek. We got a big release from Reflection. I don't know.
43:03 What are you guys looking forward to most with the model releases coming out soon? >> Expecting some smaller I'm looking for some like efficient model. Actually, Reflection's model is it's really efficient. I'm I'm thinking like maybe in the future we have large size model, right? two trillion, three trillion, and it's capable enough for people to do powerful stuffs. But we also have models that offer good enough performance, but it's like extremely efficient that company won't care about its cost and still do some certain thing well, like reflection that they release their models. Makes sense, guys. Okay. Well, thanks for coming on whirlwind tour of everything GLM 5.3 and why we don't think model releases like this mean people should sell their memory stocks and why we expect Jeans Paradox to continue to result in more people consuming a lot more tokens and a lot more HPM and a lot more bandwidth over time. Thanks for joining guys. Great job.
44:01 >> Thank you Jordan. Thank you. >> Thank you Jordan. Thanks everybody.