Can you run a frontier model yourself? | First Pass Ep. 5 with Simon Mo (Inferact)

0:00 Given this is a almost three trillion trading parameter class model, the model weights itself is a few terabytes, right? Just you think about that and it requires a minimum eight Blackwell B100 GPUs or AMD or TPU equivalent. >> We're back with our next episode of First Pass with Simon from Infract. >> Good to be here. >> Awesome. Thanks for joining us. You are our first repeat guest. We had you on a few weeks ago talking about GLM 52 and here we are a few weeks later with the next the next huge open weights release in Qwen than than you to to come on and talk about it. Maybe at a high level just like walk us through this model. Like what what stands out to you about it?

0:44 What is it good at? What is it bad at? You know, what would you what would you highlight about the model? >> Yeah, Jamie, good to be back. So, if you think about it, model just keeps releasing and when we first heard about Qwen 1.5, we really feel like it is a milestone. Why is it a milestone? Because first of all, it is a big model and it is a very very practical model that will score high on the leaderboard, right? Over the last few weeks there's a lot of hype or discussion built around this model. It is a 2.7 2.8 trillion parameter model. This is like first of its class that came out this big and it is above or equal Opus 4.8 and even sometimes matching or exceeding Fable 5 level. If you don't believe me, just look at Arena. Arena has all sorts of data, all sorts of task, front-end web design, agent decoding traces or like you do something like a a just general knowledge work or chat oriented workload.

1:47 On Arena, you can see Qwen 1.5 officially has some really really good ranking. And now what we mean here is we have a frontier intelligence that are open weight available. >> Yeah, wow. It's pretty awesome. And you wrote a great blog post on it the other day. We'll We'll link to it in in the notes. I think everyone likes to think about open weights models is oh, they're way cheaper. We're going to you know, get intelligence at a at a really low price.

2:13 And you know, K3 isn't isn't really that. It's a huge model. It costs more. I think what you argued is it's it's less the cost that matters with these open weight models, but it's more of the operational freedom you get. Maybe talk to us about like why open weight models are important and and what like what what what is this freedom you get when you use them? >> The freedom is a lot, right? There are of course just the the classic things people talk about security, compliance, and you can you there's you can configure your own guardrail and all that part, but it's also about just be able to control the performance and cost profile of this model. What do I mean by that is I have seen company now trying to deploy K3 internally for the coding agent workload. And their initial thought is that okay, maybe can use this to replace Opus and Fable. They can, but additionally, they can also go so much far and beyond for some of the most premium expensive task they would like to maybe have fastest token available.

3:13 You can run 300 400 tokens per second per user today with VRAM on this K3 model on a on Nvidia GB300 rack. And for that purposes, maybe for some of the most challenging project when someone is pressing against deadline, we can get this level of fast token, which is faster than fast mode across OpenAI and Anthropic, which is typically about 200 or 150 or 100 tokens per second. This just accelerate and your freedom so much. And on the other hand, if I have lower priority batch processing job, you don't need a very very fast token. You can have the largest throughput available. And this kind of operational freedom and this kind of cost predictability is something that really, really matters when you're becoming owning your own frontier intelligence.

4:05 >> I mean, this is definitely one of the benefits of open weight models that you can, you know, it becomes part of your engineering surface and it gives you a lot of different degrees of freedom. Could you expand on some of the advantages of that and maybe some of the disadvantages of that? >> Of course, the advantage, as I mentioned, is very much you can leverage it. You can build on top of it as well. You can fine-tune it. You can also leverage the whole ecosystem that's going to operate and optimize against this model to expect improvement overall, right? So, for a proprietary API, typically you don't expect the cost will drop over time or it will maybe performance you will have some degradation, availability will have some issue. But for owning your own intelligence with this model, you can just do so much more. You are you will be able to really be able to control and understand what's going on and have a clear expectation of it. But then the downside, of course, becomes, well, it is actually fairly complex to set up. It is non-trivial, unlike other previous open weight model you can run on your laptop, on your Mac mini, or even sometimes on your phone. This is a true data center ready architecture. You need to run it in the data center. You need to work with deployment inference partners to set it up. But fundamentally, it's about freedom and flexibility and the choice you make.

5:25 >> But maybe real quick, what are some of the, you know, maybe just to get a little in the weeds, what are some of the specifics about what makes it hard to deploy? >> Given this is a almost 3 trillion parameter class model, the model weights itself is a few terabytes, right? Just think about that and it requires a minimum eight block will be 300 GPUs or Nvidia TPU equivalent. And this is a naturally a multi-node setup. So, for any practical deployment, you're going to need to orchestrate a few nodes of the most expensive GPU out there. Number one is availability. Can you get it up?

6:04 Number two is reliability. Can you make sure it runs 24/7? Number three is optimization, right? Just be able to run this model doesn't mean you can run this model efficiently and under a cost cost target. And then number four is, how do you continue to optimize against it for the workload? Because all these are what Frontier proprietary API is able to do for you without you even worry about it. But now, if you're coming to only your intelligence, those are all very operationally heavy, but we're pretty glad to see the whole ecosystem is ready for it. So, yesterday upon launch, you have 10 or even more inference partners just sign up and be able to run this model, and really helps the ecosystem.

6:47 >> Yeah, well, one of the other advantages is data retention and being able to set your own data policies by controlling the model weights. How are you seeing your users and customers deal with that? >> One famous example now is for the model you especially deploy internally or have a dedicated deployment, you can enforce zero data retention typically with a provider, right? But even if you're running yourself, you're able to have the freedom to collect or not collect, and very much have the true attestation that just there's no data leaving the premise.

7:24 And this is in comparison to company sending their data to OpenAI or Anthropic, and really worry about a potential downfall or even just avoid like outside of contractual obligations, right? If you look at table five, your data must be retained for duration. While for your own intelligence, you can control however you want. And then finally for guardrail is about can you use this model for the practical use cases. When Hugging Face is investigating the security incidents that involve OpenAI, they couldn't use OpenAI model, they couldn't use Anthropic model, they have to turn to their own self-hosted open weight model for help to investigate the security breach and be able to effectively identify what's going on. And this is where having the flexibility and having a way to really bypass the guardrail and or even set your own guardrail there will be very useful.

8:20 >> Yeah. This is a hard question to answer, but I'll ask it anyways. So, do you think these these models are more or less secure? Right, on one hand they can be more secure, but there's also a lot of pitfalls you can have to if you're deploying them yourselves to make them less secure. Like are they inherently more or less secure or is the framing of of my question wrong? >> Models fundamentally are just right, taking your input and generating text as output. And most of the responsible model lab, I would say, they are very serious about their red teaming effort and they're trying to make sure the model is right, really helping people achieving something safely. So, but in the end I think it's about setting your own guardrail, right? What even when people are using Codex for example, OpenAI Codex, there's still example of Codex just deleting the whole directory or even home directory. So, in the end it comes down to how is the developer trusting the model and how good is a harness and then how is everything a whole system composed together in the inference infrastructure to make sure that the output is secure and overall.

9:33 And the model will do its best and then our side inference engine will do its best, but in the end it's a end-to-end system problem rather than each component. >> Yeah, yeah, yeah. That's a good way to put it. You know, one other thing I want to hit on is you know, we we we've seen open source before. Open source has existed for decades. And one of the interesting things that I've I've observed and I think the whole world has observed over the last, you know, call it 10 15 years is the evolution of the licenses. From fully open and permissive, you know, Apache 2.0 MIT license to, you know, the equivalent of like a source available license. And we saw that with Elastic and and Kafka. And you know, kind of all the big open source projects today.

10:14 It feels like the K3 model had a similar, almost like source available license nature to it. Could you talk to us a little bit about the the license they chose and and some of its implications? >> I was a little bit as I a little bit said it today, I was unreasonably excited by this license in the sense that of course I'm not legal expert, but my interpretation is a this is the first time I saw a explicit call out of model as a service provider in the licensing term, right? So, what the Kimiko stream license is says essentially from my interpretation is you can free to use it however you want internally for research purposes and just general evaluation and be able to use this model effectively for your own work. However, if you choose to commercialize it, especially with derived training services or inference services, this is what they're termed model as a service provider, you will need to discuss carefully with the Moonshot team to be able to have a commercial collaboration. And this part is from my point of view is what's going to help the open source ecosystem so much because training the model is is very, very expensive and establishing this kind of commercial relationship early on and be able to sustain the model training efforts through the model open sourcing and also a very clear distribution of a distribution of revenue and sort of that aligning incentive will help so much.

11:42 And the reason here is the model inference cloud, right? The inference cloud and the model training as a service, they are doing heroic work to build on top of this model. But the base model also matters so much from the what intelligence point you're starting from. Everybody is playing and contributing to the end result of what you can see today in the end API token pricing, right? So, if that can flow back, not just stopping as the GPU, the cloud, and the a model service provider, but also a portion of you can flow back to the model trainer or the model lab, I think that's just such a much more closing the loop moment for the open way ecosystem.

12:28 >> Maybe final quick hitter here. So, does this spell the end for the big labs, the Open AIs and the Anthropic? Is open source taking over? Is is there a is there a place for both of them to coexist? Is is this a positive sum world or a zero sum world? >> Definitely positive sum, right? We also saw the Open AI signing and Meta signing the Open Way model pledge and alliance that came out a few days ago, which VLM UFRAC also signed. It's really we see open source and open way ecosystem is helping the whole progress, AGI progress forward. Why?

13:03 Because this movement really build a racetrack for everybody, right? It helped everybody to see, "Oh, wait, this is what Kimi have done well." And here is what maybe MiniMax other model has done really well. And then you can see how Kimi and MiniMax can learn from them. GLM, maybe GLM did something better here. And so, everybody is building on top of each other's work. And what does that mean? Just accelerate the whole progress so much, right? And even with Kimi as one of the first model app to train use a state-of-the-art machine learning optimizer, Muon optimizer at scale, just their learning and sharing, I'm pretty sure it's also benefiting OpenAI and Anthropic, right?

13:46 And this really helps the whole ecosystem forward. And of course, this is also a call for OpenAI and Anthropic and others to be able to open up a little bit more, right? GPT- OSS has been more than 1 year old. Let's maybe we'll see the next model from OpenAI coming out maybe with some open source soon, and really, really hoping for it. >> Yeah, awesome. Well, Simon, as always, thanks for sharing your thoughts. >> Thank you so much.