Why Does China Keep Releasing FREE AI Models?

0:00 on the China side, DeepSeek's people have talked about how they're not profit-driven. So, whatever revenue they make, they want to put that money into R&D. Given that they have making their money and I >> You mentioned a few times, and I think we should talk about it, the benefit of doing open source, right? And you called it a share R&D sort of effort here. And I remember after DeepSeek came out, I spoke with somebody who knows their stuff in AI, and they were basically like, "Look, with open source, it's every open source research house working together. When you're building a closed model like OpenAI or Anthropic you know have, you're basically building on your own.

0:42 I mean, of course, they can bring in the open source innovations, but it's just it's basically them against the world, where open source you kind of have the world against everybody or against the closed models, to be more technical about it, more more accurate. So, just talk a little bit about that cuz I think that's an important point. >> Yeah, I think a lot of even just bringing it back to the kind of the narrative around China versus US AI right now, I feel like it's really about open source versus closed source at this point, right? And I think it was really humbling to even see this morning Kimi released their weights yesterday, like last night in Asia time, and in their opening paragraph, they talked about how like we are still behind the leading most frontier, but we are inching towards it. Basically, something like I paraphrase it, but it's something like that or something in in those realms. And I think what it really shows is open source is able to somewhat now play catch-up because they are leaning into basically everyone's intelligence or everyone's R&D. And I think it's really played a large role in propelling China's open source ecosystem, and it's something a lot of the leaders we just talked about, like Baidu Ma Fong or Yang Zhiling, have really I think embodied as well as rallied behind. It's essentially in the beginning it was like a branding strategy for a lot of these labs and when I spoke to them because they said, "Look, if we want more developers on our ecosystem or on our using our our APIs, they need to know what's out there especially since we are Chinese, frankly. And they're like, if we don't put out our R&D, you're going to have a lot of accusations of this and that, but we put can be the judge of it." So, that was the initial kind of starting point of open source and how it got I think a lot of users, especially startups that might be more cost constrained and less compliance, you know, worried, getting on these labs.

2:37 I start getting on these models. And since then, essentially it's become like a nice virtuous cycle because the more developers on it, the more you learn about, you know, the, you know, the the use cases and whatnot. You can tweak it, you can make it better. So, that's really kind of the original, I think, goal was to just sell their models abroad. Now, I think from there it's really become an unintentional unintentional consequence where, you know, the learnings of each lab will now serve as essentially open learning textbook for each of these labs. And they will openly congratulate each other. I believe I think when Deep Sleep came out with something Ziai even retweeted them on Twitter on X saying, "Oh, congratulations. This is such like, you know, genius work, blah, blah, blah. Like, we will incorporate it in our in our own, you know, our R&D and our own infrastructure layer." So, so, essentially, you know, that that's really what's been driving it from the commercial sense. And I think on back on the cultural sense, you know, I think a lot of academics and researchers actually really like to be followed, cited, and it's part of that kind of academic loop. So, the people who want to go into private, they need to be cited to go back into academia.

3:46 And this is something I've spoken to I've learned from speaking to a lot of for academics. and this allows them to kind of have their work in public. So, all of that has garnered a very strong open source philosophy base in China. >> Now, a very interesting thing has happened in the ensuing days. and we're talking Monday, July 27th. This will come out Wednesday on the 29th. but this is all live and this is happening. the you if you looked at what the advantage of of let's say US and China were, China of course is this open source ecosystem. The US was leading or is leading, I would say still leading, at least when you when it comes to model intelligence. And the advantage that the United States has had is these closed you know, AI labs, OpenAI and Anthropic that have pushed the frontier forward again and again and again.

4:39 and you would think that if you were thinking what's the strategy going to be for US companies, it would be almost a fear of open source and a rededication to closed. But the exact opposite has happened this week where you know, basically led by Jensen Huang, the Nvidia CEO, seemingly every US company has come out in favor of open source. even OpenAI has signed on to this letter saying we shouldn't ban open source. Then after like days of silence, Anthropic basically had to come out with a statement said, "Hey, hey, by the way, we never called for the banning of open source." So, so Grace Grace, just help us understand what do you think about the fact that if open source is China's big advantage, how do you then explain what's going on in the US where all these US companies are coming out in full-throated vocal support of open source?

5:34 >> Well, I mean, there was a huge 180, right? And I think it was quite interesting. but you know, going back to what something we just touched on already, I think in the last year, low-key a lot of companies have been building on these open source models, right? Because essentially, if you self-host these models or you use them through these inference service providers like Fireworks, these models become yours or American, if you want to put it, right? So, the whole fear-mongering around does the data, whatever, go to China doesn't really That narrative doesn't really work anymore. So, then it was very interesting because I think at the day it was kind of like, "Okay, if we don't open source and we build the strongest open source models in the US, then actually this pie is being eaten by someone else anyway. Like, it's not like we can stop people from using Chinese open source. And even despite, you know, Kimmy K3 being said as a token hungry model, it's not as cheap as other Chinese models. The task per, I think, the task per token usage is still quite high. It's still cheaper. It's still cheaper than the most frontier models.

6:33 And it's inching towards frontier. So, it's almost like I want to cynically say it's a business decision. That's one aspect from the labs. So, it's like, "Well, we now need to compete in open source." You know, back then the margins were extremely high, right? And I think, you know, I I can't comment exactly how high the margins were because they're not disclosed, but, you know, on the China side, Deep Seek's people have talked about how they are not revenue driven. So, whatever money they Sorry, they're not profit driven. So, whatever revenue they make, they want to put that money into R&D.

7:04 Given that they have the quant fund kind of, like, you know, making their money and I think the philosophy of the founder is not so commercialized. So, I think there was like a bit of a push for open sourcing. Then beyond that is something going back to what we also touched on earlier, which is why should companies continue to pay for the best intelligence when the intelligence are kind of taking away what makes a company special. So, then you see companies more and more mindfully saying, "Okay, we shouldn't actually give all our data to Claude and GPT and then in return pay for intelligence twice." That's what the Microsoft CEO said, right? So, I think there's a bit of I think I take a step back, I'll understand where people's perspectives are coming from and what their own goals are, right?

7:49 >> That's right. Okay, I I actually want to get to what the fact that I want to get to the fact that you can now get open-source models to do a similar job as some of the frontier models in the US or close to the frontier models for a cheaper cost. What that does to the AI competition. But before I go there, you know, we're we're we're not even 30 minutes in, but we've made it 20 minutes in and I haven't brought distillation yet.

8:13 and so, I think a lot of our listeners, if you've made it to this point, have will are probably saying to themselves, well, Grace, these are nice explanations, good culture, and you know, specialization, and I can buy that at a surface level, but we we do have evidence that the labs like DeepSeek and and Moonshot have distilled. So, basically, taken the essence of the big LLMs from Anthropic and potentially OpenAI, and you know, quote-unquote taken the IP of these closed labs to build their own models. what's your perspective on that?

8:53 >> Yeah, first of all, I want to say, obviously, no one has come out publicly saying, "Hey, Grace, I've distilled a model." So, I just want to publicly say that. >> >> >> Right. There's it's more like there's there's like some research that indicates that it's happening, but like neither of these companies have said they've done it. >> Right. Right. >> So, so, I think it's really interesting. And again, just this week I was listening to the All-In podcast. I'm sure you I think even Freeberg and Sachs, these people who were quite I wouldn't say anti-China, but had a tough tough stance on the national security angle on China AI, were saying, "Look, distillation is a practice that's been widely used in product iteration and R&D, even, you know, in its Google's days." I think Freeberg was saying that.

9:31 But beyond that, what something I found the most enlightening, the framing was provided by Google DeepMind's Yutian Yu. He's a researcher at Google. And he said there's smart distillation and dumb distillation. Dumb distillation is something what, you know, the layman think of when distillation happens. Essentially, you know, I literally take, you know, model A's answer and plug it into model B essentially. And then model A will just spit out what model B will just say what spit out what model A said. And it's so obvious. It's like copycatting. And I don't think anyone is quite literally doing that because frankly, it's just too unsophisticated.

10:05 Now, there's smart distillation, which is kind of operating in the gray area. It is again not IP theft. It's not breaking the law. However, it could be breaking what is considered, you know, you know, your own terms of services. I think labs should be doing better, you know, KYCs. But in that sense, essentially, it's like what enterprises are doing in fine-tuning their own models. So, say if I'm a enterprise and I host self-host an open-source model, I'm going to fine-tune it with my own data. I'm going to build harness around it. I'm going to make it the best model for my own use case.

10:38 And often I also use the frontier model to guide that kind of less frontier model in terms of getting its homework done and sort of kind of getting to the right direction or even for a synthetic data training. So, when you're looking at smart distillation, it's a lot more murkier and it's really hard to say what is right or wrong. And I think it goes back to the mainstream narrative that we're hearing right now, even the happening in the US. What is distillation when when we also are hearing potentially, you know, the thinking machines model distilling on Chinese models.

11:09 So, it's quite funny where everything's kind of going in full circle. And it goes back to my point of open-sourcing R&D, where it's really open-sourcing R&D really pushes the whole industry forward as as a unit. And if you really believe in AI is really going to help and transform our economy, how we work, our basic infrastructure, then pushing it forward together, propelling together, will make more sense. But if you believe this layer of models should be capturing all the value and you should be selling intelligence, then of course you want closed model and you don't want people distilling your models.

11:46 >> Yeah, let's let's read from Nathan Lambert one more time just for the heck of it because I think he has a good perspective here as well and you've you highlight this actually on your sub stack, so I think it was sort of downstream from your finding Grace. He writes, "It is clearly the strongest open model ever released." writing about Kimmy K3. "It should be clear looking at this model that if adversarial distillation from the closed frontier models in the US contributed, it is at most to a relatively small degree." "AI observers who followed the distillation panic and came away with the wrong conclusion, the Chinese AI labs are only producing good models due to IP theft, are in for an awakening. The Chinese companies are extremely good at building models in the same way the leading American companies are, which I would say is even more remarkable given the fact that they can't get the latest Nvidia chips, right? So they have to work off of generations previous."

12:44 >> They couldn't get the latest chips and they couldn't get, you know, access to Fable, but the window was too short for when Kimmy K3 came out. And I think look, Nathan is the technical expert here. If he believes in that, I I would really, you know, trust his judgment. I do really appreciate and respect his work.