E26: NVIDIA Just Changed The Course of AI Forever

0:00 Today, you're joining me for a really special interview that marks 10 years of NVIDIA DGX and 20 years of CUDA, the software that makes modern AI possible. I'm joined by Charlie Boyle, vice president of DGX systems at NVIDIA and frequent guest of the channel. Charlie helped shape the evolution of AI infrastructure as we know it, and you're about to get an in-depth look at the 35 to 50x power, efficiency, and performance gains from Blackwell to Rubin and the DGX features that helped make that possible. I asked Charlie every question I could think of, and he had some surprising things to say about the future of AI factories and data centers. Your time is valuable, so let's get right into it.

0:45 Charlie, I'm so happy to be able to speak with you again, and I'm super excited to talk to you about all things DGX today. So, I know it's the 10th anniversary of DGX and the 20th anniversary of CUDA. So, let me just start by asking the absolute basics. What is a DGX system to begin with? Yeah, so it is the 10-year of DGX. We started 10 years ago today in this building. Oh, wow. A little bit different setup there. The DGX 1 was behind velvet ropes. It was our first AI supercomputer, and the mission back then is still the mission today, which is to take the best technology that NVIDIA has, build a system with it. Back then, it was one box. Now, it's a giant data center. But, to build the system, to make a vertically integrated software stack that makes AI easy to use for customers. 10 years ago, it was all about researchers. You know, can I get the first-generation AI models working?

1:44 But, now it's all about how do we make AI easy to use, cost-effective, and really deliver business value to customers around the world. And DGX is just one implementation of that. We are the reference architecture that all of our partners around this show floor and around the world use to build their AI systems today. And what separates DGX from the other ways that NVIDIA offers their systems, right? So, this big form factor versus the bladed systems that we often see Jensen reference. Walk me through what makes DGX DGX specifically, you know? We build DGX, it's our reference architecture, because we have to take all those NVIDIA components, new GPUs, new networking, new power, build that into a system, and then share that design with all of our ecosystem, so they can build systems for all of our customers. Now, the system we're standing in front of is our current generation, our latest DGX B300, so the 300 series of the Blackwell generation.

2:40 On the show floor, there's plenty of Vera Rubins. What you saw on stage today with Jensen, which you referenced as the blade systems, that's our NVL72 system. >> Yeah. >> that last year. We You and I saw it together, our Blackwell NVL72. Now, we've got our Vera Rubin NVL72. The great thing is, from generation to generation, for all of our customers out there, it's the same chassis. It's just that compute blade that changed. Now, it got a lot faster. It's 35x faster. You know, it got a lot more memory, but not without a lot more power in that. So, we're delivering tremendous new performance all in that same footprint. And the reason that we're building those, it's not just so we can sell this to customers, that we're the reference design, it's to help all of our partners. I'm looking in the background at the Dell booth right now. You know, I see our other partners all around here.

3:31 The systems that we build as reference architectures, they take out to their customers. So, there's a Dell Vera Rubin, there's a Supermicro Vera Rubin, there's an HPE Vera Rubin. All of those things started on our reference design that we built internally. I'm super proud of these. They're beautiful, they're gold. We help thousands of customers around the world, but they're helping tens of thousands, hundreds of thousands of customers around the world with AI. >> Yeah. And beautiful and powerful, right?

3:55 So, uh Vera Rubin NVL 72, you know, uh how many GPUs are in this one? So, in each of these there's eight. Yeah. And so, in these four there's four in the rack here. So, in this rack there's 32 GPUs. >> Yeah. In the Vera Rubin, in that same space, there's 72. >> Yeah. And they're all connected with NVLink networking. So, all of those 72 GPUs, there's actually 18 different compute trays in there. All act as one big GPU. And the reason you need that is for that massive agentic workflow that Jensen was talking about.

4:29 >> Yeah. You know, it's not just a chatbot anymore. I'm asking you a question. It reads a PDF. It's a whole workflow. Like, go build me a compiler. Yeah. You know, I need a system that's a rack level to go do that work and to come back in a reasonable amount of time that's cost-effective for me. Yeah. And that's the generation on generation efficiency that we have is every year that efficiency, you know, Jensen talked about 35x. Well, that's just not 35x faster. That means for the same job, for the same thing that was impossible or too expensive for you to do last year, it's now 35 times less expensive to do it.

5:03 >> Yeah. When somebody chooses between a system with 72 GPUs and 32 GPUs, what what's what's the reason to go with this system? So, it it's all about the specific workload and and where you are. >> Yeah. You know, we started our AI systems, you know, I don't build them as DGX, but our partners build them, you know, Yeah. put PCI cards into a standard x86 server. Uh-huh. Some of our customers, their AI workload works great on that. This eight-way form factor something that we introduced 10 years ago. The original DGX-1 was the very first eight-way system. And the funny thing was, 10 years ago, when I would talk to customers, their number one question is, "What am I possibly going to do with eight GPUs? Can I virtualize it?" Now, people have one application that takes thousands of GPUs.

5:45 >> Yeah. But the reason you choose one of these systems is where your application is. So this is a very standard form factor that you know every not only every OEM system but every cloud the eight-way Nvidia GPU server is up until the Blackwell generation the gold standard that everyone had. And then for new really large memory workloads cuz the the difference is these are four different computers but the applications that you run on that would just use the memory in one of these computers. With the NVL 72 you have the memory of all those 72 GPUs connected with one NVLink so your application can see that as one giant GPU. So whereas this your application would see eight GPUs together in a memory context the NVL 72 it's 72 GPUs. So I can do a much bigger agentic workload. I can do trillion parameter context. I can do applications that weren't possible before that technology. That's amazing. Actually let's let's double click on that for a second. So one of the things that got announced during the keynote was the BlueField 4 STX reference memory architecture right? So explain what that is at a high level and what that means for these systems going forward. Right.

6:56 So you know kind of going back in history when we introduced our A100 that was the first time and Jensen showed it in that history video that was fabulous in the keynote in A100 we started something new that we called the SuperPod. And so that was our first DGX SuperPod which was a reference architecture of a number of these nodes connected together with at the time InfiniBand plus storage. And so customers would buy an AI factory in that pod format. And that was 32 of these systems together and you kind of put those together you know to build your AI factory. Well as AI has gotten a lot more powerful it's not just enough to have just GPUs anymore. So what we talked about in the keynote was a brand new pod. So it's the NVL 72 systems.

7:44 It's our Vera rack as well because a gentic AI needs a lot of CPU processing power for all the sandboxing, for all the testing that it does. So, I need NVL 72 Vera Rubins. I need racks of Vera systems for all the compute work. And there's a new class of storage that's needed and that's what we did with STX. Very similar to what we did 10 years ago with DGX, we came out with a reference architecture for the industry to accelerate a new form of application.

8:14 And all this a gentic workflow needs high-speed storage context that can either store the context of what the workflow that you want it to do. It could offload certain things because AI, the power of AI, it needs data. You need to be close to the data. And so, we're working with all of our storage partners so that they can take their storage stack, all of our great partners like NetApp and Vast and DDN and HPE, their storage stacks, what they've got decades of investment will run on top of that STX reference architecture all in that same AI pod. And so, as enterprise customers looking to deploy AI, they're not going to buy STX from Nvidia. They're going to buy that STX design from the storage partners they're already working with today. Nvidia's innovating on the STX platform to help all of our storage partners to bring better speed, better efficiency, better token economics to that entire pod with the STX design.

9:12 >> Yeah. Help me understand what the STX design even enables. So, like from a workload perspective, if I'm thinking about running an AI agent before, I was storing a lot of that context in like high-bandwidth memory close to the GPU, right? Now, what does that let me do? Bigger workloads faster? Like help me understand. Do all of the above. When you think about the new a gentic workloads, it's more beyond, you know, the things that we were doing just even a year or so ago where a job would run for a minute, maybe 5 minutes. You know, one example that I think we all saw in the news was they had an energetic workload from scratch build a C compiler. That took a week.

9:49 Now, in that I couldn't possibly store all that context in GPU memory all at once. So, I needed something that was very close to the GPU, something that was very accelerated to store all that context to move things back and forth. You know, so that's one very long-running use case of it. But, the other part of accelerating, you know, your token economics on that workload, especially in today's, you know, storage world with everything that's going on in the market, if I can make your tokens process 5x faster because I'm putting that storage optimized closer to the GPUs, well, I can do 5x more work on the same amount of storage that I just bought. And so, it's it's not only great for our storage partners, but it's great for our customers who are trying to put all these things in their data center.

10:33 Less physical infrastructure means it's more power efficient, means I can use more power for processing. It's lower cost cuz I can get more work done with the same physical footprint. So, all in all it's a win-win, but it all builds in that same pod architecture. >> Yeah. Speaking of which, so power efficiency I think is something I'd love to talk to you about. Um, during the keynote, you know, there was a lot of talk about Vera, the CPU, and Rubin, the GPU. Help me understand how those two um new architectures, like, you know, the new Vera-Rubin architecture uh affects the DGX systems going forward. What is the performance jump from the Blackwell version of DGX to the Vera-Rubin version of DGX? So, you know, as as Jensen put it up in the keynote, you know, 35x on >> 35x energetic workloads. Now, the the funny thing is cuz last year we had a 35x as well, and even talked about it in the keynote, the you know, the SemiAnalysis, when they ran it, it was 50x. And they they What did they say? They said, "Jensen, you're sandbagging." Well, it's funny is most people think when we put out those numbers like that's the most cherry-picked number possible out there, but I see real numbers like that from customers even in the Hopper to Blackwell generation. I had a customer that was seeing a 50 to 100 x speed up.

11:47 And for them, that meant for the same system they had, they could get 50 more clients on that same infrastructure. Like So, they could serve more customers, bring more people on board at the same cost, the same power efficiency. And so, when you see that 35x in Vera Rubin, you can take that in two ways. Like I can do more work faster, or I can save a lot of money. And most of our customers do both.

12:13 And a big thing that Jensen talked about towards the end in the new DSX gigascale AI factory, uh he talked about dynamic power and Max-Q. And now, most of most of the people watching this today aren't going out and building a gigascale AI factory tomorrow, right? But you know, I've been in the data center industry for more years than I'd care to remember at this point, but in many decades. But what does everyone do in a data center when you're building it? You provision for the power that's on the nameplate on the back of the server. And what that does is you're over-provisioning the power because your entire racks and racks of systems are never running at 100% all at the same time. But for safety reasons, everyone says, "Well, no, you know, like it could happen." And humans can't turn the knobs fast enough if everything does happen hit at the same time. So, that's what we talked about in the new DSX design for gigascale, but that translates all the way down to a customer buying two racks of NVL472.

13:13 It's that dynamic power management that you just tell it how how much power you have available to you, and if one rack is using you know, both racks using 100% of that, great. If one of the racks isn't using all of that, it can speed up the other rack. And because that's AI built into the chip, built into the power management, brand new in Vera Rubin, cuz it's both the CPU and the GPU working together, that power slashing, I can make every watt I pay for turn into real tokens.

13:44 Whereas today, anyone would tell you with over provisioning, the average is 60% of the energy coming into the data center is actually doing useful work. That other 40% is over provisioned, it's heat loss, it's all those other things, because nobody ever felt safe pushing that limit because there weren't automatic controls. And that's brand new in the Vera Rubin architecture. It starts with the chip, it goes all the way through the software and the telemetry, so that as a customer, you set that number. We put things in the power systems, capacitors, everything needed so that you can feel safe for that. So, you're getting the value out of every watt you're spending. So, it's not just a 35x improvement in terms of performance, but it sounds like there's also like a 67% improvement in the amount of power you can use that you had provisioned, right? From that 60% all the way to the 100%.

14:34 >> Yeah. And because any data center operator would tell you like, "Oh my god, I you know, like, if you hit 100%, bad things happen today." >> Sure, yeah. But, if I can have those automatic controls and I can believe in it, and that's why we're investing. So, I just not only that pretty picture that you saw of DSX in the render, in the simulation, that's being built in Northern Virginia. So, we're going to build that and run that for our own use, but that same design, we can show customers not only that it works on paper, but that we're running that 24/7 at 100%. And when the public utility says, "Hey, I need you to not be at 100%," they can send us a signal and the system automatically reacts to that.

15:13 So, it's not only what we do in the data center, everyone's talking about worldwide power, that the interfaces, the things that we're pioneering in there aren't just for our own things like, "Hey, it's hot. People need more air conditioning in their house. Hey, data center, can you turn it down a little bit?" They send a signal and it automatically works and we're still optimizing the work coming out of this. Wow, I feel like that's really slept on.

15:32 I didn't hear I certainly didn't hear enough about that in the keynote, so I'm really happy you highlighted that. I think that's a really huge benefit to especially since most data centers are power constrained today, right? Yeah, no matter no matter what size you are, you only have so much power. Whether it's your home, you know, whether it's your data center, like you've only got so much power, but at a data center level, however much power you pay for, whether you use it all or not, you're still paying for it. And so, that's the tremendous advantage in the Vera Rubin generation. We had to put a lot into the hardware itself. We had some of it in the Blackwell generation. We could smooth things out a little bit. Yeah, we talked about it last year. That was a new innovation in the power shelves, but now it's all the way from the chip to the power shelves to the rack to the data center. That's huge. That's huge. I think that's a feature that I'd love to talk more about, but one of the things I want to ask just because I know we're short on time. Um so, 10th year of DGX, 20th year of CUDA. You've seen the system evolve so much over generation after generation. Is there another feature that you're like really proud of, really pumped to talk about, you know, that's you've seen evolve sort of from the ground up? I I don't It It's less of a feature in the system. It's how our customers use these. Because the you know, the the biggest thing and you know, one of the things, you know, I've talked to people in this about this in the past, but it's still true today.

16:48 Every system we put out, within a year of putting that system out, just in software, the system usually gets up to 2x faster, which is like completely opposite of consumer electronics. Like your phone gets slower every year. But because the optimizations that we do in CUDA and because of Tensor Core, that 20 years of CUDA, it's application compatible. That very first DGX-1 that was running on the show floor here 10 years ago, the application that was running on that would run on this thing today. So, like when NVIDIA releases a TensorRT LLM update that makes it twice as fast as inference. Yeah. All these Regardless of the generation, yeah.

17:22 Everyone gets it in that generation, and you know, that's something, you know, it's it's a little bit of the unsung hero. Our customers talk about it, but it's one of those things that like you can't see on day one. The numbers that we put up in the keynote, fantastic numbers today. When we revisit that 6 months from now, 9 months from now, they're just going to get better. And, you know, I I guess the, you know, from a feature perspective, it's not a feature that I'm looking forward to.

17:45 It's all the new things that our customers are doing with this is a genetic workload. We talked about open claw and doing it safely. Like that is the most exciting thing as just a general like technology user. Like I'm sure you've had this idea. I'm, you know, I've definitely had the idea to like, oh, I wish I had a program that could do X. And it's just, you know, an average everyday user I'm like, I could probably code that or I could call a friend, but it's like, nah, I I never do that. But now that we can safely take open claw, build a software application, and sandbox it. We're doing that actually in the park. We got open claw on stall fest with safe software to help people build their own applications. That's the thing that's exciting me the most is that everyone at this show, everyone at home, every business user that ever had an idea that used to say like, hey, I wish I just had a little software application that did X.

18:35 Well, if you can think about that now, with the technology that's available today, you can make that happen. So, you know, that's super exciting now, and what I can't wait for is like next year everyone showing the examples of like what they did on their systems that they got from us this year. Like what was new and unexpected that like nobody thought of that like changed the way they did their day-to-day work or their day-to-day life. I'm super excited for that. Charlie, thank you so much for your time. A huge thank you to Charlie for walking us through Nvidia's DGX systems, their role in the AI revolution, and the huge gains from Blackwell to Rubin. 35 to 50x performance in a single generation redefines what's possible across training, inference, and opens the doors for entirely new kinds of AI workloads.

19:21 And to me, that's a future worth investing in. Thank you to the Nvidia team for flying us out to California, for supplying us with press passes for GTC, and for making this interview possible. And of course, thank you for watching and supporting the channel. Without you, I would never get these kinds of opportunities in the first place. And if you want to see what else I'm investing in, check out this video next. Either way, thanks for watching, and until next time, this is Ticker Symbol You. My name is Alex, reminding you that the best investment you can make is in you.