transcribe

E22: NVIDIA'S HUGE AI Announcements Will Change Everything

Ticker Symbol: YOU · 24m · transcribed May 2026
More from Ticker Symbol: YOU Business
𝕏 Share ▶ YouTube 📥 PDF 🤖 .md

Transcript

0:00 I'm excited to share this exclusive interview with the investing community. Most people think of Nvidia as a hardware company that builds chips to train massive AI models, but you're about to get an inside look at a very different side of the story. I'm joined by Joe Dalaire, product lead of AI infrastructure at Nvidia. Joe spent the last 4 years deploying the hardware and software behind some of the most powerful AI models on the planet, and he shared a few surprising insights about where AI is headed next. But, that's just one of the many technologies that I'll be covering live at GTC in a few weeks. GTC is Nvidia's massive AI conference, showcasing the biggest breakthroughs in robotics and self-driving cars, AI agents and the chips that power them, and a whole lot more. And anyone who signs up for a free online session at GTC with my link can win an Nvidia RTX 5090 graphics card.

0:52 Just attend any session, take a screenshot as proof, and send it to me after the conference using the links below. GTC should be on every investor's radar, and so should Nvidia's ecosystem for AI inference. Your time is valuable, so let's get right into it. I'm so happy to be here with you. Thanks for taking the time, by the way. >> Okay. Jensen talked about a lot of awesome things at the keynote, and one of the things that he talked about in detail is that Nvidia actually co-designed six different chips for the Vera Rubin generation.

1:23 That's a lot to go through, so I'd love to go through all of it with you, starting from the GPU itself and working all the way up to the rack-scale system level, if that's okay. So, let's let's just start with uh Rubin itself. What's the difference between Blackwell and Rubin? Oh, so there's several different things about Rubin that are are different than Blackwell. So, we have the six chips that you talked about. Uh all of it co-designed together. So, what we did is we looked at the data center requirements, and we worked our way backwards and said, "What do we need in all these six different chips to make sure that we get the best performance, the best energy efficiency, the lowest cost. Yeah. So, that's what the fundamental thing about Rubin is this extreme co-design. All these chips manufactured together, designed together, working in concert for the best performance.

2:11 >> And when you say you looked at the data center requirements, are those being driven by AI models today? Or what like what's driving those requirements? >> Absolutely. Models are definitely the thing that are driving this compute demand. And MOE models in particular, mixture of experts, where they're generating many, many tokens factors more tokens because of the reasoning that they do. Also, the model sizes are growing as well. So, they're getting more intelligence from model size, from reasoning.

2:40 So, that is just generating a tremendous amount of compute demand. And Rubin is designed to address that. Got it. So, talk to me about the difference between Blackwell and Rubin, the GPUs specifically in terms of power and performance. So, in terms of power and performance, for inference workloads, we will see up to 10x better performance on Rubin versus Blackwell. Wow. 10x performance per watt. So, that means that So, at given fixed latency, you can see with those parado charts that we've shown in Jensen's keynotes, at a particular latency, a very, you know, high latency, that's a very uh good for users of the model. So, yeah, the 10x performance is across the rack scale. Is it at the rack scale?

3:25 >> Rack scale architecture. So, here we have the Blackwell Ultra generation compute tray. And I can show you what what we have here in terms of the components and their breakdown. So, we have two superchips. Two superchips, okay. >> Superchips have two Blackwell Ultra GPUs, and then one Grace CPU on on one superchip, and then there's two of them together, so four GPUs, two a uh we also have ConnectX-8 super NICs that are also part of this superchip. Uh and that's going to be an important distinction when we talk about Vera Rubin later and how those have been moved. Um but yeah, you can see that this is a hybrid cooled. Okay.

4:06 >> So, these are cold plates doing the liquid cooling on the superchips and all their components. And then on the bottom half of of the tray or the front half, uh I should say, this is air cooled. So, these are all What I'm actually looking at is the tops of all fans, right? >> Eight fans here. >> Got it. So, eight fans and then we have a uh BlueField DPU uh that is part of this tray as well. That is for the north-south traffic uh connecting the storage, getting the data in to the the the compute rack so that it feeds the the GPUs. Got it. So, the yeah, the DPU brings data in and out. Yes. And then all the process all the magic happens in the superchips themselves. Got it. So, there's two kinds of network traffic.

4:48 North-south is inside the same rack. East-west is connecting multiple racks. Is that how we should think about it? >> That's That's the proper way to think of it, yes. Yes. I thought NVIDIA was just a GPU designer, but Grace is a CPU, right? So, what is the CPU do? So, the CPU it handles a lot of the management. So, for example, like when you're doing uh you're trying to use inference and you want your your model to make uh some code for you. And you want it to make maybe it makes a little application, a Python application. It needs to run that. Grace CPU can actually run that application. The GPU wouldn't run an application that's generated by by a model.

5:27 Um but it is also doing uh other kinds of things like database analytics and those types of functions that are more CPU-friendly. Uh it's able to accelerate those types of Oh, so really the whole idea is kind of like you have the GPUs do what they're the best at, then you have the CPU to do things obviously that CPUs are much better at GPUs at so that you can sort of spread out the work over the right chip for the job, right?

5:52 >> You also mentioned something called a DPU. Can you walk us through what a DPU >> So, DPU, BlueField DPU, data processing unit. That's going to handle some of the north-south traffic. North-south traffic, yep. And when you're connected to storage that's on a different rack, there's going to be compression, encryption, that's all going to be managed by the DPU that we have in BlueField BlueField 3. And the goal for that is just to make sure the CPU and the GPU aren't doing those things. That's correct. Offloading Offloading all those functions from the CPU and the GPU, accelerating those functions in hardware, so that you get the fastest data access to feed the GPUs. That makes a lot of sense. Okay, so those are three of the six chips so far, right? The CPU, the GPU, and the DPU. And the And the ConnectX.

6:39 >> Yeah, talk to me a little more about that. >> ConnectX-8, this is your east-west connectivity. So, this is your supernic for connecting east-west. It also has in-line encryption, those types of functions for the east-west traffic that's going to be connecting between rack to rack of GPU racks. Got it. So, we have the GPU, the CPU, the DPU, and the ConnectX on this board. That's correct. Where are the other two chips? So, the NVLink switch is the the other chip. And there there's a two here on this switch tray.

7:12 This is NVLink 5 or the fifth generation of NVLink. And these are these are communicating to the NVLink network at 1,800 GB per second. 1,800 GB >> 1.8 TB per second. So, very high speed, and that's really going to be the the central nervous system of a Blackwell GB200 NVL72. Got it. So, So, these are two completely different trays, right? So, this This compute tray, that's where the magic happens in terms of crunching the numbers. And then this is the switch tray, which I think you mentioned earlier is all about just connecting all the GPUs together. So, it connects all the GPUs together. Uh there's several of these trays within a rack. Yeah. Uh all the GPUs are 72 GPUs.

7:58 They have 72 GPUs in a rack. And it's all-to-all connectivity. So, every GPU has to be able to talk to every other GPU at full bandwidth. And that's what the switches achieve. So, 1.8 terabytes per second, any GPU talking to any other GPU. Is that why it's called a compute fabric? Like when I think when I draw a network diagram of That's Okay, got it. So. So, yeah. They call it a compute fabric not just because it's connecting all the GPUs to each other. There's also some compute functions in our NVLink switch chips. So, we call that all reduce or collective operations where in training when certain operations need to be shared across the network, instead of sending it to all the GPUs, it will do some of those operations within the switch. Oh, wow. Okay, so the switch isn't just connecting things, it's actually also doing some Some computation as well. That's awesome.

8:51 Okay, so I think we've covered five of the chips now, right? Is that correct? That's correct. Where What's the sixth chip? Six is the uh Spectrum-X uh What's Can we try to take a look at those racks? Yeah, let's go take a look. There's 10 trays up top. Those are the compute trays. Nine networking trays. Nine NVLink switch trays, I should say. And their job is to connect all the GPUs in the 10 above and the eight below compute trays together, right? That's correct. So, what's up there then?

9:24 So, that that is the top-of-rack uh 1 gigabit switch for telemetry. That's telemetry? That's just telemetry. It's just system management, managing functions. It's low-speed Ethernet. It's just a uh It's a just a management system for the rack itself. It doesn't It's not processing the compute data for AI. >> It's managing if a GPU goes down. It's like Help me understand what telemetry means and what that >> Telemetry means like I'm just looking at the the functions of the rack itself.

9:50 I'm looking at its uptime. I'm looking at >> Health and status, I guess. >> Health and status checking, yes. Diagnostics would also >> And you mentioned that there's another kind of rack that would sit next to this. So yeah, you will have your your group of compute racks, GB200 compute racks, and then you would also have racks dedicated to Spectrum-X east-west network switches. We don't have that here, but though that's how the the function would be like a we call it a pod. You have maybe eight GB200 racks, and then you'll have a few switch racks with Spectrum-X.

10:27 >> Yeah. So that's a great overview of the Blackwell system, right? >> That's right. Now, I want to understand how what things changed from Blackwell to Rubin. Okay. Can we go over there and look at that? >> at the at the trays. So this is Looking at the components up here on the wall, we talked about in the compute tray, the BlueField DPU, BlueField 4. So that There you can see it on the wall. The that that board is part of the module system that slides in and out of the compute tray for serviceability.

10:59 And then all likewise, the ConnectX-9 is there in the middle. And there's two ConnectX-9s that are on that board for a total of eight in every compute tray. So every GPU is fed 1.6 terabits per second for the ConnectX-9s. And then we have the the Spectrum-X photonics co-packaged optics. This is really, really cool. Yeah, what is that? So instead of having SFP pluggable modules for for the optics, they're actually built onto the chip itself. Okay.

11:34 with it. So, this has a a huge gain in energy efficiency, uh reliability, uh and this factors more in terms of of those two factors. >> So, before we would have fiber optic transceivers. The fiber optic optical transceivers. >> Yeah. So, the fiber optic cables on either end, and those transceivers have lasers in them. That's correct. >> That need power, right? Like and that's what you're getting rid of And we're putting them packaging on the with the chip.

12:01 >> What does that actually mean in terms of like performance or power gains? So, in terms of performance, the performance would be the same. But, it's going to be the uh the power reduction and the uh reliability improvement. Cuz uh those pluggable lasers can be very, you know, sometimes very unreliable. They have to be swapped out very frequently. But, if it's co-packaged here uh on on the chip, the reliability goes up like uh I think 10x better reliability.

12:29 >> So, it's a huge difference. And where in the rack does that live? So, that would be in its own switch tray uh or a switch server. And that's a separate rack. >> That's the side rack, right? >> That's the separate rack that's separate from the the NVE 72. So, that's the east-west traffic switch rack. Awesome. So, Quantum MX, uh there's also uh for InfiniBand, which is a an alternative to Ethernet. There's also a co-packaged optics for Quantum InfiniBand as well.

12:57 >> So, those two chips are equivalent. One is for Spectrum-X Ethernet, one is for Quantum InfiniBand. >> That's correct. And then you also have a Spectrum-X Ethernet photonics switch. So, that is the uh the co-packaged optics chip is in there in the Ethernet photonics switch. So, that's where the photonics part is, the co-packaged optics. Got it. But, these these go in the side car. These go into uh switch racks. Yeah. Got it. As well as that one, right? If you're doing Quantum InfiniBand as your east-west traffic protocol, then you would use the Infiniband as a side rack.

13:31 >> So, these are Sorry. These are equivalents. One for Infiniband, one for Ethernet, right? Correct. Got it. Yeah, that's right. So, what we kind of just talked about is what I would say is the current state of the art for data centers, right? Blackwell Ultra is the one that's sort of the best in class in data centers right now. And then Jensen announced Vera Rubin, the six chips we just talked about. We talked about the Blackwell versions. This is a substantially different compute tray than the one we just saw. Can you walk us through all the differences? Oh, yeah, there's plenty. So, uh what we've done is overall, it's a modular design. Okay. So, that means that there there's bays here and these can just slide out and slide in and just lock and latch. So, there's not a bunch of wires and cabling to do all the connectivity between all the components that are on the tray.

14:22 Also, the hosing as well, that's been streamlined. >> Yeah. So, there's a manifold in the in the middle, um and it manages a lot of the uh distribution of liquid. So, overall on the GB300, there was 43 hoses. There was a bay of fans here cuz it was a hybrid cooled. Uh the the bottom half of GB300 was was fan cooled. This is we have eliminated that uh and because we're 100% liquid cooled now. So, eight fans goes to zero fans, zero hoses.

14:52 And then there's a bunch of cables that have been removed as well. Uh so, it's cable free. So, this I I'm trying to even piece together what I'm looking at. So, these would be where the two super chips were in the last generation. >> the super chips. They slide in and out. They latch in. Uh so, you have the two Rubins, uh one Vera on them. So, uh one other important point is because it's modular now and we have all these bays that slide in and out and it's all connectivity with connectors instead of cabling, putting this together and doing assembly on it is like 20 times faster.

15:26 >> Sure. So, something that would take 2 hours to assemble the GB300 rack, now you can do in 5 minutes on this particular rack. >> And that's And that's just assembly, right? Like if I have a maintenance issue and I need to >> for maintenance, right? The The amount of speed that you can do serviceability increases that manyfold as well. No, it makes a ton of sense, right? If I don't have all these wires and hoses, I can just snap things out, fix fix whatever the issue is, snap it back in.

15:53 And it's modular like so we'll talk about some of the other pieces down here. So, two super chips, Reuben, Vera. We also have the CX 9's, ConnectX-9, the next generation of that super nick are over on these in boards in modules. So, before they were connected to the bottom of the super chip on GB300, but now they're their own module and cards slide in and out. So, you can service different components now separately. Yeah. >> And then BlueField 4, the new generation of the DPU, is also a module here that slides in and out. Got it. So, this is not just about performance, it's also about more uptime, right? So, that's another multiplier on the overall output of an AI factory is how much uptime you We call that good put. Like you want the the the amount of time that you're actually producing tokens, you want to maximize that.

16:45 >> Yeah. That makes sense. So, okay, this is the equivalent compute tray. That's right. And then there's also an equivalent switch tray, right? That's correct. And this looks a lot more streamlined, too. So, walk me through the changes here. So, in terms of the changes here, you know, we have the the switches at the top, 100% liquid cooled. There's four switch chips. This is NVLink 6. Okay. >> Sixth generation NVLink, twice the speed of what we had in the Blackwall.

17:11 >> Twice the speed. Wow. So, now it's 3.6 terabytes per second. And that's just going to help us with our that performance I talked about, 10x performance per watt or per megawatt per gigawatt, whatever value you want. Uh, that's the increase in NVLink speed is part of that contributes to that along with some other GPU features that we can talk about as well. And are there so is it the same number of total GPUs in a Blackwell rack versus a Rubin rack?

17:41 >> It is. So, it's a NBL72s. The 72 signifies the GPU count. So, GB200 NBL72 uh, and now we have Vera Rubin NBL72. Same GPU count. Um, and it also makes it so it's very compatible for our customers to to move from one to the other. Uh, and that's part of the goal of having the same GPU count, same kind of MGX rack architecture. Um, so that that's just makes it easier for our customers. The ecosystem is, you know, been working with these racks for two generations now. Now we have a third generation. They're just going to be able to work very fast and deploy uh, at a very high rate with our end customers.

18:19 >> No, it makes total sense. Okay, can we go look at a Vera Rubin rack now? >> Yes. So, this is the Vera Rubin. Uh, this is the Vera Rubin NBL72 rack. You can see that there's, you know, it's very similar in in form and in look to the GB200. The the most uh, the biggest difference is on the compute trays you'll see there's no vents. So, there was vents on the GB200 cuz the bottom half of the compute tray still had fans.

18:48 Okay, yeah. And then we got rid of those fans. It's all 100% liquid cooled on the compute trays. That's why you see in the face plate you don't see those vents anymore. Got it. Uh, but overall still, you know, still the nine uh, switch trays, still 10 compute trays on top and the eight on the bottom. Same kind of still telemetry on top. Still top of rack telemetry with the one gig switch on top. >> Now here's the big question, right? From Blackwell to Rubin at the rack level, talk to me about the performance gains at the rack level.

19:20 >> Performance gain at rack level is the 10x. 10x? >> The 10x more tokens per second per megawatt or per watt uh then that's going to be a rack level kind of uh performance metric. And that's with a mixture of expert model, something like Kimmy K2 thinking, uh which is very large model over a trillion parameters. Uh and that is going to fit and be uh optimized in a single rack uh with, you know, thanks to NVLink switch, the experts in a mixture of expert model are distributed across the 72 GPUs. And uh that can uh factors more performance in tokens per second.

20:00 So, here we have the Kyber rack. So, this would be for the Rubin Ultra generation. Subsequent to Rubin, which is a 2026 product, in 2027 we'll have uh Rubin Ultra. So, that's going to be a different rack architecture than we've had for the previous three generations. Uh we're putting much more compute. Yeah, I'm noticing a lot more trays in this one. So, we have 18 compute trays in each of these canisters. So, there's four canisters, up to 72 GPUs in each of the canisters.

20:32 So, you would have 288. 288 So, moving from 144 to 288 or is that 72? >> 72 >> to 288. Okay, so it's a 4x increase in GPUs. >> So, each of these canisters, the four I talked about, is equivalent to the whole rack over here. >> So, there's four racks worth of GPUs >> racks of NBL72s worth of uh compute in here. So, very uh high compute density. >> Yeah. Um and that's why the architecture is different. It's a blade type of architecture rather than a tray architecture. Uh so, we have 18 uh compute blades uh in each of the canisters. Excuse me, sorry. These are all compute then? This is all compute on the front.

21:12 >> Yeah. On the back is where the switch blades are. For the for the NVLink connectivity. Got it. And so, that's What is the performance leap that you guys are expecting from Rubin to Rubin Ultra in the Kyber rack? So, we haven't given any of the performance yet on Rubin Ultra, but it's going to be factors more performance as as usual between our generations. Just because you're going to have inc- performance increases at the chip level, at the superchip level, at the rack level, and you're going to have four times as many >> extreme co-design all all again, right?

21:47 Extreme co-design, all the chips being designed for for greater performance, working in concert, being designed from scratch together. Are we expecting extreme co-design of all six chips for every generation from now on? We should expect to see six new chips? So, for every generation, there's going to be a new generation of GPU for for every year. Now, whether all six are going to be co-designed every year, that's that's probably not going to be the case, but you're going to see at the for the flagship starting of each generation like Rubin, six new chips, some other new chips that go with Rubin Ultra, but not the entirety of all six.

22:26 >> we might see the Vera CPU, but the Rubin Ultra GPU. >> Exactly. >> Got it. Exactly. Got it. Yeah. I'm super excited for this. I can't wait to see what this looks like. When when can we expect to learn a little more about this? Is this something that we'll learn about this year, next year? So, yeah, it'll be something that Jensen talks about, you know, in the in the coming year. Uh, I don't have a specific date, but yeah.

22:49 I'm super excited for it, man. What are you looking forward to the most? Like, what excites you the most as you see like this rapid evolution year over year and generation over generation? So, the the amount of in- innovation at the with the extreme co-design, that's what's most impressive. Yeah. Right. So, there's only so much, and Jensen talked about this, that you can do moving from one GPU generation to a next. Process technology can only improve so much.

23:16 You know, it's not factors more improvement in in the number of transistors that you can go from one generation to the next. So, for example, between Vera Rubin and Blackwell, it's about 70% more transistors. Yeah. In terms of all the different chips that we we co-design. But, we're getting the 10x more performance per watt. So, if you were just Moore's law, it would only be a 70% jump, not a 1,000% jump. From Yeah, not a 10x, yeah. So, so this kind of all these different chips being designed together, working together to maximize that performance, that's the most amazing thing about this generation and the future generations. Yeah. That's really exciting.

23:57 Thanks so much for your time. A huge thank you to Joe Balaware for breaking down Nvidia's Blackwell ecosystem, giving us an inside look at Rubin, and explaining how it will all make AI models faster, smarter, and more efficient. Not just language models, but everything from image and video models to medicine, robotics, and so much more. And if you want to really understand the science behind this stock, join me at Nvidia GTC. You can register for free with my links below, and jump into as many online sessions as you like. I'll announce the winner of that RTX 5090 giveaway a few days after the conference, so make sure to enter.

24:33 Another huge thank you to Nvidia for sponsoring my travel and my media access to cover GTC live, and to you for supporting the channel. Thanks for watching, and until next time, this is ticker symbol you. My name is Alex, reminding you that the best investment you can make is in you.

Summary

Joe Dalaire from Nvidia provides an in-depth look at the advancements in AI infrastructure, particularly focusing on the new Vera Rubin architecture compared to the previous Blackwell generation. The discussion highlights significant performance improvements, innovative chip designs, and the implications for AI model efficiency and capabilities.

- Nvidia's Vera Rubin architecture features six co-designed chips, enhancing performance and energy efficiency.
- The new architecture promises up to 10x better performance per watt compared to the Blackwell generation.
- AI model demands, especially from mixture of experts models, are driving the need for increased compute capabilities.
- The modular design of the Vera Rubin system allows for rapid assembly and maintenance, significantly reducing downtime.
- Key components include GPUs, CPUs, DPUs, and high-speed NVLink switches, all optimized for AI workloads.
- Future generations, like Rubin Ultra, are expected to further enhance performance with extreme co-design principles.
- The upcoming GTC conference will showcase these innovations and offer insights into Nvidia's evolving ecosystem for AI inference.
© transcribe · For agents Built with care and craft by Gokul Rajaram