M5 Ultra… Apple Wasn’t Messing Around

0:00 Well, the M5 Ultra is finally here and it's right here on my desk right next to the M3 Ultra. Apple says it has local AI performance up to 4.3 times faster than the previous generation. Two times faster storage, 1.3 times faster CPU speeds. Sounds pretty good, but is that real or is that just marketing? Well, according to the spec sheet, the new memory bandwidth is 1.2 terabytes per second. Going to test that out, of course. M5 Ultra is what I have here.

0:28 That one starts at 5499. And if you configure it the way I have it here, you're going to need to bump that GPU to 80 cores. You're going to need to bump the memory to 256 GB. 512 is coming next month and you're going to need to bump the storage to 8 TB. I mean, you don't need to, but that's what I have here, so I thought I'd give you the price, which comes to 14,299. Not only do you get a performance bump, apparently, but you also, apparently, get a price bump. Quite a bit. Apple loaned me this one. It's nice to be recognized after covering them for many years and buying my own. I bought this one, though. And by the way, the reason both of these machines are on my desk right now is because of you. You folks watching this channel and supporting me allows me to get real hardware to do these real tests for you instead of just reading the numbers off a spec sheet.

1:14 So, thank you. Look at all those cores. We got 36 of them. By the way, there's no efficiency cores in the Mac Studio here. We have 24 performance cores and 12 super cores. Now, to keep this fair, both machines ran the exact same builds of llama.cpp and mlx and I used the exact same model files on both. By the way, llama.cpp is the cross-platform library that works on pretty much all the systems, not only Apple, and is very popular. mlx is Apple's own stack for running AI. Typically, mlx has better performance, but not always.

1:48 Now, if you're buying this machine just for developer tasks, let me tell you something, it's overkill. Here's the speedometer score. 47 is the M3 Ultra score. We got 60.2 on the M5 Ultra. I just did a M6 mini review and yeah, it destroys that score by quite a lot. Now, this is the M5 chip. That's the M6 chip. So, a single core of that is going to be much faster in the M6 family than the M5 family. All right, quick CPU test for devs. Since this thing has 36 cores, let's use all of them. Here we've got a large.NET compilation with 100,000 name spaces and classes. And I ran the compilation, we've got 71.7 seconds on the M3 Ultra, 52.4 on I think that's the fastest I've ever seen it on the M5 Ultra. That's a compilation. What about Python, which is an interpreted test? So, let's run that.

2:41 And boom. All right, now this is Mandelbrot test, which is basically doing the Mandelbrot algorithm in Python. Whole Oh my gosh. What? this is two times faster. Wow. Okay, 10.25 seconds on the M3 Ultra, 5.8 seconds. This is crazy. I've never seen it that fast. It's okay. It'll get the job done. Storage is supposed to be twice as fast and I just ran a more first disk mark. M3 Ultra sequential speed, so this is like copying a large file back and forth, 6,500 megabytes per second, so 6.5 gigabytes per second for read and 2.7 for write. Wow, the M5 Ultra, 14,888 read and almost 20,000 in write. Wow, that is crazy. That's for sequential.

3:35 Now, for compilation, for developer-related things, you're going to want to look at the random numbers, too, because you're copying lots of little files back and forth, but that's actually more than two times faster, also. So, yeah, two times faster is a conservative label of by marketing team is actually faster than that. This matters for AI as well. When you're loading a 140 GB model, it's going to read it off the disk much faster than this one. Editor Alex here. This number turned out to be way too low and I noticed it later. This is actually re-running it. So, the right speed looks a lot better and this is on 8 TB drive by the way. So, both of them are 8 TB. All right, let's switch to local LLM talk. For local LLMs, there's really two numbers I care about. Prompt processing or PP as sometimes it's referred to as and token generation or TG. These are terms that I didn't invent. This is from llama.cpp world. They like to have fun over there, okay? PP prompt processing is the model reading your prompt and it leans on compute or the GPU cores. So, the more cores you have, the faster those cores, the better. And token generation TG, that's the model writing the answer. And that one leans on memory bandwidth. So, both of those things were improved in the M5 Ultra. But, it's not just an incremental improvement. It's not a small improvement because in the M3 Ultra in the M3 generation, the GPU cores was just GPU cores. They were just regular old cores. But, now with the M5 generation, we have for every GPU core, there's a neural accelerator that lives inside of it. That's dedicated matrix multiplication or AI hardware inside of every single core. Now, check this out.

5:20 On the M3 Ultra, llama.cpp starts up and says has tensor equals false. Same exact build on the M5 Ultra, has tensor is true. So, llama.cpp supports metal. It's actually one of the first things llama.cpp supported. And now it has specific paths that the software takes for the M5 family of chips to take advantage of those neural accelerators. Quick word from the sponsor and we'll take a look at memory bandwidth. Every developer account leaves another piece of your information online. Data brokers collect those pieces, build a profile, and then sell it. Making your data harder to find makes you harder to target. Privacy laws in many countries let you tell data brokers to remove the information they have about you. The hard part is finding all those brokers, then submitting every request, and checking again when your information eventually comes back. Incogni handles those requests automatically and then keeps checking up and following up on them. My dashboard showed over 231 databases with my details in them. Most of those are already gone. And if I spot a specific page with my data on it, I send Incogni the URL through custom removals and they take it from there.

6:27 Basically, find the data, remove it, and keep it removed. For some added reassurance, Deloitte verified Incogni's data removal processes. Now, there are limits. It's not going to be deleting government records or social posts, but it can tackle data brokers and eligible pages. And you can try it with a 30-day money-back guarantee. So, make your personal information harder to find with Incogni. Go to incogni.com/alexziskind and use code alexziskind for 60% off an annual plan. And now, back to the video.

6:55 So, 1.2 terabytes per second memory bandwidth on the M5 Ultra? I wanted to measure it. There's a good old benchmark for that called STREAM, sustained memory bandwidth in high-performance computers. It's old, but it still works. So, I pulled it down. You can do the same. You can compile it, and boom. Now, the M3 Ultra, 819 GB per second is what Apple claims. We're getting 312.9 GB per second for the triad and 583.6 GB per second on the M5 Ultra. So, it is quite a lot higher than the M3 Ultra.

7:27 And the reason for the lower number is because STREAM is actually a CPU-related test. So, it's running on the CPU memory bandwidth. So, let's measure the memory bandwidth from the GPU side, which is what matters for AI. Memory bandwidth microbenchmark. And let's go. That's better. 724 GB per second on the M3 Ultra. Apple says 819, so that's about 89 90%. And the M5 Ultra, 1,039 GB per second. Out of 1.2 TB, that's about 85 86%. So that's 1.4 times the actual measured memory bandwidth over the M3 Ultra. Does that mean that token generation is going to be 1.4 times faster?

8:07 >> >> All right, let's start big. Deep Seek V4 Flash is 284 billion parameters. By the way, this is a pretty big model and it's pretty popular now. And we're getting 37 tokens per second on the M3 Ultra, 53 on the M5 Ultra. And that is about 1 and 1/2 times faster. Kind of lines up, right? So that's token generation, that's memory bandwidth bound, remember that. What about prompt processing? That thing that the M3 generation was so good at token generation. I'm saying generation a lot. It had really good memory bandwidth. It was impressive for a box like this. But prompt processing was always the issue of the M3s and the M4s until we got those neural accelerators. So what about prompt processing now? 483 tokens per second on the M3 Ultra and 1,485 on the M5 Ultra. That's three times. Now we're talking. But Apple said four.

9:06 Now, different models behave differently and you might remember this one from the M5 Max video that I did a few months ago. And at that time, the M3 Ultra was king at 82 tokens per second. That's GPT- OSS 120B, 120 billion parameter model. Now that was in LM Studio, not pure Llama CPP. Now we're getting 90. I tested it today again. Now we're getting 90 tokens per second. And on the M5 Ultra, 130 tokens per second. That's fast. For a 120 billion parameter model, this is not Qwen 34B like I usually would show on the channel.

9:38 But, we're going to skip that today. Prompt processing goes from 1,300 to about 3,200 tokens per second. Now, watch this. With 64,000 tokens already in context, I gave it 32,000 more. That took 80 seconds on the M3 Ultra and 56 seconds on the M5 Ultra. So, that's 1.4 times. Now, DeepSeek and GPT- 4 SS are both MoE models or mixture of experts. With those kind of models, each token only goes through a few small experts. Most of it sits idle. In a dense model, it's every weight, every token. So, I added Quen 3.8 27B, a pretty popular model, not as many parameters, but it is dense, so it's heavy. Right? Density is the same thing. I took science class. I know what density is. Token generation on the M3 Ultra, 37 tokens per second, 54 tokens per second on the M5 Ultra. So, that's 1.47 times. Basically, we're getting the same kind of scale on all kinds of models here. Token generation is basically following Apple's spec.

10:49 Now, prompt processing, 430 on the M3 Ultra and on the M5 Ultra, 1,800. Holy cow, that's kind of like four times, right? It's actually more than four times. I also gave them both a 14,000 token prompt, equivalent to about a handful of source files if you're doing code, just to see if it falls apart if I'm doing a real code base instead of just a benchmark. And I timed the prompt processing. That's 33 seconds on the M3 Ultra and eight seconds on the M5 Ultra. Apple did say it's up to four times faster prompt processing and I got 4.2. So, Apple, again, was not lying.

11:27 They did it again. Now, don't get me wrong, I'm impressed, but before you get your credit card out, and it is going to be a big credit card bill, that four times needs a long prompt. At about 1,700 tokens, it's 3.4 times. It takes about 4,500 tokens to hit the four time scale, which shouldn't be a problem if you're using this machine professionally. However, on a short prompt, around 350 tokens or so, it's only 1.6 times. It's still an improvement, but if you're mostly chatting, then you won't really feel the improvement. If you're feeding it a few files, then you will. It's also not much more efficient. During generation, the chip reported about 45 W versus 36 on the M3 Ultra. And that's GPU plus CPU, not wall power. So, tokens per watt only went up to about 12%. This is basically the machine idling, 12 11 or 12 W for the M3 Ultra, and about eight or nine, up to 10 W on the M5 Ultra. However, during heavy intense GPU work, I got it pegging right now. That's a GPU history chart right now on both of them, all the way to 100% usage. And the M3 Ultra is hitting almost 200 W, the M5 over 400 W.

12:45 That is crazy. Wow. And that's pretty stable there. Wow, look at that. So, I started hearing the M5 Ultra first. It's definitely much much more audible. You can probably hear it from my microphone. That's all M5 Ultra right now. I'm not even hearing the M3 Ultra. I'm hearing the M5 Ultra. And look at the temperatures. You can just tell from the bright orange of the M5 Ultra that it's heating up quite a bit more. So, I'm about 43 44 degrees on the M5 Ultra.

13:22 And considerably lower on the M3 Ultra. And then there's memory. Everything I ran today fits in 256 gigs. Ooh, ooh, that's so much toastier than the M3s were. Wow. Okay, that gets toasty. Wow. Now, there's some bigger models out there and I showed plenty of examples of my tower of 512 GB Mac Studios with the M3 Ultra, including clustering, which I'll come back to later on in Mac Studio format and in Mac Mini format as well. So, stay tuned for that. Obviously, the 512 GB models of M5 Ultra are going to be very expensive, but they're going to be able to run much larger models, too, and faster. So, bottom line, token generation 1 and 1/2 times faster, nice upgrade. Prompt processing two to four times faster. For coding agents, you will feel that. So, this was a first look. I've got more testing to do, of course, just to give you a little preview with eight requests at once, this thing puts out 134 tokens per second against M3 Ultra's 75. I'm also going to dive a little deeper into MLX versus Llama CPP and 8-bit models. So, stay tuned for that. Make sure you subscribe so you don't miss it.

14:34 And if you want to check out the M5 Max and its capabilities, that video is right over here. Thanks for watching and I'll see you in the next one. >>