I Gave Two AI Supercomputers a Real Job

0:00 So, I recently covered the Camino Grando and the DJX station, but I have Wendell here today visiting me and he's going to show us Turnstone, which is how to use these machines in a real scenario. >> I like doing developer and developer things. It's fun. >> Let's talk about the hardware a little bit. You haven't had a chance to test the Grand yet, but you will. I'm going to give it to Wendell, so watch his channel for some updates. He's going to go deep.

0:24 >> It's eight RTX Pro 6000s. And this is kind of the question, isn't it? eight RTX Pro 6000s versus DGX station, >> right? There's a reason why you don't hear them right now, but they are actually running and they're busy. It's getting toasty in here. We have 6,000 watts of heat generation in here. This is nuts. I've got the Grandondo. It's using four plugs as I showed in my other video and the DJX station is running as well off of the Jackary.

0:51 It's hilarious, I know, but it works. We're down to 86% and it's right now idling at 319 watts. There's so many watts. What are you doing with all these? And that's down to 86% in just about 45 to 50 minutes. You can see why this door is closed. It's kind of loud in there. By the way, these machines are running headless. I got the DJX station up and running remotely through its management port. This machine is completely off right now, but I could still get into it. How? Well, that's through the management port. system 10G Ethernet port goes here. And this is the management port. And here I can control it. I can check the sensors, the LEDs, the logs, update the BIOS. And here's the KVM. So I can power the machine on from here. And there it is. It's turning on just like any other desktop. You can still hear me over it. And this is pretty much what it sounds like all the time, even when it's cranking away. I don't expect one AI prompt to build an entire project for me. In reality, I'm constantly moving between models depending on the job. GPT for research, claude for coding, Gemini for massive context, nano banana, midjourney, flux for images, and then I've got seed dance and cling for video. That's why chat lm by Abacus Aai makes sense. It brings day one support for the latest GPT, Claw, Gemini, Grock, Deepseek, and more in one place the moment they drop. Pick any model from the interface or let route LLM automatically choose the best model for each prompt. Create professional presentations with graphs and charts and deep research detailed content. Need human sounding copy? Human eyes rewrites text to defeat AI detectors. Need visuals? Pick frontier or open- source models. And when you need more than chat, Abacus AI agent can help build complex apps and websites, connect payments, or run 247 agents that keep working through longer tasks. The best part is app hosting, back-end database, and off support comes with the subscription. All that starts at just $10 a month. Way cheaper than paying for all those subscriptions separately.

2:57 Check out chatlm.abacus.ai or click the link below. We're going to turn to Turnstone and we're actually going to give it a couple of projects that I want done. Cloning the repo. In this current working directory, you'll find Turnstone. Read the instructions and set this up on this Mac. Boom. This is how I work now. >> I worry about the future. >> Wait, it just deleted my home directory. >> Woo! Dangerously doing things. >> I didn't get it, did I? No, I didn't.

3:29 Okay, I have a scenario. I run sessions with cloud code and I run it on multiple DJX sparks just to kind of distribute the load, but I want that automated. I want to go to one thing and tell it go start a session. You pick what's the best place to run it. This is where Turnstone will come in, right? I thought so. Oh man, that's going to be nice. I'm going to have a fleet. >> So, the first thing we want to do is set up a model. If you put in the base URL, if you got something local, if you know the IP address off top of your head, >> you could load up two models. Actually, that's not a bad idea.

4:00 >> Let's see if this model is loaded. We're connected and it's enabled. >> And you can add more models. You can have a mix of models that are good at really different things, but you can also configure Turnstone. Like in general, this kind of request goes to this model. These are the personas that it ships with, but you can build your own. Some of them have access to MCP model context protocol. So like if you want to run MCP servers, it can inventory them and use them. So you have engineer, executive, manager, orchestrator, researcher, scribe, and writer. Like depending on what your writing preferences are, you can teach it. It's like no m dashes or no extra bits for you. which is >> this is scribe.

4:34 >> Mhm. >> I need that. >> Yeah. You can also save memories. So you can say whenever I tell you to do this, remember to do that and it'll just build memory that over time. >> That's really cool. Can you tell me about these nodes? What are these? Are these basically just virtual machines that are running here inside Docker or what? >> Yes. And they're very limited right now because we haven't mounted a workspace or anything like that. So if you had a shared workspace, it's mounted at SLworkspace on the nodes by default. And this manager orchestrator is going to create workers with different personas for that kind of a task, including a manager and an evaluator. And that will help bring the project to completion. I have a bunch of repositories on my GitHub. And one of the things I like to do is I have to set up a lot of machines here, Linux, Macs, Windows, and I created these repositories. I don't have one for Mac yet. Is to bootstrap a bunch of developer tools and set it up as a developer machine. So I have this Windows one. Problem is this works on x86 machines but not on armbbased machines. And I ran into this recently in a recent video. This was a problem for me. So I want turn stone to turn every stone overh.

5:38 >> Okay. Okay. >> Been hanging out too much. >> And figure out how to create another branch for me to do a setup on arm. Let's turn over a leaf and stone and not leave any stones unturned. This is getting bad. >> Click dashboard for the workspace. And so paste it in, but then you got to explain what you want. You can't just sort of grunt in the general direction of the repository. >> I want you to look at this repository and create a separate branch that'll handle Windows on ARM situations because there's certain software that this installs that are only compatible with x86. Python, node are being some of the examples. I want you to go through and figure out every single example of a software that's installed and create the ARM equivalent of this. So before we run this, where is this running >> on those nodes?

6:31 >> On the nodes, does it matter which node? >> It'll schedule it itself. >> And the nodes are using an LLM. >> The nodes are connected through the model service >> which is going to use the Grando. >> Mhm. >> And also create an auditor that'll check the work. I'll start by exploring the repository to understand what it does. And there it goes. >> All right. While that's working, go and let's go and create another task and another task and another task cuz didn't you have some other tasks you wanted to do like find a bug or something?

6:56 >> I do. Where's the new task button? >> you just I think >> issue issue issue number one. I have this repository called code needle which is actually a benchmark for code for LLMs. So we're going to do a little bit of a meta thing and I'm going to have it evaluate this and look for bugs. specifically. There's one bug that it has that I want fixed. Fix the generated test files before they use many more tokens that their name claims. Boom.

7:24 Let's go. And now we have two tasks running at the same time all on that ground machine. And for some things, DJX Station is better. For other things, 8 RTX Pro 6000s is better. >> It just depends on what you're doing. >> Here's an example of a single user generated speed. >> GPTOS has 120B, and that's because it's running entirely from HBM. That is an absurd speed. >> HBM is the high bandwidth memory that's inside the DGX station. The Grounder does not have that. There's only 252 GB of HBM in the station. So, you have to keep your models lower than that.

7:57 However, the station does have a total of 748 GB of memory. That's unified. If you spill over to the Unified memory, then it'll still run. It'll just be a little bit slower. And you can see that later when we run bigger models. It's like 600 gigabytes per second of LPDDR5 and that's pretty fast. And the link speed and some of the other stuff there is also pretty fast. It also depends on the model. So like if you have like the new mixture of experts models where they have Ingram offloading also seems to be set up for something like the station's memory model because the Ingrams are fine running from slower memory. That HBM3e that's 7 terabytes per second.

8:34 That's an absurd amount of memory bandwidth in the 252 GB. whereas, you know, GDDR7 is a tiny fraction of that. Then the RTX Pro 6000s have to talk to each other over 16 lanes of PCIe Gen 5. That's only 64 GB a second each way. That's just pedestrian. >> So, here's Deepseek V4.1 Flash. And this one has recipes for the GB300, which is the DJX Station GPU, and it's FP8. Notice the size. 614 GB. See, again, that's a little misleading because some of the weights are perfectly fine running from HBM and some of them are perfectly fine running from the LPDDR5 because it's the new architecture.

9:10 Mixture of experts runs better. You've guys probably experienced that. This is another level in terms of there's more knobs and tunables that we can use. So, we're not suffering as bad because we've only got 252 GB of HBM3 even though it is an absurd number of weight, >> right? But the difference between the Grando and the Station is that this 614 can all fit on the RTX 6000. >> True. >> And we can see that here in the results.

9:36 So there's Deepseek V4 Flash, but I also included the numbers from Grandos's V4.1 here with DSpark. 307 tokens per second there. So it is faster there. >> Oh, just by a little bit. >> Just by a little bit. By a little bit. But still, this is HBM versus nonHBM memory. >> That's where the difference really shows up. It's exciting because it's different and weird. Now, prefill that prefill number on that HBM is always ludicrously absurd and there's no prompt processing.

10:04 You're never going to beat HBM. >> So, one of the things that these machines are good for is not just using single communication like a chat, but also allowing multiple users to use them at the same time. So, that's where multi-user throughput comes in. I like multi-user throughput and real world testing because this is like if you're going to buy one of these machines, chances are you're supporting a developer team or you're doing something that's privacy focused because you have to buy I mean like cloud tokens are really not comparatively very expensive compared to these machines. You're going to have to keep it busy like 24 hours a day. Look at the many users of throughput. 3,800 tokens per second.

10:39 Just over 2,000 tokens per second on Neotron 3 Super 12B. >> That's nuts. >> These are useful models. You can do a lot with them. These cloud models are incredible and how far they have come in such a short time is incredible, but you really don't need, you know, like Claude Opus doing all of your really basic stuff. Claude Opus can be told with a harness system, you have all these workers and they're pretty good. You can manage them. And so you have Claude Opus spending tokens at a much lower rate and then it has eight or 10 workers, agents that are running on this hardware that's not burning cloud tokens. And those models are smart enough. Neatron 3120 12B is pretty good for like unit tests, integration tests, dev like real dev work with deep context and they can be managed by a smarter model and that's the real unlock here.

11:26 >> Exactly. By the way, I will have these numbers posted in a link down below if you want to go check that out. >> Oh, video generation also like we're doing we're doing like code and programming, but like station OMG. >> Yes, I tried H3. Oh my gosh, this is insane. This is like a whole new level. It does run a little bit faster on the DJX station than the Grand as you can see. Quite a bit faster because H3 is small. So, it's going to run much faster on the HBM. I asked it to create a 5-second clip and it sounds exactly the same just sitting here. There's no more noise. There is a little bit more heat.

12:01 I can actually feel the difference. Each one took 2 minutes to make with H3. Yes. Okay. Hey, we saw something really good on the screen. All right, make him do something crazier. >> Oh, I have it. >> What? What? All right. All of them knock out spouts.

12:33 >> Oh my gosh, that is nuts. What is THIS ONE? >> LET US GO. Fun can be had for sure. So, it's showing us the tasks and then the subtasks are all subdivided there. So, you can go and dig into each one of them and you have visibility into everything that it's doing, which is kind of cool. >> Yeah, I love this for auditability sake. >> It found the solution. Look at that. >> Neat. And it only used 15,000 tokens.

13:01 >> That's not bad at all. It tells me what each of the subtasks are using. >> Personally, I like running the workers on DGX Spark. So, I have cluster of two Spark, the Dell GB10, Promax, GB10, Promax with GB10. >> Hang on. Hang on. >> It's so loud. Yeah. This is what my Turnstone cluster is running on. I have the Judge model running on here and I have a smaller version of Deepseek0731 running on here. And then the big machine has the big model, but the small models and the Judge model run on that. Great.

13:36 >> And for like a small team of like up to four people, that's great. So do you map a node to a spark or is that not exactly a onetoone? >> So you run docker on the spark and you can give it full access or you can just run it from the shell. So like where we were doing stuff with python you can just run it and then it's one to one. >> Okay. >> But it can run tasks locally on the real Spark hardware at that point. So it's got a lot more access than it does in a Docker sandbox.

13:59 >> That's cool. That's what I'm going to have to set up for my clusters. >> Yep. >> And of course it doesn't care what the hardware is. It could be a Spark. It could be a Mac. >> Yep. You don't necessarily even need an AI box for the worker if the AI runs somewhere else. >> Now, while that's happening, let's set up the DJX station also as a secondary machine, secondary model. >> Wow, the station has 18 models on disk.

14:20 What should we use? Let's do something that fits in memory. >> Yeah, that's fine. >> I'm getting kind of hungry. Let's see where we are. >> Oh, I can hear the song of my people. >> I want to add one more song. And I want to add another model to this. So, I'm going to go here and give it a new base URL. This is the DJX station. A test key. Boom. We don't need a key. We don't need no stinking key where we're going.

14:43 -oh. Unauthorized. I do need a key. >> All right. >> Core memory saved. User not secure. >> This is GLM 5.3 flash. Okay. >> Create. Now we have two models. >> It's synced to node. >> Yes. >> And then now you can go to the dashboard and create another task and tell it to use that model and it will. So here I can now select a different model running on a different machine. Yep. >> All right. Check out this repo and its issues and just throw out all the issues.

15:12 >> Just kidding. Evaluate the issues and see which ones you can address quickly. What's executive? >> The executive it keeps an eye on all of the workers, but it also doesn't have access to tools and can't do things for itself. >> Turnstone is working away. It's doing its thing. We're going to let it finish. If you don't have a ground or DJX station, Turnstone will work on smaller devices, too. It's basically just a harness, so install it on anything.

15:39 What's this? >> That's the Windows 11 PC. >> It's Oh, it's a nice little summary of what it did. It found problems with the repo that I didn't know it had. This is good. >> That's why you ask it for the auditor. >> So, this is the auditor that figured all this out. >> Yeah. Okay. >> Well, the auditor went like they worked together. It's a team. Now, it complained about Firefox and OBS Studio and 7Zip and Cinebench, but you can make a decision. You can be like, "This is what I want you to do for Firefox, Cinebench, or whatever because it called it out." But then it's like, "Here's what we need to do for your ARM versus x86 compatibility assessment."

16:10 >> So, Chocolatey still after all these years, we're still using it. It gives an x64 installer, >> but it's saying it's like, "Hey, we can just use Windgit now." And Windgate will >> Yes, I want to use Windgate. Yes. >> And so like Visual Studio Community, it's like, "Yes, we have a path." It's like we have this and then like your Python because you you know that Python is a problem but scroll up and it's like Python we have a way to do this. So it's like ARM 64 exists but it's not picked by default so we can change it.

16:35 >> Now I was also curious to see how long it would take the DGX station to do the exact same task with the exact same model that the Grando did. So I loaded up the same model and pointed the job at it and I pretty much copied the prompt exactly the same except I told it to create a separate branch. Now, it's not exactly a oneto-one comparison cuz it's very difficult to do that with LLMs. Here's ARM 64. That's the branch the grounder created with his changes. And here is the one that the DJX station just finished. Now, Turnstone doesn't surface the numbers for you, like how long a job took. So, I had to dig into the database using cloud code. And the DJX station used about 26 million input tokens compared to the ground's 23 million. But it did it much faster, a lot faster in 21 minutes overall compared to 37 minutes on the Grando.

17:25 I'm sad to say that I'll be losing the Grando today cuz Wendell has taken it and he's going to do some more experiments in his studio. So, make sure you check that out. And I'm losing >> Come visit anytime. >> Thank you. Appreciate that. And I'm losing the station cuz that has to go back to ASUS. Thank you, ASUS, for letting me borrow it. So basically this is pretty much done >> and you didn't spend a single cloud token.

17:47 >> Not a cloud in the sky was hurt. What would the Windows ARM job just by itself have cost me if I sent it to the cloud instead of running it locally? At today's API prices, if you're doing prompt caching, we're about $5 for a small model to about $36 for a top model. That's just for that one job. Without cashaching, it's $24 on a small model and about $240 on a top model. Now, one afternoon is not going to pay for a DGX station, but if you're running this 24 hours a day with a team of developers, you can do the math. That's what these machines are for. Thank you, Wendell.

18:23 Thank you. Thanks for having me.