This New Engine Runs Local AI Using 10x Less RAM! (Cactus)

0:00 This is Cactus. It's a low-latency inference engine designed to treat mobile and edge devices as first-class citizens. So, usually when we try to run AI models on edge devices, they feel heavy and battery draining and prone to getting killed by the mobile operating system's memory manager. But, Cactus is trying to solve this problem because it's built specifically for the constraints of the neural processing units and limited RAM. So, today we're going to look at Cactus, see how it works, and test it out on an edge device to see how it performs.

0:34 So, let's dive into it. The biggest bottleneck for local AI isn't actually compute, it's the memory overhead. On a standard mobile device, the operating system is extremely aggressive about killing apps that spike in RAM usage. But, Cactus solves this by using a zero-copy memory mapping. Instead of the usual approach where you load everything to RAM, Cactus maps model weights directly from storage.

1:05 It's a zero-copy system that only pulls specific tensors into the active compute cycle as they are needed. You get the reasoning power of a large model without the risk of the operating system shutting your app down. And to achieve this, they've even transitioned away from the traditional GGUF format and have their own proprietary {dot}cact format that allows this mapping to be effective on edge devices. But, the real heavy lifting happens in the NPU or the neural processing unit. While most local engines default to the GPU, Cactus is built to be NPU first. If you've looked at modern chips from Apple, Qualcomm, or MediaTek, they all have dedicated silicon just for neural networks. Cactus communicates with these units directly, bypassing the usual translation layers that slow down your inference. And they've actually optimized specific models to take full advantage of these matrix multiplication units. If you head over to the Cactus dashboard, you'll see a list of NPU optimized models ready for download. And another cool feature Cactus has is the hybrid router. Now, the reality is that on edge devices, local models, no matter how optimized, eventually hit that reasoning ceiling.

2:22 And this is where the hybrid router comes in. Instead of forcing you to choose between a fast but limited local model and a smart but expensive cloud model, Cactus can handle both and swap between them. It uses a confidence-based routing system, and if you ask it a simple question, it stays on the NPU because it's fast, private, and costs you nothing. But if the local model senses that the task is too complex or require a massive context window, it automatically hands the specific request off to a frontier model on the cloud.

2:55 Your code stays the same, the engine just manages the failover in the background. So, it's a production-ready way to keep costs low without sacrificing the user experience when things get complicated. Now, all that sounds cool, but I want to try it out for myself. So, on their landing page, they have this demo where they show how you can do a real-time transcription with about 100 ms latency on an edge device. So, I went ahead and vibe coded a little Swift app using their Swift Cactus package that supports running a real-time transcription using their Parakeet speech model locally and a Gemini model on the cloud. So, let's try it out. As you can see, locally, we are averaging about 260 ms of latency with live streaming. And mind you, I'm running this on an older iPhone model, the 12 Pro. So, for an older model like this one, I think this performance on edge is pretty good. And if we switch to cloud, Cactus switches to Gemini 2.5 flash as the cloud alternative. And for some reason, they don't have the same Parakeet model on their cloud side, so I was forced to use Gemini. And we can see here that this is averaging at about 2,000 milliseconds for a 3-second batch transcription. And I guess this is to be expected because it is doing a round trip to the data server. But realistically, most of the time you would end up using the on-Edge transcription anyway. But the cloud option is useful for other tasks like heavy image analysis or something else that would be a heavier task. So there you have it, folks. That is the Cactus engine in a nutshell. I think they are doing something really interesting here.

4:36 I like how they are thinking about on-Edge optimization using a custom MPU-friendly architecture. And I like the fact that they offer so many SDKs and so many models for all sorts of multimodal tasks. And I'm really curious to see how their product evolves, so I'll be keeping an eye on their progress for sure. But what do you folks think about Cactus? Have you tried it? Let us know in the comments section down below. And folks, if you like these types of breakdowns, please let me know by smashing that like button underneath the video. And also, don't forget to subscribe to our channel. This has been Andres from Better Stack, and I will see you in the next videos.

5:17 >>