Cluster your computers to run bigger models

A model that doesn’t fit on any one of your devices can sometimes run across several of them. Free tools such as EXO and llama.cpp RPC split the model so each device holds part of it, and pass the work between them over your network. You get the memory of all of them together. Whether you also get speed depends on how the model is split and how fast the devices and the connection are.

1. Two speeds, two different limits

Every answer has two phases (two speeds). First the model reads your whole prompt at once: that is limited by raw compute, and it’s the pause before the first word. Then it writes one token at a time, reading all of its active weights for every token: that is limited by memory bandwidth. A cluster can change each phase differently, so always ask which one a speed-up figure is about.

EXO’s run with a DGX Spark and a Mac Studio shows the two phases apart: the Spark, with more compute, reads the prompt; the Mac, with faster memory, types.

2. Two ways to split a model

• Pipeline split (layers in turn): each device holds a run of layers, and each token passes from one device to the next. Only one device works at a time, so typing is no faster than the devices’ average, and each hop over the network adds a little delay. Its value is memory: a 70B model can run on a Mac plus a gaming PC that couldn’t run it apart. llama.cpp RPC, Mesh LLM, prima.cpp and Pooled split this way.
• Tensor split (every layer shared): every device holds a slice of every layer, and they all work on each token at once, then swap results twice per layer. That can make typing faster with each device you add, but only on a very low-latency link. EXO and MLX distributed do it over Thunderbolt 5 with RDMA between Macs; Distributed Llama does it on wired Ethernet for small boards; llama.cpp RPC and vLLM do it between NVIDIA machines.

Here are both at work. The detailed animations add our estimates for real devices: where each word’s time goes in a pipeline, and what a slow link does to a tensor split.

Mixture-of-experts models gain less from a tensor split than dense ones: in Apple’s and EXO’s published runs, four Mac Studios typed a dense 123B model 2.9 times faster than one, but a 358B mixture-of-experts model only 1.6 times faster.

3. The connection matters

• Wi-Fi adds a few milliseconds every time a token crosses it, and seconds to every long prompt. It works for experiments; it’s the slowest choice.
• Gigabit or 2.5 Gb Ethernet is fine for a pipeline split, and enough for Raspberry Pi clusters with Distributed Llama.
• 10 Gb Ethernet and Thunderbolt cables carry long prompts quickly. Two Macs can simply be joined with a Thunderbolt cable.
• Thunderbolt 5 with RDMA (macOS 26.2 or later, enabled once from macOS Recovery, a cable between every pair of Macs) cuts the delay of each message to a few microseconds. It’s what makes a tensor split worth it on Macs.

4. The slowest device sets the pace

In a pipeline split, every token has to pass through every device that holds part of the model. A Raspberry Pi 4 reads its memory more than 40 times slower than an M4 Pro MacBook, so a model that puts even a few gigabytes on the Pi types at seconds per word. Use a slow device only when a model fits nowhere else, and otherwise leave it out. Our cluster builder shows which of your devices each model needs and which ones only slow things down.

5. What the headline demos really measured

• Four 512 GB Mac Studios wired with Thunderbolt 5 RDMA run 670B to 1T-parameter models at 4 to 8 bits: about 32 tokens per second for DeepSeek V3.1 and 28 for Kimi K2 Thinking in the runs EXO publishes. That is real, and it is about $38,000 of Macs, not the devices most people have lying around.
• EXO’s DGX Spark + Mac Studio demo read the prompt on the Spark (four times the Mac’s compute) and typed on the Mac (three times the Spark’s memory speed). The 2.8× speed-up came from an 8,192-token prompt with a 32-token answer. The typing itself was exactly as fast as the Mac alone: the whole gain was in the wait for the first word. EXO’s own arithmetic says the transfer only hides behind the compute on 10 Gb Ethernet when the prompt runs to roughly 5,000 to 10,000 tokens or more, so it suits long prompts with short answers. It was a research build: in the public EXO app, Macs and Linux machines can’t join one cluster.
• The biggest open models are still beyond any desk. A 2.8-trillion-parameter model such as Kimi K3 ships in native 4-bit and needs about 4.8 TB of memory to run as released, more than twice what four 512 GB Mac Studios hold. Squeezing it further with another 4-bit quantization stacks one lossy compression on another, so what runs is no longer quite the same model.

6. Setting one up

• Only Macs: install EXO on each (brew install --cask exo), open it, and the Macs find each other. Use the same EXO version everywhere, and Thunderbolt cables if you can.
• A mix of Macs, PCs and boards: install llama.cpp on each, start ggml-rpc-server --host 0.0.0.0 -c on every helper, then run llama-server on the main machine with --rpc and the helpers’ addresses. Keep it on your home network: it has no password (llama.cpp RPC).
• Two to eight identical Raspberry Pis or mini PCs: Distributed Llama on a gigabit switch, with a power-of-two number of boards.
• Prefer an app with tokens over addresses? Mesh LLM joins machines with an invite token; Pooled runs in browser tabs, iPhones included.
• Ollama can’t cluster; llama.cpp RPC runs the same GGUF files.

A quick checklist

• Does the model fit on one device? Then don’t cluster: one device is faster.
• Put the most memory on the device with the fastest memory, and leave slow devices out.
• Use a cable, not Wi-Fi.
• For long prompts with short answers (a big codebase, a long report), a cluster with a high-compute device helps most.

Try it with your own devices in the cluster builder, or start from an example cluster. Sources: The Stack’s video “EXO: Can Frontier AI Models Now Run Locally?”, EXO’s README and blog, Apple’s MLX documentation, and each tool’s own repository (all tools).