llama.cpp RPC

by ggml-org (llama.cpp) · MIT · 133,581 GitHub stars · Active · v0.9.0, Nov 2026

llama.cpp's own way to use other machines' graphics cards, Macs and processors as extra devices for one GGUF model, over the network.

Mac: Yes
NVIDIA PC: Yes
AMD PC: Yes
Linux, processor only: Yes
Raspberry Pi and boards: Yes
Windows: Yes
Android: Partly
iPhone, iPad: No

Go to llama.cpp RPC Source code

At a glance

Best for: Mixed devices you already own, a Mac, a gaming PC, a mini PC or a Jetson, when one GGUF model won't fit on any of them alone.
Runs on: Any llama.cpp build with RPC turned on can be a worker: Metal, CUDA, ROCm, Vulkan, SYCL or CPU. The official downloads for macOS, Linux and Windows include it, so a Mac, an NVIDIA PC and a Raspberry Pi can share one model.
How it splits a model: Layers in proportion to each device's free memory by default (you can set the shares). Since autumn 2026 also a tensor split (-sm tensor) with NCCL on NVIDIA machines.
Connections: Ordinary TCP over Ethernet or Wi-Fi. RDMA when both ends support it: Thunderbolt 5 between Macs on macOS 26.2+, or RoCE network cards on Linux (such as the DGX Spark's ConnectX port).
How devices join: Manual: you list every worker's address.
Models: GGUF, any architecture llama.cpp supports.
API: OpenAI compatible (llama-server).
Rules: Start ggml-rpc-server on each worker; the main machine lists them with --rpc. Only the main machine needs the model file; the -c cache keeps workers from downloading it again.

Get started

macOS

brew install llama.cpp

Linux

curl -LsSf https://llama.app/install.sh | sh

Windows

winget install llama.cpp

Then:

ggml-rpc-server --host 0.0.0.0 --port 50052 -c   # on each worker
llama-server -hf <model-GGUF> -ngl 99 --rpc 192.168.1.10:50052,192.168.1.11:50052   # on the main machine

Published speeds

Qwen3.6 27B Q4 on 1, 2 and 4 Mac Studios (M3 Ultra): 24.4 tok/s alone; split across 2, 22.5 over Thunderbolt RDMA and 18.0 over TCP. Splitting a model that already fits only adds overhead. Source
Qwen3.8 27B Q4 on 1 and 2 DGX Sparks with the tensor split: 13.0 and 23.5 tok/s. Source
Qwen3.5 122B on five 16 GB AMD BC-250 boards over 2.5 Gb Ethernet: about 29 tok/s (community build). Source

Measured by the project or its users, for one person chatting. Our own estimates for any set of devices are in the cluster builder.

Limits to know

Its own docs call it a proof of concept that is fragile and insecure: never expose a worker to an open network.
No discovery or passwords; you type every address.
Over plain TCP each extra machine slows typing a little for models that already fit on one.
Some models crash a worker that can't run one of their operations.

Open issues worth reading:

Example clusters it suits

Sources:

All clustering tools

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.