prima.cpp

At a glance

Best for

Getting a 30B-70B model running at all on a few ordinary laptops, desktops and Android phones whose memory together is still too small.

Runs on

macOS (Metal), Linux (CUDA or processor) and Android through Termux can mix in one cluster. No Windows, no Vulkan or ROCm yet; Raspberry Pi-type boards are untested.

How it splits a model

Pipeline in a ring, with its Halda solver deciding how many layers each device takes (and leaving out devices that would slow the rest).

Connections

Made for home Wi-Fi: it barely needs bandwidth, but latency matters.

How devices join

Manual: each device is started with its place in the ring.

Models

GGUF (tested with Llama, Qwen 2.5, QwQ and DeepSeek R1 distills).

API

OpenAI compatible (llama-server on the first device).

Rules

Every device needs its own copy of the model file on a fast disk.

Get started

brew install gcc make cmake fio git wget highs zeromq curl, then make -j

Then:

./llama-server -m model.gguf --world 2 --rank 0 --master 192.168.1.2 --next 192.168.1.3 --prefetch   # device 1
./llama-cli -m model.gguf --world 2 --rank 1 --master 192.168.1.2 --next 192.168.1.2 --prefetch   # device 2

Published speeds

• Llama 3 70B Q4 on a Mac M1, two Linux PCs (RTX 3070 and RTX 2080 Ti) and a phone on Wi-Fi: about 1.5 tok/s, where llama.cpp on the best PC managed 0.1. Source
• QwQ 32B on the same home cluster: about 11 tok/s. Source

Measured by the project or its users, for one person chatting. Our own estimates for any set of devices are in the cluster builder.

Limits to know

• The open code hasn't changed since June 2026, and its makers' newer runtime is closed.
• No Windows.
• Few model families tested.
• Our estimates don't model its disk streaming, so for models bigger than the devices' memory it may run where we say it won't (slowly).

Open issues worth reading:

• A worker dying mid-answer hangs the serverprima.cpp issue #1 (opens in a new tab)