Distributed Llama

by Bartłomiej Tadych (b4rtaz) · MIT · 3,132 GitHub stars · Maintained · v0.16.5, Feb 2026

Splits every layer of a model across 2, 4 or 8 machines that work in parallel on each word, so a stack of small boards types faster than one.

Mac: Partly
NVIDIA PC: Partly
AMD PC: Partly
Linux, processor only: Yes
Raspberry Pi and boards: Yes
Windows: Yes
Android: No
iPhone, iPad: No

Go to Distributed Llama Source code

At a glance

Best for: Identical small machines on a wired switch, such as four Raspberry Pi 5s or a few Mac minis, where adding boards really speeds up typing.
Runs on: Built for ARM and x86 processors on Linux, macOS and Windows. Graphics cards only through experimental Vulkan, which can be slower than the processor.
How it splits a model: Tensor only: each machine holds a slice of every layer, and they swap results for every layer of every word.
Connections: Wired Ethernet (a gigabit switch is enough for Raspberry Pis) or Thunderbolt networking; Wi-Fi is too slow for its per-layer syncing.
How devices join: Manual: start a worker on each machine and list them on the main one.
Models: Its own converted format (4-bit Q40 or full precision); ready-made Llama 3.x, DeepSeek R1 distill and Qwen3 models, and a converter for Llama, Mistral and Qwen3.
API: OpenAI compatible (dllama-api).
Rules: 1, 2, 4 or 8 machines, and no more than the model's KV heads.

Get started

macOS

git clone https://github.com/b4rtaz/distributed-llama && cd distributed-llama && make dllama dllama-api

Linux

git clone https://github.com/b4rtaz/distributed-llama && cd distributed-llama && make dllama dllama-api

Then:

./dllama worker --port 9999 --nthreads 4   # on each worker
python3 launch.py llama3_2_3b_instruct_q40   # on the main machine, then dllama-api with --workers

Published speeds

Llama 3.2 3B on 1, 2 and 4 Raspberry Pi 5s over gigabit Ethernet: 5.95, 10.3 and 13.7 tok/s. Source
Qwen3 30B-A3B on four 8 GB Raspberry Pi 5s: 13.0 tok/s. Source
Llama 3.3 70B on four 24 GB Mac minis (M4 Pro, processor only): 6.5 tok/s over Thunderbolt, 6.0 over 10 Gb Ethernet. Source

Measured by the project or its users, for one person chatting. Our own estimates for any set of devices are in the cluster builder.

Limits to know

No GGUF: models must be in its own format.
Needs a wired network.
Machines should be alike: the slowest one sets the pace.

Open issues worth reading:

Qwen3 MoE models don't run on 8 nodesdistributed-llama issue #270 (opens in a new tab)

Sources:

All clustering tools

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.