Mesh LLM

At a glance

Best for

A household or office with mixed Macs and NVIDIA or AMD PCs that wants one shared local endpoint, and now and then a model too big for any one machine.

Runs on

Ready-made downloads for macOS (Metal), Linux (CPU, CUDA, ROCm, Vulkan, ARM64) and Windows. RTX 50 cards on Windows need WSL2.

How it splits a model

Pipeline only: whole layers per machine. The planner skips peers more than 80 ms away and adds machines only until the model fits.

Connections

Encrypted QUIC between peers (iroh), on a LAN or across the internet with relays; no RDMA.

How devices join

Invite token for a private mesh; an optional public mesh is found automatically.

Models

GGUF, packaged so each machine downloads only its own layers; splitting is limited to certified model families.

API

OpenAI compatible.

Rules

Every machine uses the same layer-package of the model. An invite token is connection information, not a password.

Get started

brew install Mesh-LLM/tap/mesh-llm

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

Then:

mesh-llm serve --model hf://meshllm/<repo> --split   # on the first machine
mesh-llm serve --join <token>   # on each other machine

Published speeds

• GLM 4.7 Flash Q4 over Wi-Fi: 68 tok/s on one Mac, 21 split across two (an M4 Max and a Mac mini M4) and about 13 across three (April 2026, before the current split runtime). Source

Measured by the project or its users, for one person chatting. Our own estimates for any set of devices are in the cluster builder.

Limits to know

• Every stage adds a network hop to each word, so a split model types slower than one machine would.
• Its makers call it experimental distributed-systems software.

Open issues worth reading:

• Placement ignores machine-to-machine latencymesh-llm issue #1020 (opens in a new tab)