LocalAI (P2P workers)

At a glance

Best for

People already running LocalAI who want workers to join with a token instead of typing addresses.

Runs on

Containers for CPU, CUDA, ROCm, Intel and Vulkan, Jetson and DGX Spark images, and a Mac app; Windows through containers.

How it splits a model

Worker mode splits one llama.cpp model by memory (llama.cpp RPC); federated mode only sends whole requests to whichever machine is free.

Connections

llama.cpp RPC over its peer-to-peer overlay, or MLX over TCP or Thunderbolt RDMA.

How devices join

A shared token (libp2p).

Models

GGUF in worker mode; MLX with MLX distributed.

API

OpenAI, Anthropic and Ollama compatible.

Rules

One model at a time in worker mode, and workers must join before the model loads.

Get started

docker run -p 8080:8080 localai/localai:latest

Then:

local-ai run --p2p   # main machine; copy the token
TOKEN=<token> local-ai p2p-worker p2p-llama-cpp-rpc   # each worker

Limits to know

• Its docs call the distributed modes experimental.
• Speeds are the same as llama.cpp RPC, which our estimates use.

Open issues worth reading:

• v4.12.1 prints an outdated worker commandLocalAI issue #13032 (opens in a new tab)

We don’t estimate speeds for LocalAI (P2P workers): it uses llama.cpp RPC underneath, so see llama.cpp RPC’s figures.