Apps · Inference server

LocalAI

by Ettore Di Giacinto (mudler) · MIT · 49,957 GitHub stars

OpenAI-compatible server that bundles llama.cpp for GGUF models and a reranker API in one install.

macOSWindows (WSL2)Linux

Get LocalAI

At a glance

Best for
One local endpoint for a GGUF guard model and a reranker side by side.
Inference engine
  • llama.cpp
  • rerankers (Python)
  • Transformers
GPU support
  • NVIDIA CUDA
  • AMD ROCm
  • Intel oneAPI / SYCL
  • Vulkan (AMD, Intel, NVIDIA)
  • Apple Metal
  • CPU only
Opens
  • GGUF
  • Safetensors (original weights)
Features
  • OpenAI-compatible API
  • Web chat UI
  • HTTP API
Platforms
Mac app (DMG) for Apple silicon; Docker images for Linux and Windows (WSL2) with CUDA, ROCm, Intel, Vulkan or CPU.
Speed
Speed follows the backend it loads: llama.cpp for GGUF, PyTorch for rerankers.

Sources: github.com · localai.io

Install

macOS

Download LocalAI.dmg from GitHub releases

Linux

docker run -ti -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12

Windows (WSL2)

Docker Desktop with WSL2, then the Linux docker command

30 models on this site run in LocalAI

Decision models it can load: rerankers, classifiers and scorers with a supported architecture. Open a model for its speed and memory on your hardware.

Model Made by Type Size Opens it as Quality tier
CompassJudger-2 32B OpenCompass Decision 33B GGUF 86
Qwen3Guard Gen 8B Alibaba Decision 8.2B GGUF 86
Granite Guardian 4.1 8B IBM Decision 8.4B GGUF 84
Llama 3.1 Nemotron Safety Guard 8B v3 NVIDIA Decision 8.0B GGUF 84
Qwen3Guard Gen 4B Alibaba Decision 4.4B GGUF 84
gpt-oss-safeguard 120B OpenAI Decision 117B (5.1B active) GGUF 80
Granite Guardian 3.3 8B IBM Decision 8.2B GGUF 80
Selene 1 Mini 8B Atla Decision 8.0B GGUF 80
DynaGuard 8B University of Maryland Decision 8.2B GGUF 79
CompassJudger-2 7B OpenCompass Decision 7.6B GGUF 78
Nemotron Content Safety Reasoning 4B NVIDIA Decision 4.3B GGUF 78
gpt-oss-safeguard 20B OpenAI Decision 22B (3.6B active) GGUF 76
NuExtract 2.0 4B NuMind Decision 3.8B GGUF 76
Prometheus 2 8x7B Prometheus Eval Decision 47B (13B active) GGUF 74
Qwen3Guard Gen 0.6B Alibaba Decision 750M GGUF 74
BGE Reranker v2 M3 BAAI Decision 570M Original weights 72
DynaGuard 1.7B University of Maryland Decision 2.0B GGUF 70
GTE Reranker ModernBERT Base Alibaba Decision 150M Original weights 70
Jina Reranker v2 Base Multilingual Jina AI Decision 280M Original weights 70
NuExtract 2.0 2B NuMind Decision 2.2B GGUF 70
Granite Embedding Reranker English R2 IBM Decision 150M Original weights 68
Llama Guard 3 8B Meta Decision 8.0B GGUF 68
Prometheus 2 7B Prometheus Eval Decision 7.2B GGUF 68
GLIDER Patronus AI Decision 3.8B GGUF 66
ShieldGemma 27B Google Decision 27B GGUF 66
Flow Judge v0.1 Flow AI Decision 3.8B GGUF 64
ShieldGemma 9B Google Decision 9.2B GGUF 64
ShieldGemma 2B Google Decision 2.6B GGUF 58
Llama Guard 3 1B Meta Decision 1.5B GGUF 56
ms-marco MiniLM L6 v2 Sentence Transformers Decision 23M Original weights 52

Alternatives

LM Studio Ollama llama.cpp vLLM Jan Text Embeddings Inference MLX LM SGLang GLiNER Lemonade Infinity Unsloth Desktop Sentence Transformers Docker Model Runner Atomic Chat

Which app should I use?

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.