Apps · Command line + local API

Docker Model Runner

by Docker · Apache-2.0 · 658 GitHub stars

Runs GGUF or safetensors models inside Docker with an OpenAI-compatible endpoint for your containers.

macOSWindowsLinux

Get Docker Model Runner

At a glance

Best for
Developers who already build with Docker and Compose.
Inference engine
  • llama.cpp
  • vLLM
GPU support
  • Apple Metal
  • NVIDIA CUDA
  • AMD ROCm
  • Vulkan (AMD, Intel, NVIDIA)
  • CPU only
Opens
  • GGUF
  • Safetensors (original weights)
Features
  • OpenAI-compatible API
  • GPU + RAM split for big models
Platforms
Docker Desktop on Apple silicon Macs and Windows (NVIDIA), Docker Engine on Linux (CUDA, ROCm, Vulkan). The vLLM engine needs NVIDIA on Linux or WSL2.

Sources: docs.docker.com

Install

macOS

Docker Desktop 4.40+

Windows

Docker Desktop 4.41+

Linux

Docker Engine with the model runner plugin

562 models on this site run in Docker Model Runner

Based on the files published for each model: GGUF, Safetensors (original weights). Open a model to see the exact command or file for your quantization.

Model Made by Type Size Opens it as Quality tier
CompassJudger-2 32B OpenCompass Decision 33B GGUF 86
Qwen3Guard Gen 8B Alibaba Decision 8.2B GGUF 86
Shieldstral 1.0 3B Mistral AI Decision 3.9B Original weights 86
Granite Guardian 4.1 8B IBM Decision 8.4B GGUF 84
Llama 3.1 Nemotron Safety Guard 8B v3 NVIDIA Decision 8.0B GGUF 84
Qwen3Guard Gen 4B Alibaba Decision 4.4B GGUF 84
gpt-oss-safeguard 120B OpenAI Decision 117B (5.1B active) GGUF 80
Granite Guardian 3.3 8B IBM Decision 8.2B GGUF 80
Selene 1 Mini 8B Atla Decision 8.0B GGUF 80
WildGuard 7B Ai2 Decision 7.3B Original weights 80
DynaGuard 8B University of Maryland Decision 8.2B GGUF 79
CompassJudger-2 7B OpenCompass Decision 7.6B GGUF 78
Nemotron Content Safety Reasoning 4B NVIDIA Decision 4.3B GGUF 78
gpt-oss-safeguard 20B OpenAI Decision 22B (3.6B active) GGUF 76
NuExtract 2.0 4B NuMind Decision 3.8B GGUF 76
Prometheus 2 8x7B Prometheus Eval Decision 47B (13B active) GGUF 74
Qwen3Guard Gen 0.6B Alibaba Decision 750M GGUF 74
DynaGuard 1.7B University of Maryland Decision 2.0B GGUF 70
NuExtract 2.0 2B NuMind Decision 2.2B GGUF 70
Llama Guard 3 8B Meta Decision 8.0B GGUF 68
Prometheus 2 7B Prometheus Eval Decision 7.2B GGUF 68
GLIDER Patronus AI Decision 3.8B GGUF 66
Llama Guard 4 12B Meta Decision 12B Original weights 66
ShieldGemma 27B Google Decision 27B GGUF 66
Flow Judge v0.1 Flow AI Decision 3.8B GGUF 64
ShieldGemma 9B Google Decision 9.2B GGUF 64
ShieldGemma 2B Google Decision 2.6B GGUF 58
Llama Guard 3 1B Meta Decision 1.5B GGUF 56
MiMo V2.6 Pro (1T-A42B) Xiaomi LLM 1,024B (42.0B active) GGUF 99
Qwen3.8 Max (2.4T-A95B) Alibaba LLM 2,446B (95.0B active) GGUF 98.3
GLM-5.3 (753B-A40B) Zhipu AI LLM 753B (40.0B active) GGUF 97.8
MiMo V2.6 Flash (310B-A15B) Xiaomi LLM 311B (15.0B active) GGUF 97.2
DeepSeek V4 Pro (1.6T-A49B) DeepSeek LLM 1,651B (49.0B active) GGUF 96.8
GLM-5.2 (753B-A40B) Zhipu AI LLM 753B (40.0B active) GGUF 96.3
Hunyuan Hy4 Preview (780B) Tencent LLM 780B (55.0B active) GGUF 96.3
Qwen3.8 Flash Next (125B-A6B) Alibaba LLM 180B (6.0B active) GGUF 96.5
DeepSeek V4 Pro (April 2026) DeepSeek LLM 1,599B (49.0B active) Original weights 95
GLM-5.1 (744B-A40B) Zhipu AI LLM 754B (40.0B active) GGUF 95.2
GLM-5.3 Flash (320B-A18B) Zhipu AI LLM 321B (18.0B active) GGUF 95.4
Kimi K2.6 (1T) Moonshot AI LLM 1,027B (32.0B active) GGUF 95.7
Qwen3.5 397B-A17B Alibaba LLM 403B (17.0B active) GGUF 95
DeepSeek V4 Flash (284B-A13B) DeepSeek LLM 304B (13.0B active) GGUF 94
GLM-5 (744B-A40B) Zhipu AI LLM 754B (40.0B active) GGUF 94.7
Hunyuan Hy3 (295B-A21B) Tencent LLM 299B (21.0B active) GGUF 94.4
Kimi K2.5 (1T) Moonshot AI LLM 1,027B (32.0B active) GGUF 94.3
Kimi K2.7 Code (1T) Moonshot AI LLM 1,027B (32.0B active) GGUF 94.3
MiniMax M3 (427B) MiniMax LLM 427B (25.0B active) GGUF 94.8
Ornith 1.5 397B-A17B DeepReinforce LLM 403B (17.0B active) GGUF 94
Qwen3.6 27B Alibaba LLM 28B GGUF 94
DeepSeek R1 0528 (671B) DeepSeek LLM 685B (37.0B active) GGUF 93
DeepSeek V3.2 (671B) DeepSeek LLM 685B (37.0B active) GGUF 93.7
GLM-4.7 (355B-A32B) Zhipu AI LLM 358B (32.0B active) GGUF 93.5
Kimi K2 Thinking (1T) Moonshot AI LLM 1,026B (32.0B active) GGUF 93
MiniMax M2.7 (230B-A10B) MiniMax LLM 229B (10.0B active) GGUF 93.5
Nemotron 3 Ultra 550B-A55B NVIDIA LLM 561B (55.0B active) GGUF 93.2
DeepSeek R1 (671B) DeepSeek LLM 671B (37.0B active) GGUF 92
DeepSeek V3.2 Exp DeepSeek LLM 685B (37.0B active) Original weights 92.5
DeepSeek V4 Flash (April 2026) DeepSeek LLM 291B (13.0B active) GGUF 92
Falcon 180B Chat TII LLM 180B GGUF 92
Gemma 4 26B-A4B Google LLM 26B (3.8B active) GGUF 92.5
MiniMax M2.5 (230B-A10B) MiniMax LLM 229B (10.0B active) GGUF 92.5
Qwen3.5 122B-A10B Alibaba LLM 125B (10.0B active) GGUF 92
Step 3.5 Flash (196B-A11B) StepFun LLM 199B (11.0B active) GGUF 92
DeepSeek V3.1 (671B) DeepSeek LLM 685B (37.0B active) GGUF 91
DeepSeek V3.1 Terminus DeepSeek LLM 685B (37.0B active) GGUF 91.5
Gemma 4 31B Google LLM 31B GGUF 91
Kimi K2 Instruct 0905 (1T) Moonshot AI LLM 1,026B (32.0B active) GGUF 91
MiniMax M2.1 (230B-A10B) MiniMax LLM 229B (10.0B active) GGUF 91
Qwen3 235B-A22B Thinking 2507 Alibaba LLM 235B (22.0B active) GGUF 91
Qwen3 Coder 480B-A35B Alibaba LLM 480B (35.0B active) GGUF 91
Kimi K3 (2.8T) Moonshot AI LLM 2,780B (104.0B active) GGUF 90
Ling 1T Ant Group (inclusionAI) LLM 1,000B (50.0B active) GGUF 90
MiMo V2 Flash (309B-A15B) Xiaomi LLM 310B (15.0B active) GGUF 90.9
Mistral Large 3 (675B) Mistral AI LLM 675B (41.0B active) GGUF 90
Qwen3 235B-A22B Instruct 2507 Alibaba LLM 235B (22.0B active) GGUF 90
Solar Open 2 250B Upstage LLM 250B (15.0B active) Original weights 90
Apodex 1.1 Mini (35B-A3B) Apodex LLM 36B (3.0B active) GGUF 89
Kimi K2 Instruct (July 2025) Moonshot AI LLM 1,026B (32.0B active) GGUF 89
MiniMax M2 (230B-A10B) MiniMax LLM 229B (10.0B active) GGUF 89
Qwen3-VL 235B-A22B Instruct Alibaba LLM 236B (22.0B active) GGUF 89
Fara 1.5 27B Microsoft LLM 27B GGUF 88
GLM-4.7 Flash (30B-A3B) Zhipu AI LLM 31B (3.0B active) GGUF 88
K-EXAONE 236B-A23B LG AI Research LLM 237B (23.0B active) GGUF 88
LongCat Flash Chat (560B) Meituan LLM 562B (27.0B active) GGUF 88
Nemotron 3 Super 120B-A12B NVIDIA LLM 124B (12.0B active) GGUF 88
Qwen3 235B-A22B Alibaba LLM 235B (22.0B active) GGUF 88
Agents-A1 35B-A3B InternScience LLM 35B (3.0B active) GGUF 87
Qwen AgentWorld 35B-A3B Alibaba LLM 35B (3.0B active) GGUF 87
Devstral 2 (123B) Mistral AI LLM 125B GGUF 86
InternVL3.5 241B-A28B Shanghai AI Lab LLM 241B (28.0B active) GGUF 86
Laguna S 2.1 (118B-A8B) Poolside LLM 118B (8.0B active) GGUF 86
Llama 3.1 Nemotron Ultra 253B v1 NVIDIA LLM 253B GGUF 86
Qwen3 Coder Next (80B-A3B) Alibaba LLM 80B (3.0B active) GGUF 86
Qwen3-Next 80B-A3B Thinking Alibaba LLM 81B (3.0B active) GGUF 86
UI-Mate 27B Tencent LLM 27B GGUF 86
Baichuan M3 235B-A22B Baichuan LLM 235B (22.0B active) GGUF 84
GLM-4.6 (355B-A32B) Zhipu AI LLM 357B (32.0B active) GGUF 84
gpt-oss 120B OpenAI LLM 117B (5.1B active) GGUF 84
Qwen3 Next 80B-A3B Instruct Alibaba LLM 81B (3.0B active) GGUF 84
DeepSeek V3 0324 (671B) DeepSeek LLM 685B (37.0B active) GGUF 83
Gemma 4 12B Google LLM 12B GGUF 83
GLM-4.6V Zhipu AI LLM 108B (12.0B active) GGUF 83
MiniMax-M1 80K MiniMax LLM 456B (45.9B active) Original weights 83
Command A Reasoning (111B) Cohere LLM 111B GGUF 82
DeepSeek R1 Distill Llama 70B DeepSeek LLM 71B GGUF 82
EXAONE 4.5 33B LG AI Research LLM 34B GGUF 82
GLM-4.5 (355B-A32B) Zhipu AI LLM 358B (32.0B active) GGUF 82
Leanstral 119B-A6B Mistral AI LLM 119B (6.5B active) Original weights 82
Ling 3.0 Flash (127B-A5B) Ant Group (inclusionAI) LLM 127B (5.1B active) GGUF 82
Llama 2 70B Chat Meta LLM 69B GGUF 82
Llama 3.3 70B Instruct Meta LLM 71B GGUF 82
MiniMax-M1 40K MiniMax LLM 456B (45.9B active) Original weights 82
Command A (111B) Cohere LLM 111B GGUF 80
ERNIE 4.5 300B-A47B Baidu LLM 300B (47.0B active) GGUF 80
GLM-4.5V Zhipu AI LLM 108B (12.0B active) GGUF 80
Hermes 4 405B Nous Research LLM 406B GGUF 80
Kimi Dev 72B Moonshot AI LLM 73B GGUF 80
Llama 3.3 Nemotron Super 49B v1.5 NVIDIA LLM 50B GGUF 80
Mistral Large 2.1 (2411) Mistral AI LLM 123B GGUF 80
Mistral Medium 3.5 (128B) Mistral AI LLM 128B GGUF 80.4
Mistral Small 4 (119B-A6B) Mistral AI LLM 119B (6.0B active) GGUF 80.6
North Mini Code 1.0 (30B-A3B) Cohere LLM 30B (3.0B active) GGUF 80
Seed-OSS 36B Instruct ByteDance LLM 36B GGUF 80
Solar Open 100B Upstage LLM 103B (12.0B active) GGUF 80
Trinity Large Thinking (400B-A13B) Arcee AI LLM 399B (13.0B active) GGUF 80.3
Hermes 4.3 36B Nous Research LLM 36B GGUF 79
DiffusionGemma 26B-A4B IT Google LLM 26B (4.0B active) GGUF 78
GLM-4.5 Air (106B-A12B) Zhipu AI LLM 110B (12.0B active) GGUF 78
Mistral Large 2 (123B) Mistral AI LLM 123B GGUF 78
Pixtral Large 124B (2411) Mistral AI LLM 124B Original weights 78
Qwen3 30B-A3B Thinking 2507 Alibaba LLM 31B (3.3B active) GGUF 78
Qwen3 Coder 30B-A3B Alibaba LLM 31B (3.3B active) GGUF 78
Qwen3-VL 32B Instruct Alibaba LLM 33B GGUF 78
QwQ 32B Alibaba LLM 33B GGUF 78
Ring Flash 2.0 (100B-A6B) Ant Group (inclusionAI) LLM 103B (6.1B active) GGUF 78
Nemotron 3 Nano 30B-A3B NVIDIA LLM 32B (3.5B active) GGUF 77
Qwen3 30B-A3B Instruct 2507 Alibaba LLM 31B (3.3B active) GGUF 77
Devstral Small 2 (24B) Mistral AI LLM 24B GGUF 76
dots.llm1 Instruct rednote LLM 143B (14.0B active) GGUF 76
Granite 4.2 30B IBM LLM 29B GGUF 76
InternVL3.5 38B Shanghai AI Lab LLM 38B GGUF 76
Ling Flash 2.0 (100B-A6B) Ant Group (inclusionAI) LLM 103B (6.1B active) GGUF 76
MiniMax-Text-01 MiniMax LLM 456B (45.9B active) Original weights 76
Olmo 3.1 32B Think Ai2 LLM 32B GGUF 76
OpenReasoning Nemotron 32B NVIDIA LLM 33B GGUF 76
Qwen3 32B Alibaba LLM 33B GGUF 76
Qwen3-VL 30B-A3B Thinking Alibaba LLM 31B (3.3B active) GGUF 76
Qwen3.5 9B Alibaba LLM 9.7B GGUF 76
Skywork OR1 32B Skywork LLM 33B GGUF 76
Tülu 3 405B Ai2 LLM 406B GGUF 76
DeepSeek V3 (671B) DeepSeek LLM 671B (37.0B active) GGUF 75
Sarvam 105B Sarvam AI LLM 106B (10.0B active) GGUF 75
DeepSeek Coder V2 (236B) DeepSeek LLM 236B (21.0B active) GGUF 74
EXAONE Deep 32B LG AI Research LLM 32B GGUF 74
GLM-4 32B 0414 Zhipu AI LLM 33B GGUF 74
GLM-Z1 32B (0414) Zhipu AI LLM 33B GGUF 74
Hermes 4 70B Nous Research LLM 71B GGUF 74
Hunyuan A13B (80B-A13B) Tencent LLM 80B (13.0B active) GGUF 74
InternVL3 78B Shanghai AI Lab LLM 78B GGUF 74
Kimi Linear 48B-A3B Moonshot AI LLM 49B (3.0B active) GGUF 74
Qwen3 30B-A3B Alibaba LLM 31B (3.3B active) GGUF 74
Qwen3-VL 30B-A3B Instruct Alibaba LLM 31B (3.3B active) GGUF 74
Qwen2.5 72B Instruct Alibaba LLM 73B GGUF 73
Qwen2.5-VL 72B Instruct Alibaba LLM 73B GGUF 73
Apriel 1.5 15B Thinker ServiceNow LLM 15B GGUF 72
DeepSeek Coder V2 Instruct (0614) DeepSeek LLM 236B (21.0B active) GGUF 72
DeepSeek R1 Distill Qwen 32B DeepSeek LLM 33B GGUF 72
DeepSeek V2.5 1210 DeepSeek LLM 236B (21.0B active) GGUF 72
EXAONE 4.0 32B LG AI Research LLM 32B GGUF 72
Gemma 3 27B IT Google LLM 27B GGUF 72
Granite 4.1 30B IBM LLM 29B GGUF 72
Llama 3.1 405B Instruct Meta LLM 406B GGUF 72
Llama 4 Maverick (17B x 128E) Meta LLM 402B (17.0B active) GGUF 72
Llama 4 Scout 17B-16E Meta LLM 109B (17.0B active) GGUF 72
Magistral Small 1.2 (24B) Mistral AI LLM 24B GGUF 72
Nemotron 3.5 Lightning 30B-A3B NVIDIA LLM 32B (3.0B active) GGUF 72
Olmo 3.1 32B Instruct Ai2 LLM 32B GGUF 72
Phi-4 Reasoning Plus (14B) Microsoft LLM 15B GGUF 72
Qwen3.6 35B-A3B Alibaba LLM 36B (3.0B active) GGUF 72
Tülu 3 70B Ai2 LLM 71B GGUF 72
Nemotron Cascade 2 30B-A3B NVIDIA LLM 32B (3.5B active) GGUF 71
Ornith 1.0 35B-A3B DeepReinforce LLM 36B (3.0B active) GGUF 71
Ornith 1.5 35B-A3B DeepReinforce LLM 36B (3.0B active) GGUF 71
Qwen3.5 35B-A3B Alibaba LLM 36B (3.0B active) GGUF 71
Baichuan M2 32B Baichuan LLM 33B GGUF 70
Command R+ (08-2024) 104B Cohere LLM 104B GGUF 70
DeepSeek V2.5 DeepSeek LLM 236B (21.0B active) GGUF 70
Devstral Small 1.1 (24B) Mistral AI LLM 24B GGUF 70
ERNIE 4.5 21B-A3B Thinking Baidu LLM 22B (3.0B active) GGUF 70
gpt-oss 20B OpenAI LLM 21B (3.6B active) GGUF 70
InternVL3 38B Shanghai AI Lab LLM 38B GGUF 70
Jamba 1.5 Large AI21 Labs LLM 399B (94.0B active) Original weights 70
Laguna XS 2.1 (33B-A3B) Poolside LLM 33B (3.0B active) GGUF 70
Llama 3.1 70B Instruct Meta LLM 71B GGUF 70
Magistral Small 2507 Mistral AI LLM 24B GGUF 70
MedGemma 27B Text IT Google LLM 27B GGUF 70
OpenHands LM 32B All Hands AI LLM 33B GGUF 70
Phi-4 Reasoning (14B) Microsoft LLM 15B GGUF 70
Phi-4 Reasoning Vision 15B Microsoft LLM 15B Original weights 70
Qwen2.5 32B Instruct Alibaba LLM 33B GGUF 70
Qwen3-Omni 30B-A3B Instruct Alibaba LLM 35B (3.3B active) GGUF 70
Qwen3.5 27B Alibaba LLM 28B GGUF 70
Qwen3.8 27B Alibaba LLM 28B GGUF 70
Sarvam 30B Sarvam AI LLM 32B (2.4B active) GGUF 70
TMax 27B Ai2 LLM 27B Original weights 70
UI-Mate 9B Tencent LLM 9.4B GGUF 70
UI-Venus 2 9B Ant Group (inclusionAI) LLM 9.4B GGUF 70
Gemma 4 E4B Google LLM 8.0B GGUF 68
Granite 4.2 8B IBM LLM 8.8B GGUF 68
Hermes 3 (Llama 3.1 70B) Nous Research LLM 71B GGUF 68
Magistral Small 2506 Mistral AI LLM 24B GGUF 68
Mistral Small 3.2 24B Mistral AI LLM 24B GGUF 68
QVQ 72B Preview Alibaba LLM 73B GGUF 68
Qwen2.5-VL 32B Instruct Alibaba LLM 33B GGUF 68
Qwen3 14B Alibaba LLM 15B GGUF 68
Qwen2.5 Coder 32B Instruct Alibaba LLM 33B GGUF 67
AceReason Nemotron 14B NVIDIA LLM 15B GGUF 66
Apertus v1.5 70B Swiss AI LLM 72B Original weights 66
Apriel Nemotron 15B Thinker ServiceNow LLM 15B GGUF 66
Devstral Small 2505 Mistral AI LLM 24B GGUF 66
Hermes 4 14B Nous Research LLM 15B GGUF 66
Llama 3.1 Nemotron 70B Instruct NVIDIA LLM 71B GGUF 66
Llama 3.2 90B Vision Instruct Meta LLM 89B Original weights 66
Ministral 3 14B Reasoning Mistral AI LLM 14B GGUF 66
Mistral Small 3.1 24B Mistral AI LLM 24B GGUF 66
OpenReasoning Nemotron 14B NVIDIA LLM 15B GGUF 66
Qwen2.5 Coder 14B Instruct Alibaba LLM 15B GGUF 66
QwQ 32B Preview Alibaba LLM 33B GGUF 66
Reka Flash 3.1 (21B) Reka AI LLM 21B GGUF 66
Trinity Mini (26B-A3B) Arcee AI LLM 26B (3.0B active) GGUF 66
Command R+ 104B (04-2024) Cohere LLM 104B GGUF 64
DeepCoder 14B Preview Agentica LLM 15B GGUF 64
ERNIE 4.5 21B-A3B Baidu LLM 22B (3.0B active) GGUF 64
Falcon-H1 34B Instruct TII LLM 34B GGUF 64
InternVL3 14B Shanghai AI Lab LLM 15B GGUF 64
Ministral 3 14B Mistral AI LLM 14B GGUF 64
Mistral Small 3 24B (2501) Mistral AI LLM 24B GGUF 64
Phi-4 (14B) Microsoft LLM 15B GGUF 64
Qwen2 72B Instruct Alibaba LLM 73B GGUF 64
Qwen2-VL 72B Instruct Alibaba LLM 73B GGUF 64
Codestral 22B Mistral AI LLM 22B GGUF 62
Command R (08-2024) 32B Cohere LLM 32B GGUF 62
DeepSeek R1 0528 Qwen3 8B DeepSeek LLM 8.2B GGUF 62
DeepSeek R1 Distill Qwen 14B DeepSeek LLM 15B GGUF 62
DeepSeek V2 Chat DeepSeek LLM 236B (21.0B active) GGUF 62
Dolphin Mistral 24B Venice Edition Cognitive Computations LLM 24B GGUF 62
Gemma 2 27B IT Google LLM 27B GGUF 62
Gemma 4 E2B Google LLM 5.1B GGUF 62
Granite 4.0 H Small (32B-A9B) IBM LLM 32B (9.0B active) GGUF 62
InternVL3.5 8B Shanghai AI Lab LLM 8.5B GGUF 62
JanusCoder 14B Shanghai AI Lab LLM 15B GGUF 62
Qwen2.5 14B Instruct Alibaba LLM 15B GGUF 62
Qwen3-VL 8B Instruct Alibaba LLM 8.8B GGUF 62
Qwen3.5 4B Alibaba LLM 4.7B GGUF 62
Voxtral Small 24B Mistral AI LLM 24B GGUF 62
TMax 4B Ai2 LLM 4.2B Original weights 61
Agents-A1 4B InternScience LLM 4.5B GGUF 60
Apertus 70B Instruct Swiss AI LLM 71B GGUF 60
Aya Expanse 32B Cohere LLM 32B GGUF 60
Aya Vision 32B Cohere LLM 33B Original weights 60
Dolphin 3.0 R1 Mistral 24B Cognitive Computations LLM 24B GGUF 60
Gemma 3 12B IT Google LLM 12B GGUF 60
GLM-4.1V 9B Thinking Zhipu AI LLM 10B GGUF 60
Granite 4.1 8B IBM LLM 8.8B GGUF 60
Kimi-VL A3B Thinking 2506 Moonshot AI LLM 16B (3.0B active) GGUF 60
Ling 3.0 Tiny (8B-A1.3B) Ant Group (inclusionAI) LLM 7.9B (1.3B active) GGUF 60
MiniCPM-V 4.5 8B OpenBMB LLM 8.7B GGUF 60
Nemotron Nano 9B v2 NVIDIA LLM 8.9B GGUF 60
OLMo 2 32B Instruct Ai2 LLM 32B GGUF 60
Qwen2.5 14B Instruct 1M Alibaba LLM 15B GGUF 60
Qwen3 8B Alibaba LLM 8.2B GGUF 60
Solar Pro Preview 22B Upstage LLM 22B GGUF 60
Fara 1.5 4B Microsoft LLM 4.5B GGUF 58
GLM-Z1 9B (0414) Zhipu AI LLM 9.4B GGUF 58
Llama 3 70B Instruct Meta LLM 71B GGUF 58
MiMo-VL 7B RL Xiaomi LLM 8.3B GGUF 58
Qwen2.5-Math 72B Instruct Alibaba LLM 73B GGUF 58
Seed-Coder 8B Reasoning ByteDance LLM 8.3B GGUF 58
EXAONE Deep 7.8B LG AI Research LLM 7.8B GGUF 56
Falcon-H1R 7B TII LLM 7.6B GGUF 56
InternVL3 8B Shanghai AI Lab LLM 7.9B GGUF 56
LFM2 24B-A2B Liquid AI LLM 24B (2.3B active) GGUF 56
Mage-VL 4.7B Microsoft LLM 4.7B Original weights 56
Ministral 3 8B Mistral AI LLM 8.9B GGUF 56
Phi-3.5 MoE (42B-A6.6B) Microsoft LLM 42B (6.6B active) GGUF 56
Qwen3 4B Thinking 2507 Alibaba LLM 4.0B GGUF 56
Seed-Coder 8B Instruct ByteDance LLM 8.3B GGUF 56
Mixtral 8x22B Instruct Mistral AI LLM 141B (39.0B active) GGUF 55
Qwen2.5 Coder 7B Instruct Alibaba LLM 7.6B GGUF 55
Command R 35B (v01) Cohere LLM 35B GGUF 54
Llama 3.1 Nemotron Nano 8B v1 NVIDIA LLM 8.0B GGUF 54
MiniCPM4 8B OpenBMB LLM 8.2B GGUF 54
Mistral Small 22B (2409) Mistral AI LLM 22B GGUF 54
Olmo 3 7B Think Ai2 LLM 7.3B GGUF 54
OpenReasoning Nemotron 7B NVIDIA LLM 7.6B GGUF 54
Qwen3 4B Instruct 2507 Alibaba LLM 4.0B GGUF 54
Qwen3-VL 4B Instruct Alibaba LLM 4.4B GGUF 54
DeepSeek Coder V2 Lite (16B) DeepSeek LLM 16B (2.4B active) GGUF 52
Falcon 3 10B Instruct TII LLM 10B GGUF 52
GLM-4 9B 0414 Zhipu AI LLM 9.4B GGUF 52
Granite 4.2 3B IBM LLM 3.7B GGUF 52
Hunyuan 7B Instruct Tencent LLM 7.5B GGUF 52
LFM2.5 8B-A1B Liquid AI LLM 8.5B (1.5B active) GGUF 52
Ling Mini 2.0 (16B-A1.4B) Ant Group (inclusionAI) LLM 16B (1.4B active) GGUF 52
MiniCPM-o 2.6 8B OpenBMB LLM 8.7B GGUF 52
Mistral Nemo 12B Instruct Mistral AI LLM 12B GGUF 52
Nous Hermes 2 Mixtral 8x7B DPO Nous Research LLM 47B (12.9B active) GGUF 52
Qwen1.5 110B Chat Alibaba LLM 111B GGUF 52
Qwen2.5 7B Instruct Alibaba LLM 7.6B GGUF 52
Qwen2.5-VL 7B Instruct Alibaba LLM 8.3B GGUF 52
Aya 23 35B Cohere LLM 35B GGUF 50
DeepSeek R1 Distill Llama 8B DeepSeek LLM 8.0B GGUF 50
Falcon-H1 7B Instruct TII LLM 7.6B GGUF 50
Fara 7B Microsoft LLM 8.3B GGUF 50
Gemma 2 9B IT Google LLM 9.2B GGUF 50
Grok-1 (314B) xAI LLM 314B (86.0B active) GGUF 50
InternLM2.5 20B Chat Shanghai AI Lab LLM 20B GGUF 50
InternLM3 8B Instruct Shanghai AI Lab LLM 8.8B GGUF 50
InternVL2.5 8B Shanghai AI Lab LLM 8.1B Original weights 50
Jamba 1.5 Mini AI21 Labs LLM 52B (12.0B active) GGUF 50
Kimi-VL A3B Instruct Moonshot AI LLM 16B (3.0B active) GGUF 50
Mixtral 8x7B Instruct Mistral AI LLM 47B (12.9B active) GGUF 50
Nemotron 3 Nano 4B NVIDIA LLM 4.0B GGUF 50
Olmo 3 7B Instruct Ai2 LLM 7.3B GGUF 50
Pixtral 12B (2409) Mistral AI LLM 12B Original weights 50
Qwen2 57B-A14B Instruct Alibaba LLM 57B (14.0B active) GGUF 50
Qwen2.5 7B Instruct 1M Alibaba LLM 7.6B GGUF 50
Qwen3 4B Alibaba LLM 4.0B GGUF 50
Snowflake Arctic Instruct Snowflake LLM 479B (17.0B active) Original weights 50
Yi 1.5 34B Chat 01.AI LLM 34B GGUF 50
GLM-4 9B Chat (2024) Zhipu AI LLM 9.4B GGUF 48
GLM-4 9B Chat 1M Zhipu AI LLM 9.5B GGUF 48
Hermes 3 (Llama 3.1 8B) Nous Research LLM 8.0B GGUF 48
Llama 3.1 8B Instruct Meta LLM 8.0B GGUF 48
MiniCPM-V 2.6 8B OpenBMB LLM 8.1B GGUF 48
Apertus v1.5 8B Swiss AI LLM 8.9B Original weights 46
CodeGeeX4 All 9B Zhipu AI LLM 9.4B GGUF 46
DeepHermes 3 Llama 3 8B Preview Nous Research LLM 8.0B GGUF 46
DeepSeek R1 Distill Qwen 7B DeepSeek LLM 7.6B GGUF 46
Dolphin 3.0 Llama 3.1 8B Cognitive Computations LLM 8.0B GGUF 46
EXAONE 3.5 7.8B LG AI Research LLM 7.8B GGUF 46
Falcon 3 7B Instruct TII LLM 7.5B GGUF 46
Gemma 3n E4B Google LLM 7.9B GGUF 46
Granite 3.2 8B Instruct IBM LLM 8.2B GGUF 46
Granite 3.3 8B IBM LLM 8.2B GGUF 46
Granite 4.1 3B IBM LLM 3.4B GGUF 46
Ministral 8B (2410) Mistral AI LLM 8.0B GGUF 46
Moonlight 16B-A3B Instruct Moonshot AI LLM 16B (3.0B active) Original weights 46
Phi-3 Medium (14B) Microsoft LLM 14B GGUF 46
Phi-3 Medium 128K Instruct Microsoft LLM 14B GGUF 46
Phi-4 Mini Reasoning (3.8B) Microsoft LLM 3.8B GGUF 46
Qwen2.5-Omni 7B Alibaba LLM 11B GGUF 46
Tülu 3 8B Ai2 LLM 8.0B GGUF 46
Gemma 3 4B IT Google LLM 4.3B GGUF 45
Ornith 1.0 9B DeepReinforce LLM 9.7B GGUF 45
Ornith 1.5 9B DeepReinforce LLM 9.7B GGUF 45
Aya Vision 8B Cohere LLM 8.6B Original weights 44
Code Llama 70B Instruct Meta LLM 69B GGUF 44
Command R7B Cohere LLM 8.0B GGUF 44
DeepSeek Coder 33B Instruct DeepSeek LLM 33B GGUF 44
Dolphin 2.9.4 Llama 3.1 8B Cognitive Computations LLM 8.0B GGUF 44
Granite 3.1 8B Instruct IBM LLM 8.2B GGUF 44
Granite 4.0 H Tiny (7B-A1B) IBM LLM 6.9B (1.0B active) GGUF 44
Hermes 2 Pro Llama 3 8B Nous Research LLM 8.0B GGUF 44
InternVL2 8B Shanghai AI Lab LLM 8.1B Original weights 44
Jamba Reasoning 3B AI21 Labs LLM 3.2B GGUF 44
LFM2.5 2.6B Liquid AI LLM 2.7B GGUF 44
LFM2.5-VL 3B Liquid AI LLM 3.1B GGUF 44
Llama 3.1 Nemotron Nano 4B v1.1 NVIDIA LLM 4.5B GGUF 44
LLaVA 1.6 34B LLaVA LLM 35B Original weights 44
MedGemma 4B IT Google LLM 4.3B GGUF 44
Ministral 3 3B Mistral AI LLM 3.9B GGUF 44
North Micro Vision Cohere LLM 2.5B Original weights 44
OLMo 2 13B Instruct Ai2 LLM 14B GGUF 44
OpenChat 3.6 8B OpenChat LLM 8.0B GGUF 44
Phi-3 Small 8K Instruct Microsoft LLM 7.4B Original weights 44
Phi-4 Mini (3.8B) Microsoft LLM 3.8B GGUF 44
Phi-4 Multimodal Instruct Microsoft LLM 5.6B Original weights 44
Qwen1.5 32B Chat Alibaba LLM 33B GGUF 44
Qwen2 7B Instruct Alibaba LLM 7.6B GGUF 44
Qwen2-VL 7B Instruct Alibaba LLM 8.3B GGUF 44
TMax 9B Ai2 LLM 9.0B Original weights 44
MiMo 7B RL Xiaomi LLM 7.8B Original weights 43
Command R7B Arabic Cohere LLM 8.0B Original weights 42
Hunyuan 4B Instruct Tencent LLM 4.2B GGUF 42
InternLM2 Chat 20B Shanghai AI Lab LLM 20B GGUF 42
Jamba2 3B AI21 Labs LLM 3.0B GGUF 42
Llama 3 8B Instruct Meta LLM 8.0B GGUF 42
Llama 3.2 11B Vision Instruct Meta LLM 11B Original weights 42
Mistral-NeMo-Minitron 8B Instruct NVIDIA LLM 8.4B GGUF 42
SmolLM3 3B Hugging Face LLM 3.1B GGUF 42
AFM 4.5B Arcee AI LLM 4.6B GGUF 40
Apertus 8B Instruct Swiss AI LLM 8.1B GGUF 40
Aya Expanse 8B Cohere LLM 8.0B GGUF 40
DeepSeek LLM 67B Chat DeepSeek LLM 67B GGUF 40
DeepSeek Prover V2 7B DeepSeek LLM 6.9B GGUF 40
DeepSeek V2 Lite Chat DeepSeek LLM 16B (2.4B active) GGUF 40
Granite 3.0 8B Instruct IBM LLM 8.2B GGUF 40
Granite 4.0 Micro (3B) IBM LLM 3.4B GGUF 40
Granite Code 34B Instruct IBM LLM 34B GGUF 40
InternLM2 Math Plus 20B Shanghai AI Lab LLM 20B GGUF 40
InternLM2.5 7B Chat Shanghai AI Lab LLM 7.7B GGUF 40
Nous Hermes 2 SOLAR 10.7B Nous Research LLM 11B GGUF 40
Phi-3.5 Mini Instruct Microsoft LLM 3.8B GGUF 40
Qwen3.5 2B Alibaba LLM 2.3B GGUF 40
SOLAR 10.7B Instruct Upstage LLM 11B GGUF 40
StarCoder2 15B Instruct BigCode LLM 16B GGUF 40
TMax 2B Ai2 LLM 1.9B Original weights 40
Unlimited OCR (3B-A0.6B) Baidu LLM 3.3B (0.6B active) Original weights 40
Yi 34B Chat 01.AI LLM 34B GGUF 40
Yi Coder 9B Chat 01.AI LLM 8.8B GGUF 40
Code Llama 34B Instruct Meta LLM 34B GGUF 38
Dolphin 2.9 Llama 3 8B Cognitive Computations LLM 8.0B GGUF 38
EXAONE 3.0 7.8B Instruct LG AI Research LLM 7.8B GGUF 38
Falcon3 Mamba 7B Instruct TII LLM 7.3B GGUF 38
Granite 4.0 H Micro IBM LLM 3.2B GGUF 38
LFM2 8B-A1B Liquid AI LLM 8.3B (1.5B active) GGUF 38
MiniCPM3 4B OpenBMB LLM 4.1B GGUF 38
Mistral 7B Instruct v0.3 Mistral AI LLM 7.3B GGUF 38
Phi-3.5 Vision Instruct Microsoft LLM 4.2B Original weights 38
Zamba2 7B Instruct Zyphra LLM 7.4B Original weights 38
Codestral Mamba 7B Mistral AI LLM 7.3B Original weights 36
Falcon-H1 3B Instruct TII LLM 3.2B GGUF 36
Gemma 3n E2B Google LLM 5.4B GGUF 36
Granite Code 20B Instruct IBM LLM 20B GGUF 36
Hermes 2 Pro Mistral 7B Nous Research LLM 7.2B GGUF 36
OLMo 2 7B Instruct Ai2 LLM 7.3B GGUF 36
Phi-3 Mini 128K Instruct Microsoft LLM 3.8B GGUF 36
Phi-3 Mini 4K Instruct Microsoft LLM 3.8B GGUF 36
Qwen2.5 Coder 3B Instruct Alibaba LLM 3.1B GGUF 36
Qwen2.5-Math 7B Instruct Alibaba LLM 7.6B GGUF 36
Qwen3-VL 2B Instruct Alibaba LLM 2.1B GGUF 36
Starling LM 7B Beta Nexusflow LLM 7.2B GGUF 36
Voxtral Mini 3B Mistral AI LLM 4.7B GGUF 36
Yi 1.5 9B Chat 01.AI LLM 8.8B GGUF 36
Llama 3.2 3B Instruct Meta LLM 3.2B GGUF 35
OpenChat 3.5 (0106) OpenChat LLM 7.2B GGUF 35
Aya 23 8B Cohere LLM 8.0B GGUF 34
CodeGemma 7B Google LLM 8.5B GGUF 34
DeepSeek Coder 6.7B Instruct DeepSeek LLM 6.7B GGUF 34
LFM2 2.6B Liquid AI LLM 2.6B GGUF 34
Mathstral 7B Mistral AI LLM 7.3B GGUF 34
Mistral 7B Instruct v0.2 Mistral AI LLM 7.2B GGUF 34
Nous Hermes 2 Mistral 7B DPO Nous Research LLM 7.2B GGUF 34
OpenChat 3.5 (1210) OpenChat LLM 7.2B GGUF 34
Phi-3 Vision 128K Instruct Microsoft LLM 4.2B Original weights 34
Qwen1.5 14B Chat Alibaba LLM 14B GGUF 34
Qwen2.5 3B Instruct Alibaba LLM 3.1B GGUF 34
Qwen2.5-VL 3B Instruct Alibaba LLM 3.8B GGUF 34
Granite Code 8B Instruct (128K) IBM LLM 8.1B GGUF 32
Hermes 3 Llama 3.2 3B Nous Research LLM 3.2B GGUF 32
InternLM2 Math Plus 7B Shanghai AI Lab LLM 7.7B GGUF 32
LFM2.5 1.2B Instruct Liquid AI LLM 1.2B GGUF 32
OpenHermes 2.5 Mistral 7B Teknium LLM 7.2B GGUF 32
StarCoder2 15B (base) BigCode LLM 16B GGUF 32
Starling LM 7B Alpha Berkeley LLM 7.2B GGUF 32
Code Llama 13B Instruct Meta LLM 13B GGUF 30
EXAONE 3.5 2.4B LG AI Research LLM 2.4B GGUF 30
EXAONE 4.0 1.2B LG AI Research LLM 1.3B GGUF 30
Falcon 3 3B Instruct TII LLM 3.2B GGUF 30
Falcon Mamba 7B Instruct TII LLM 7.3B GGUF 30
Granite 3.2 2B Instruct IBM LLM 2.5B GGUF 30
Granite 3.3 2B IBM LLM 2.5B GGUF 30
Mistral 7B Instruct v0.1 Mistral AI LLM 7.2B GGUF 30
Qwen2.5-Omni 3B Alibaba LLM 5.5B GGUF 30
Qwen3 1.7B Alibaba LLM 2.0B GGUF 30
SQLCoder 7B v2 Defog LLM 6.7B GGUF 30
Gemma 1.1 7B IT Google LLM 8.5B GGUF 28
Gemma 2 2B Google LLM 2.6B GGUF 28
Granite 3.1 2B Instruct IBM LLM 2.5B GGUF 28
Granite 4.0 H 1B IBM LLM 1.5B GGUF 28
LLaVA 1.6 Mistral 7B LLaVA LLM 7.6B Original weights 28
Neural Chat 7B v3.3 Intel LLM 7.2B GGUF 28
Orca 2 13B Microsoft LLM 13B GGUF 28
StableLM 2 12B Chat Stability AI LLM 12B GGUF 28
Yi 1.5 6B Chat 01.AI LLM 6.1B GGUF 28
Baichuan 2 13B Chat Baichuan LLM 14B GGUF 26
DeepScaleR 1.5B Preview Agentica LLM 1.8B GGUF 26
DeepSeek MoE 16B Chat DeepSeek LLM 16B (2.8B active) GGUF 26
Falcon-H1 1.5B Instruct TII LLM 1.6B GGUF 26
Gemma 7B IT Google LLM 8.5B GGUF 26
Granite 3.0 2B Instruct IBM LLM 2.6B GGUF 26
Granite Vision 3.2 2B IBM LLM 3.0B GGUF 26
LFM2 1.2B Liquid AI LLM 1.2B GGUF 26
Llama 2 13B Chat Meta LLM 13B GGUF 26
OLMoE 1B-7B Instruct Ai2 LLM 6.9B (1.3B active) GGUF 26
Qwen1.5 7B Chat Alibaba LLM 7.7B GGUF 26
Qwen2.5 Coder 1.5B Instruct Alibaba LLM 1.5B GGUF 26
Qwen3.5 0.8B Alibaba LLM 0.9B GGUF 26
StarCoder 15B BigCode LLM 16B GGUF 26
StarCoder2 3B BigCode LLM 3.0B GGUF 26
WizardMath 7B v1.1 WizardLM LLM 7.2B GGUF 26
Zephyr 7B Beta Hugging Face LLM 7.2B GGUF 26
Code Llama 7B Instruct Meta LLM 6.7B GGUF 24
LFM2-VL 1.6B Liquid AI LLM 1.6B GGUF 24
Qwen1.5 MoE A2.7B Chat Alibaba LLM 14B (2.7B active) Original weights 24
Qwen2-VL 2B Instruct Alibaba LLM 2.2B GGUF 24
Qwen2.5 1.5B Instruct Alibaba LLM 1.5B GGUF 24
SmolVLM2 2.2B Instruct Hugging Face LLM 2.3B GGUF 24
StarCoder2 7B BigCode LLM 7.2B GGUF 24
DeepSeek R1 Distill Qwen 1.5B DeepSeek LLM 1.8B GGUF 22
Hunyuan 1.8B Instruct Tencent LLM 1.8B GGUF 22
Llama 3.2 1B Instruct Meta LLM 1.2B GGUF 22
LLaVA 1.5 13B LLaVA LLM 13B Original weights 22
Nemotron Mini 4B Instruct NVIDIA LLM 4.2B GGUF 22
Orca 2 7B Microsoft LLM 6.7B GGUF 22
SmolVLM 2.2B Instruct Hugging Face LLM 2.3B GGUF 22
WizardLM 13B v1.2 WizardLM LLM 13B GGUF 22
Zamba2 2.7B Instruct Zyphra LLM 2.7B Original weights 22
Baichuan 2 7B Chat Baichuan LLM 7.5B GGUF 20
Gemma 3 1B Google LLM 1.0B GGUF 20
Granite 3.1 3B-A800M Instruct IBM LLM 3.3B (0.8B active) GGUF 20
Granite Code 3B Instruct (128K) IBM LLM 3.5B GGUF 20
Llama 2 7B Chat Meta LLM 6.7B GGUF 20
Qwen2.5-Math 1.5B Instruct Alibaba LLM 1.5B GGUF 20
SmolLM2 1.7B Instruct Hugging Face LLM 1.7B GGUF 20
Stable Code Instruct 3B Stability AI LLM 2.8B GGUF 20
Vicuna 13B v1.5 LMSYS LLM 13B GGUF 20
Yi 6B Chat 01.AI LLM 6.1B GGUF 20
DeepSeek LLM 7B Chat DeepSeek LLM 6.9B GGUF 18
Granite 3.0 3B-A800M Instruct IBM LLM 3.4B (0.8B active) GGUF 18
InternLM2.5 1.8B Chat Shanghai AI Lab LLM 1.9B GGUF 18
LFM2 700M Liquid AI LLM 0.7B GGUF 18
Falcon 3 1B Instruct TII LLM 1.7B GGUF 16
Falcon 40B Instruct TII LLM 42B GGUF 16
LLaVA 1.5 7B LLaVA LLM 7.1B GGUF 16
Phi-2 2.7B Microsoft LLM 2.8B GGUF 16
Qwen2 1.5B Instruct Alibaba LLM 1.5B GGUF 16
StableLM Zephyr 3B Stability AI LLM 2.8B GGUF 16
Yi Coder 1.5B Chat 01.AI LLM 1.5B GGUF 16
CodeGemma 2B Google LLM 2.5B GGUF 14
Falcon-H1 0.5B Instruct TII LLM 0.5B GGUF 14
LFM2.5 350M Liquid AI LLM 0.4B GGUF 14
OLMo 2 1B Instruct Ai2 LLM 1.5B GGUF 14
OLMo 7B Instruct Ai2 LLM 6.9B Original weights 14
Qwen3 0.6B Alibaba LLM 0.8B GGUF 14
SmolLM 1.7B Instruct Hugging Face LLM 1.7B GGUF 14
Vicuna 7B v1.5 LMSYS LLM 6.7B GGUF 14
DeepSeek Coder 1.3B Instruct DeepSeek LLM 1.4B GGUF 12
Gemma 2B IT Google LLM 2.5B GGUF 12
Granite 4.0 H 350M IBM LLM 0.3B GGUF 12
InternLM 7B Shanghai AI Lab LLM 7.3B Original weights 12
LFM2 350M Liquid AI LLM 0.4B GGUF 12
LFM2-VL 450M Liquid AI LLM 0.5B GGUF 12
LFM2.5 230M Liquid AI LLM 0.2B GGUF 10
Qwen2.5 0.5B Instruct Alibaba LLM 0.5B GGUF 10
SmolVLM 500M Instruct Hugging Face LLM 0.5B GGUF 10
StableLM 2 1.6B Chat Stability AI LLM 1.6B GGUF 10
StableLM 2 Zephyr 1.6B Stability AI LLM 1.6B GGUF 10
Hunyuan 0.5B Instruct Tencent LLM 0.5B GGUF 8
Qwen2.5 Coder 0.5B Instruct Alibaba LLM 0.5B GGUF 8
SmolLM2 360M Instruct Hugging Face LLM 0.4B GGUF 8
ERNIE 4.5 0.3B Baidu LLM 0.4B GGUF 6
Falcon 7B Instruct TII LLM 7.2B GGUF 6
Gemma 3 270M Google LLM 0.3B GGUF 6
Qwen2 0.5B Instruct Alibaba LLM 0.5B GGUF 6
SmolVLM 256M Instruct Hugging Face LLM 0.3B GGUF 6
TinyLlama 1.1B Chat TinyLlama LLM 1.1B GGUF 6
SmolLM 360M Instruct Hugging Face LLM 0.4B GGUF 5
SmolLM2 135M Instruct Hugging Face LLM 0.1B GGUF 4
Falcon-H1 Tiny 90M Instruct TII LLM 0.1B GGUF 3
SmolLM 135M Instruct Hugging Face LLM 0.1B GGUF 2

Alternatives

LM Studio Ollama llama.cpp vLLM Jan Text Embeddings Inference MLX LM SGLang GLiNER Lemonade Infinity Unsloth Desktop Sentence Transformers LocalAI Atomic Chat

Which app should I use?

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.