Apps · Inference server

SGLang

by LMSYS / SGLang project · Apache-2.0 · 38,156 GitHub stars

Production server like vLLM with fast prefix caching and structured output, for NVIDIA and AMD GPUs.

Windows (WSL2)Linux

Get SGLang

At a glance

Best for
Agent and RAG backends with many requests that share long prompts, on GPUs with room for the full model.
Inference engine
  • SGLang
GPU support
  • NVIDIA CUDA
  • AMD ROCm
Opens
  • Safetensors (original weights)
  • AWQ
  • GPTQ
  • FP8
Features
  • OpenAI-compatible API
  • Multiple GPUs
  • Many users at once
  • Image input
  • Tool calling
Platforms
Linux with NVIDIA or AMD GPUs; Windows through WSL2 or Docker.

Sources: docs.sglang.io

Install

Linux

pip install sglang

569 models on this site run in SGLang

Based on the files published for each model: Safetensors (original weights), AWQ, GPTQ, FP8. Open a model to see the exact command or file for your quantization.

Model Made by Type Size Opens it as Quality tier
Qwen3 Reranker 4B Alibaba Decision 4.0B Original weights 90
Qwen3 Reranker 8B Alibaba Decision 8.2B Original weights 90
Skywork Reward V2 Llama 3.1 8B Skywork Decision 7.5B Original weights 90
CompassJudger-2 32B OpenCompass Decision 33B Original weights 86
Qwen3Guard Gen 8B Alibaba Decision 8.2B Original weights 86
Shieldstral 1.0 3B Mistral AI Decision 3.9B Original weights 86
Granite Guardian 4.1 8B IBM Decision 8.4B Original weights 84
Llama 3.1 Nemotron Safety Guard 8B v3 NVIDIA Decision 8.0B Original weights 84
Qwen3Guard Gen 4B Alibaba Decision 4.4B Original weights 84
Skywork Reward V2 Qwen3 8B Skywork Decision 7.6B Original weights 84
gpt-oss-safeguard 120B OpenAI Decision 117B (5.1B active) Original weights 80
Granite Guardian 3.3 8B IBM Decision 8.2B Original weights 80
Qwen3 Reranker 0.6B Alibaba Decision 600M Original weights 80
Selene 1 Mini 8B Atla Decision 8.0B Original weights 80
WildGuard 7B Ai2 Decision 7.3B Original weights 80
DynaGuard 8B University of Maryland Decision 8.2B Original weights 79
CompassJudger-2 7B OpenCompass Decision 7.6B Original weights 78
Nemotron Content Safety Reasoning 4B NVIDIA Decision 4.3B Original weights 78
gpt-oss-safeguard 20B OpenAI Decision 22B (3.6B active) Original weights 76
NuExtract 2.0 4B NuMind Decision 3.8B Original weights 76
Prometheus 2 8x7B Prometheus Eval Decision 47B (13B active) Original weights 74
Qwen3Guard Gen 0.6B Alibaba Decision 750M Original weights 74
BGE Reranker v2 M3 BAAI Decision 570M Original weights 72
Skywork Reward V2 Qwen3 0.6B Skywork Decision 600M Original weights 72
DynaGuard 1.7B University of Maryland Decision 2.0B Original weights 70
Jina Reranker v2 Base Multilingual Jina AI Decision 280M Original weights 70
NuExtract 2.0 2B NuMind Decision 2.2B Original weights 70
Llama Guard 3 8B Meta Decision 8.0B Original weights 68
Prometheus 2 7B Prometheus Eval Decision 7.2B Original weights 68
GLIDER Patronus AI Decision 3.8B Original weights 66
Llama Guard 4 12B Meta Decision 12B Original weights 66
ShieldGemma 27B Google Decision 27B Original weights 66
Flow Judge v0.1 Flow AI Decision 3.8B Original weights 64
ShieldGemma 9B Google Decision 9.2B Original weights 64
ShieldGemma 2B Google Decision 2.6B Original weights 58
Llama Guard 3 1B Meta Decision 1.5B Original weights 56
MiMo V2.6 Pro (1T-A42B) Xiaomi LLM 1,024B (42.0B active) Original weights 99
Qwen3.8 Max (2.4T-A95B) Alibaba LLM 2,446B (95.0B active) Original weights 98.3
GLM-5.3 (753B-A40B) Zhipu AI LLM 753B (40.0B active) Original weights 97.8
MiMo V2.6 Flash (310B-A15B) Xiaomi LLM 311B (15.0B active) Original weights 97.2
DeepSeek V4 Pro (1.6T-A49B) DeepSeek LLM 1,651B (49.0B active) Original weights 96.8
GLM-5.2 (753B-A40B) Zhipu AI LLM 753B (40.0B active) Original weights 96.3
Hunyuan Hy4 Preview (780B) Tencent LLM 780B (55.0B active) Original weights 96.3
Qwen3.8 Flash Next (125B-A6B) Alibaba LLM 180B (6.0B active) Original weights 96.5
DeepSeek V4 Pro (April 2026) DeepSeek LLM 1,599B (49.0B active) Original weights 95
GLM-5.1 (744B-A40B) Zhipu AI LLM 754B (40.0B active) Original weights 95.2
GLM-5.3 Flash (320B-A18B) Zhipu AI LLM 321B (18.0B active) Original weights 95.4
Kimi K2.6 (1T) Moonshot AI LLM 1,027B (32.0B active) Original weights 95.7
Qwen3.5 397B-A17B Alibaba LLM 403B (17.0B active) Original weights 95
DeepSeek V4 Flash (284B-A13B) DeepSeek LLM 304B (13.0B active) Original weights 94
GLM-5 (744B-A40B) Zhipu AI LLM 754B (40.0B active) Original weights 94.7
Hunyuan Hy3 (295B-A21B) Tencent LLM 299B (21.0B active) Original weights 94.4
Kimi K2.5 (1T) Moonshot AI LLM 1,027B (32.0B active) Original weights 94.3
Kimi K2.7 Code (1T) Moonshot AI LLM 1,027B (32.0B active) Original weights 94.3
MiniMax M3 (427B) MiniMax LLM 427B (25.0B active) Original weights 94.8
Ornith 1.5 397B-A17B DeepReinforce LLM 403B (17.0B active) Original weights 94
Qwen3.6 27B Alibaba LLM 28B Original weights 94
DeepSeek R1 0528 (671B) DeepSeek LLM 685B (37.0B active) Original weights 93
DeepSeek V3.2 (671B) DeepSeek LLM 685B (37.0B active) Original weights 93.7
GLM-4.7 (355B-A32B) Zhipu AI LLM 358B (32.0B active) Original weights 93.5
Kimi K2 Thinking (1T) Moonshot AI LLM 1,026B (32.0B active) Original weights 93
MiniMax M2.7 (230B-A10B) MiniMax LLM 229B (10.0B active) Original weights 93.5
Nemotron 3 Ultra 550B-A55B NVIDIA LLM 561B (55.0B active) Original weights 93.2
DeepSeek R1 (671B) DeepSeek LLM 671B (37.0B active) Original weights 92
DeepSeek V3.2 Exp DeepSeek LLM 685B (37.0B active) Original weights 92.5
DeepSeek V4 Flash (April 2026) DeepSeek LLM 291B (13.0B active) Original weights 92
Falcon 180B Chat TII LLM 180B Original weights 92
Gemma 4 26B-A4B Google LLM 26B (3.8B active) Original weights 92.5
MiniMax M2.5 (230B-A10B) MiniMax LLM 229B (10.0B active) Original weights 92.5
Qwen3.5 122B-A10B Alibaba LLM 125B (10.0B active) Original weights 92
Step 3.5 Flash (196B-A11B) StepFun LLM 199B (11.0B active) Original weights 92
DeepSeek V3.1 (671B) DeepSeek LLM 685B (37.0B active) Original weights 91
DeepSeek V3.1 Terminus DeepSeek LLM 685B (37.0B active) Original weights 91.5
Gemma 4 31B Google LLM 31B Original weights 91
Kimi K2 Instruct 0905 (1T) Moonshot AI LLM 1,026B (32.0B active) Original weights 91
MiniMax M2.1 (230B-A10B) MiniMax LLM 229B (10.0B active) Original weights 91
Qwen3 235B-A22B Thinking 2507 Alibaba LLM 235B (22.0B active) Original weights 91
Qwen3 Coder 480B-A35B Alibaba LLM 480B (35.0B active) Original weights 91
Kimi K3 (2.8T) Moonshot AI LLM 2,780B (104.0B active) Original weights 90
Ling 1T Ant Group (inclusionAI) LLM 1,000B (50.0B active) Original weights 90
MiMo V2 Flash (309B-A15B) Xiaomi LLM 310B (15.0B active) Original weights 90.9
Mistral Large 3 (675B) Mistral AI LLM 675B (41.0B active) Original weights 90
Qwen3 235B-A22B Instruct 2507 Alibaba LLM 235B (22.0B active) Original weights 90
Solar Open 2 250B Upstage LLM 250B (15.0B active) Original weights 90
Apodex 1.1 Mini (35B-A3B) Apodex LLM 36B (3.0B active) Original weights 89
Kimi K2 Instruct (July 2025) Moonshot AI LLM 1,026B (32.0B active) Original weights 89
MiniMax M2 (230B-A10B) MiniMax LLM 229B (10.0B active) Original weights 89
Qwen3-VL 235B-A22B Instruct Alibaba LLM 236B (22.0B active) Original weights 89
Fara 1.5 27B Microsoft LLM 27B Original weights 88
GLM-4.7 Flash (30B-A3B) Zhipu AI LLM 31B (3.0B active) Original weights 88
K-EXAONE 236B-A23B LG AI Research LLM 237B (23.0B active) Original weights 88
LongCat Flash Chat (560B) Meituan LLM 562B (27.0B active) Original weights 88
Nemotron 3 Super 120B-A12B NVIDIA LLM 124B (12.0B active) Original weights 88
Qwen3 235B-A22B Alibaba LLM 235B (22.0B active) Original weights 88
Agents-A1 35B-A3B InternScience LLM 35B (3.0B active) Original weights 87
Qwen AgentWorld 35B-A3B Alibaba LLM 35B (3.0B active) Original weights 87
Devstral 2 (123B) Mistral AI LLM 125B Original weights 86
InternVL3.5 241B-A28B Shanghai AI Lab LLM 241B (28.0B active) Original weights 86
Laguna S 2.1 (118B-A8B) Poolside LLM 118B (8.0B active) Original weights 86
Llama 3.1 Nemotron Ultra 253B v1 NVIDIA LLM 253B Original weights 86
Qwen3 Coder Next (80B-A3B) Alibaba LLM 80B (3.0B active) Original weights 86
Qwen3-Next 80B-A3B Thinking Alibaba LLM 81B (3.0B active) Original weights 86
UI-Mate 27B Tencent LLM 27B Original weights 86
Baichuan M3 235B-A22B Baichuan LLM 235B (22.0B active) Original weights 84
GLM-4.6 (355B-A32B) Zhipu AI LLM 357B (32.0B active) Original weights 84
gpt-oss 120B OpenAI LLM 117B (5.1B active) Original weights 84
Qwen3 Next 80B-A3B Instruct Alibaba LLM 81B (3.0B active) Original weights 84
DeepSeek V3 0324 (671B) DeepSeek LLM 685B (37.0B active) Original weights 83
Gemma 4 12B Google LLM 12B Original weights 83
GLM-4.6V Zhipu AI LLM 108B (12.0B active) Original weights 83
MiniMax-M1 80K MiniMax LLM 456B (45.9B active) Original weights 83
Command A Reasoning (111B) Cohere LLM 111B Original weights 82
DeepSeek R1 Distill Llama 70B DeepSeek LLM 71B Original weights 82
EXAONE 4.5 33B LG AI Research LLM 34B Original weights 82
GLM-4.5 (355B-A32B) Zhipu AI LLM 358B (32.0B active) Original weights 82
Leanstral 119B-A6B Mistral AI LLM 119B (6.5B active) Original weights 82
Ling 3.0 Flash (127B-A5B) Ant Group (inclusionAI) LLM 127B (5.1B active) Original weights 82
Llama 2 70B Chat Meta LLM 69B Original weights 82
Llama 3.3 70B Instruct Meta LLM 71B Original weights 82
MiniMax-M1 40K MiniMax LLM 456B (45.9B active) Original weights 82
Command A (111B) Cohere LLM 111B Original weights 80
ERNIE 4.5 300B-A47B Baidu LLM 300B (47.0B active) Original weights 80
GLM-4.5V Zhipu AI LLM 108B (12.0B active) Original weights 80
Hermes 4 405B Nous Research LLM 406B Original weights 80
Kimi Dev 72B Moonshot AI LLM 73B Original weights 80
Llama 3.3 Nemotron Super 49B v1.5 NVIDIA LLM 50B Original weights 80
Mistral Large 2.1 (2411) Mistral AI LLM 123B Original weights 80
Mistral Medium 3.5 (128B) Mistral AI LLM 128B Original weights 80.4
Mistral Small 4 (119B-A6B) Mistral AI LLM 119B (6.0B active) Original weights 80.6
North Mini Code 1.0 (30B-A3B) Cohere LLM 30B (3.0B active) Original weights 80
Seed-OSS 36B Instruct ByteDance LLM 36B Original weights 80
Solar Open 100B Upstage LLM 103B (12.0B active) Original weights 80
Trinity Large Thinking (400B-A13B) Arcee AI LLM 399B (13.0B active) Original weights 80.3
Hermes 4.3 36B Nous Research LLM 36B Original weights 79
GLM-4.5 Air (106B-A12B) Zhipu AI LLM 110B (12.0B active) Original weights 78
Mistral Large 2 (123B) Mistral AI LLM 123B Original weights 78
Pixtral Large 124B (2411) Mistral AI LLM 124B Original weights 78
Qwen3 30B-A3B Thinking 2507 Alibaba LLM 31B (3.3B active) Original weights 78
Qwen3 Coder 30B-A3B Alibaba LLM 31B (3.3B active) Original weights 78
Qwen3-VL 32B Instruct Alibaba LLM 33B Original weights 78
QwQ 32B Alibaba LLM 33B Original weights 78
Ring Flash 2.0 (100B-A6B) Ant Group (inclusionAI) LLM 103B (6.1B active) Original weights 78
Nemotron 3 Nano 30B-A3B NVIDIA LLM 32B (3.5B active) Original weights 77
Qwen3 30B-A3B Instruct 2507 Alibaba LLM 31B (3.3B active) Original weights 77
Devstral Small 2 (24B) Mistral AI LLM 24B Original weights 76
dots.llm1 Instruct rednote LLM 143B (14.0B active) Original weights 76
Granite 4.2 30B IBM LLM 29B Original weights 76
InternVL3.5 38B Shanghai AI Lab LLM 38B Original weights 76
Ling Flash 2.0 (100B-A6B) Ant Group (inclusionAI) LLM 103B (6.1B active) Original weights 76
MiniMax-Text-01 MiniMax LLM 456B (45.9B active) Original weights 76
Olmo 3.1 32B Think Ai2 LLM 32B Original weights 76
OpenReasoning Nemotron 32B NVIDIA LLM 33B Original weights 76
Qwen3 32B Alibaba LLM 33B Original weights 76
Qwen3-VL 30B-A3B Thinking Alibaba LLM 31B (3.3B active) Original weights 76
Qwen3.5 9B Alibaba LLM 9.7B Original weights 76
Skywork OR1 32B Skywork LLM 33B Original weights 76
Tülu 3 405B Ai2 LLM 406B Original weights 76
DeepSeek V3 (671B) DeepSeek LLM 671B (37.0B active) Original weights 75
Sarvam 105B Sarvam AI LLM 106B (10.0B active) Original weights 75
DeepSeek Coder V2 (236B) DeepSeek LLM 236B (21.0B active) Original weights 74
EXAONE Deep 32B LG AI Research LLM 32B Original weights 74
GLM-4 32B 0414 Zhipu AI LLM 33B Original weights 74
GLM-Z1 32B (0414) Zhipu AI LLM 33B Original weights 74
Hermes 4 70B Nous Research LLM 71B Original weights 74
Hunyuan A13B (80B-A13B) Tencent LLM 80B (13.0B active) Original weights 74
InternVL3 78B Shanghai AI Lab LLM 78B Original weights 74
Kimi Linear 48B-A3B Moonshot AI LLM 49B (3.0B active) Original weights 74
Qwen3 30B-A3B Alibaba LLM 31B (3.3B active) Original weights 74
Qwen3-VL 30B-A3B Instruct Alibaba LLM 31B (3.3B active) Original weights 74
Qwen2.5 72B Instruct Alibaba LLM 73B Original weights 73
Qwen2.5-VL 72B Instruct Alibaba LLM 73B Original weights 73
Apriel 1.5 15B Thinker ServiceNow LLM 15B Original weights 72
DeepSeek Coder V2 Instruct (0614) DeepSeek LLM 236B (21.0B active) Original weights 72
DeepSeek R1 Distill Qwen 32B DeepSeek LLM 33B Original weights 72
DeepSeek V2.5 1210 DeepSeek LLM 236B (21.0B active) Original weights 72
EXAONE 4.0 32B LG AI Research LLM 32B Original weights 72
Gemma 3 27B IT Google LLM 27B Original weights 72
Granite 4.1 30B IBM LLM 29B Original weights 72
Llama 3.1 405B Instruct Meta LLM 406B Original weights 72
Llama 4 Maverick (17B x 128E) Meta LLM 402B (17.0B active) Original weights 72
Llama 4 Scout 17B-16E Meta LLM 109B (17.0B active) Original weights 72
Magistral Small 1.2 (24B) Mistral AI LLM 24B Original weights 72
Nemotron 3.5 Lightning 30B-A3B NVIDIA LLM 32B (3.0B active) Original weights 72
Olmo 3.1 32B Instruct Ai2 LLM 32B Original weights 72
Phi-4 Reasoning Plus (14B) Microsoft LLM 15B Original weights 72
Qwen3.6 35B-A3B Alibaba LLM 36B (3.0B active) Original weights 72
Tülu 3 70B Ai2 LLM 71B Original weights 72
Nemotron Cascade 2 30B-A3B NVIDIA LLM 32B (3.5B active) Original weights 71
Ornith 1.0 35B-A3B DeepReinforce LLM 36B (3.0B active) Original weights 71
Ornith 1.5 35B-A3B DeepReinforce LLM 36B (3.0B active) Original weights 71
Qwen3.5 35B-A3B Alibaba LLM 36B (3.0B active) Original weights 71
Baichuan M2 32B Baichuan LLM 33B Original weights 70
Command R+ (08-2024) 104B Cohere LLM 104B Original weights 70
DeepSeek V2.5 DeepSeek LLM 236B (21.0B active) Original weights 70
Devstral Small 1.1 (24B) Mistral AI LLM 24B Original weights 70
ERNIE 4.5 21B-A3B Thinking Baidu LLM 22B (3.0B active) Original weights 70
gpt-oss 20B OpenAI LLM 21B (3.6B active) Original weights 70
InternVL3 38B Shanghai AI Lab LLM 38B Original weights 70
Jamba 1.5 Large AI21 Labs LLM 399B (94.0B active) Original weights 70
Laguna XS 2.1 (33B-A3B) Poolside LLM 33B (3.0B active) Original weights 70
Llama 3.1 70B Instruct Meta LLM 71B Original weights 70
Magistral Small 2507 Mistral AI LLM 24B Original weights 70
MedGemma 27B Text IT Google LLM 27B Original weights 70
OpenHands LM 32B All Hands AI LLM 33B Original weights 70
Phi-4 Reasoning (14B) Microsoft LLM 15B Original weights 70
Phi-4 Reasoning Vision 15B Microsoft LLM 15B Original weights 70
Qwen2.5 32B Instruct Alibaba LLM 33B Original weights 70
Qwen3-Omni 30B-A3B Instruct Alibaba LLM 35B (3.3B active) Original weights 70
Qwen3.5 27B Alibaba LLM 28B Original weights 70
Qwen3.8 27B Alibaba LLM 28B Original weights 70
Sarvam 30B Sarvam AI LLM 32B (2.4B active) Original weights 70
TMax 27B Ai2 LLM 27B Original weights 70
UI-Mate 9B Tencent LLM 9.4B Original weights 70
UI-Venus 2 9B Ant Group (inclusionAI) LLM 9.4B Original weights 70
Gemma 4 E4B Google LLM 8.0B Original weights 68
Granite 4.2 8B IBM LLM 8.8B Original weights 68
Hermes 3 (Llama 3.1 70B) Nous Research LLM 71B Original weights 68
Magistral Small 2506 Mistral AI LLM 24B Original weights 68
Mistral Small 3.2 24B Mistral AI LLM 24B Original weights 68
QVQ 72B Preview Alibaba LLM 73B Original weights 68
Qwen2.5-VL 32B Instruct Alibaba LLM 33B Original weights 68
Qwen3 14B Alibaba LLM 15B Original weights 68
Qwen2.5 Coder 32B Instruct Alibaba LLM 33B Original weights 67
AceReason Nemotron 14B NVIDIA LLM 15B Original weights 66
Apertus v1.5 70B Swiss AI LLM 72B Original weights 66
Apriel Nemotron 15B Thinker ServiceNow LLM 15B Original weights 66
Devstral Small 2505 Mistral AI LLM 24B Original weights 66
Hermes 4 14B Nous Research LLM 15B Original weights 66
Llama 3.1 Nemotron 70B Instruct NVIDIA LLM 71B Original weights 66
Llama 3.2 90B Vision Instruct Meta LLM 89B Original weights 66
Ministral 3 14B Reasoning Mistral AI LLM 14B Original weights 66
Mistral Small 3.1 24B Mistral AI LLM 24B Original weights 66
OpenReasoning Nemotron 14B NVIDIA LLM 15B Original weights 66
Qwen2.5 Coder 14B Instruct Alibaba LLM 15B Original weights 66
QwQ 32B Preview Alibaba LLM 33B Original weights 66
Reka Flash 3.1 (21B) Reka AI LLM 21B Original weights 66
Trinity Mini (26B-A3B) Arcee AI LLM 26B (3.0B active) Original weights 66
Command R+ 104B (04-2024) Cohere LLM 104B Original weights 64
DeepCoder 14B Preview Agentica LLM 15B Original weights 64
ERNIE 4.5 21B-A3B Baidu LLM 22B (3.0B active) Original weights 64
Falcon-H1 34B Instruct TII LLM 34B Original weights 64
InternVL3 14B Shanghai AI Lab LLM 15B Original weights 64
Ministral 3 14B Mistral AI LLM 14B Original weights 64
Mistral Small 3 24B (2501) Mistral AI LLM 24B Original weights 64
Phi-4 (14B) Microsoft LLM 15B Original weights 64
Qwen2 72B Instruct Alibaba LLM 73B Original weights 64
Qwen2-VL 72B Instruct Alibaba LLM 73B Original weights 64
Codestral 22B Mistral AI LLM 22B Original weights 62
Command R (08-2024) 32B Cohere LLM 32B Original weights 62
DeepSeek R1 0528 Qwen3 8B DeepSeek LLM 8.2B Original weights 62
DeepSeek R1 Distill Qwen 14B DeepSeek LLM 15B Original weights 62
DeepSeek V2 Chat DeepSeek LLM 236B (21.0B active) Original weights 62
Dolphin Mistral 24B Venice Edition Cognitive Computations LLM 24B Original weights 62
Gemma 2 27B IT Google LLM 27B Original weights 62
Gemma 4 E2B Google LLM 5.1B Original weights 62
Granite 4.0 H Small (32B-A9B) IBM LLM 32B (9.0B active) Original weights 62
InternVL3.5 8B Shanghai AI Lab LLM 8.5B Original weights 62
JanusCoder 14B Shanghai AI Lab LLM 15B Original weights 62
Qwen2.5 14B Instruct Alibaba LLM 15B Original weights 62
Qwen3-VL 8B Instruct Alibaba LLM 8.8B Original weights 62
Qwen3.5 4B Alibaba LLM 4.7B Original weights 62
Voxtral Small 24B Mistral AI LLM 24B Original weights 62
TMax 4B Ai2 LLM 4.2B Original weights 61
Agents-A1 4B InternScience LLM 4.5B Original weights 60
Apertus 70B Instruct Swiss AI LLM 71B Original weights 60
Aya Expanse 32B Cohere LLM 32B Original weights 60
Aya Vision 32B Cohere LLM 33B Original weights 60
Dolphin 3.0 R1 Mistral 24B Cognitive Computations LLM 24B Original weights 60
Gemma 3 12B IT Google LLM 12B Original weights 60
GLM-4.1V 9B Thinking Zhipu AI LLM 10B Original weights 60
Granite 4.1 8B IBM LLM 8.8B Original weights 60
Kimi-VL A3B Thinking 2506 Moonshot AI LLM 16B (3.0B active) Original weights 60
Ling 3.0 Tiny (8B-A1.3B) Ant Group (inclusionAI) LLM 7.9B (1.3B active) Original weights 60
MiniCPM-V 4.5 8B OpenBMB LLM 8.7B Original weights 60
Nemotron Nano 9B v2 NVIDIA LLM 8.9B Original weights 60
OLMo 2 32B Instruct Ai2 LLM 32B Original weights 60
Qwen2.5 14B Instruct 1M Alibaba LLM 15B Original weights 60
Qwen3 8B Alibaba LLM 8.2B Original weights 60
Solar Pro Preview 22B Upstage LLM 22B Original weights 60
Fara 1.5 4B Microsoft LLM 4.5B Original weights 58
GLM-Z1 9B (0414) Zhipu AI LLM 9.4B Original weights 58
Llama 3 70B Instruct Meta LLM 71B Original weights 58
MiMo-VL 7B RL Xiaomi LLM 8.3B Original weights 58
Qwen2.5-Math 72B Instruct Alibaba LLM 73B Original weights 58
Seed-Coder 8B Reasoning ByteDance LLM 8.3B Original weights 58
EXAONE Deep 7.8B LG AI Research LLM 7.8B Original weights 56
Falcon-H1R 7B TII LLM 7.6B Original weights 56
InternVL3 8B Shanghai AI Lab LLM 7.9B Original weights 56
LFM2 24B-A2B Liquid AI LLM 24B (2.3B active) Original weights 56
Mage-VL 4.7B Microsoft LLM 4.7B Original weights 56
Ministral 3 8B Mistral AI LLM 8.9B Original weights 56
Phi-3.5 MoE (42B-A6.6B) Microsoft LLM 42B (6.6B active) Original weights 56
Qwen3 4B Thinking 2507 Alibaba LLM 4.0B Original weights 56
Seed-Coder 8B Instruct ByteDance LLM 8.3B Original weights 56
Mixtral 8x22B Instruct Mistral AI LLM 141B (39.0B active) Original weights 55
Qwen2.5 Coder 7B Instruct Alibaba LLM 7.6B Original weights 55
Command R 35B (v01) Cohere LLM 35B Original weights 54
Llama 3.1 Nemotron Nano 8B v1 NVIDIA LLM 8.0B Original weights 54
MiniCPM4 8B OpenBMB LLM 8.2B Original weights 54
Mistral Small 22B (2409) Mistral AI LLM 22B Original weights 54
Olmo 3 7B Think Ai2 LLM 7.3B Original weights 54
OpenReasoning Nemotron 7B NVIDIA LLM 7.6B Original weights 54
Qwen3 4B Instruct 2507 Alibaba LLM 4.0B Original weights 54
Qwen3-VL 4B Instruct Alibaba LLM 4.4B Original weights 54
DeepSeek Coder V2 Lite (16B) DeepSeek LLM 16B (2.4B active) Original weights 52
Falcon 3 10B Instruct TII LLM 10B Original weights 52
GLM-4 9B 0414 Zhipu AI LLM 9.4B Original weights 52
Granite 4.2 3B IBM LLM 3.7B Original weights 52
Hunyuan 7B Instruct Tencent LLM 7.5B Original weights 52
LFM2.5 8B-A1B Liquid AI LLM 8.5B (1.5B active) Original weights 52
Ling Mini 2.0 (16B-A1.4B) Ant Group (inclusionAI) LLM 16B (1.4B active) Original weights 52
MiniCPM-o 2.6 8B OpenBMB LLM 8.7B Original weights 52
Mistral Nemo 12B Instruct Mistral AI LLM 12B Original weights 52
Nous Hermes 2 Mixtral 8x7B DPO Nous Research LLM 47B (12.9B active) Original weights 52
Qwen1.5 110B Chat Alibaba LLM 111B Original weights 52
Qwen2.5 7B Instruct Alibaba LLM 7.6B Original weights 52
Qwen2.5-VL 7B Instruct Alibaba LLM 8.3B Original weights 52
Aya 23 35B Cohere LLM 35B Original weights 50
DeepSeek R1 Distill Llama 8B DeepSeek LLM 8.0B Original weights 50
Falcon-H1 7B Instruct TII LLM 7.6B Original weights 50
Fara 7B Microsoft LLM 8.3B Original weights 50
Gemma 2 9B IT Google LLM 9.2B Original weights 50
Grok-1 (314B) xAI LLM 314B (86.0B active) Original weights 50
InternLM2.5 20B Chat Shanghai AI Lab LLM 20B Original weights 50
InternLM3 8B Instruct Shanghai AI Lab LLM 8.8B Original weights 50
InternVL2.5 8B Shanghai AI Lab LLM 8.1B Original weights 50
Jamba 1.5 Mini AI21 Labs LLM 52B (12.0B active) Original weights 50
Kimi-VL A3B Instruct Moonshot AI LLM 16B (3.0B active) Original weights 50
Mixtral 8x7B Instruct Mistral AI LLM 47B (12.9B active) Original weights 50
Nemotron 3 Nano 4B NVIDIA LLM 4.0B Original weights 50
Olmo 3 7B Instruct Ai2 LLM 7.3B Original weights 50
Pixtral 12B (2409) Mistral AI LLM 12B Original weights 50
Qwen2 57B-A14B Instruct Alibaba LLM 57B (14.0B active) Original weights 50
Qwen2.5 7B Instruct 1M Alibaba LLM 7.6B Original weights 50
Qwen3 4B Alibaba LLM 4.0B Original weights 50
Snowflake Arctic Instruct Snowflake LLM 479B (17.0B active) Original weights 50
Yi 1.5 34B Chat 01.AI LLM 34B Original weights 50
GLM-4 9B Chat (2024) Zhipu AI LLM 9.4B Original weights 48
GLM-4 9B Chat 1M Zhipu AI LLM 9.5B Original weights 48
Hermes 3 (Llama 3.1 8B) Nous Research LLM 8.0B Original weights 48
Llama 3.1 8B Instruct Meta LLM 8.0B Original weights 48
MiniCPM-V 2.6 8B OpenBMB LLM 8.1B Original weights 48
Apertus v1.5 8B Swiss AI LLM 8.9B Original weights 46
CodeGeeX4 All 9B Zhipu AI LLM 9.4B Original weights 46
DeepHermes 3 Llama 3 8B Preview Nous Research LLM 8.0B Original weights 46
DeepSeek R1 Distill Qwen 7B DeepSeek LLM 7.6B Original weights 46
Dolphin 3.0 Llama 3.1 8B Cognitive Computations LLM 8.0B Original weights 46
EXAONE 3.5 7.8B LG AI Research LLM 7.8B Original weights 46
Falcon 3 7B Instruct TII LLM 7.5B Original weights 46
Gemma 3n E4B Google LLM 7.9B Original weights 46
Granite 3.2 8B Instruct IBM LLM 8.2B Original weights 46
Granite 3.3 8B IBM LLM 8.2B Original weights 46
Granite 4.1 3B IBM LLM 3.4B Original weights 46
Ministral 8B (2410) Mistral AI LLM 8.0B Original weights 46
Moonlight 16B-A3B Instruct Moonshot AI LLM 16B (3.0B active) Original weights 46
Phi-3 Medium (14B) Microsoft LLM 14B Original weights 46
Phi-3 Medium 128K Instruct Microsoft LLM 14B Original weights 46
Phi-4 Mini Reasoning (3.8B) Microsoft LLM 3.8B Original weights 46
Qwen2.5-Omni 7B Alibaba LLM 11B Original weights 46
Tülu 3 8B Ai2 LLM 8.0B Original weights 46
Gemma 3 4B IT Google LLM 4.3B Original weights 45
Ornith 1.0 9B DeepReinforce LLM 9.7B Original weights 45
Ornith 1.5 9B DeepReinforce LLM 9.7B Original weights 45
Aya Vision 8B Cohere LLM 8.6B Original weights 44
Code Llama 70B Instruct Meta LLM 69B Original weights 44
Command R7B Cohere LLM 8.0B Original weights 44
DeepSeek Coder 33B Instruct DeepSeek LLM 33B Original weights 44
Dolphin 2.9.4 Llama 3.1 8B Cognitive Computations LLM 8.0B Original weights 44
Granite 3.1 8B Instruct IBM LLM 8.2B Original weights 44
Granite 4.0 H Tiny (7B-A1B) IBM LLM 6.9B (1.0B active) Original weights 44
Hermes 2 Pro Llama 3 8B Nous Research LLM 8.0B Original weights 44
InternVL2 8B Shanghai AI Lab LLM 8.1B Original weights 44
Jamba Reasoning 3B AI21 Labs LLM 3.2B Original weights 44
LFM2.5 2.6B Liquid AI LLM 2.7B Original weights 44
LFM2.5-VL 3B Liquid AI LLM 3.1B Original weights 44
Llama 3.1 Nemotron Nano 4B v1.1 NVIDIA LLM 4.5B Original weights 44
LLaVA 1.6 34B LLaVA LLM 35B Original weights 44
MedGemma 4B IT Google LLM 4.3B Original weights 44
Ministral 3 3B Mistral AI LLM 3.9B Original weights 44
North Micro Vision Cohere LLM 2.5B Original weights 44
OLMo 2 13B Instruct Ai2 LLM 14B Original weights 44
OpenChat 3.6 8B OpenChat LLM 8.0B Original weights 44
Phi-3 Small 8K Instruct Microsoft LLM 7.4B Original weights 44
Phi-4 Mini (3.8B) Microsoft LLM 3.8B Original weights 44
Phi-4 Multimodal Instruct Microsoft LLM 5.6B Original weights 44
Qwen1.5 32B Chat Alibaba LLM 33B Original weights 44
Qwen2 7B Instruct Alibaba LLM 7.6B Original weights 44
Qwen2-VL 7B Instruct Alibaba LLM 8.3B Original weights 44
TMax 9B Ai2 LLM 9.0B Original weights 44
MiMo 7B RL Xiaomi LLM 7.8B Original weights 43
Command R7B Arabic Cohere LLM 8.0B Original weights 42
Hunyuan 4B Instruct Tencent LLM 4.2B Original weights 42
InternLM2 Chat 20B Shanghai AI Lab LLM 20B Original weights 42
Jamba2 3B AI21 Labs LLM 3.0B Original weights 42
Llama 3 8B Instruct Meta LLM 8.0B Original weights 42
Llama 3.2 11B Vision Instruct Meta LLM 11B Original weights 42
Mistral-NeMo-Minitron 8B Instruct NVIDIA LLM 8.4B Original weights 42
SmolLM3 3B Hugging Face LLM 3.1B Original weights 42
AFM 4.5B Arcee AI LLM 4.6B Original weights 40
Apertus 8B Instruct Swiss AI LLM 8.1B Original weights 40
Aya Expanse 8B Cohere LLM 8.0B Original weights 40
DeepSeek LLM 67B Chat DeepSeek LLM 67B Original weights 40
DeepSeek Prover V2 7B DeepSeek LLM 6.9B Original weights 40
DeepSeek V2 Lite Chat DeepSeek LLM 16B (2.4B active) Original weights 40
Granite 3.0 8B Instruct IBM LLM 8.2B Original weights 40
Granite 4.0 Micro (3B) IBM LLM 3.4B Original weights 40
Granite Code 34B Instruct IBM LLM 34B Original weights 40
InternLM2 Math Plus 20B Shanghai AI Lab LLM 20B Original weights 40
InternLM2.5 7B Chat Shanghai AI Lab LLM 7.7B Original weights 40
Nous Hermes 2 SOLAR 10.7B Nous Research LLM 11B Original weights 40
Phi-3.5 Mini Instruct Microsoft LLM 3.8B Original weights 40
Qwen3.5 2B Alibaba LLM 2.3B Original weights 40
SOLAR 10.7B Instruct Upstage LLM 11B Original weights 40
StarCoder2 15B Instruct BigCode LLM 16B Original weights 40
TMax 2B Ai2 LLM 1.9B Original weights 40
Unlimited OCR (3B-A0.6B) Baidu LLM 3.3B (0.6B active) Original weights 40
Yi 34B Chat 01.AI LLM 34B Original weights 40
Yi Coder 9B Chat 01.AI LLM 8.8B Original weights 40
Code Llama 34B Instruct Meta LLM 34B Original weights 38
Dolphin 2.9 Llama 3 8B Cognitive Computations LLM 8.0B Original weights 38
EXAONE 3.0 7.8B Instruct LG AI Research LLM 7.8B Original weights 38
Falcon3 Mamba 7B Instruct TII LLM 7.3B Original weights 38
Granite 4.0 H Micro IBM LLM 3.2B Original weights 38
LFM2 8B-A1B Liquid AI LLM 8.3B (1.5B active) Original weights 38
MiniCPM3 4B OpenBMB LLM 4.1B Original weights 38
Mistral 7B Instruct v0.3 Mistral AI LLM 7.3B Original weights 38
Phi-3.5 Vision Instruct Microsoft LLM 4.2B Original weights 38
Zamba2 7B Instruct Zyphra LLM 7.4B Original weights 38
Codestral Mamba 7B Mistral AI LLM 7.3B Original weights 36
Falcon-H1 3B Instruct TII LLM 3.2B Original weights 36
Gemma 3n E2B Google LLM 5.4B Original weights 36
Granite Code 20B Instruct IBM LLM 20B Original weights 36
Hermes 2 Pro Mistral 7B Nous Research LLM 7.2B Original weights 36
OLMo 2 7B Instruct Ai2 LLM 7.3B Original weights 36
Phi-3 Mini 128K Instruct Microsoft LLM 3.8B Original weights 36
Phi-3 Mini 4K Instruct Microsoft LLM 3.8B Original weights 36
Qwen2.5 Coder 3B Instruct Alibaba LLM 3.1B Original weights 36
Qwen2.5-Math 7B Instruct Alibaba LLM 7.6B Original weights 36
Qwen3-VL 2B Instruct Alibaba LLM 2.1B Original weights 36
Starling LM 7B Beta Nexusflow LLM 7.2B Original weights 36
Voxtral Mini 3B Mistral AI LLM 4.7B Original weights 36
Yi 1.5 9B Chat 01.AI LLM 8.8B Original weights 36
Llama 3.2 3B Instruct Meta LLM 3.2B Original weights 35
OpenChat 3.5 (0106) OpenChat LLM 7.2B Original weights 35
Aya 23 8B Cohere LLM 8.0B Original weights 34
CodeGemma 7B Google LLM 8.5B Original weights 34
DeepSeek Coder 6.7B Instruct DeepSeek LLM 6.7B Original weights 34
LFM2 2.6B Liquid AI LLM 2.6B Original weights 34
Mathstral 7B Mistral AI LLM 7.3B Original weights 34
Mistral 7B Instruct v0.2 Mistral AI LLM 7.2B Original weights 34
Nous Hermes 2 Mistral 7B DPO Nous Research LLM 7.2B Original weights 34
OpenChat 3.5 (1210) OpenChat LLM 7.2B Original weights 34
Phi-3 Vision 128K Instruct Microsoft LLM 4.2B Original weights 34
Qwen1.5 14B Chat Alibaba LLM 14B Original weights 34
Qwen2.5 3B Instruct Alibaba LLM 3.1B Original weights 34
Qwen2.5-VL 3B Instruct Alibaba LLM 3.8B Original weights 34
Granite Code 8B Instruct (128K) IBM LLM 8.1B Original weights 32
Hermes 3 Llama 3.2 3B Nous Research LLM 3.2B Original weights 32
InternLM2 Math Plus 7B Shanghai AI Lab LLM 7.7B Original weights 32
LFM2.5 1.2B Instruct Liquid AI LLM 1.2B Original weights 32
OpenHermes 2.5 Mistral 7B Teknium LLM 7.2B Original weights 32
StarCoder2 15B (base) BigCode LLM 16B Original weights 32
Starling LM 7B Alpha Berkeley LLM 7.2B Original weights 32
Code Llama 13B Instruct Meta LLM 13B Original weights 30
EXAONE 3.5 2.4B LG AI Research LLM 2.4B Original weights 30
EXAONE 4.0 1.2B LG AI Research LLM 1.3B Original weights 30
Falcon 3 3B Instruct TII LLM 3.2B Original weights 30
Falcon Mamba 7B Instruct TII LLM 7.3B Original weights 30
Granite 3.2 2B Instruct IBM LLM 2.5B Original weights 30
Granite 3.3 2B IBM LLM 2.5B Original weights 30
Mistral 7B Instruct v0.1 Mistral AI LLM 7.2B Original weights 30
Qwen2.5-Omni 3B Alibaba LLM 5.5B Original weights 30
Qwen3 1.7B Alibaba LLM 2.0B Original weights 30
SQLCoder 7B v2 Defog LLM 6.7B Original weights 30
Gemma 1.1 7B IT Google LLM 8.5B Original weights 28
Gemma 2 2B Google LLM 2.6B Original weights 28
Granite 3.1 2B Instruct IBM LLM 2.5B Original weights 28
Granite 4.0 H 1B IBM LLM 1.5B Original weights 28
LLaVA 1.6 Mistral 7B LLaVA LLM 7.6B Original weights 28
Neural Chat 7B v3.3 Intel LLM 7.2B Original weights 28
Orca 2 13B Microsoft LLM 13B Original weights 28
StableLM 2 12B Chat Stability AI LLM 12B Original weights 28
Yi 1.5 6B Chat 01.AI LLM 6.1B Original weights 28
Baichuan 2 13B Chat Baichuan LLM 14B Original weights 26
DeepScaleR 1.5B Preview Agentica LLM 1.8B Original weights 26
DeepSeek MoE 16B Chat DeepSeek LLM 16B (2.8B active) Original weights 26
Falcon-H1 1.5B Instruct TII LLM 1.6B Original weights 26
Gemma 7B IT Google LLM 8.5B Original weights 26
Granite 3.0 2B Instruct IBM LLM 2.6B Original weights 26
Granite Vision 3.2 2B IBM LLM 3.0B Original weights 26
LFM2 1.2B Liquid AI LLM 1.2B Original weights 26
Llama 2 13B Chat Meta LLM 13B Original weights 26
OLMoE 1B-7B Instruct Ai2 LLM 6.9B (1.3B active) Original weights 26
Qwen1.5 7B Chat Alibaba LLM 7.7B Original weights 26
Qwen2.5 Coder 1.5B Instruct Alibaba LLM 1.5B Original weights 26
Qwen3.5 0.8B Alibaba LLM 0.9B Original weights 26
StarCoder 15B BigCode LLM 16B Original weights 26
StarCoder2 3B BigCode LLM 3.0B Original weights 26
WizardMath 7B v1.1 WizardLM LLM 7.2B Original weights 26
Zephyr 7B Beta Hugging Face LLM 7.2B Original weights 26
Code Llama 7B Instruct Meta LLM 6.7B Original weights 24
LFM2-VL 1.6B Liquid AI LLM 1.6B Original weights 24
Qwen1.5 MoE A2.7B Chat Alibaba LLM 14B (2.7B active) Original weights 24
Qwen2-VL 2B Instruct Alibaba LLM 2.2B Original weights 24
Qwen2.5 1.5B Instruct Alibaba LLM 1.5B Original weights 24
SmolVLM2 2.2B Instruct Hugging Face LLM 2.3B Original weights 24
StarCoder2 7B BigCode LLM 7.2B Original weights 24
DeepSeek R1 Distill Qwen 1.5B DeepSeek LLM 1.8B Original weights 22
Hunyuan 1.8B Instruct Tencent LLM 1.8B Original weights 22
Llama 3.2 1B Instruct Meta LLM 1.2B Original weights 22
LLaVA 1.5 13B LLaVA LLM 13B Original weights 22
Nemotron Mini 4B Instruct NVIDIA LLM 4.2B Original weights 22
Orca 2 7B Microsoft LLM 6.7B Original weights 22
SmolVLM 2.2B Instruct Hugging Face LLM 2.3B Original weights 22
WizardLM 13B v1.2 WizardLM LLM 13B Original weights 22
Zamba2 2.7B Instruct Zyphra LLM 2.7B Original weights 22
Baichuan 2 7B Chat Baichuan LLM 7.5B Original weights 20
Gemma 3 1B Google LLM 1.0B Original weights 20
Granite 3.1 3B-A800M Instruct IBM LLM 3.3B (0.8B active) Original weights 20
Granite Code 3B Instruct (128K) IBM LLM 3.5B Original weights 20
Llama 2 7B Chat Meta LLM 6.7B Original weights 20
Qwen2.5-Math 1.5B Instruct Alibaba LLM 1.5B Original weights 20
SmolLM2 1.7B Instruct Hugging Face LLM 1.7B Original weights 20
Stable Code Instruct 3B Stability AI LLM 2.8B Original weights 20
Vicuna 13B v1.5 LMSYS LLM 13B Original weights 20
Yi 6B Chat 01.AI LLM 6.1B Original weights 20
DeepSeek LLM 7B Chat DeepSeek LLM 6.9B Original weights 18
Granite 3.0 3B-A800M Instruct IBM LLM 3.4B (0.8B active) Original weights 18
InternLM2.5 1.8B Chat Shanghai AI Lab LLM 1.9B Original weights 18
LFM2 700M Liquid AI LLM 0.7B Original weights 18
Falcon 3 1B Instruct TII LLM 1.7B Original weights 16
Falcon 40B Instruct TII LLM 42B Original weights 16
LLaVA 1.5 7B LLaVA LLM 7.1B Original weights 16
Phi-2 2.7B Microsoft LLM 2.8B Original weights 16
Qwen2 1.5B Instruct Alibaba LLM 1.5B Original weights 16
StableLM Zephyr 3B Stability AI LLM 2.8B Original weights 16
Yi Coder 1.5B Chat 01.AI LLM 1.5B Original weights 16
CodeGemma 2B Google LLM 2.5B Original weights 14
Falcon-H1 0.5B Instruct TII LLM 0.5B Original weights 14
LFM2.5 350M Liquid AI LLM 0.4B Original weights 14
OLMo 2 1B Instruct Ai2 LLM 1.5B Original weights 14
OLMo 7B Instruct Ai2 LLM 6.9B Original weights 14
Qwen3 0.6B Alibaba LLM 0.8B Original weights 14
SmolLM 1.7B Instruct Hugging Face LLM 1.7B Original weights 14
Vicuna 7B v1.5 LMSYS LLM 6.7B Original weights 14
DeepSeek Coder 1.3B Instruct DeepSeek LLM 1.4B Original weights 12
Gemma 2B IT Google LLM 2.5B Original weights 12
Granite 4.0 H 350M IBM LLM 0.3B Original weights 12
InternLM 7B Shanghai AI Lab LLM 7.3B Original weights 12
LFM2 350M Liquid AI LLM 0.4B Original weights 12
LFM2-VL 450M Liquid AI LLM 0.5B Original weights 12
LFM2.5 230M Liquid AI LLM 0.2B Original weights 10
Qwen2.5 0.5B Instruct Alibaba LLM 0.5B Original weights 10
SmolVLM 500M Instruct Hugging Face LLM 0.5B Original weights 10
StableLM 2 1.6B Chat Stability AI LLM 1.6B Original weights 10
StableLM 2 Zephyr 1.6B Stability AI LLM 1.6B Original weights 10
Hunyuan 0.5B Instruct Tencent LLM 0.5B Original weights 8
Qwen2.5 Coder 0.5B Instruct Alibaba LLM 0.5B Original weights 8
SmolLM2 360M Instruct Hugging Face LLM 0.4B Original weights 8
ERNIE 4.5 0.3B Baidu LLM 0.4B Original weights 6
Falcon 7B Instruct TII LLM 7.2B Original weights 6
Gemma 3 270M Google LLM 0.3B Original weights 6
Qwen2 0.5B Instruct Alibaba LLM 0.5B Original weights 6
SmolVLM 256M Instruct Hugging Face LLM 0.3B Original weights 6
TinyLlama 1.1B Chat TinyLlama LLM 1.1B Original weights 6
SmolLM 360M Instruct Hugging Face LLM 0.4B Original weights 5
SmolLM2 135M Instruct Hugging Face LLM 0.1B Original weights 4
Falcon-H1 Tiny 90M Instruct TII LLM 0.1B Original weights 3
SmolLM 135M Instruct Hugging Face LLM 0.1B Original weights 2

Alternatives

LM Studio Ollama llama.cpp vLLM Jan Text Embeddings Inference MLX LM GLiNER Lemonade Infinity Unsloth Desktop Sentence Transformers Docker Model Runner LocalAI Atomic Chat

Which app should I use?

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.