LLMs · AI21 Labs

Jamba 1.5 Large

Name: Jamba 1.5 Large
Author: AI21 Labs

Parameters: 399B (94.0B active)
Architecture: Mixture of experts
Max context: 256K tokens
License: jamba-open-model
Released: 2024-08

How to run it

Original weights · Hugging Face

Full-precision files from AI21 Labs, for fine-tuning, vLLM, Transformers.

huggingface.co/ai21labs/AI21-Jamba-Large-1.5 (opens in a new tab)

Free Hugging Face account needed: accept the license on the model page and the download starts right away.

Links checked 5 days ago against Hugging Face, Ollama and GitHub.

Memory needed

Weights plus KV cache and runtime overhead. Leave a little extra free for the operating system and other apps.

KV cache: hybrid Mamba-Transformer: only the attention layers keep a KV cache.

Quantization	Quality	Weights	Total at 8K context	Total at 32K context	Download
`Q2_K`	Low	159 GB	160 GB	161 GB	All files (opens in a new tab)
`Q3_K_M`	Low	195 GB	196 GB	197 GB	All files (opens in a new tab)
`Q4_K_M`	Good	243 GB	244 GB	245 GB	All files (opens in a new tab)
`Q5_K_M`	High	283 GB	284 GB	285 GB	All files (opens in a new tab)
`Q6_K`	High	327 GB	328 GB	329 GB	All files (opens in a new tab)
`Q8_0`	Very high	422 GB	423 GB	424 GB	All files (opens in a new tab)
`F16`	Full	797 GB	798 GB	799 GB	Original weights (opens in a new tab)

Best hardware for Jamba 1.5 Large

Grade · points	Hardware	Quant	Memory	Speed	Fit
75 ptsGreat (A) · 75 of 100 points: Model quality 30.6/45, Memory fit 25.0/25, Speed 4.6/15, Memory use 15.0/15	Apple M3 Ultra (512 GB)	`Q5_K_M`	315 GB / 384 GB	3.8 tok/s	Runs well
67 ptsGood (B) · 67 of 100 points: Model quality 22.1/45, Memory fit 24.0/25, Speed 6.5/15, Memory use 15.0/15	Apple M3 Ultra (256 GB)	`Q2_K`	176 GB / 192 GB	7.6 tok/s	Tight fit
46 ptsUsable (C) · 46 of 100 points: Model quality 22.1/45, Memory fit 4.0/25, Speed 5.9/15, Memory use 15.0/15	Apple M2 Ultra (192 GB)	`Q2_K`	172 GB / 144 GB	6.3 tok/s	Heavy offload

Check it on your own hardware

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.