Before anything else, we compare the memory a model needs (weights, KV cache and a little overhead) with the memory it can use on your hardware. The ratio decides the label.
| Label | Needs, as a share of usable memory | What it means |
|---|---|---|
| Fits in memory | up to 85% | Comfortable, with room for a longer context. |
| Tight fit | 85–100% | It fits, but with little room. A longer context or another app can tip it over. |
| Offloads to RAM | 100–110% | A few layers run from system RAM, which is slower. |
| Heavy RAM offload | 110–125% | A large share runs from RAM. Expect a big drop in speed. |
| Won’t fit | over 125% | Too big even with offload. |
Why offload is slow
Apps built on llama.cpp (LM Studio, Ollama, Jan and others) can keep some layers on the GPU and run the rest on the CPU from system RAM. Every token has to pass through all layers, and the layers in RAM are read at a fraction of VRAM speed, so a small overflow costs more speed than its size suggests (bandwidth). vLLM, SGLang and MLX do not offload: the whole model must fit.
On a Mac or other shared-memory machine there is no separate pool to spill into. A model that is over the GPU’s share is labelled over GPU memory limit instead (Macs).
What to change
- Drop a quantization step, for example Q6_K to Q4_K_M. Biggest effect for the least quality loss (guide).
- Shorten the context. The KV cache is part of the total (guide), and you can shrink it further (KV cache quantization).
- Choose a smaller model or an MoE model of similar quality (MoE).
- Close other GPU-hungry apps. Browsers and games hold VRAM.
A tight fit is not a problem to fix: it works, at full speed. Just leave the context alone. Use the memory calculator to see how many tokens of context you can add before it tips over.