Once the model fits, the next thing to grow is the KV cache (why it grows). Two settings in llama.cpp-based apps keep it in check.
Flash attention
Flash attention computes attention in a way that avoids building a large intermediate table, which saves memory and often time at long context. Many apps turn it on by default for supported hardware. You mostly meet it because quantizing the V half of the cache needs it.
KV cache quantization
By default the cache is stored at 16 bits. Storing it at 8 bits (q8_0) roughly halves it, and 4 bits (q4_0) cuts it to roughly a quarter. One published example for a 32B model at 32K tokens: about 8.6 GB at 16 bits, 4.3 GB at 8 bits and 2.1 GB at 4 bits. The 8-bit setting usually costs very little quality; 4-bit can show up as small errors on long, exacting tasks, so try 8-bit first.
| App | How to set it |
|---|---|
| llama.cpp | llama-server -m model.gguf -c 32768 -fa on -ctk q8_0 -ctv q8_0. The flags are -ctk and -ctv for the K and V cache types and -fa for flash attention. |
| Ollama | Set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 before starting the server. It applies to every model that server loads. |
| LM Studio, Jan and others | Look for flash attention and cache type options in the model’s load settings; availability depends on the version. |
What it does not do
- It does not help when the weights do not fit. Use a smaller quantization or model first (offloading).
- It saves little at short context. At 4K to 8K tokens the cache is already small.
- Our memory estimates assume a 16-bit cache. If you quantize it, you will have more headroom than the memory calculator shows.