While a model reads and writes, it keeps a KV cache: the attention keys and values for every token so far, in every layer. It lets the model avoid recomputing the past, and it grows in step with the conversation.
KV cache ≈ 2 × layers × KV heads × head size × bytes × tokensBecause it scales with tokens, doubling the context doubles this part of the memory bill. The weights stay the same. For a long document, a long chat or a coding agent that keeps many files in view, the KV cache can rival the model itself.
What to do about it
- Use the context you need. Most chat fits in 8K tokens; set 32K or more only for long documents.
- Pick models with efficient attention. Grouped-query attention and mixture-of-experts designs keep the cache small. Our estimates use each model’s real layer and head counts.
- Leave headroom. The runtime and operating system need memory too.
See the exact number for any model and context length in the memory calculator.