- Token
- A chunk of text a model reads or writes: a short word, part of a longer word, or punctuation. In English, 100 tokens is roughly 75 words.
- Parameters (7B, 70B)
- The learned numbers inside a model, counted in billions (B). More parameters usually means more capability and more memory.
- Weights
- Another word for the parameters, and for the file that stores them.
- Quantization
- Storing the weights in fewer bits to save memory at a small cost in quality. Q4_K_M is the usual default.
- GGUF, MLX, safetensors
- File formats for models. GGUF is the most widely supported for local apps; MLX is for Apple silicon; safetensors is the original weights for servers.
- VRAM
- Memory built into a graphics card. Models run fastest when they fit entirely in it.
- Unified memory
- One pool of memory shared by the CPU and GPU, as on a Mac. Part of it can be used for a model.
- Offload
- Running part of a model from system RAM because it does not fit in VRAM. It works, but it is slower.
- Context length
- How many tokens the model can consider at once: your prompt, the conversation so far and its reply.
- KV cache
- Memory that stores the model’s working notes about every token so far. It grows with context length.
- Prefill and decode
- Prefill is reading your prompt; decode is writing the reply one token at a time. They have different speed limits.
- Tokens per second
- How fast the model writes. About 5 to 8 matches comfortable reading; more is faster than you can read.
- Time to first token
- The wait between sending a prompt and seeing the first word. Dominated by prefill.
- Memory bandwidth
- How quickly the processor can read from memory, in GB per second. It sets the ceiling on decode speed.
- Dense and MoE
- A dense model uses all its parameters for every token. A mixture-of-experts (MoE) model uses only a few “experts” per token, so it is faster than its size suggests.
- Active parameters
- For an MoE model, the parameters actually used per token, for example 3B active of 30B total.
- Instruct and base
- A base model continues text. An instruct (or chat) model is trained to follow instructions and is what most people want.
- Reasoning model
- A model trained to write out its thinking before answering. Better at hard problems, slower and hungrier for tokens.
- Fine-tune
- A model further trained on extra data for a purpose or style. Fine-tunes appear as their own entries.
- Embedding
- A list of numbers that captures the meaning of a piece of text, used for search and for chatting with documents.
- RAG
- Retrieval-augmented generation: look up relevant passages first, then let the model answer from them.
- Flash attention
- A faster, more memory-efficient way to compute attention. Usually on by default, and required for some KV cache settings.
- Speculative decoding
- A small draft model guesses ahead and the big model checks the guesses, for faster output with the same result.
- Open weights
- The model files can be downloaded and run by anyone, under a license. Not always the same as open source.
- OpenAI-compatible API
- A local server that speaks the same protocol as OpenAI’s, so existing tools can use your local model.
New to all this? Start with Run your first local LLM.