Most local-model problems fall into a handful of causes. Work down the list that matches your symptom.
It is very slow
- Is the GPU being used? In the app, look for a GPU layers or offload setting and make sure it is not zero. Watch GPU memory use while it runs. A CPU-only build, or missing drivers, is the most common cause.
- Does the model fit? If part of it runs from system RAM, speed drops sharply (offloading). Compare against your hardware on My hardware.
- Is the context huge? Reading a very long prompt takes time, and a large cache can push the model out of fast memory (two speeds).
- Is something else using the GPU? Browsers, games and other apps hold video memory.
- Is the machine throttling? Laptops on battery or with poor cooling slow down under sustained load. Plug in and check temperatures.
It runs out of memory or crashes while loading
- Pick a smaller quantization or model (quantization).
- Lower the context length (why), or quantize the cache (how).
- Close other apps. On a Mac, remember the GPU can use only part of the memory (Macs).
- Check the memory calculator before downloading.
It works at first, then fails in a long chat
The KV cache grows with every token. A model that fit at the start can run out as the conversation grows. Start a new chat, or reduce the context setting so older text is dropped earlier.
The output is odd
- Repeating, rambling or ignoring instructions. Check you are using an instruct model, not a base model, and that the app picked the right chat template for it. Try a slightly higher quantization.
- Forgetting the start of the conversation. The context window is too small for your input. Raise it.
- Gibberish. A corrupted download, a model newer than your app understands, or very low quantization. Update the app, re-download, and compare the file size with the source.
- Mistakes at very low bit-widths. Below 4 bits, quality drops quickly. Prefer a smaller model at 4 bits.
It will not download or load
- Some models are gated: accept the terms on Hugging Face and sign in (licenses).
- Check free disk space. Large models are tens of gigabytes.
- “Unsupported architecture” usually means the app is too old. Update it.
- Make sure the app can read the file type (formats).
Still stuck? Start again from the first-model guide with a small model, to separate a hardware problem from a model problem.