Guides
Local AI, explained
Short, practical explainers on running models offline: memory, speed, hardware, apps, privacy and more. Each one links to the calculators that apply it to your hardware.
Start here
- Run your first local LLM in 10 minutes Check your hardware, pick a model that fits, install an app and start chatting, with nothing leaving your machine.
- How the grades and Runs ratings work What the S to D grade means, how “Runs great” is decided, and how the match score ranks models for your hardware.
- Local AI glossary Plain-English definitions of the terms on this site: tokens, quantization, offload, MoE, context, VRAM and more.
Memory and speed
- Quantization explained: Q4_K_M, Q8_0 and friends What the letters mean, how much memory each level saves, and which one to download.
- VRAM, unified memory and system RAM Which memory a model actually lives in on a GeForce, a Mac or a mini PC, and what happens when it doesn’t fit.
- Why long context costs memory (the KV cache) A model that fits at 8K tokens can run out of room at 64K. Here is why, and how to estimate it.
- How many tokens per second do you need? What a token is, how speed feels at 5, 10 and 30 tokens per second, and why reading the prompt is a separate speed.
- Memory bandwidth: the real speed limit Why tokens per second is roughly bandwidth divided by model size, and what that means when you compare machines.
- “Tight fit” and “Offloads to RAM”, decoded What the memory labels mean, how close to the limit each one is, and what to change when you see them.
- Mixture-of-experts models: big on disk, fast to run Why a 30B model with 3B active can feel like a small model, what it still costs in memory, and how expert offload helps.
- Stretch your context: flash attention and KV cache quantization Two settings that cut the memory a long conversation needs, with the flags for Ollama and llama.cpp.
- Speculative decoding: a free speed-up How a small draft model lets a big one write faster with identical output, when it helps and when it does not.
Hardware
- Choosing a graphics card for local AI Why memory size comes first, how to read bandwidth, and what to do about two cards, used cards and laptops.
- Local AI on a Mac: unified memory, MLX and the GPU limit How much of a Mac’s memory a model can use, why MLX helps, and how to raise the limit safely.
- Running without a graphics card What to expect from a CPU-only PC or an older laptop, which models still feel good, and how to get the most out of RAM.
Apps and setup
- GGUF, MLX or safetensors: which file do you need? The three formats you will meet on Hugging Face and the apps that open each one.
- Ollama, LM Studio, llama.cpp or vLLM: which app? The four kinds of app on this site, who each suits, and how to pick without installing them all.
- Use your local model from other apps (the local API) Most local apps can pose as an OpenAI-style server on your own machine. Here is how to point tools at it.
- Set up a local coding assistant What a coding model needs that a chat model does not: context, tool use and a steady speed. Plus the agents that work with local models.
Using models well
- Picking a model for the job: chat, coding, reasoning, agents How to read the task tags, why size is not everything, and a simple way to shortlist three models.
- Reasoning (“thinking”) models: smarter, slower, hungrier Why a model that thinks before it answers needs more tokens and more context, and when it is worth it.
- Chat with your own documents (local RAG) How retrieval works, which models do the embedding, the settings that matter, and the apps that do it for you.
- Open-weight licenses in plain English What “open weights” does and does not mean, and how to read Apache 2.0, MIT and the custom community licenses.
Images, video and small models
- Local image generation: what your GPU needs Why image models are sized differently from chat models, what the precisions mean, and why resolution changes memory.
- Local video generation: what to expect Why video is the hardest local workload, what the memory modes mean, and how to budget time and memory.
- Small “decision” models: classifiers, safety filters and rerankers Models that return a label or a probability instead of prose: where they are useful and why they run almost anywhere.
Privacy, cost and troubleshooting
- What “local” and “offline” really guarantee What stays on your machine, what an app can still send out, and how to run fully air-gapped.
- Local or cloud: when does owning hardware pay off? A simple way to compare a one-off hardware cost with pay-per-token pricing, including the costs people forget.
- Troubleshooting: slow, out of memory or odd output A checklist for the most common problems, from a model that crawls to one that repeats itself or crashes on load.
Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.