Single-board computers run small language models well enough for a voice assistant, a home-automation helper or a robot that has to work offline. What a board can run comes down to three things: how much memory it has, how fast that memory is, and whether anything besides the CPU can do the work.
1. Most ARM boards run models on the CPU
On a Raspberry Pi, a Rockchip RK3588 or RK3576 board, an Orange Pi AIpro or a BeagleBone, llama.cpp and Ollama run the model on the CPU cores. Generation speed is set by how fast those cores can read the model from memory for every token (bandwidth). A Raspberry Pi 5’s memory is rated at about 17 GB/s and an RK3588’s at about 34 GB/s. In published llama.cpp tests that works out to roughly 11 to 21 tokens per second for Llama 3.2 1B and about 5 for the 3B model on a Pi 5, and about 11 for the 3B model on an RK3588. Boards with older Cortex-A53 or A55 cores (Coral Dev Board, Arduino UNO Q and Portenta X8, RK3566) read memory much more slowly, so only the smallest models are usable.
2. Jetson is different: it has a real GPU
NVIDIA’s Jetson boards run models on an Ampere or Blackwell GPU with CUDA, using memory shared with the CPU. A Jetson Orin Nano Super (8 GB, 102 GB/s) runs 3B to 4B models quickly; the AGX Orin (32 or 64 GB, 205 GB/s) and AGX Thor (128 GB, 273 GB/s) run much bigger ones. Ollama, llama.cpp and NVIDIA’s containers for vLLM and SGLang all work. On Orin Nano and Orin NX, flash JetPack 6.2 with a “Super” configuration to get the faster memory clock.
3. NPUs and AI HATs: mostly for vision
- Hailo-8 and Hailo-8L (the Raspberry Pi AI Kit and AI HAT+, 13 to 26 TOPS) and Google’s Edge TPU (Coral, 4 TOPS) run compiled vision models such as object detection. They have no room for a language model, and llama.cpp can’t use them.
- The Raspberry Pi AI HAT+ 2 (Hailo-10H, 40 TOPS, with its own 8 GB) runs a few 1.5B language models through Hailo’s own Ollama build. In CNX Software’s review it was slower than the Pi 5’s own CPU; its strength is vision while the CPU stays free.
- Rockchip’s NPU (6 TOPS on the RK3588 and RK3576) runs converted 8-bit models with Rockchip’s RKLLM toolkit. Generation is still limited by memory speed, so it isn’t much faster than the CPU, but it leaves the CPU free and draws less power.
- Huawei’s Ascend (Orange Pi AIpro, 20 TOPS) and TI’s C7x and MMA (BeagleBone AI-64, 8 TOPS) need their makers’ own toolchains; llama.cpp runs on the CPU cores instead.
A TOPS figure measures 8-bit multiply speed for vision networks. For language models, memory size and speed matter far more.
Setting one up
- Memory sets the ceiling. Boards with 4 GB fit models up to about 3B; 8 GB and up fit 7B to 8B models, though on a CPU those generate at only 2 to 5 tokens per second.
- Use a 64-bit OS (Raspberry Pi OS 64-bit, Armbian, Ubuntu) and an app with an ARM Linux build: Ollama, llama.cpp, LM Studio, llamafile or LocalAI.
- Keep it cool. Generating text keeps every core busy; without a fan or a large heatsink the chip slows itself down within a minute or two.
- Put models on fast storage. An NVMe SSD loads a model in seconds; a microSD card takes much longer.
- Keep prompts short. Reading a long prompt on a CPU is slow (two speeds).
- x86 boards (LattePanda, UDOO BOLT, UP Squared) are small PCs: they run Windows or Linux and use their Intel or AMD graphics like a mini PC (no graphics card).
See every board under Hardware, or press Detect on My hardware: browsers on a Raspberry Pi, Rockchip board or Jetson report their graphics chip, so we can usually tell which board it is.