Apps · Inference server

Text Embeddings Inference

by Hugging Face · Apache-2.0 · 5,087 GitHub stars

Hugging Face’s fast server for rerankers and text classifiers: BERT, XLM-RoBERTa and ModernBERT models behind /rerank and /predict.

macOSWindows (WSL2)Linux

Get Text Embeddings Inference

At a glance

Best for
Serving an encoder reranker or classifier (BGE Reranker, GTE, MiniLM) at low latency with batching.
Inference engine
  • TEI (Rust, Candle)
GPU support
  • NVIDIA CUDA
  • Apple Metal
  • CPU only
Opens
  • Safetensors (original weights)
Features
  • HTTP API
  • Many users at once
Platforms
Apple silicon with Metal through Homebrew; Linux and Windows (WSL2) through Docker images for CPU or NVIDIA GPUs.
Speed
Built for encoders: dynamic batching and Flash Attention keep each rerank or classify call to a few milliseconds on a GPU.

Sources: github.com · huggingface.co

Install

macOS

brew install text-embeddings-inference

Linux

docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-embeddings-inference:latest --model-id BAAI/bge-reranker-v2-m3

Windows (WSL2)

Docker Desktop with WSL2, then the Linux docker command

6 models on this site run in Text Embeddings Inference

Decision models it can load: rerankers, classifiers and scorers with a supported architecture. Open a model for its speed and memory on your hardware.

Model Made by Type Size Opens it as Quality tier
BGE Reranker v2 M3 BAAI Decision 570M Original weights 72
GTE Reranker ModernBERT Base Alibaba Decision 150M Original weights 70
Jina Reranker v2 Base Multilingual Jina AI Decision 280M Original weights 70
ModernBERT Large Zero-shot v2.0 Moritz Laurer Decision 400M Original weights 70
Granite Embedding Reranker English R2 IBM Decision 150M Original weights 68
ms-marco MiniLM L6 v2 Sentence Transformers Decision 23M Original weights 52

Alternatives

LM Studio Ollama llama.cpp vLLM Jan MLX LM SGLang GLiNER Lemonade Infinity Unsloth Desktop Sentence Transformers Docker Model Runner LocalAI Atomic Chat

Which app should I use?

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.