Apps · Inference server

Infinity

by Michael Feil · MIT · 2,968 GitHub stars

Python server for rerankers and classifiers with an OpenAI-style API; runs any BERT-style sequence-classification model.

macOSWindows (WSL2)Linux

Get Infinity

At a glance

Best for
Running rerankers and classifiers (including DeBERTa models TEI does not cover) behind one local API.
Inference engine
  • PyTorch
  • ONNX
  • CTranslate2
GPU support
  • NVIDIA CUDA
  • AMD ROCm
  • Apple Metal
  • CPU only
Opens
  • Safetensors (original weights)
  • ONNX
Features
  • HTTP API
  • Many users at once
Platforms
pip on macOS and Linux (Apple MPS, NVIDIA CUDA, AMD ROCm or CPU); Docker images for Linux and Windows (WSL2).
Speed
Dynamic batching; the ONNX and CTranslate2 engines speed up BERT-size models on a CPU.

Sources: github.com

Install

macOS

pip install "infinity-emb[all]"

Linux

pip install "infinity-emb[all]"

Windows (WSL2)

docker run -it -p 7997:7997 michaelf34/infinity:latest v2 --model-id {hf}

12 models on this site run in Infinity

Decision models it can load: rerankers, classifiers and scorers with a supported architecture. Open a model for its speed and memory on your hardware.

Model Made by Type Size Opens it as Quality tier
Llama Prompt Guard 2 86M Meta Decision 280M Original weights 76
BGE Reranker v2 M3 BAAI Decision 570M Original weights 72
DeBERTa v3 Large Zero-shot v2.0 Moritz Laurer Decision 440M Original weights 72
GTE Reranker ModernBERT Base Alibaba Decision 150M Original weights 70
Jina Reranker v2 Base Multilingual Jina AI Decision 280M Original weights 70
ModernBERT Large Zero-shot v2.0 Moritz Laurer Decision 400M Original weights 70
Granite Embedding Reranker English R2 IBM Decision 150M Original weights 68
Llama Prompt Guard 2 22M Meta Decision 71M Original weights 68
ProtectAI DeBERTa v3 Prompt Injection v2 Protect AI Decision 180M Original weights 66
DeBERTa v3 Base Zero-shot v2.0 Moritz Laurer Decision 180M Original weights 64
BART Large MNLI Meta Decision 410M Original weights 56
ms-marco MiniLM L6 v2 Sentence Transformers Decision 23M Original weights 52

Alternatives

LM Studio Ollama llama.cpp vLLM Jan Text Embeddings Inference MLX LM SGLang GLiNER Lemonade Unsloth Desktop Sentence Transformers Docker Model Runner LocalAI Atomic Chat

Which app should I use?

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.