
LocalAI
A self-hosted stand-in for the OpenAI API — chat to speech to video — over swappable local inference engines
What is LocalAI?
Point any OpenAI client at it and it answers: chat with tool calling, embeddings, image generation, transcription and speech, a reranker, even the realtime speech-to-speech API — plus imitations of the Anthropic, ElevenLabs and Ollama APIs. A small core detects your hardware and pulls inference backends on demand — llama.cpp, vLLM, MLX, whisper.cpp among many — a web gallery installs models by click, autonomous agents with MCP support are embedded, and API keys, OIDC and quotas cover multiple users. The honest comparison: Ollama stays simpler for chatting with a model on a laptop, a bare vLLM is leaner for one model at maximum throughput, and there is no native Windows build — containers or WSL only.
What can you do with LocalAI?
- The whole OpenAI surface, not just chat — Chat with tool calling and constrained grammars, embeddings, images, transcription, speech synthesis, reranking and the realtime speech-to-speech API over WebSocket — one deployment answers clients written for any of them, and it speaks the Anthropic, ElevenLabs and Ollama APIs besides.
- Backends arrive on demand, matched to your machine — The core is a small binary or container; each engine ships as a separate image pulled when a model needs it, and the project states GPU capability is detected automatically. CUDA, ROCm, Intel oneAPI, Vulkan, Apple Metal and Jetson each have their own build.
- Models install from a gallery, or from anywhere — The web UI's gallery indexes roughly 1,700 configurations installed by click; the CLI takes the same names, or pulls straight from Hugging Face, Ollama and OCI registries by URI.
- Agents and multi-user machinery are built in — The embedded LocalAGI layer gives agents tools, per-agent document collections and MCP servers from the web UI or JSON config, with skills off until enabled; API keys, OIDC sign-on, quotas and role-based access arrived for shared deployments.
- Two ways to outgrow one machine — Peer-to-peer federation load-balances across instances, worker mode shards a single llama.cpp model's weights across machines — with the documented limits that it serves one model and workers cannot join mid-inference — and a separate distributed mode routes across a fleet by available VRAM.
Before you choose LocalAI
- There is no native Windows build — containers or WSL only — and the request has been the tracker's most-reacted open issue since May 2024.Reported in#2368
- For raw single-model throughput it is a management layer, not the engine: it can run vLLM for you, but running vLLM directly is leaner when that is all you need.
- Development is heavily concentrated in the founder — thousands of commits to the next contributor's hundred-odd — though a second maintainer is now named.
Star history
19 Aug to 30 Oct · +1.5k
Frequently asked questions
Is LocalAI free for commercial use?
LocalAI is released under the MIT licence — OSI-approved open source, which permits commercial use.
How can LocalAI be deployed?
LocalAI is available as Runs locally / Self-hosted.