← Back to all projects
llama.cpp
Inference in portable C++, from laptops to single-board machines
OfficialMIT
What is llama.cpp?
The quantisation work here is what put usable models on hardware without a datacentre GPU, and much of the local-model ecosystem is built on top of it. Working with it directly means dealing with build flags and hardware specifics that higher-level wrappers hide from you.
What can you do with llama.cpp?
- Run a model in one command — llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF pulls the GGUF straight from Hugging Face and drops you into an interactive session, with no Python environment involved.
- Serve an OpenAI-compatible endpoint — llama serve -hf <repo> starts a REST server that existing OpenAI clients can be pointed at, and ships a built-in web UI for trying the model.
- Fit models larger than VRAM — Integer quantisation from 1.5-bit through 8-bit, combined with CPU+GPU hybrid inference, lets you offload only part of a model to the GPU and keep the rest in system RAM.
- Target the hardware you have — One source tree covers Metal and ARM NEON on Apple silicon, CUDA on NVIDIA, HIP on AMD, MUSA on Moore Threads, plus Vulkan, SYCL, OpenCL, WebGPU and CANN backends.
- Constrain output with GBNF grammars — A GBNF grammar file forces generation to match a syntax you define, which is how you get reliably structured output out of a small local model.
Before you choose llama.cpp
- Check that the model you plan to serve reuses its earlier work between turns. There is an open report of the server reprocessing the whole prompt each time, which both costs the work again and loses what came before.Reported in#21831
Star history
15 Aug to 30 Oct · +8.9k
124k133k
Frequently asked questions
Is llama.cpp free for commercial use?
llama.cpp is released under the MIT licence — OSI-approved open source, which permits commercial use.
How can llama.cpp be deployed?
llama.cpp is available as Runs locally / Self-hosted.