llama.cpp vs vLLM
Both are catalogued under Model Serving. The figures come from the GitHub API; the assessments are ours.
llama.cppInference in portable C++, from laptops to single-board machinesOfficialMIT
vLLMHigh-throughput inference server with an OpenAI-compatible APIOfficialApache-2.0At a glance
| At a glance | llama.cpp | vLLM |
|---|---|---|
| License | MIT | Apache-2.0 |
| Languages | C++, C | Python, CUDA |
| Deployment | Runs locally / Self-hosted | Self-hosted / Runs locally |
| Maturity | Established | Established |
| Stars | 124k | 89.1k |
| Star growth over the last 7 days | +30 ★ | +23 ★ |
| Forks | 21.7k | 20.7k |
| Open issues | 2k | 6.7k |
| Last commit | 15 Aug 2026 | 15 Aug 2026 |
| Activity | Active | Active |
What each one does
llama.cpp
The quantisation work here is what put usable models on hardware without a datacentre GPU, and much of the local-model ecosystem is built on top of it. Working with it directly means dealing with build flags and hardware specifics that higher-level wrappers hide from you.
Full entry →vLLM
Paged attention and continuous batching let one GPU serve far more concurrent requests than a naive loop, which matters because agents are unusually chatty. The OpenAI-compatible endpoint means most agent frameworks point at it with a base-URL change. Expect real operational work around GPU memory sizing and model loading times.
Full entry →What you can do
llama.cpp
- Run a model in one command — llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF pulls the GGUF straight from Hugging Face and drops you into an interactive session, with no Python environment involved.
- Serve an OpenAI-compatible endpoint — llama serve -hf <repo> starts a REST server that existing OpenAI clients can be pointed at, and ships a built-in web UI for trying the model.
- Fit models larger than VRAM — Integer quantisation from 1.5-bit through 8-bit, combined with CPU+GPU hybrid inference, lets you offload only part of a model to the GPU and keep the rest in system RAM.
- Target the hardware you have — One source tree covers Metal and ARM NEON on Apple silicon, CUDA on NVIDIA, HIP on AMD, MUSA on Moore Threads, plus Vulkan, SYCL, OpenCL, WebGPU and CANN backends.
- Constrain output with GBNF grammars — A GBNF grammar file forces generation to match a syntax you define, which is how you get reliably structured output out of a small local model.
vLLM
- Serve more concurrent agents per GPU — Paged attention and continuous batching keep many requests in flight on a single card instead of running them through a naive one-at-a-time loop.
- Point existing clients at it — Most agent frameworks switch over to the OpenAI-compatible API server with a base-URL change and no client rewrite, and the same server also speaks the Anthropic Messages API and gRPC.
- Run it on the hardware you have —
uv pip install vllminstalls the server, which runs on NVIDIA, AMD and Intel GPUs and on x86/ARM/PowerPC CPUs, with hardware plugins covering Google TPUs, Intel Gaudi, Huawei Ascend and Apple Silicon. - Shrink a model to fit the card — Quantized weights are served directly — FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF and compressed-tensors are all on the supported list — so the memory a model needs is something you choose rather than a fixed property of the checkpoint.
- Constrain output and parse tool calls — Structured output generation runs through
xgrammarorguidance, and the server ships tool-calling and reasoning parsers for the models that emit them.