llama.cpp vs SGLang
Both are catalogued under Model Serving. The figures come from the GitHub API; the assessments are ours.
llama.cppInference in portable C++, from laptops to single-board machinesOfficialMIT
SGLangServing engine tuned for prefix reuse and structured outputOfficialApache-2.0At a glance
| At a glance | llama.cpp | SGLang |
|---|---|---|
| License | MIT | Apache-2.0 |
| Languages | C++, C | Python |
| Deployment | Runs locally / Self-hosted | Self-hosted |
| Maturity | Established | Established |
| Stars | 124k | 31.9k |
| Star growth over the last 7 days | +30 ★ | +10 ★ |
| Forks | 21.7k | 7.9k |
| Open issues | 2k | 4.9k |
| Last commit | 15 Aug 2026 | 15 Aug 2026 |
| Activity | Active | Active |
What each one does
llama.cpp
The quantisation work here is what put usable models on hardware without a datacentre GPU, and much of the local-model ecosystem is built on top of it. Working with it directly means dealing with build flags and hardware specifics that higher-level wrappers hide from you.
Full entry →SGLang
Its prefix cache pays off precisely in the agent case, where many calls share a long, identical system prompt. Also strong at constrained decoding when responses must match a schema. The main alternative in this space is vLLM; benchmark both on your own traffic shape.
Full entry →What you can do
llama.cpp
- Run a model in one command — llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF pulls the GGUF straight from Hugging Face and drops you into an interactive session, with no Python environment involved.
- Serve an OpenAI-compatible endpoint — llama serve -hf <repo> starts a REST server that existing OpenAI clients can be pointed at, and ships a built-in web UI for trying the model.
- Fit models larger than VRAM — Integer quantisation from 1.5-bit through 8-bit, combined with CPU+GPU hybrid inference, lets you offload only part of a model to the GPU and keep the rest in system RAM.
- Target the hardware you have — One source tree covers Metal and ARM NEON on Apple silicon, CUDA on NVIDIA, HIP on AMD, MUSA on Moore Threads, plus Vulkan, SYCL, OpenCL, WebGPU and CANN backends.
- Constrain output with GBNF grammars — A GBNF grammar file forces generation to match a syntax you define, which is how you get reliably structured output out of a small local model.
SGLang
- Reuse a shared system prompt — RadixAttention prefix caching retains the KV state for identical leading tokens, so repeated agent calls that all carry one long system prompt skip recomputing it.
- Constrain output to a schema — Constrained decoding forces generated responses to match a supplied schema instead of relying on retries to repair malformed tool-call JSON.
- Run it beyond NVIDIA GPUs — The PyPI package is
sglang, and besides NVIDIA GB200/B300/H100/A100 the runtime targets AMD MI355/MI300, Intel Xeon CPUs, Google TPUs and Ascend NPUs. - Reach it with an OpenAI client — The server is compatible with OpenAI APIs and with most Hugging Face models, so code already written against the OpenAI SDK reaches it without a rewrite.
- Reuse the engine for RL rollouts — SGLang has native RL integrations and is used as the rollout backend by post-training frameworks including AReaL, Miles, slime, Tunix and verl.