← Back to all projects

vLLM
High-throughput inference server with an OpenAI-compatible API
OfficialApache-2.0
What is vLLM?
Paged attention and continuous batching let one GPU serve far more concurrent requests than a naive loop, which matters because agents are unusually chatty. The OpenAI-compatible endpoint means most agent frameworks point at it with a base-URL change. Expect real operational work around GPU memory sizing and model loading times.
What can you do with vLLM?
- Serve more concurrent agents per GPU — Paged attention and continuous batching keep many requests in flight on a single card instead of running them through a naive one-at-a-time loop.
- Point existing clients at it — Most agent frameworks switch over to the OpenAI-compatible API server with a base-URL change and no client rewrite, and the same server also speaks the Anthropic Messages API and gRPC.
- Run it on the hardware you have —
uv pip install vllminstalls the server, which runs on NVIDIA, AMD and Intel GPUs and on x86/ARM/PowerPC CPUs, with hardware plugins covering Google TPUs, Intel Gaudi, Huawei Ascend and Apple Silicon. - Shrink a model to fit the card — Quantized weights are served directly — FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF and compressed-tensors are all on the supported list — so the memory a model needs is something you choose rather than a fixed property of the checkpoint.
- Constrain output and parse tool calls — Structured output generation runs through
xgrammarorguidance, and the server ships tool-calling and reasoning parsers for the models that emit them.
Before you choose vLLM
- The supported-hardware list describes the server, not every model on it. New models can need features only the latest cards have, and getting one onto older but still-supported hardware is open work, not a setting.Reported in#50576
Star history
15 Aug to 30 Oct · +6.8k
89.1k95.9k
Frequently asked questions
Is vLLM free for commercial use?
vLLM is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.
How can vLLM be deployed?
vLLM is available as Self-hosted / Runs locally.