← Back to all projects

vLLM

High-throughput inference server with an OpenAI-compatible API

OfficialApache-2.0
Stars
95.9k
Forks
24.2k
Open issues
9.7k
Last commit
30 Oct 2026

What is vLLM?

Paged attention and continuous batching let one GPU serve far more concurrent requests than a naive loop, which matters because agents are unusually chatty. The OpenAI-compatible endpoint means most agent frameworks point at it with a base-URL change. Expect real operational work around GPU memory sizing and model loading times.

What can you do with vLLM?

  • Serve more concurrent agents per GPU — Paged attention and continuous batching keep many requests in flight on a single card instead of running them through a naive one-at-a-time loop.
  • Point existing clients at it — Most agent frameworks switch over to the OpenAI-compatible API server with a base-URL change and no client rewrite, and the same server also speaks the Anthropic Messages API and gRPC.
  • Run it on the hardware you have — uv pip install vllm installs the server, which runs on NVIDIA, AMD and Intel GPUs and on x86/ARM/PowerPC CPUs, with hardware plugins covering Google TPUs, Intel Gaudi, Huawei Ascend and Apple Silicon.
  • Shrink a model to fit the card — Quantized weights are served directly — FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF and compressed-tensors are all on the supported list — so the memory a model needs is something you choose rather than a fixed property of the checkpoint.
  • Constrain output and parse tool calls — Structured output generation runs through xgrammar or guidance, and the server ships tool-calling and reasoning parsers for the models that emit them.

Before you choose vLLM

  • The supported-hardware list describes the server, not every model on it. New models can need features only the latest cards have, and getting one onto older but still-supported hardware is open work, not a setting.Reported in#50576

Star history

15 Aug to 30 Oct · +6.8k

89.1k95.9k

Frequently asked questions

Is vLLM free for commercial use?

vLLM is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can vLLM be deployed?

vLLM is available as Self-hosted / Runs locally.