Model Serving

OllamaModel Serving

Handles model download, quantisation and an OpenAI-compatible server so a local model is a single command away. The obvious starting point for development against local inference. For serving many concurrent users, a throughput-oriented server will do considerably better.

llama.cppModel Serving

The quantisation work here is what put usable models on hardware without a datacentre GPU, and much of the local-model ecosystem is built on top of it. Working with it directly means dealing with build flags and hardware specifics that higher-level wrappers hide from you.

vLLMModel Serving

Paged attention and continuous batching let one GPU serve far more concurrent requests than a naive loop, which matters because agents are unusually chatty. The OpenAI-compatible endpoint means most agent frameworks point at it with a base-URL change. Expect real operational work around GPU memory sizing and model loading times.

SGLangModel Serving

Its prefix cache pays off precisely in the agent case, where many calls share a long, identical system prompt. Also strong at constrained decoding when responses must match a schema. The main alternative in this space is vLLM; benchmark both on your own traffic shape.