Ollama vs vLLM: Which Should Serve Your AI Agents in Production?

Use Ollama when one person (or one process) talks to the model at a time: local development, prototyping agents on your laptop, personal assistants, offline tools, and demos. Use vLLM when multiple requests hit the model concurrently: production agent APIs, customer-facing chat, internal tools with more than a handful of simultaneous users, and batch pipelines. The exceptions cut both ways. A small internal tool with 3-5 occasional users can live on Ollama indefinitely if latency spikes are acceptable. And a solo developer with a CUDA Linux box who wants production-realistic behavior can run vLLM from day one. But as a default: Ollama for dev, vLLM for production.

This post is the decision guide. If you have already decided on vLLM and want the deployment walkthrough with Docker Compose, nginx, and monitoring, go straight to our companion guide on deploying LangGraph with vLLM to production in 30 minutes. Here we cover the why: what the two tools actually are under the hood, honest benchmark numbers with hardware assumptions stated, a decision tree by scenario, the agent-specific concurrency math most comparisons miss, and the migration path between them.

One framing note before we start. This is not a "which tool is better" question. Ollama and vLLM optimize for opposite goals. Ollama optimizes for time-to-first-token-on-your-machine: one command, any hardware, model running in minutes. vLLM optimizes for aggregate throughput per GPU dollar: squeeze the maximum concurrent tokens out of a CUDA card. Judging Ollama on throughput or vLLM on ease of setup misses the point of both.

Ollama is a model runner built for local use. Historically it wrapped llama.cpp; newer versions run some model families on Ollama's own engine built on the same underlying ggml library, but the design philosophy is unchanged. It runs GGUF-quantized models, manages downloads through a Docker-like registry (ollama pull llama3.1), handles model loading and unloading automatically, and runs on macOS (Apple Silicon via Metal), Windows, and Linux, on NVIDIA GPUs, AMD GPUs, or plain CPU. It exposes both a native API and an OpenAI-compatible endpoint, and supports tool calling for models that support it. Setup is genuinely one command.

vLLM is an inference engine built for serving. It came out of UC Berkeley research and introduced PagedAttention, a memory management technique that changed how the industry serves LLMs. It loads models directly from Hugging Face in FP16/BF16 or quantized formats (AWQ, GPTQ, FP8, INT8), batches concurrent requests continuously, splits large models across GPUs with tensor parallelism, and exposes an OpenAI-compatible server with tool-call parsing for agent workloads. It wants a Linux machine with a CUDA GPU (ROCm support exists but is secondary), and its configuration surface is an order of magnitude larger than Ollama's.

Both tools sit in the same slot of a self-hosted LLM agent stack: the inference layer your agent framework talks to. Everything above them (LangGraph, CrewAI, your FastAPI service) is identical either way, which is exactly what makes the choice low-risk and reversible, as we will see in the migration section.

The performance gap between these tools is not about code quality. It comes from two architectural decisions: how they manage GPU memory for the KV cache, and how they schedule concurrent requests.

KV cache memory. When an LLM generates text, it caches attention keys and values for every token in the context. A naive server reserves one contiguous memory block per request, sized for the maximum possible context. Most of that reservation sits empty, so a GPU that could theoretically serve 20 requests runs out of "reserved" memory after 4. vLLM's PagedAttention treats the KV cache like an operating system treats RAM: it splits the cache into small pages and allocates them on demand as each sequence grows. Memory waste drops from 60-80% to a few percent, which means far more concurrent sequences fit on the same GPU. Ollama, by contrast, allocates memory statically when a model loads, sized by OLLAMA_NUM_PARALLEL times the context length. Simple and predictable, but nothing is shared or paged.

Request scheduling. Ollama processes requests up to its parallelism limit (OLLAMA_NUM_PARALLEL, which defaults to 1, or 4 when memory allows) and queues the rest FIFO up to OLLAMA_MAX_QUEUE (default 512). Requests beyond the parallel limit wait their turn, so under load the GPU behaves close to sequentially and per-user latency stacks up linearly. vLLM uses continuous batching: instead of waiting for a batch of requests to all finish before starting the next batch, it operates at the granularity of individual token steps. The moment any sequence finishes, its slot in the running batch is handed to a waiting request on the very next iteration. The GPU stays saturated, and adding concurrent users barely moves per-user latency until you hit the memory ceiling.

Add tensor parallelism (splitting one model's weights across 2, 4, or 8 GPUs, a single flag in vLLM) and you get the full picture: Ollama is a lovingly polished single-lane road, vLLM is a highway interchange. Both get one car to the destination at about the same speed. The difference appears when there are forty cars.

Here is the full side-by-side. Details on the benchmark rows follow in the next section.

Two rows deserve emphasis for agent builders. First, tool calling: both support it, but vLLM requires you to pick the right parser flag for your model family (--tool-call-parser hermes for Qwen, llama3_json for Llama, and so on) plus --enable-auto-tool-choice. Get this wrong and your agent framework receives tool calls as plain text. Second, model management: Ollama's ability to hot-swap between models is genuinely great for development, where you might compare three models in an hour. vLLM serves one model per process, which is what you want in production anyway. Our roundup of the best open-source models for agents covers which models are worth serving in the first place.

Benchmark numbers without hardware context are marketing. Here is a synthesis of published 2025-2026 comparisons, including Red Hat's deep-dive benchmark and independent tests, normalized to what you should expect on a single 24GB consumer or workstation GPU (RTX 4090 class) serving an 8B model:

The caveats, because they matter:

Quantization is not held constant. Ollama defaults to Q4_K_M GGUF; vLLM benchmarks often run FP16 or AWQ. A 4-bit model is smaller and faster per token but slightly lower quality, so some published gaps understate or overstate depending on setup. Hardware changes everything. On an H100 with FP8, vLLM's advantage grows; on a CPU-only box, vLLM does not meaningfully run and Ollama wins by default. Ollama keeps improving. Recent versions handle parallelism better than the versions in older benchmarks, so treat any "26x" headline number as a stress-test extreme, not a typical gap. Latency and throughput trade off. vLLM's aggregate numbers come from batching; an individual request in a full batch generates tokens somewhat slower than it would alone.

The honest summary: for one user, pick whichever is easier for you (that is Ollama). From roughly 4-8 truly concurrent requests onward, vLLM pulls decisively ahead, and past 20 concurrent requests it is not a contest. If you want to sanity-check what your own hardware and expected load imply, our AI stack builder walks through the sizing questions interactively.

The quantization ecosystems are the hidden fork in the road, because they determine which hardware each tool actually fits.

Ollama's world is GGUF. GGUF is the llama.cpp format, designed for mixed CPU/GPU execution and memory-mapped loading. It is why Ollama runs a 8B model on a MacBook Air, offloads half a 70B model to CPU RAM when VRAM runs out, and works on machines with no GPU at all. Q4_K_M is the pragmatic default: roughly 4.5 bits per weight with minimal quality loss for most agent tasks.

vLLM's world is GPU-native formats. AWQ and GPTQ are 4-bit weight quantization schemes with CUDA kernels (Marlin kernels on Ampere and newer) tuned for batched inference. FP8 is the modern choice on Hopper (H100) and Ada Lovelace GPUs: near-lossless quality with hardware-accelerated math. vLLM can technically load GGUF files, but support is experimental and throughput is poor; GGUF was designed for llama.cpp's execution model, not paged batching. If you are moving a model from Ollama to vLLM, do not carry the GGUF file over. Pull the AWQ or FP8 variant of the same model from Hugging Face instead.

Hardware fit falls out directly:

For a broader look at picking and provisioning the server itself, see our guide to running AI agents on your own server.

Most Ollama-vs-vLLM comparisons assume a chat app: one user message equals one LLM request. Agent systems break that assumption badly, and this is the single most important section of this post if you are building agents.

A typical ReAct-style agent handling one user message does something like: plan (1 LLM call), call a tool, interpret the result (1 call), call another tool, interpret (1 call), maybe reflect or retry (1-2 calls), then write the final answer (1 call). That is 5-8 LLM calls per user message, routinely more for multi-step research or a RAG agent that retrieves, grades, and re-queries. Multi-agent systems multiply again: a supervisor delegating to three workers can burn 15-30 calls on one task. If you have built graphs in the style of our LangGraph tutorial, count the nodes that invoke the model and you will see this immediately.

Now run the numbers. An internal tool with 10 users, each triggering an agent run every few minutes, sounds like a trivial load. But each run holds a sequence of LLM calls, tool loops overlap across users, and several agents inevitably sit in their reasoning loops simultaneously. Your "10 user" app is really a 30-80 concurrent-request workload at peak. On Ollama with OLLAMA_NUM_PARALLEL=4, most of those calls queue. Queued calls inside an agent loop are especially painful because they compound: a run needing six sequential LLM calls, each waiting 20 seconds in queue, takes two extra minutes. Users experience the agent as broken, not slow.

There is a second, subtler effect: prefix caching. Agent calls within one run share a long common prefix (system prompt, tool definitions, accumulated conversation). vLLM's automatic prefix caching reuses the KV cache for those shared tokens across requests, cutting prefill work substantially for exactly the repetitive-prefix pattern agents produce. Ollama caches context within a session but has nothing comparable across a batch of concurrent sequences.

The practical rule: estimate concurrency from LLM calls in flight, not from user count. Take peak simultaneous agent runs, multiply by the average depth of your loop, and if the result is above about 8, you are in vLLM territory even if your user count says otherwise.

Work through these scenarios top to bottom and stop at the first match.

Scenario 1: Solo developer on a laptop. You are prototyping agents, testing prompts, comparing models. Ollama, no contest. One command to install, instant model switching, runs on your Mac or gaming PC, and your agent framework will not know the difference later. This is also the right answer for personal automations and offline tools that only you use.

Scenario 2: Small team internal tool, under ~5 truly concurrent users. A support-ticket summarizer or internal Q&A bot with light, bursty usage. Ollama is defensible if occasional 30-60 second waits during overlap are acceptable and you value zero ops. Set OLLAMA_NUM_PARALLEL=4, size RAM for parallel slots times context length, and monitor queue behavior. The moment people complain about slowness during busy hours, that is your signal to migrate, not to buy a bigger GPU.

Scenario 3: Customer-facing agent, any real concurrency. Paying users, SLAs, or more than ~8 concurrent LLM calls (remember the agent multiplier above). vLLM, full stop. This also applies to internal tools that became load-bearing. Typical hardware: an RTX 4090 (24GB) serves an 8B model at FP16 or a 14B at AWQ for dozens of concurrent users; an A6000 (48GB) or dual 4090s handle 32B-70B quantized; an H100 (80GB) runs 70B AWQ or FP8 with headroom. The full deployment recipe, from Docker Compose through nginx and Grafana, is in our LangGraph + vLLM production guide.

Scenario 4: Batch and offline pipelines. Nightly document processing, dataset generation, bulk evaluation. vLLM, because throughput per GPU-hour is the whole game and continuous batching is built for exactly this. Ollama would leave most of the GPU idle.

Scenario 5: You have no CUDA GPU at all. Mac-only shop, or CPU servers. Ollama by elimination for local work, and honestly consider a managed API for production (see the "when neither" section below). Do not fight vLLM onto unsupported hardware.

If you are still unsure after this list, default to the two-phase pattern: Ollama now, vLLM when concurrency arrives. The next section shows why that transition costs almost nothing.

Because both servers speak the OpenAI API, migrating an agent from Ollama to vLLM is not a rewrite. It is a configuration change. Here is the entire diff in a LangChain/LangGraph agent:

from langchain_openai import ChatOpenAI

# Development: Ollama on your laptop
llm = ChatOpenAI(
    base_url="http://localhost:11434/v1",   # Ollama
    api_key="ollama",                       # dummy, Ollama ignores it
    model="qwen2.5:14b-instruct",
)

# Production: vLLM on your GPU server
llm = ChatOpenAI(
    base_url="http://your-server:8000/v1",  # vLLM - the only real change
    api_key="not-needed",
    model="Qwen/Qwen2.5-14B-Instruct-AWQ",  # HF id of the same model, AWQ variant
)

In practice you make base_url and model environment variables from day one (LLM_BASE_URL, LLM_MODEL), and the "migration" becomes editing your .env file. The same pattern works in the OpenAI SDK directly, CrewAI, AutoGen, and every framework that accepts a custom base URL.

Three real gotchas to check during the switch:

1. Model identity changes. Ollama model tags (qwen2.5:14b) map to Hugging Face repos in vLLM, and you should switch from the GGUF quantization to an AWQ, GPTQ, or FP8 variant. Same weights family, different container. Re-run your eval set after switching; a Q4_K_M and an AWQ quantization of the same model are close but not bit-identical.

2. Tool calling needs explicit flags in vLLM. Where Ollama detects tool support from the model, vLLM needs --enable-auto-tool-choice --tool-call-parser <parser> matched to your model family. If your agent suddenly stops calling tools after migration, this flag is the culprit 90% of the time.

3. Sampling defaults differ. Pin temperature, max_tokens, and any stop sequences explicitly in your code rather than relying on server defaults, so behavior stays constant across backends.

This portability is also your insurance policy in the other direction: the same one-line swap points your agent at OpenAI, Anthropic-compatible gateways, or any hosted endpoint if self-hosting stops making sense. Vendor lock-in at the inference layer is now a choice, not a default.

An honest decision guide has to include the option of not self-hosting at all. The math is simple and frequently ignored: a capable GPU server costs roughly $700-1,500/month rented (A6000/A100 class) or a five-figure sum bought, plus real engineering hours for setup, upgrades, and on-call. Small-model API pricing (GPT-4o-mini class, Llama 70B on hosted providers) sits around $0.15-0.60 per million tokens blended. If your agents process under roughly 1-2 million tokens per day, the API bill is tens of dollars a month and self-hosting cannot compete on cost alone. The crossover where a dedicated GPU pays for itself typically arrives in the 5-20 million tokens/day range, or earlier if you have hard data-privacy requirements that make managed APIs a non-starter regardless of price. We work through the full spreadsheet in self-hosting vs the OpenAI API: the real cost math.

Within the self-hosted world, two other engines deserve a mention so you know when to look past both Ollama and vLLM:

SGLang is vLLM's closest competitor and beats it on some workloads, particularly ones heavy on structured output (JSON schemas, constrained decoding) and repeated prefixes, thanks to RadixAttention caching. It is a legitimate choice for large agent deployments; the operational story is similar to vLLM's, so everything in this post about "vLLM territory" applies. llama.cpp's own server (llama-server) is what to use when you want Ollama-class hardware flexibility without Ollama's abstractions: slightly more control, slightly more setup, same GGUF ecosystem and same single-user sweet spot. TGI (Hugging Face's Text Generation Inference) powered many early deployments but has shifted into maintenance mode, so we do not recommend it for new builds in 2026.

The short version: Ollama and vLLM remain the two defaults for good reason. Reach for SGLang when you are already at vLLM scale and structured output dominates your workload; reach for llama.cpp server when Ollama feels too magical; reach for a managed API when your token volume does not justify a GPU.

Shipping Ollama to production because it worked in dev. The most common failure mode by far. Everything feels fine in testing because you are one user; the first day with real traffic, requests queue behind OLLAMA_NUM_PARALLEL and p95 latency goes from 3 seconds to 90. Ollama is not "bad in production", it is single-user infrastructure being asked to do multi-user work.

Benchmarking with one request and extrapolating. Single-request tok/s is the one metric where the tools tie. Always load-test at your realistic concurrency (use the agent multiplier from earlier) before committing to hardware.

Carrying GGUF files into vLLM. It loads, it runs, and it wastes most of what you paid for. Use the AWQ, GPTQ, or FP8 release of the model instead.

Forgetting vLLM's tool-calling flags. An agent that reasons fine but never executes tools after a migration almost always means a missing --tool-call-parser. Check this first, not your prompt.

Ignoring KV cache memory when sizing. Model weights are only part of VRAM. Long agent contexts times concurrent sequences can eat as much memory as the model itself. In vLLM, watch gpu_cache_usage_perc; in Ollama, remember RAM scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.

Self-hosting at 100k tokens a day. Prestige-driven infrastructure. Run the cost math honestly; a GPU idling at 3% utilization is the most expensive way to serve an agent.

Skipping monitoring because "it is just one server". The three metrics that catch nearly every incident (p95 latency, GPU/cache memory, error rate) take an hour to set up with vLLM's built-in Prometheus endpoint. The monitoring section of our vLLM deployment guide has the exact dashboard.

Ollama and vLLM are not rivals so much as stages of the same journey. Prototype your agent against Ollama on whatever machine you have. Wire base_url and model through environment variables from the first commit. When concurrent load arrives (and with agents, it arrives at a fraction of the user count you expect), stand up vLLM on a CUDA server, swap two environment variables, re-run your evals, and ship. Neither decision locks you in, and the OpenAI-compatible layer both tools share means even the managed-API escape hatch stays open.

Where to go from here depends on which side of the fork you are on:

Ready to deploy? Follow the step-by-step LangGraph + vLLM production deployment guide: Docker Compose, nginx with SSL, systemd, and Grafana monitoring in about 30 minutes. Pair it with our open-source agent model roundup to pick what to serve, and the AI stack builder to sanity-check your architecture choices.

Want to go deeper on the engineering? Our Production-Grade Agent Engineering course covers the full lifecycle: agent design, evaluation, inference serving trade-offs like the ones in this post, and the operational patterns that keep agent systems alive under real traffic.

Rather have it built for you? If your team needs a production agent system, self-hosted or hybrid, without spending a quarter learning inference infrastructure, work with us. We have deployed this exact stack, both engines included, more times than we can count, and we are happy to tell you when the honest answer is "just use an API".