vLLM Tutorial: Serve LLMs with PagedAttention and an OpenAI API

Chappan Express's warehouse once stored every order on its own reserved shelf, sized for the largest order anyone could ever place. Most orders were small, so most shelves stood half empty, and on busy days new orders waited even though the floor was full of free space.

The fix was numbered trays. Each order gets trays one at a time as it grows, and a register records which trays belong to which order. No space is reserved in advance, and nothing is wasted between orders.

vLLM does the same with GPU memory. The trays are KV cache blocks and the register is a block table.

What is vLLM?

vLLM is an open-source library and server for fast LLM inference that manages the KV cache in fixed-size pages (PagedAttention), schedules requests with continuous batching, and serves models through an OpenAI-compatible API.

It began as a research project at UC Berkeley and is now a widely used community project. It supports most popular open-weight model families, multiple GPU types, quantised weights and multi-GPU serving.

For Chappan Express, vLLM is how the self-hosted model behind Chappan Help reaches the app. The app keeps calling the alias support-chat-v1; vLLM answers at that name. This lesson applies the ideas from LLM inference optimisation, and the commands follow the vLLM documentation as of November 2026.

How does vLLM work?

Every decode step, the scheduler decides which requests run, and the cache manager gives each one just enough blocks.

1. API server. Accepts /v1/chat/completions and /v1/completions requests in the OpenAI format, applies the model's chat template and tokenises.
2. Scheduler. Keeps waiting and running queues. After every step it admits new requests if there is cache space, which is continuous batching.
3. KV cache manager. Splits the reserved GPU memory into blocks of a fixed number of tokens and assigns blocks to requests as they grow.
4. Model runner. Executes one batched forward pass for all running requests and returns one new token for each.
5. Detokeniser and streamer. Turns tokens back into text and streams them to clients.

What is PagedAttention?

Before paging, servers reserved a contiguous chunk of memory per request, big enough for the maximum context length. If a chat was allowed 8,192 tokens but used 900, the rest sat idle. PagedAttention stores each request's keys and values in small blocks (16 tokens is common) that can live anywhere in memory, with a block table mapping positions to blocks.

Three benefits follow. Waste is limited to the last partly filled block. Many more requests fit at once. And identical prefixes, like Chappan Help's system prompt, can point at the same blocks, which is how prefix caching shares memory.

Same three chats, same GPU. Paging turns reserved-but-empty space into room for more users.

import math
import numpy as np

rng = np.random.default_rng(4)
max_len, block = 8192, 16
kv_per_token = 128 * 1024                        # bytes, 8B-class model with GQA
budget = 50 * 1024 ** 3                          # 50 GiB set aside for KV cache
used = rng.integers(300, 3000, 1000)             # real tokens per chat

contig_per_req = max_len * kv_per_token          # reserve the maximum for everyone
paged_per_req = np.array([math.ceil(u / block) * block for u in used]) * kv_per_token

fit_contig = budget // contig_per_req
fit_paged = int(budget // paged_per_req.mean())
waste_contig = 1 - used.mean() / max_len
waste_paged = 1 - used.sum() * kv_per_token / paged_per_req.sum()
print(f"contiguous: {fit_contig:4d} chats fit, {waste_contig:.0%} of reserved memory unused")
print(f"paged     : {fit_paged:4d} chats fit, {waste_paged:.1%} of allocated memory unused")

Reserving the maximum fits only 50 chats and wastes most of the memory. Paging fits several times more, wasting under one per cent. That is the main reason vLLM delivers high throughput.

Install and start a vLLM server

vLLM runs on Linux with a supported GPU. Install it in a fresh virtual environment, then serve a model from the Hugging Face Hub. Replace <model-id> with the open-weight model you chose, as in running LLMs locally.

uv venv --python 3.12 && source .venv/bin/activate
uv pip install vllm --torch-backend=auto

vllm serve <model-id> \
  --served-model-name support-chat-v1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --enable-prefix-caching \
  --api-key "$VLLM_API_KEY"

The server listens on port 8000 by default. Because the API matches OpenAI's, the app can use the standard OpenAI Python client with a different base URL.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_VLLM_API_KEY")

stream = client.chat.completions.create(
    model="support-chat-v1",                    # the --served-model-name alias
    messages=[
        {"role": "system", "content": "You are Chappan Help. Refunds for missing items within 48 hours."},
        {"role": "user", "content": "My raita was missing yesterday. Can I get a refund?"},
    ],
    temperature=0.2,
    max_tokens=200,
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Sampling settings such as temperature and max tokens work as described in generation parameters.

The vLLM flags that matter most

Plan capacity before launch

How many GPUs does Chappan Help need at dinner peak? Combine the KV budget with Little's law: concurrent requests = arrival rate × average time each request is active.

import math

gpu_gib, util, weights_gib, overhead_gib = 80, 0.90, 15, 4
kv_gib_per_token = 128 / 1024 / 1024             # 128 KiB per token
avg_tokens = 2600                                 # prompt + reply, from logs

kv_budget = gpu_gib * util - weights_gib - overhead_gib
seqs_by_memory = int(kv_budget / (avg_tokens * kv_gib_per_token))
seqs_per_gpu = min(seqs_by_memory, 64)            # also capped by --max-num-seqs

peak_chats_per_s = 9                              # dinner peak arrivals
avg_active_s = 5.5                                # TTFT + streaming time per turn
concurrent = peak_chats_per_s * avg_active_s
gpus = math.ceil(concurrent / seqs_per_gpu * 1.3) # 30% headroom

print(f"KV budget per GPU     : {kv_budget:.1f} GiB -> {seqs_by_memory} sequences by memory")
print(f"Sequences per GPU used: {seqs_per_gpu}")
print(f"Concurrent at peak    : {concurrent:.0f}")
print(f"GPUs needed with headroom: {gpus}")

This is a starting estimate, not a promise. Latency targets may cap batch size below what memory allows, so confirm it with a load test.

Run vLLM in Docker and load-test it

In production you run the official container, the same pattern as in Docker for machine learning. The image tag should be pinned to a tested version rather than latest.

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:<pinned-version> \
  --model <model-id> --served-model-name support-chat-v1 --api-key "$VLLM_API_KEY"

# replay realistic traffic: long prompts, short replies, 9 requests per second
vllm bench serve --base-url http://localhost:8000 --model <model-id> \
  --dataset-name random --random-input-len 2400 --random-output-len 150 \
  --num-prompts 1000 --request-rate 9 --metric-percentiles 50,95,99

The benchmark reports TTFT, time per output token and inter-token latency at the percentiles you request. Raise the request rate until p95 TTFT crosses your target; the rate just below that is your real capacity per GPU. For scaling several replicas, see Kubernetes for ML engineers.

Batch jobs without a server

For nightly work, such as summarising the day's 6,000 support chats for the quality team, skip HTTP and use the offline API, which batches everything for maximum throughput.

from vllm import LLM, SamplingParams

llm = LLM(model="<model-id>", max_model_len=8192)
params = SamplingParams(temperature=0.0, max_tokens=120)
prompts = [f"Summarise this support chat in one line:\n{chat}" for chat in todays_chats]
for out in llm.generate(prompts, params):
    print(out.outputs[0].text)

Monitor vLLM in production

vLLM exposes Prometheus metrics at /metrics. The ones to put on a dashboard first:

• vllm:time_to_first_token_seconds and vllm:inter_token_latency_seconds (histograms) for user-facing speed.
• vllm:e2e_request_latency_seconds for whole-request time.
• vllm:num_requests_running and vllm:num_requests_waiting for load and queueing.
• vllm:kv_cache_usage_perc for how full the block pool is.
• vllm:prefix_cache_hits divided by vllm:prefix_cache_queries for how well prefix caching works.

A rising waiting queue while cache usage sits near 1.0 means you need more GPUs or shorter contexts. Metric names occasionally change between releases, so check the metrics page for your version. Request-level tracing is covered next, in LLM observability.

Common vLLM mistakes

• Leaving --max-model-len at the model's huge maximum. It shrinks how many sequences the scheduler admits. Set what you need.
• Running without --api-key on a public address. Anyone can use your GPU. Keep vLLM behind a gateway or private network too.
• Using latest image tags. Upgrades can change defaults and metric names. Pin and test.
• Changing the system prompt per user at the start. Prefix caching needs an identical prefix; put personal details after the shared part.
• Skipping evaluation after switching engines or quantisation. Rerun the golden set; chat templates and precision can change outputs.

Key takeaways

• vLLM is an open-source LLM serving engine with an OpenAI-compatible API.
• PagedAttention allocates the KV cache in blocks, fitting many more concurrent requests; continuous batching keeps the GPU busy.
• Start with vllm serve, then tune memory share, context length, concurrent sequences and an API key.
• Plan with KV maths and Little's law, then confirm with vllm bench serve at realistic traffic.
• Watch TTFT, waiting requests and KV cache usage in production.