AI · Nov 18, 2026 · 12 min read

High-Throughput LLM Serving with vLLM: PagedAttention and LoRA

Practical guide to vLLM 0.5.4 featuring PagedAttention, AWQ/GPTQ quantization, dynamic LoRA serving, FastAPI integration, Prometheus metrics, and Kubernetes deployment with exact commands and configur

High-Throughput LLM Serving with vLLM: PagedAttention and LoRA

Understanding PagedAttention and Its Impact on KV Cache Management

vLLM solves the memory fragmentation problem that plagues traditional LLM inference. In standard attention implementations, the key-value cache for each request is allocated as one contiguous block. When a request finishes, that block is freed, leaving holes in GPU memory that cannot be reused by new requests of different lengths. This leads to low utilization even when total memory appears available.

PagedAttention treats the KV cache like virtual memory in an operating system. The cache is split into fixed-size blocks (default block size is 16 tokens). A block table maps logical KV positions to physical blocks. When a sequence grows, vLLM allocates additional blocks from a free list instead of requiring one large contiguous allocation. This eliminates fragmentation and allows the engine to maintain a much higher number of concurrent sequences.

On an A100 40GB with Llama-3-8B, continuous batching combined with PagedAttention typically supports 4-6× more concurrent users than Hugging Face Transformers with the same memory budget. The exact gain depends on average response length and prompt length. For 512-token prompts and 256-token generations, you can comfortably run 128 concurrent sequences where a naive implementation would struggle beyond 32.

The main tunable parameters are block_size (16 is optimal for most models), gpu_memory_utilization (0.9 is a safe production default), and swap_space for CPU offloading when running very large batches. Setting gpu_memory_utilization to 0.95 can squeeze extra sequences but risks CUDA OOM during peak load if generation lengths spike.

Installing vLLM 0.5.4 with CUDA 12.1

Begin with a clean Python 3.11 environment. vLLM 0.5.4 requires CUDA 12.1 or 11.8. The following commands install the library with AWQ and GPTQ support:

pip install vllm==0.5.4
pip install "vllm[awq,gptq]"==0.5.4

For ROCm users targeting AMD Instinct MI250, use the ROCm 6.0 build instead. Installation takes 3-7 minutes depending on internet speed because it compiles custom CUDA kernels for FlashAttention-2 and PagedAttention.

Verify the installation with a quick test that loads a tiny model and runs one forward pass. This confirms both the Python bindings and GPU kernels are working before moving to larger models.

from vllm import LLM, SamplingParams
llm = LLM(model="facebook/opt-125m")
sampling_params = SamplingParams(temperature=0.7, max_tokens=128)
output = llm.generate("The capital of France is", sampling_params)
print(output[0].outputs[0].text)

Launching the OpenAI-Compatible Server

vLLM ships with a production-ready OpenAI-compatible endpoint. This is the fastest way to get started. The command below launches Llama-3-8B with 4-bit GPTQ quantization, tensor parallelism of 2, and an OpenAI-compatible API on port 8000:

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Llama-3-8B-GPTQ \
  --quantization gptq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 8192 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 128 \
  --port 8000

The --max-num-batched-tokens value should be roughly 4× the product of expected concurrent requests and average prompt length. Setting it too high wastes memory on padding; too low limits throughput. For production traffic with 70 % cache hit rate on prefixes, 16384 is a good starting point on two A100s.

The server exposes /v1/completions, /v1/chat/completions, and /v1/models. It also supports the /v1/embeddings endpoint when using embedding models. Request logging is disabled by default; enable it with --log-requests only during debugging because it adds measurable overhead at 500+ RPM.

Continuous Batching Explained with Code

Continuous batching (also called dynamic batching) is the second major innovation in vLLM. Instead of waiting for an entire batch to finish before starting the next, vLLM adds new requests to the current running batch every step. It also swaps out finished sequences immediately.

This removes the "straggler" problem where one long generation holds up many short ones. The scheduler runs every token step and decides which sequences to continue, which to pause, and which new ones to add. The Python code to use the engine directly demonstrates this behavior:

from vllm import AsyncLLMEngine
from vllm.engine.arg_utils import AsyncEngineArgs

engine_args = AsyncEngineArgs(
    model="meta-llama/Meta-Llama-3-8B",
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,
    max_num_batched_tokens=4096,
    max_num_seqs=64
)

engine = AsyncLLMEngine.from_engine_args(engine_args)

async def generate(prompt: str):
    results_generator = engine.generate(
        prompt,
        SamplingParams(temperature=0.8, max_tokens=512),
        request_id="req-001"
    )
    async for request_output in results_generator:
        yield request_output.outputs[0].text

Notice that engine.generate returns an async generator. You can have hundreds of these generators running concurrently. The engine internally manages the continuous batch scheduler. In benchmarks with mixed 128- and 2048-token generations, continuous batching improves throughput by 2.3× compared to static batching.

Quantization Options: AWQ, GPTQ, FP8 and Their Trade-offs

vLLM 0.5.4 supports multiple 4-bit and 8-bit quantization formats. GPTQ and AWQ both reduce a 16-bit Llama-3-70B from 140 GB to roughly 35 GB, enabling it to run on a single A100 40GB with room for KV cache.

AWQ generally preserves accuracy better on instruction-tuned models because it searches for optimal scaling factors per channel. GPTQ is faster to quantize. For both, perplexity degradation on WikiText is usually under 0.3 points compared to FP16.

FP8 is newer and supported on Hopper GPUs (H100). It offers higher throughput than INT4 because it avoids dequantization overhead during GEMM. However, it requires careful activation scaling. The command to launch with AWQ is:

--model casperhansen/llama-3-70b-awq \
--quantization awq \
--max-model-len 4096

Memory savings are concrete: a 70B AWQ model uses 38 GB for weights. With gpu_memory_utilization=0.85 you still have 6 GB left for KV cache on an A100 40GB after CUDA context overhead. This translates to roughly 1100 tokens of KV cache at FP16 precision.

Serving Multiple LoRA Adapters Dynamically

One of vLLM's strongest production features is native LoRA serving. You can load many adapters (dozens) and switch between them per request without reloading the base model. This is critical for SaaS products that offer domain-specific fine-tunes.

First convert your Hugging Face PEFT adapters to vLLM format using the built-in script, then launch the server with the adapter list:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B \
  --enable-lora \
  --lora-modules "customer1=/path/to/adapter1" "support=/path/to/support-adapter" \
  --max-lora-rank 64 \
  --max-num-seqs 64

In the request, specify the adapter with the lora_name field in the extra body. The engine keeps a small set of active adapters in GPU memory and pages others in from CPU. With 4 adapters of rank 32 on Llama-3-8B, the additional memory overhead is approximately 800 MB per active adapter. Keep the number of simultaneously loaded adapters under 8 on A100 to avoid memory pressure.

Switching latency between adapters is under 30 ms when they are already paged in. Cold adapter loading takes 300-800 ms depending on rank and disk speed. For latency-sensitive applications, use a small set of hot adapters and accept the cold-start penalty for rare ones.

Building a FastAPI Wrapper with Custom Business Logic

Many teams wrap the vLLM OpenAI server behind FastAPI to add authentication, usage quotas, prompt templating, and post-processing. Here is a production-ready skeleton using vLLM's Python engine directly so you can add middleware:

from fastapi import FastAPI, Request
from vllm import AsyncLLMEngine, SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
import time

app = FastAPI()
engine_args = AsyncEngineArgs(model="TheBloke/Llama-3-8B-GPTQ", quantization="gptq", gpu_memory_utilization=0.88)
engine = AsyncLLMEngine.from_engine_args(engine_args)

@app.post("/generate")
async def generate(request: Request):
    data = await request.json()
    prompt = data["prompt"]
    start = time.perf_counter()
    
    sampling_params = SamplingParams(
        temperature=data.get("temperature", 0.7),
        max_tokens=data.get("max_tokens", 512),
        stop=data.get("stop")
    )
    
    request_id = f"req-{int(time.time()*1000)}"
    async for output in engine.generate(prompt, sampling_params, request_id):
        if output.finished:
            latency = time.perf_counter() - start
            return {
                "text": output.outputs[0].text,
                "tokens": output.outputs[0].token_ids,
                "latency_ms": round(latency * 1000, 2)
            }

This pattern lets you integrate with existing auth (FastAPI Users, Auth.js, etc.), add Pydantic validation, emit custom Prometheus metrics per customer, and apply output guards before returning to the client. The overhead of the wrapper is typically 2-4 ms compared to hitting the vLLM server directly.

Monitoring with Prometheus: Key Metrics and Grafana Dashboards

vLLM exposes Prometheus metrics at /metrics when launched with the OpenAI server. Critical metrics to alert on are:

  • vllm:num_requests_waiting — should stay under 8 in healthy systems
  • vllm:gpu_cache_usage_perc — target 75-85 %
  • vllm:avg_prompt_throughput_tokens_per_sec
  • vllm:avg_generation_throughput_tokens_per_sec

Configure Prometheus scrape interval to 15 seconds. A useful Grafana panel calculates effective tokens per dollar by combining throughput with instance cost. For an A100 on-demand at $2.50/hour, you want at least 1800 output tokens per second across the cluster to achieve acceptable unit economics.

Add custom middleware in the FastAPI wrapper to emit vllm:requests_total{customer="acme"} and vllm:request_duration_seconds histograms. These let you implement per-customer rate limiting and cost allocation.

Kubernetes Deployment with Helm and Horizontal Pod Autoscaler

vLLM provides an official Helm chart. Create a values.yaml with these production settings:

replicaCount: 2
image:
  repository: vllm/vllm-openai
  tag: "0.5.4"
resources:
  limits:
    nvidia.com/gpu: 2
podAnnotations:
  prometheus.io/scrape: "true"
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 8
  targetCPUUtilizationPercentage: 65
  targetMemoryUtilizationPercentage: 80

The HPA should target both CPU and custom metrics (queue length from Prometheus). Because each pod requires 2 GPUs, use the NVIDIA GPU operator with time-slicing disabled. For multi-tenancy, run separate deployments per model size rather than trying to share GPUs across unrelated workloads.

Startup time for a 70B model is 45-90 seconds. Configure Kubernetes readiness probes with httpGet on /health (available in vLLM 0.5.4+) and initial delay of 60 seconds. Liveness probes are not recommended because killing a busy inference pod drops in-flight requests.

Trade-offs: Latency, Throughput, Cost and Operational Complexity

vLLM maximizes throughput at the expense of slightly higher tail latency compared to single-request synchronous servers. Median time to first token is usually 40-80 ms higher because of batching. The 95th percentile latency can be tuned by lowering max_num_seqs at the cost of overall tokens per second.

Compared with Hugging Face Text Generation Inference (TGI), vLLM typically delivers 1.7-2.4× more output tokens per second on the same hardware for Llama models. TGI offers easier integration with some enterprise features like better token streaming control and built-in watermarking. vLLM's advantage grows as batch size increases.

Memory is the primary constraint. A rule of thumb is that KV cache memory in bytes equals 2 × num_layers × num_heads × head_dim × block_size × num_blocks × 2 (for K and V). With block size 16 on Llama-3-8B (32 layers, 32 heads, 128 dim), each block consumes about 0.5 MB. Plan capacity by dividing available KV memory by average tokens per request.

Operational complexity increases because you must monitor both GPU cache usage and request queue depth. Under-provisioning causes requests to queue; over-provisioning wastes expensive GPUs. The sweet spot for most workloads is 80 % GPU memory utilization and average queue length of 4-6.

Implementing Structured Output with vLLM and Outlines

vLLM integrates with the Outlines library for regex and JSON constrained generation. This is more reliable than asking the model to output JSON. Install outlines==0.0.46 and use it with the vLLM engine:

from outlines import models, generate
from pydantic import BaseModel

class Extraction(BaseModel):
    company: str
    amount: float
    date: str

model = models.vllm(engine)
generator = generate.json(model, Extraction)

result = generator("Extract from this invoice: Acme Corp paid $1250 on 2024-10-12")
print(result)

The integration works by biasing the logits at each step to only valid tokens according to the JSON schema. This approach achieves greater than 99 % valid JSON rate versus 60-75 % with simple prompting. The performance cost is 8-15 % reduction in tokens per second because of the extra logit mask computation.

Common Pitfalls and How to Resolve Them

The most frequent issue is CUDA OOM during high load. Reduce gpu_memory_utilization by 0.05 increments until stable. Another common error is setting max-model-len higher than the model actually supports after quantization, which causes immediate crashes on first request.

Prefix caching is enabled by default in vLLM 0.5.4 but only works when using the same sampling_params. Different temperatures or stop sequences break the cache. For maximum cache hit rate in chat applications, standardize the system prompt and sampling parameters across all users of the same adapter.

When using LoRA, make sure all adapters were trained with the exact same base model and tokenizer. Even small tokenizer differences cause silent quality degradation. Always validate new adapters with a small test suite before promoting to production.

Network bandwidth becomes the bottleneck when scaling to many replicas. A single H100 generating at 180 tokens/s with 70B model can push 3-4 Gbps of output data. Plan your cluster networking accordingly and consider using FP8 to reduce payload size.

When to Choose vLLM Versus Alternatives

Choose vLLM when your primary goal is maximum tokens per GPU per dollar and your traffic has variable sequence lengths. It is the current leader for high-throughput chat, RAG, and agent workloads. Use Hugging Face TGI when you need deeper integration with the Hugging Face ecosystem, easier model hub deployment, or specific quantization formats not yet supported in vLLM.

For latency-critical applications under 10 ms time-to-first-token, consider vLLM with speculative decoding (enabled with --speculative-model and a small draft model). This can cut latency by 30-40 % on supported hardware.

The combination of PagedAttention, continuous batching, and native LoRA support makes vLLM the most practical choice for most production LLM services in 2024-2025. Start with the OpenAI-compatible server, add a thin FastAPI wrapper for business logic, instrument with Prometheus, and scale horizontally with Kubernetes. The patterns shown above have been validated across multiple production deployments running at thousands of requests per minute.

Keep reading

Related Articles