Scaling inference with vLLM

The real bottleneck in serving a language model is often not compute, but how memory is managed. In a naive server, the KV cache for each request is reserved as one contiguous block sized for the maximum sequence length. Since most requests never reach that maximum, the bulk of expensive GPU memory sits idle 'just in case.'

vLLM's fix is borrowed from operating systems: virtual memory and paging. The KV cache is split into small fixed-size pages instead of one huge contiguous block. Each sequence sees logical pages that may sit scattered across physical memory. Fragmentation drops to near zero.

In practice this means many more concurrent requests on the same GPU. Memory is spent on what's actually used, not on the worst case, and that small accounting change decides the capacity of the whole service.

Paging brings a bonus: requests that start the same way (say, all sharing one system prompt) physically share those pages. With prefix caching the same tokens aren't recomputed over and over; both memory and compute are paid once.

Picture a chat app: every request is preceded by the same long system instruction. Without sharing, it's reprocessed each time. In vLLM it's computed once, then everyone looks at the same pages. Below, the navy cells are that shared prefix, and each request's own-coloured cells are its unique continuation. Tap a legend item to highlight the shared prefix or a single request.

An inference request splits into two characters. In prefill the whole prompt is fed to the model at once; all tokens are processed in parallel (fast), and it saturates the GPU. In decode the answer is produced one token at a time: each token depends on the last, sequential and relatively slow.

The split matters because the two phases have different appetites: one hungers for compute, the other for memory. That's exactly why systems like llm-d disaggregate them onto separate machines: give each job the hardware it loves.

The second big idea is in scheduling. A fixed batch waits for its slowest request to finish; short answers complete early but the cores spin idle. vLLM instead rebuilds the batch at every decode step: a finished request leaves, a queued one takes its slot immediately.

This is called continuous batching, and its effect is striking: the GPU never waits for 'the batch to drain,' utilization stays high, and latency shrinks for everyone in the queue. The two diagrams below show it: first the gaps of a fixed batch, then how a single decode step is rebuilt.

One of the most elegant speed tricks at the core of vLLM is speculative decoding. Token generation is inherently sequential and slow: each token waits for the previous one. Speculative decoding lets a small, fast 'draft' model guess several tokens ahead; the large 'target' model verifies all of them in a single pass.

The correct prefix is accepted as-is and cut at the first miss. When the draft guesses well you produce several tokens in one pass; when it guesses badly you lose nothing, because verification guarantees quality. The result: same output, fewer rounds. The slide on stage summed it up in three steps.

The second half of that session was Multi-LoRA serving. Instead of hosting a separate giant model per specialty, small LoRA adapters are attached on top of a single base model. vLLM can serve them at once: same GPU, same base weights, but each request is answered with its own adapter.

Cost of one model, flexibility of dozens. One request calls the 'code' adapter while another calls 'SQL', yet the base weights stay as a single copy in memory. It's the most practical way to keep dozens of specialized models alive on one GPU.

On the NVIDIA side, Mireille Fares's session focused on lightening the model itself. Quantization represents weights with fewer bits: from FP16 to FP8, even INT4. Fewer bits means less memory and faster memory access.

The trick is keeping quality: smart quantization schemes shrink a model to half or a quarter with almost no loss of accuracy. A smaller model fits on more GPUs and answers faster, and combined with speculative decoding and PagedAttention, the gains compound.

Edoardo Vacchi stepped beyond a single GPU into the cluster. llm-d scales vLLM on Kubernetes: incoming requests pass through a KV-cache-aware router, prefill and decode workloads can be split onto separate nodes, and the cache is shared across replicas.

A model is no longer a process, it's a service topology. The router knows which request is close to which cache; so requests sharing a prefix land on the same replica and recomputation is avoided. Every trick of the single machine, now spread across the cluster.

Goin's close showed where the ecosystem is heading: vLLM isn't just a fast engine, it's an evolving platform. The roadmap on stage pointed to five stops.

A direction running from the core engine to raw performance, to distributed serving, to multimodal models and online reinforcement learning. In other words, everything we discussed today is just the first few stops of a much larger story.

Tufan Küpeli (BeyondGuard) took the less-discussed but critical side: the moment you expose an LLM to production, you also expose an attack surface. As much as speed and capacity, what goes in and what comes out is an engineering problem too.

The talk gathered around three layers, and all three are the responsibility of the system, not the model.