vLLM architecture overview

源码版本v0.25.1

In one sentence

vLLM is a high-throughput LLM inference engine. v0.25 has switched to the v1 architecture: LLMEngine is a thin shell; the real core is EngineCore (which can run in a ZMQ background process, EngineCoreProc), exposed via EngineCoreClient (Inproc/MP/AsyncMP). At runtime, Scheduler maintains waiting/running queues and does continuous batching and preemption; GPUModelRunner.execute_model runs the forward pass on Workers; KVCacheManager + BlockPool implement PagedAttention + prefix caching; AttentionBackend abstracts attention across multiple backends.

Layering

One sentence per layer

• Startup and configuration: EngineArgs parses all startup parameters and derives VllmConfig; LLM is the offline inference class; AsyncLLM is the async entry point.
• Engine mainline v1: LLMEngine is a thin shell; EngineCore is the real core (composing scheduler + worker); EngineCoreProc wraps it as a ZMQ background process.
• EngineCoreClient: InprocClient (same process) / MPClient (multi-process) / AsyncMPClient — uniformly exposes add_request/abort and other interfaces to upper layers.
• Scheduler: Scheduler maintains waiting/running queues; schedule decides which requests to run this step; supports preemption and prefill throttle.
• Executor: Executor abstracts execution backends — UniProc/Multiproc/Ray — responsible for spinning up Workers and the distributed environment.
• Worker and ModelRunner: Worker holds the model and KV cache; GPUModelRunner.execute_model runs one forward pass; ubatch handles micro-batching; cudagraph accelerates execution.
• Model loading: DefaultModelLoader loads weights; TP linear layers (ColumnParallel/QKVParallel/RowParallel) do the sharding; quant layers handle quantization.
• KV Cache: KVCacheManager manages block lifecycle; KVCacheCoordinator (Hybrid/Unitary/NoPrefix) sets the strategy; BlockPool handles allocation + prefix cache index.
• Attention backends: AttentionBackend abstraction; FlashAttention/FlashInfer/Triton/MLA/Mamba are swappable.
• Sampling: Sampler forward/sample turns logits into tokens; LogitsProcessor does preprocessing.
• Prefix cache and distributed: prefix cache hits reuse KV blocks; GroupCoordinator manages TP/PP communication; OpenAI serving exposes the HTTP API.

Why this layering

vLLM fully decouples "how to configure" (engine args), "how to schedule" (scheduler), "how to execute" (executor + worker), "how to manage GPU memory" (KV cache), "how to compute attention" (attention backend), "how to communicate" (distributed), and "how to expose" (serving). This way swapping attention backends does not affect scheduling; swapping execution backends (single-GPU/Ray) does not affect KV cache logic; adding prefix cache does not require touching model code. EngineCore is the single hub — all requests flow through it.

Common misreadings

• "vLLM's core is PagedAttention" — PagedAttention is the key technique behind KV cache management, but the core of the v1 architecture is the trio of EngineCore + Scheduler + continuous batching. PagedAttention is only the implementation of the KV cache layer.
• "LLMEngine is the engine" — In v1, LLMEngine is a thin shell; the real work happens in EngineCore. LLMEngine mainly does parameter validation and request forwarding.
• "Continuous batching and PagedAttention are the same thing" — They are not. Continuous batching is a scheduling policy (requests can be added/removed each step); PagedAttention is memory management (block-based KV cache). The two complement each other but are independent.

Recommended reading order

Start with Startup and configuration, then LLMEngine and EngineCore, then proceed in order: Scheduler → Executor → Worker and ModelRunner → KV Cache → Attention → Sampling → Distributed and serving.

See official docs: vLLM docs · design docs · GitHub