Prefix Caching: KV block reuse across requests

源码版本v0.25.1

Responsibilities

Many requests share a common prefix: system prompt, chat template, function definitions, few-shot examples. The K/V for these tokens has already been computed, so if the next request can reuse them directly, a whole prefill is saved. Prefix caching is exactly this: each KV block is fingerprinted by a "content hash", and when the next request arrives it first looks up the hash; on a hit, the existing block is referenced instead of recomputed.

The enable switch is CacheConfig.enable_prefix_caching, default True(cache.py:93). EngineCore.__init__ constructs request_block_hasher when enable_prefix_caching is on or there is a KV connector(core.py:210-219). It is essentially a closure returned by get_request_block_hasher(kv_cache_utils.py:673) that slices the request's token sequence by hash_block_size and hashes it in a chained fashion. The hash of each block feeds "the parent block's hash + the current block's token ids + extra_keys(LoRA / multimodal)" into the hash function(kv_cache_utils.py:577-604), so a block's fingerprint is the fingerprint of "the entire prefix from the start up to and including this block".

KVCacheManager delegates hit lookup to KVCacheCoordinator.find_longest_cache_hit(kv_cache_coordinator.py:364), which dispatches to the unitary manager. On a hit, allocate_slots reuses these blocks and only allocates new blocks for the miss portion. Scheduler.schedule calls get_computed_blocks(kv_cache_manager.py:206) before scheduling each request, reducing the token budget on a hit.

Design motivation

• Block-level instead of token-level reuse: KV cache is split by PagedAttention blocks, so naturally the block is the minimum unit and the hash is computed per block — coarse-grained and fast to look up.
• Chained hash preserves prefix semantics: hash_block_tokens(parent_hash, tokens, extra_keys)(kv_cache_utils.py:577) makes each block's fingerprint implicitly include the entire prefix, so there is no false hit of the form "different middle, same end".
• Multiple hash algorithms available: prefix_caching_hash_algo supports sha256 / sha256_cbor / xxhash / xxhash_cbor(cache.py:95-110). xxhash is faster but not cryptographically secure; be cautious in multi-tenant scenarios.
• Dedicated coordinator for the disabled path: KVCacheCoordinatorNoPrefixCache(kv_cache_coordinator.py:377) makes find_longest_cache_hit return ((), 0) directly, so when enable_caching=False no hash computation is done at all.
• Multimodal / LoRA extra_keys: generate_block_hash_extra_keys(kv_cache_utils.py:714) uses multimodal inputs and LoRA IDs as additional hash keys, preventing different images from occupying the same block slot.
• Non-causal attention layers force-disabled: EngineCore._initialize_kv_caches flips enable_prefix_caching to False when it detects a non_causal=True spec(core.py:255-269), because the bidirectional attention of a Prefix LM would corrupt prefix-hit semantics.
• Skip reading prefix cache: Request.get_skip_reading_prefix_cache(request.py:266-277) makes prompt logprobs / pooling-all requests explicitly skip the prefix, since they require a full recompute.
• Watermark to prevent thrash: KVCacheManager.watermark_blocks(kv_cache_manager.py:164-167) keeps a small number of free blocks reserved to avoid high-frequency preemption.

Key files

• enable_prefix_caching:93 — default True, whether prefix caching is enabled at startup.
• prefix_caching_hash_algo:95 — pick one of sha256 / sha256_cbor / xxhash / xxhash_cbor.
• EngineCore constructs request_block_hasher:210-219 — picks the hash function + init_none_hash + get_request_block_hasher.
• non-causal attention force-disabled:255-269 — enable_prefix_caching = False for Prefix LM / bidirectional attention.
• Request.update_block_hashes:237-240 — calls _block_hasher(self) to hash newly filled blocks.
• get_skip_reading_prefix_cache:266-277 — prompt logprobs / pooling explicitly skip hits.
• hash_block_tokens:577 — chained hashing: hash_function((parent_hash, token_ids_tuple, extra_keys)).
• get_request_block_hasher:673 — returns a closure that hashes only full blocks and chains parent hashes.
• get_computed_blocks:206 — entry point: request arrives -> look up the longest hit -> return (KVCacheBlocks, num_computed_tokens).
• allocate_slots:248 — reuses the hit portion and pulls new blocks from BlockPool for the miss portion.
• KVCacheCoordinatorNoPrefixCache:377 — coordinator stub for the disabled path.
• UnitaryKVCacheCoordinator.find_longest_cache_hit:480 — hit lookup for single-KV-group models.
• Scheduler calls get_computed_blocks:724-726 — gets the hit count during scheduling to shrink num_new_tokens.

Data flow

When a request arrives, Scheduler first calls get_computed_blocks. Internally this goes through the coordinator's find_longest_cache_hit, which walks the block_hashes list and asks BlockPool for each block whether one with the same hash already exists:

# vllm/v1/core/kv_cache_manager.py L206-L246
def get_computed_blocks(self, request: Request) -> tuple[KVCacheBlocks, int]:
    """Get the computed (cached) blocks for the request.
    Note that the computed blocks must be full.
    """
    # We skip finding the prefix cache hit when prefix caching is
    # disabled or the request is marked as skipping kv cache read
    # (which happens when the request requires prompt logprobs
    # or calls a pooling model with all pooling).
    if not self.enable_caching or request.skip_reading_prefix_cache:
        return self.empty_kv_cache_blocks, 0

    # NOTE: When all tokens hit the cache, we must recompute the last token
    # to obtain logits. Thus, set max_cache_hit_length to prompt_length - 1.
    # This can trigger recomputation of an entire block, rather than just
    # the single last token, because allocate_slots() requires
    # num_computed_tokens to be block-size aligned. Removing this limitation
    # could slightly improve performance in the future.
    max_cache_hit_length = request.num_tokens - 1
    computed_blocks, num_new_computed_tokens = (
        self.coordinator.find_longest_cache_hit(
            request.block_hashes, max_cache_hit_length
        )
    )

    if self.log_stats:
        assert self.prefix_cache_stats is not None
        self.prefix_cache_stats.record(
            num_tokens=request.num_tokens,
            num_hits=num_new_computed_tokens,
            preempted=request.num_preemptions > 0,
        )

    return self.create_kv_cache_blocks(computed_blocks), num_new_computed_tokens

Each block's hash is computed inside the request's own update_block_hashes. request_block_hasher is a closure that hashes only full blocks and chains the parent hash:

# vllm/v1/core/kv_cache_utils.py L687-L728
def request_block_hasher(request: Request) -> list[BlockHash]:
    start_token_idx = len(request.block_hashes) * hash_block_size
    num_tokens = request.num_tokens

    if start_token_idx + hash_block_size > num_tokens:
        # Early stop when there no new full blocks created.
        return []

    curr_mm_idx = 0
    if start_token_idx > 0:
        # Set curr_mm_idx = -1 to indicate the last mm input.
        # Note that since we reach to this branch only when the block is
        # completed with generated tokens, we only need to consider the
        # last mm input.
        curr_mm_idx = -1

    prev_block_hash_value = (
        request.block_hashes[-1] if request.block_hashes else None
    )
    new_block_hashes: list[BlockHash] = []
    while True:
        end_token_idx = start_token_idx + hash_block_size
        if end_token_idx > num_tokens:
            # We only hash full blocks
            break

        # MM and LoRA requests need extra keys for block-hash computation.
        extra_keys, curr_mm_idx = generate_block_hash_extra_keys(
            request, start_token_idx, end_token_idx, curr_mm_idx
        )

        # Compute the hash of the current block
        block_tokens = request.all_token_ids[start_token_idx:end_token_idx]
        block_hash = hash_block_tokens(
            caching_hash_fn, prev_block_hash_value, block_tokens, extra_keys
        )

        new_block_hashes.append(block_hash)
        start_token_idx += hash_block_size
        prev_block_hash_value = block_hash

    return new_block_hashes

After Scheduler gets the hit count, it subtracts the hit portion from num_new_tokens, and allocate_slots only takes the missed blocks from the block pool. The hit blocks have their reference count incremented, but the tokens are not recomputed.

Boundaries and failure modes

• Non-causal attention layers force-disable prefix caching: EngineCore._initialize_kv_caches flips enable_prefix_caching = False when it detects non_causal=True(core.py:265-269). Otherwise prefix hits under bidirectional attention would read incorrect KV.
• Prompt logprobs / pooling skip hits: When Request.get_skip_reading_prefix_cache returns True, get_computed_blocks returns empty directly(kv_cache_manager.py:222-223), because these scenarios must recompute the prompt KV.
• Full hit also recomputes the last token: max_cache_hit_length = request.num_tokens - 1(kv_cache_manager.py:231), otherwise there are no logits to sample.
• xxhash is not cryptographically secure: prefix_caching_hash_algo = "xxhash" is faster but has a theoretically higher collision probability; in multi-tenant scenarios it may leak prefix information(cache.py:101-108).
• Dedicated stub for the disabled path: KVCacheCoordinatorNoPrefixCache.find_longest_cache_hit returns ((), 0)(kv_cache_coordinator.py:416-424), so when enable_caching=False no hash computation is wasted.
• BlockPool LRU eviction: A reused hit block has its reference count incremented and decremented only when the request ends. Blocks with no long-term references are LRU-reclaimed for new requests(kv_cache_manager.py:508).

Summary

Prefix caching is one of vLLM's key high-throughput mechanisms: KV blocks are chained-hashed by "parent hash + token ids + extra_keys" to reuse computation results for shared prefixes across requests. EngineCore decides at init whether to construct a hasher based on enable_prefix_caching, and Scheduler.schedule calls get_computed_blocks each time to look up hits. Reused blocks are managed in /kv-cache/kv-cache-manager; for BlockPool details see /kv-cache/coordinator-blockpool; for the engine core that ties this together see /engine/engine-core.

See official docs: vLLM docs · README