KVCacheManager: request-level KV cache lifecycle

源码版本v0.25.1

Responsibilities

KVCacheManager is the KV cache interface that the scheduler sees. It wraps actions like "allocate N blocks for this request", "this request finished, return its blocks", "see how much prefix cache this prompt hit" into three high-level APIs — allocate_slots / free / get_computed_blocks. Internally, it delegates coordination across different KV cache types (full attention / sliding window / Mamba state, etc.) to KVCacheCoordinator, and physical block pool management to BlockPool. The scheduler just takes the immutable KVCacheBlocks result (kv_cache_manager.py:29-50) and doesn't need to know whether the model underneath is single-type or hybrid.

At construction time, KVCacheManager.__init__ sets up the coordinator via get_kv_cache_coordinator and attaches block_pool / num_kv_cache_groups / kv_cache_config to itself (kv_cache_manager.py:147-183). The watermark (watermark_blocks = int(watermark * num_blocks)) is a margin reserved for already-scheduled requests; it only applies to WAITING / PREEMPTED status, so a new incoming request can't evict a half-finished running one (kv_cache_manager.py:164-167)(kv_cache_manager.py:367-374). The manager itself holds almost no per-request state; all req_to_blocks live inside SingleTypeKVCacheManager.

Design motivation

Why does the manager look so thin, with all the real logic in the coordinator?

• Layered responsibilities: KVCacheManager owns the request-facing semantics (allocate_slots / free / get_computed_blocks); KVCacheCoordinator owns the KV-cache-group-facing coordination (how multiple groups allocate together in a hybrid model); BlockPool owns the physical-block-facing pooling. Three layers, each minding its own business; the manager is the glue.
• KVCacheBlocks is the stable interface: blocks[i][j] means the j-th block of the i-th KV cache group; get_block_ids() converts to tuple[list[int], ...] for the attention backend (kv_cache_manager.py:73-88). __add__ lets two allocate_slots results be concatenated (<SrcLink path="vllm/v1/core/kv_cache_manager.py" lines="52-59" label="kv_cache_manager.py"/>), and new_empty returns a placeholder for "no blocks".
• Deferred cache commit: The delay_cache_blocks=True branch of allocate_slots skips cache_blocks during P/D async transfer (kv_cache_manager.py:448-451); hashing happens only after remote KV has actually arrived, avoiding premature caching that would cause false hits.
• Watermark only applies to waiting requests: Watermark is added only when has_scheduled_reqs and request.status in (WAITING, PREEMPTED); incremental allocation for already-running requests doesn't need reservation (kv_cache_manager.py:367-374).
• Full-sequence admission check: When full_sequence_must_fit=True, the manager first estimates how many blocks the entire prompt needs; if it exceeds capacity, it returns None directly, preventing chunked prefill from letting a request in and then deadlocking (kv_cache_manager.py:376-391).
• reserved_blocks for async KV connectors: In-flight prefill sequences depend on certain blocks; a newly arriving async connector load cannot grab these. When required_blocks > available_blocks, allocation is rejected (kv_cache_manager.py:420-426).

Key files

• KVCacheBlocks dataclass:29-50 — immutable result on the scheduler side; blocks[i][j] = the j-th block of the i-th group.
• KVCacheBlocks.__add__:52-59 — concatenates two allocation results; commonly used for incremental allocation.
• KVCacheBlocks.get_block_ids:73-88 — converts to tuple[list[int], ...] for the attention backend.
• KVCacheManager.__init__:114-183 — sets up the coordinator, computes watermark_blocks, builds an empty_kv_cache_blocks cache to avoid GC overhead.
• get_computed_blocks:206-246 — calls coordinator.find_longest_cache_hit for prefix hits, returns (KVCacheBlocks, num_new_computed_tokens).
• allocate_slots:248-343 — three-stage: release skipped blocks → handle prefix tokens → allocate blocks for new tokens; returns None on failure.
• full_sequence_must_fit:376-391 — admission check: prompt block count + watermark must be ≤ free block count.
• required_blocks gate:410-426 — available = free - reserved; if required > available, allocation is rejected.
• allocate_new + cache_blocks:428-464 — delegates new-block allocation to the coordinator + calls cache_blocks to commit hashes.
• free:466-474 — calls coordinator.free(request_id); releases in reverse order so the tail block is evicted first.
• pop_blocks_for_free:495-506 — takes the per-request bookkeeping but doesn't return to BlockPool; used for P/D transfer.
• reset_prefix_cache:516-530 — invalidates the entire pool cache after RLHF changes weights.

Data flow

For each request at each step, the scheduler first calls get_computed_blocks to check prefix hits, then allocate_slots to carve out blocks for new tokens. Below is the part of allocate_slots that actually does the work, after watermark + admission checks:

if (
    new_computed_block_list is not self.empty_kv_cache_blocks.blocks
    or num_external_computed_tokens > 0
):
    # Append the new computed blocks to the request blocks until now to
    # avoid the case where the new blocks cannot be allocated.
    self.coordinator.allocate_new_computed_blocks(
        request_id=request.request_id,
        new_computed_blocks=new_computed_block_list,
        num_local_computed_tokens=num_local_computed_tokens,
        num_external_computed_tokens=num_external_computed_tokens,
    )

new_blocks = self.coordinator.allocate_new_blocks(
    request.request_id,
    num_tokens_need_slot,
    num_tokens_main_model,
    num_encoder_tokens,
)

# P/D: delay caching blocks if we have to recv from
# remote. Update state for locally cached blocks.
if not self.enable_caching or delay_cache_blocks:
    return self.create_kv_cache_blocks(new_blocks)

(kv_cache_manager.py:428-451) It first absorbs the prefix-hit blocks (incrementing their ref_cnt), then has the coordinator allocate physical blocks for the new tokens, and unless caching is deferred, calls cache_blocks to hash the just-filled blocks into the prefix cache (see Coordinator and BlockPool). The returned KVCacheBlocks is converted to a block_table via get_block_ids and stuffed into the block table for the attention kernel to use.

Boundaries and failure modes

• num_new_tokens == 0 and no external tokens: raises ValueError; the caller must ensure either there are new tokens to compute or external computed tokens to store (kv_cache_manager.py:346-350).
• get_computed_blocks is off by one on full hit: max_cache_hit_length = request.num_tokens - 1, because on a full hit the last token still has to be recomputed for logits, so the cache-hit length caps at prompt_length - 1 (kv_cache_manager.py:225-231).
• Requests that skip prefix cache: When request.skip_reading_prefix_cache=True or enable_caching=False, get_computed_blocks returns empty directly (kv_cache_manager.py:222-223), e.g. requests that need prompt logprobs.
• reset_prefix_cache refused: When num_used_blocks != 1 (only the null block is left), some blocks haven't been released yet, so the reset fails and returns False (kv_cache_manager.py:516-530).
• pop_blocks_for_free doesn't return to the pool: The caller must call block_pool.free_blocks in reverse order itself, otherwise blocks leak (kv_cache_manager.py:495-506).
• take_events annotates group metadata: A BlockStored event only carries group_idx; the manager uses kv_cache_event_metadata to fill in kind and sliding_window (kv_cache_manager.py:566-589).

Summary

KVCacheManager is the KV cache facade the scheduler sees; the actual workers — KVCacheCoordinator + BlockPool — are dissected in Coordinator and BlockPool. Allocated block ids ultimately land in the block table, where the attention kernel uses them to locate physical KV cache; the model weights themselves are loaded along the DefaultModelLoader line, decoupled from this layer.

See official docs: vLLM docs · README