KVCacheCoordinator + BlockPool: coordination and block pool for hybrid models

源码版本v0.25.1

Responsibilities

KVCacheCoordinator is a vLLM v1 coordinator layer sitting on top of multiple KV cache types (full attention / sliding window / Mamba state / MLA, etc.). Each KV cache group is paired with a SingleTypeKVCacheManager (e.g. FullAttentionManager / SlidingWindowManager / RSWAManager). The coordinator splits actions like "allocate N tokens' worth of blocks", "find the longest prefix hit", "release a request", "cache a filled block" by group and dispatches them to the corresponding single-type manager. BlockPool is the shared physical block pool: all groups share the same set of KVCacheBlock instances — only the hash key carries a group_id to distinguish them, so the same block can be cached by multiple groups at once (e.g. the same token in both full attention and sliding window).

At construction time, KVCacheCoordinator.__init__ directly creates a BlockPool, then get_manager_for_kv_cache_spec spins up a single-type manager for each group (kv_cache_coordinator.py:91-120). The concrete strategy is chosen by the get_kv_cache_coordinator factory: with caching disabled it returns KVCacheCoordinatorNoPrefixCache; with a single group, UnitaryKVCacheCoordinator; with multiple groups, HybridKVCacheCoordinator (kv_cache_coordinator.py:782-822). BlockPool itself maintains a FreeKVCacheBlockQueue (a doubly-linked list) for LRU eviction and a cached_block_hash_to_block hash table as the prefix-cache index (block_pool.py:144-197).

Design motivation

Why split the coordinator into NoPrefix / Unitary / Hybrid?

• No-cache fallback: KVCacheCoordinatorNoPrefixCache supports any number of groups (including 0) and does not implement any prefix-cache-related interface — find_longest_cache_hit returns (empty blocks, 0), get_num_common_prefix_blocks returns all zeros (kv_cache_coordinator.py:413-424). This way disabling prefix caching never touches any caching path.
• Single-group fast path: UnitaryKVCacheCoordinator assumes hash_block_size == block_size and only one group exists (kv_cache_coordinator.py:471-476). find_longest_cache_hit calls the single manager directly without merging (kv_cache_coordinator.py:480-496), avoiding the spec-group merge machinery of the hybrid path.
• Hybrid aggregates by spec: HybridKVCacheCoordinator.verify_and_split_kv_cache_groups merges multiple groups sharing the same spec into a SpecGroup and does cache-hit lookup together (kv_cache_coordinator.py:560-588), placing full attention first to give subsequent groups a tight upper bound (kv_cache_coordinator.py:590-595).
• Shared BlockPool: All groups share num_gpu_blocks physical blocks. Caches of different groups use make_block_hash_with_group_id(block_hash, group_id) to compose the hash; get_cached_block looks up each group_id in the list separately (block_pool.py:199-224). A BlockHashToBlockMap lets one hash map to blocks in multiple groups.
• LRU + tail-block-first eviction: FreeKVCacheBlockQueue is a doubly-linked list enabling O(1) middle removal; free pushes in reverse order so tail blocks get evicted first (kv_cache_utils.py:179-197). free_blocks also evicts blocks without a hash before blocks with a hash (block_pool.py:622-635).
• null_block placeholder: block_id=0 is the null_block, used by remove_skipped_blocks to replace blocks outside the sliding window or preempted blocks. Its ref_cnt is not tracked; freeing must skip it carefully (block_pool.py:188-192).

Key files

• KVCacheCoordinator.__init__:61-128 — Abstract base class: creates the BlockPool, spins up single-type managers, reads VLLM_PREFIX_CACHE_RETENTION_INTERVAL.
• get_num_blocks_to_allocate:130-185 — Sums across single-type managers; cross-attention takes a separate path.
• allocate_new_blocks:233-266 — Calls manager.allocate_new_blocks per group; encoder tokens handled separately.
• cache_blocks:268-283 — Calls each manager's cache_blocks, with retention_interval.
• remove_skipped_blocks:331-352 — Replaces blocks outside the sliding window with null_block; R-SWA uses num_prompt_tokens to decide the gap.
• KVCacheCoordinatorNoPrefixCache:377-424 — Used when prefix caching is disabled; all hits return empty.
• UnitaryKVCacheCoordinator:427-496 — Single-group fast path.
• SpecGroup + HybridKVCacheCoordinator:499-588 — SpecGroup NamedTuple + per-spec group aggregation.
• find_longest_cache_hit_per_group:742-779 — Looks up hits per group independently, returns (blocks_per_group, hit_lengths_per_group).
• get_kv_cache_coordinator:782-822 — Factory: picks one of NoPrefix / Unitary / Hybrid.
• BlockPool.__init__:144-197 — Creates FreeKVCacheBlockQueue, cached_block_hash_to_block, and null_block.
• BlockPool.get_cached_block:199-224 — Looks up blocks for the same hash across a list of group_ids; any miss returns None.
• BlockPool.cache_full_blocks:226-263 — Hashes filled blocks and writes them into cached_block_hash_to_block; supports block_mask to skip blocks that will never hit.
• BlockPool.get_new_blocks:542-572 — Pops N blocks from the head of the free queue, evicting old cached hashes along the way.
• BlockPool.touch:597-612 — ref_cnt += 1; when ref_cnt goes from 0 to 1, the block is removed from the free queue.
• BlockPool.free_blocks:614-635 — Blocks without a hash are prepend_last (evicted first); blocks with a hash are append-ed to the LRU tail.
• BlockPool.reset_prefix_cache:656-690 — Only allowed to clear when only the null block is in use.
• KVCacheBlock:118-176 — Block metadata: block_id / ref_cnt / _block_hash / list pointers / is_null.
• FreeKVCacheBlockQueue:179-197 — Doubly-linked list, O(1) middle removal, LRU order.

Data flow

BlockPool.get_new_blocks is the real entry point for physical allocation — pops N blocks from the head of the free queue, evicts old hashes one by one, and sets ref_cnt to 1:

def get_new_blocks(self, num_blocks: int) -> list[KVCacheBlock]:
    if num_blocks > self.get_num_free_blocks():
        raise ValueError(f"Cannot get {num_blocks} free blocks from the pool")

    ret: list[KVCacheBlock] = self.free_block_queue.popleft_n(num_blocks)

    if self.enable_caching:
        for block in ret:
            self._maybe_evict_cached_block(block)
            assert block.ref_cnt == 0
            block.ref_cnt += 1
            if self.metrics_collector:
                self.metrics_collector.on_block_allocated(block)
    else:
        for block in ret:
            assert block.ref_cnt == 0
            block.ref_cnt += 1
            if self.metrics_collector:
                self.metrics_collector.on_block_allocated(block)
    return ret

(block_pool.py:542-572) Called by SingleTypeKVCacheManager.allocate_new_blocks (single_type_kv_cache_manager.py:279-311), which appends the new blocks to req_to_blocks[request_id]. Release goes through BlockPool.free_blocks: ref_cnt is decremented by 1, and blocks whose ref_cnt reaches zero are split into "with hash" / "without hash" streams and pushed back to the tail of the free queue (block_pool.py:614-635). When a request's prompt fills a block, cache_full_blocks hashes the block content and writes it into cached_block_hash_to_block; the next request with the same prefix calls get_cached_block and hits directly — touch increments ref_cnt without recomputing (block_pool.py:199-224) (block_pool.py:597-612).

Boundaries and failure modes

• Exceeds free block count: get_new_blocks does not pre-check; it directly raise ValueErrors (block_pool.py:553-554). Callers (KVCacheManager.allocate_slots) must compute the requirement first via get_num_blocks_to_allocate (kv_cache_coordinator.py:130-185).
• Hybrid does not support DCP / PCP: assert dcp_world_size == 1 and pcp_world_size == 1 (kv_cache_coordinator.py:556-557).
• Unitary requires hash_block_size == block_size: assert not enable_caching or hash_block_size == self.block_size (kv_cache_coordinator.py:471-473).
• Hybrid requires at least two attention groups: assert len(self.attention_groups) > 1 (kv_cache_coordinator.py:586-588).
• Setting a hash on an already-hashed block: set_block_hash uses an assert to guard against double-setting (kv_cache_utils.py:153-157).
• reset_prefix_cache refuses to clear: If anything besides the null block is in use, it logger.warnings and returns False (block_pool.py:665-672).
• evict_blocks accepts external block_ids: Out-of-range block ids reported by a KV connector are caught immediately by an assert (block_pool.py:647-652).
• retention_interval validation: _validate_prefix_cache_retention_interval requires a positive integer multiple of scheduler_block_size (kv_cache_coordinator.py:31-58).

Summary

Coordinator + BlockPool form the physical layer underneath KVCacheManager: the coordinator dispatches to single-type managers by policy, while BlockPool manages the physical block pool and the prefix-cache index. The result of block allocation eventually becomes a list of block_ids, which block table wraps into tensors the attention kernel can consume directly.

See official docs: vLLM docs · README