EngineCore: the true core of the v1 engine

源码版本v0.25.1

Responsibilities

In the v1 architecture LLMEngine is only a thin shell: it uses InputProcessor to turn incoming prompts into EngineCoreRequest, and OutputProcessor to translate EngineCoreOutputs back to RequestOutput, while all the real work is delegated to EngineCore. The line self.engine_core = EngineCoreClient.make_client(...) in LLMEngine.__init__(llm_engine.py:105-111) is where the EngineCore actually enters the picture.

EngineCore itself assembles three things: model_executor (the executor, i.e. an instance of executor_class, responsible for running forward passes across workers), scheduler (the scheduler, responsible for the waiting/running queues, continuous batching, and preemption), and KVCacheManager (the KV cache manager, which uses _initialize_kv_caches to determine how much memory each card can give to the KV cache after profiling). The constructor initializes these three components first(core.py:122-158), then sets the step function pointer to self.step or self.step_with_batch_queue(core.py:221-223), and the loop afterwards just calls it repeatedly.

step() does one thing: schedule, run forward, reclaim outputs(core.py:479-508). Specifically, scheduler.schedule() returns a SchedulerOutput, model_executor.execute_model(..., non_block=True) immediately returns a future, while the scheduler computes the grammar bitmask in parallel; once result() on the future is ready, it goes to scheduler.update_from_output() which writes tokens back to requests and produces EngineCoreOutputs. This is one iteration of v1, and all upper-layer APIs (LLM.generate, AsyncLLMEngine.generate) ultimately land on these few dozen lines.

Design motivation

Why split LLMEngine / EngineCore / EngineCoreProc / EngineCoreClient into four layers?

• Process isolation: EngineCoreProc is a subclass of EngineCore(core.py:896-897) that runs in a separate background process, so a crash in the front-end Python does not take down workers mid-forward, and a worker OOM does not take the API process with it.
• ZMQ decoupling: the front and back ends are bridged via a ZMQ socket + two queue.Queues(core.py:915-916). ZMQ releases the GIL during socket IO, allowing real overlap with GPU forward; input and output each have their own thread(core.py:980-1001), so serialization/deserialization can hide behind the forward pass.
• In-process fallback: when multiprocessing is off, InprocClient just does self.engine_core = EngineCore(...)(core_client.py:286-287), and get_output() calls step_fn() in place, preserving the v0-style synchronous usage of LLMEngine.add_request() / .step().
• Async entry: AsyncMPClient combines multiprocessing + asyncio; in the server scenario, AsyncLLM uses it to wrap EngineCore into an await-able object so the front end does not block.
• DP scaling: run_engine_core spawns one EngineCoreProc per DP rank when there are multiple DP ranks(core.py:1192-1200) — MoE uses DPEngineCoreProc, non-MoE is treated as an independent DP=1, and the front end is load-balanced by DPLBAsyncMPClient.

Key files

• EngineCore class:96-97 — class EngineCore: definition, docstring states Inner loop of vLLM's Engine.
• EngineCore.__init__:99-237 — assembles model_executor, _initialize_kv_caches, Scheduler, batch_queue, and the prefix caching hasher.
• _initialize_kv_caches:240-295 — fetches get_kv_cache_specs, profiles memory, assembles KVCacheConfig.
• EngineCore.step:479-508 — the core 30 lines of scheduling + execute_model + update_from_output.
• step_with_batch_queue:519-535 — for PP scenarios, uses a batch queue to asynchronously overlap multiple batches, prioritizing filling the queue.
• EngineCoreProc class:896-897 — ZMQ-wrapper for running EngineCore in background process.
• run_engine_core:1154-1224 — process entry, starts EngineCoreProc, registers SIGTERM/SIGINT, calls run_busy_loop.
• run_busy_loop:1259-1267 — while self._handle_shutdown(): _process_input_queue(); _process_engine_step().
• _process_engine_step:1300-1317 — calls step_fn(), pushes outputs into output_queue, calls post_step.
• LLMEngine.make_client:105-111 — LLMEngine selects Inproc/MP/AsyncMP via EngineCoreClient.make_client.
• EngineCoreClient.make_client:83-105 — chooses one of sync multiprocess / async multiprocess / in-process.

Data flow

When a request comes in from the client, it is first processed by InputProcessor in LLMEngine into an EngineCoreRequest, then dropped via engine_core.add_request(...). In multiprocess mode, this call is ultimately serialized by MPClient and sent over ZMQ to the EngineCoreProc's input socket, where the process_input_sockets thread decodes it and put_nowaits it into input_queue(core.py:915-916). When the main loop run_busy_loop wakes up, it takes it out of the queue and dispatches it via _handle_client_request(core.py:1372-1405); the ADD type ultimately lands at self.scheduler.add_request(request)(core.py:372-403).

# vllm/v1/engine/core.py L1259-L1267
def run_busy_loop(self):
    """Core busy loop of the EngineCore."""
    while self._handle_shutdown():
        # 1) Poll the input queue until there is work to do.
        self._process_input_queue()
        # 2) Step the engine core and return the outputs.
        self._process_engine_step()

    raise SystemExit

_process_engine_step(core.py:1300-1317) takes the outputs returned by step_fn() and output_queue.put_nowait(output)s each one. The output thread then ZMQ-sends them back to the front end, where MPClient.get_output() deserializes and hands them to the engine_core.get_output() call in LLMEngine.step()(llm_engine.py:302-314); finally OutputProcessor.process_outputs stitches the token stream back into a RequestOutput.

The body of the step function is just these three segments:

# vllm/v1/engine/core.py L488-L508
if not self.scheduler.has_requests():
    return {}, False
scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
future = self.model_executor.execute_model(scheduler_output, non_block=True)
grammar_output = self.scheduler.get_grammar_bitmask(scheduler_output)
with (
    self.log_error_detail(scheduler_output),
    self.log_iteration_details(scheduler_output),
):
    model_output = future.result()
    if model_output is None:
        model_output = self.model_executor.sample_tokens(grammar_output)

# Before processing the model output, process any aborts that happened
# during the model execution.
self._process_aborts_queue()
engine_core_outputs = self.scheduler.update_from_output(
    scheduler_output, model_output
)

non_block=True makes execute_model return a future immediately, and the main thread computes the grammar bitmask in this gap — this is the key move v1 uses to truly run scheduling and the worker concurrently.

Boundaries and failure modes

• EngineCoreProc process crash: run_engine_core calls engine_core._send_engine_dead() in the except Exception branch(core.py:1229-1235), pushing an ENGINE_CORE_DEAD byte string into output_queue(core.py:1470-1474); the front-end MPClient raises on receipt rather than waiting forever.
• Executor failure callback: at construction time executor_fail_callback is registered with model_executor(core.py:124-125); when an error is raised inside a worker, the callback pushes an EXECUTOR_FAILED into input_queue, and the main loop's _handle_client_request raises RuntimeError("Executor failed.") on receipt(core.py:1400-1401).
• Shutdown modes: shutdown_timeout == 0 is abort mode, immediately marking all in-flight requests as FINISHED_ABORTED; non-zero is drain mode, waiting for all in-flight requests to finish(core.py:1329-1360). Incoming ADD and UTILITY calls during shutdown are explicitly rejected(core.py:1407-1432).
• Non-causal attention layers: some attention layers (such as Prefix LM) are tagged non_causal=True; when detected in _initialize_kv_caches, chunked prefill and prefix caching are forced off(core.py:255-269), otherwise the KV cache gets corrupted.
• batch_queue yields the CPU: when a step did not run forward but the scheduler still has unfinished requests (e.g. waiting on a remote KV transfer), time.sleep(0.001) actively yields the GIL(core.py:1311-1315), so background transfer threads do not starve.
• In-process mode has no busy loop: InprocClient.get_output calls step_fn synchronously(core_client.py:289-292), without running run_busy_loop. This means there are no input/output threads in in-process mode, and add_request enters the scheduler synchronously.

Summary

EngineCore is the engine of v1: it assembles the scheduler, executor, and KV cache together, and uses step() to repeatedly schedule + run forward + reclaim outputs. The subclass EngineCoreProc moves it into a separate process, bridging front and back ends with ZMQ + two queues, so that GPU forward and network IO truly overlap. Which client the front end uses (InprocClient, MPClient, AsyncMPClient) determines whether it runs synchronously or asynchronously, but the EngineCore layer is the same code for all front ends. To dig deeper, see /engine/llm-engine for what the thin LLMEngine shell wraps, or /engine/engine-core-proc for a deeper look at the ZMQ layer; the three client implementations are at /client/inproc-mp, and the scheduler internals are at /scheduler/scheduler.

See official docs: vLLM docs · README