LLMEngine: the synchronous thin shell of the v1 engine

源码版本v0.25.1

Responsibilities

In the v1 architecture, LLMEngine is the entry-point thin shell used by the synchronous API (LLM.generate, LLM.chat). It does not run forward passes, schedule, or allocate KV cache itself; it only glues these three things together: it uses InputProcessor to turn incoming prompts into EngineCoreRequest, uses OutputProcessor to translate EngineCoreOutputs back to RequestOutput, and delegates all real work to EngineCore. The line self.engine_core = EngineCoreClient.make_client(...) at the end of __init__(llm_engine.py:105-111) is where EngineCore actually enters the picture — make_client chooses one of InprocClient, SyncMPClient, or AsyncMPClient based on whether multiprocessing and asyncio are enabled.

This layer exists to preserve the v0-style synchronous API of LLMEngine.add_request() / .step(): in in-process mode, InprocClient directly holds an EngineCore instance(core_client.py:286-292), get_output() calls step_fn() in place, the front and back ends share memory, there is no ZMQ, no busy loop; in multiprocess mode, SyncMPClient spawns a background EngineCoreProc(core.py:896-897), and LLMEngine keeps the same synchronous interface, only with a ZMQ bridge underneath. So LLMEngine is small in code volume (448 lines), with many methods just passing the call through to engine_core (sleep/wake, profile, lora, reset_prefix_cache are all one-line pass-throughs of this kind).

Design motivation

• Preserve the v0 synchronous API: the LLM class exposes synchronous generate/chat methods, and internally uses the LLMEngine.add_request() + LLMEngine.step() loop(llm_engine.py:296-334), so v1 can run without asyncio. This is the key to replacing v0 without breaking offline inference workflows.
• Three backends, one interface: the 3-way choice logic in make_client(core_client.py:83-105) means none of LLMEngine's methods need to care whether the backend is in-process, sync multiprocess, or async multiprocess; they just need an EngineCoreClient instance, and the backend switch is decided at construction time.
• Input/output processing at the shell layer: InputProcessor(input_processor.py:36) handles multimodal, tokenization, priority, trace_headers, and LoRA; OutputProcessor(output_processor.py:417) handles detokenize, streaming chunking, stop string matching, and converting EngineCoreOutputs back to RequestOutput. Both of these dirty-data / dirty-work chores stay at the shell layer, so EngineCore only sees purely structured EngineCoreRequest / EngineCoreOutputs.
• n>1 multi-sample fan-out: when SamplingParams.n is greater than 1, add_request uses ParentRequest(parallel_sampling.py:13) to split one parent request into n child requests and feeds them into EngineCore and OutputProcessor(llm_engine.py:279-294); the output layer then aggregates the child requests' token streams back into the parent request.
• Process exit fallback: when multiprocess_mode=False, a weakref finalizer is taken on the driver model(llm_engine.py:129-133); when LLMEngine is GC'd, _cleanup_instance_caches is called to release the GPU memory pinned by byte code hooks.

Key files

• LLMEngine class:48-49 — class LLMEngine: definition, docstring self-describes as Legacy LLMEngine for backwards compatibility..
• LLMEngine.__init__:51-141 — assembles renderer / InputProcessor / OutputProcessor / EngineCoreClient, then starts StatLoggerManager and the cleanup finalizer.
• make_client call site:105-111 — EngineCoreClient.make_client(multiprocess_mode=..., asyncio_mode=False, ...) selects the backend.
• from_vllm_config:143-158 — constructs from VllmConfig; multiprocess_mode is decided by envs.VLLM_ENABLE_V1_MULTIPROCESSING.
• add_request:218-294 — validates request_id, runs input_processor.process_inputs, fans out via ParentRequest when n>1, and finally calls engine_core.add_request.
• step:296-334 — engine_core.get_output() -> output_processor.process_outputs -> abort_requests -> logger_manager.record.
• abort_request:212-216 — first marks it in OutputProcessor, then has EngineCore remove the request from the scheduler.
• sleep / wake_up:361-376 — sleep/wake across renderer + engine_core; StatLoggerManager records sleep state.
• do_log_stats_with_interval:394-401 — throttles logging using VLLM_LOG_STATS_INTERVAL, avoiding a flush on every step.
• _get_driver_model_for_cleanup:431-434 — follows the model_executor.driver_worker.model_runner.model chain to obtain the driver model for the finalizer to release GPU memory.

Data flow

A synchronous inference run is just for req in requests: add_request(...) followed by while has_unfinished_requests(): step(). The structure of step() is straightforward: pull a batch of EngineCoreOutputs from engine_core, hand them to OutputProcessor for translation, send requests that terminated early due to stop strings back for abort, and log some stats along the way.

# vllm/v1/engine/llm_engine.py L296-L314
def step(self) -> list[RequestOutput | PoolingRequestOutput]:
    if self.should_execute_dummy_batch:
        self.should_execute_dummy_batch = False
        self.engine_core.execute_dummy_batch()
        return []

    # 1) Get EngineCoreOutput from the EngineCore.
    with record_function_or_nullcontext("llm_engine step: get_output"):
        outputs = self.engine_core.get_output()

    # 2) Process EngineCoreOutputs.
    with record_function_or_nullcontext("llm_engine step: process_outputs"):
        iteration_stats = IterationStats() if self.log_stats else None
        processed_outputs = self.output_processor.process_outputs(
            outputs.outputs,
            engine_core_timestamp=outputs.timestamp,
            iteration_stats=iteration_stats,
        )
        self.output_processor.update_scheduler_stats(outputs.scheduler_stats)

There are also steps 3 and 4 afterwards(llm_engine.py:317-332): step 3 hands processed_outputs.reqs_to_abort to engine_core.abort_requests, because stop strings only become visible after detokenize and EngineCore itself does not know about them; step 4 calls record + do_log_stats_with_interval when logger_manager is non-empty and scheduler_stats are available.

add_request takes the fan-out branch when n>1(llm_engine.py:279-294):

# vllm/v1/engine/llm_engine.py L279-L294
# Fan out child requests (for n>1).
parent_req = ParentRequest(request)
for idx in range(n):
    request_id, child_params = parent_req.get_child_info(idx)
    child_request = request if idx == n - 1 else copy(request)
    child_request.request_id = request_id
    child_request.sampling_params = child_params

    # Make a new RequestState and queue.
    self.output_processor.add_request(
        child_request, prompt_text, parent_req, idx
    )
    # Add the request to EngineCore.
    self.engine_core.add_request(child_request)

return req_id

The last child reuses the original request object to save one copy; the parent request ID is generated by ParentRequest.get_child_info(idx), and once OutputProcessor has the parent reference and idx, it can aggregate the child outputs back.

Boundaries and failure modes

• EngineCoreRequest is deprecated: when add_request receives an EngineCoreRequest passed in directly, it emits a warning_once deprecation notice saying that starting from v0.18 you should use Renderer.render_cmpl() / render_chat() instead(llm_engine.py:235-248); if the passed-in request_id does not match EngineCoreRequest.request_id, the latter wins and the former is ignored.
• Dummy batch in DP mode: in data parallel external-launcher mode, when one rank has no work but other ranks are still running, has_unfinished_requests_dp sets should_execute_dummy_batch to True(llm_engine.py:197-203), and step() runs a dummy batch at the start as a placeholder(llm_engine.py:297-300), so collective communication does not hang.
• Only in-process mode exposes model_executor: the if not multiprocess_mode: branch sets self.model_executor = self.engine_core.engine_core.model_executor(llm_engine.py:123-125); in multiprocess mode this object is not accessible across processes, so downstream code accessing model_executor must ensure multiprocess_mode=False.
• Stats log throttling: do_log_stats_with_interval uses VLLM_LOG_STATS_INTERVAL to enforce a minimum interval(llm_engine.py:394-401); _last_log_time is a lazily-initialized instance attribute on first access, sidestepping the hassle of setting an initial value in __init__.
• sleep links the renderer: sleep(level>=1) first clears the renderer's multimodal cache before calling engine_core.sleep(llm_engine.py:361-367), because after wake_up the multimodal embeddings may be invalid and the renderer has to recompute them.
• EngineCore backend is decided by env vars: in from_vllm_config multiprocess_mode=envs.VLLM_ENABLE_V1_MULTIPROCESSING(llm_engine.py:157), and from_engine_args applies it once more(llm_engine.py:174-176), so the multiprocessing switch is fixed at config time and cannot be toggled at runtime.

Summary

LLMEngine is the thin shell on the v1 synchronous path: it validates inputs, uses InputProcessor / OutputProcessor to handle multimodal and streaming output, picks a backend via EngineCoreClient.make_client, and then the add_request + step loop is all that's needed. It delegates all heavy lifting to EngineCore (directly held in in-process mode, via ZMQ in multiprocess mode), and only concerns itself with shell-layer concerns: v0 compatibility, ParentRequest fan-out, stats throttling, and sleep/wake linkage. To go deeper, see /engine/engine-core for EngineCore assembly, /engine/engine-core-proc for the ZMQ background process implementation, the three client implementations at /client/inproc-mp, and the scheduler internals at /scheduler/scheduler.

See official docs: vLLM docs · README