LLM and AsyncLLM: the offline and serving entry points

源码版本v0.25.1

Responsibilities

LLM is vLLM's synchronous Python API for offline inference scenarios(llm.py:66-67), and AsyncLLM is the async wrapper used by the OpenAI API server in serving scenarios(async_llm.py:70-71). Both inherit from the EngineClient abstraction but use different EngineCoreClient implementations: LLM defaults to SyncMPClient or InprocClient, while AsyncLLM uses AsyncMPClient(or DPLBAsyncMPClient in the DP scenario). Neither class runs forward passes directly — they are containers for the trio of InputProcessor, OutputProcessor, and EngineCoreClient.

LLM.__init__(llm.py:176-222) accepts almost all EngineArgs fields as kwargs, repackaging them into an EngineArgs(model=..., tensor_parallel_size=..., ...)(llm.py:305-345), which is then handed to LLMEngine.from_engine_args(engine_args, usage_context=UsageContext.LLM_CLASS)(llm.py:349-351). LLMEngine in v1 is a thin backward-compatibility shell(llm_engine.py:48-49) that internally uses the client selected by EngineCoreClient.make_client(...)(llm_engine.py:105) as self.engine_core. So LLM.generate() ultimately bottoms out at the llm_engine.add_request + llm_engine.step loop.

AsyncLLM takes a different assembly path. build_async_engine_client_from_engine_args(api_server.py:109-154) first calls engine_args.create_engine_config() to get VllmConfig, then AsyncLLM.from_vllm_config(vllm_config, ...), which internally calls EngineCoreClient.make_async_mp_client(...)(async_llm.py:146-153) to start AsyncMPClient(or DPLBAsyncMPClient in the DP scenario). AsyncLLM has an additional long-running output_handler coroutine(async_llm.py:637-660) that keeps calling get_output_async() to pull EngineCore outputs back to the frontend.

Design motivation

• One EngineArgs schema serves both paths: LLM is constructed via EngineArgs(model=model, ...)(llm.py:305-345), and AsyncLLM comes from AsyncEngineArgs.from_cli_args(args)(api_server.py:95) — both reuse the same field definitions in /startup/engine-args, so offline and serving behave the same.
• LLM is a thin synchronous shell: LLM.generate -> _run_completion -> _render_and_add_requests -> _add_request(offline_utils.py:552-571) -> llm_engine.add_request, then _run_engine drives the loop with while has_unfinished_requests(): step_outputs = self.llm_engine.step()(offline_utils.py:594-595). This way a script can be written with a single for, and users never touch asyncio.
• AsyncLLM hides blocking in output_handler: add_request returns a RequestOutputCollector immediately(async_llm.py:280-297). The real token stream is pulled back in the background output_handler via engine_core.get_output_async() and chunk-processed(async_llm.py:656-676). VLLM_V1_OUTPUT_PROC_CHUNK_SIZE slices a large output batch to prevent the event loop from being blocked by a single oversized batch.
• LLMEngine is just a compat layer in v1: The docstring of class LLMEngine literally says Legacy LLMEngine for backwards compatibility.(llm_engine.py:48-49). Its __init__ calls EngineCoreClient.make_client(multiprocess_mode, asyncio_mode=False, ...), unifying the sync multi-process and in-process modes. make_client(core_client.py:83-105) uses two bools to select InprocClient / SyncMPClient / AsyncMPClient.
• make_async_mp_client auto-selects DP client: When data_parallel_size > 1 and data_parallel_external_lb is on, it uses DPAsyncMPClient(one client per DP rank); otherwise it uses DPLBAsyncMPClient(internal LB that dispatches requests across all DP ranks)(core_client.py:116-132). This is the v1 data-parallel entry point.
• LLM.__init__ normalizes config: When compilation_config is an int, it is auto-wrapped as CompilationConfig(mode=CompilationMode(int))(llm.py:275-282). When kv_transfer_config is a dict, it is converted to KVTransferConfig(llm.py:245-262). When worker_cls is a class, it is cloudpickle-serialized(llm.py:238-243). These conversions save SDK callers from writing boilerplate.
• from_engine_args classmethods unify entry: LLM.from_engine_args(llm.py:387-389), AsyncLLM.from_engine_args(async_llm.py:232-254), and LLMEngine.from_engine_args(llm_engine.py:161-186) all go through engine_args.create_engine_config() -> Executor.get_class(vllm_config), guaranteeing that the VllmConfig derivation process is identical across entry points.

Key files

• LLM class:66-67 — class LLM(BeamSearchOfflineMixin, PoolingOfflineMixin, OfflineInferenceMixin) definition.
• LLM.__init__:176-222 — accepts nearly 40 kwargs + **kwargs; internally converts dict/int into the corresponding config instances.
• EngineArgs assembly:305-345 — stuffs all LLM kwargs into EngineArgs(model=..., ...), then log_non_default_args.
• LLMEngine.from_engine_args:349-353 — usage_context=UsageContext.LLM_CLASS, gets llm_engine.model_config and supported_tasks.
• LLM.from_engine_args:387-389 — cls(**vars(engine_args)) directly reuses the constructor.
• LLM.generate:422-485 — validates runner_type == "generate", defaults sampling params, delegates to _run_completion.
• _add_request:552-571 — params.output_kind = RequestOutputKind.FINAL_ONLY, calls llm_engine.add_request.
• _run_engine:573-595 — while self.llm_engine.has_unfinished_requests(): step_outputs = self.llm_engine.step() synchronous loop.
• LLMEngine class:48-49 — the v1 thin shell with Legacy LLMEngine for backwards compatibility..
• make_client call:105-111 — self.engine_core = EngineCoreClient.make_client(multiprocess_mode, asyncio_mode=False, ...).
• LLMEngine.from_engine_args:161-186 — runs engine_args.create_engine_config, decides whether to enable multi-process based on VLLM_ENABLE_V1_MULTIPROCESSING.
• make_client:83-105 — two bools select InprocClient / SyncMPClient / AsyncMPClient.
• make_async_mp_client:109-132 — auto-selects DPAsyncMPClient or DPLBAsyncMPClient for the DP scenario.
• AsyncLLM class:70-71 — class AsyncLLM(EngineClient): definition.
• AsyncLLM.__init__:73-176 — assembles the trio of InputProcessor, OutputProcessor, EngineCoreClient.make_async_mp_client.
• AsyncLLM.from_vllm_config:203-229 — constructs directly from VllmConfig + Executor.get_class, the server-side entry point.
• AsyncLLM.from_engine_args:232-254 — engine_args.create_engine_config -> Executor.get_class -> cls(...).
• _run_output_handler:637-676 — background coroutine: get_output_async + output_processor.process_outputs, chunk size controlled by VLLM_V1_OUTPUT_PROC_CHUNK_SIZE.
• build_async_engine_client:78-105 — AsyncEngineArgs.from_cli_args -> build_async_engine_client_from_engine_args.
• build_async_engine_client_from_engine_args:109-154 — engine_args.create_engine_config -> AsyncLLM.from_vllm_config; in finally, async_llm.shutdown(timeout=vllm_config.shutdown_timeout).
• AsyncLLM.shutdown:259-271 — shutdown_prometheus, renderer.shutdown, engine_core.shutdown, cancel output_handler.

Data flow

Offline LLM.generate(prompts)

After generate(llm.py:422) validates runner_type, it delegates to OfflineInferenceMixin._run_completion, which calls _render_and_add_requests(offline_utils.py:523). There each prompt is processed by the renderer into an EngineInput and then _add_request is called:

# vllm/entrypoints/offline_utils.py L552-L571
def _add_request(
    self,
    prompt: EngineInput,
    params: SamplingParams | PoolingParams,
    lora_request: LoRARequest | None = None,
    priority: int = 0,
) -> str:
    if isinstance(params, SamplingParams):
        # We only care about the final output
        params.output_kind = RequestOutputKind.FINAL_ONLY

    request_id = str(next(self.request_counter))

    return self.llm_engine.add_request(
        request_id,
        prompt,
        params,
        lora_request=lora_request,
        priority=priority,
    )

Note params.output_kind = RequestOutputKind.FINAL_ONLY — the offline scenario only cares about the final result and does not want an intermediate token stream, saving a pile of chunked-streaming overhead on the EngineCore output side. After _add_request, _run_engine drives EngineCore with a synchronous loop:

# vllm/entrypoints/offline_utils.py L590-L599
# Run the engine.
outputs: list[_O] = []
total_in_toks = 0
total_out_toks = 0
while self.llm_engine.has_unfinished_requests():
    step_outputs = self.llm_engine.step()
    for output in step_outputs:
        assert isinstance(output, output_type)
        if output.finished:
            outputs.append(output)

Inside, LLMEngine.step() is just a thin wrapper around engine_core.get_output() + output_processor.process_outputs. When engine_core is InprocClient, it directly calls step_fn() synchronously; when it is SyncMPClient, it blocks on receiving ZMQ. The whole flow has no asyncio, so for output in llm.generate(...) works in a script.

Serving AsyncLLM.add_request

The server-side AsyncLLM takes an entirely different path. build_async_engine_client_from_engine_args(api_server.py:109-154) turns AsyncEngineArgs into VllmConfig, then AsyncLLM.from_vllm_config(async_llm.py:203-229). The key step in the constructor is:

# vllm/v1/engine/async_llm.py L146-L153
# EngineCore (starts the engine in background process).
self.engine_core = EngineCoreClient.make_async_mp_client(
    vllm_config=vllm_config,
    executor_class=executor_class,
    log_stats=self.log_stats,
    client_addresses=client_addresses,
    client_count=client_count,
    client_index=client_index,
)

make_async_mp_client(core_client.py:109-132) selects AsyncMPClient or DPLBAsyncMPClient based on DP topology. After this, the output_handler coroutine keeps running in the background:

# vllm/v1/engine/async_llm.py L656-L676
async def output_handler():
    try:
        while True:
            # 1) Pull EngineCoreOutputs from the EngineCore.
            outputs = await engine_core.get_output_async()
            num_outputs = len(outputs.outputs)

            iteration_stats = (
                IterationStats() if (log_stats and num_outputs) else None
            )

            # Split outputs into chunks of at most
            # VLLM_V1_OUTPUT_PROC_CHUNK_SIZE, so that we don't block the
            # event loop for too long.
            engine_core_outputs = outputs.outputs
            for start in range(0, num_outputs, chunk_size):
                end = start + chunk_size
                outputs_slice = engine_core_outputs[start:end]
                # 2) Process EngineCoreOutputs.
                processed_outputs = output_processor.process_outputs(
                    outputs_slice, outputs.timestamp, iteration_stats

On the request side, the OpenAI route calls async_llm.add_request(request_id, prompt, params, ...)(async_llm.py:280-297) and immediately gets a RequestOutputCollector; the rest is just awaiting its stream. EngineCore runs the forward pass and the output flows back via ZMQ to AsyncMPClient, where output_handler pulls it out, chunk-processes it, and pushes to the collector — the whole chain is fully async, so the API server is not blocked by a single slow request.

Boundaries and failure modes

• LLM.generate does not support non-generate runners: When runner_type != "generate", it raises directly(llm.py:465-471), suggesting --runner generate. LLM.chat and LLM.enqueue_chat do the same check(llm.py:684-689).
• LLM(data_parallel_size>1) disallows single-process: When _dp_size > 1 and it is not external_launcher and not TPU, it raises directly(llm.py:295-303), suggesting the multi-process approach in examples/features/data_parallel/data_parallel_offline.py; otherwise it will hang.
• renderer_num_workers > 1 is a no-op on offline LLM: LLM takes the synchronous renderer path; the multi-worker thread pool is only consumed in the async path of vllm serve / AsyncLLM. The offline scenario explicitly warning_once(llm.py:370-379) so users do not assume multi-threading is on.
• EngineDeadError: AsyncLLM.add_request checks self.errored at the top(async_llm.py:300-301). If the EngineCore process is already dead (a ENGINE_CORE_DEAD was sent in the background), it raises EngineDeadError(async_llm.py:1054-1058) and refuses new requests.
• async_llm.shutdown safety net: The finally block of build_async_engine_client_from_engine_args calls async_llm.shutdown(timeout=vllm_config.shutdown_timeout)(api_server.py:152-154), so even if the build throws, background processes and ZMQ sockets are cleaned up. AsyncLLM.__del__(async_llm.py:256-257) also calls shutdown as a last resort to prevent GC-time leaks.
• kv_transfer_config dict validation: When LLM.__init__ fails to convert a dict into a KVTransferConfig, it does logger.error then raises ValueError(f"Invalid 'kv_transfer_config' provided: {e}")(llm.py:253-262), wrapping the original ValidationError in a friendlier error.
• swap_space is deprecated: When swap_space appears in kwargs, it is popped out with a DeprecationWarning(llm.py:224-233). It will be removed in the future.
• Degraded mode when output_handler is not started: If asyncio.get_running_loop() raises RuntimeError (no running event loop) during __init__, output_handler is skipped(async_llm.py:170-176). In that case the user must explicitly call the start_engine_loop=True equivalent, or bring AsyncLLM into an event loop before awaiting.
• DP client selection is static: make_async_mp_client only looks at parallel_config.data_parallel_size and data_parallel_external_lb(core_client.py:126-132). Runtime LB-mode switching is not supported — the topology is fixed at startup.

Summary

LLM and AsyncLLM are vLLM's two main entry points exposed to the upper layer (scripts and OpenAI server). Both are implementations of the EngineClient abstraction but use different EngineCoreClients. LLM takes the synchronous path: internally LLMEngine + SyncMPClient/InprocClient, and the user gets a synchronous iterator in the style of for output in llm.generate(...). AsyncLLM takes the async path: internally AsyncMPClient/DPLBAsyncMPClient + a background output_handler coroutine, and each HTTP request in the API server just awaits one RequestOutputCollector. Both derive VllmConfig from EngineArgs.create_engine_config(), so the configuration side is identical; the differences are only in client selection and the event-loop model. For how EngineArgs / VllmConfig are assembled see /startup/engine-args; for how the CLI chain reaches AsyncLLM.from_vllm_config see /startup/cli; for the EngineCore background process and ZMQ internals see /engine/engine-core; for details on the three EngineCoreClient implementations see /client/inproc-mp.

See official docs: vLLM docs · README