OpenAI Serving and AsyncLLM: the async bridge between HTTP API and engine

源码版本v0.25.1

Responsibilities

What runs after vllm serve boots is a FastAPI service that listens on OpenAI-compatible endpoints like /v1/chat/completions, /v1/completions, /v1/responses, /v1/embeddings, /pooling/*, /speech_to_text/*. The HTTP-layer entry point is build_app(api_server.py:157), which registers each register_*_api_routers and installs CORS / auth / exception handlers. The engine-side entry point is AsyncLLM(async_llm.py:70), which wraps EngineCore(a separate process communicating via ZMQ) into an await-able EngineClient that runs requests in an asyncio event loop.

build_async_engine_client_from_engine_args(api_server.py:109) is the assembly point: it first calls engine_args.create_engine_config() to get VllmConfig, then AsyncLLM.from_vllm_config(...)(async_llm.py:202) starts an AsyncLLM, and EngineCoreClient.make_async_mp_client inside starts an AsyncMPClient(inter-process + asyncio). init_app_state(api_server.py:297) attaches engine_client, OpenAIServingModels, OnlineRenderer, ServingTokenization to app.state, and init_generate_state further constructs handlers like OpenAIServingChat, OpenAIServingCompletion, OpenAIServingResponses.

When an HTTP request arrives, e.g. POST /v1/chat/completions, the route calls OpenAIServingChat.create_chat_completion(serving.py:233). It uses OnlineRenderer to render the message list into a prompt, then engine_client.generate(prompt, sampling_params, request_id, ...)(serving.py:357) returns an AsyncGenerator[RequestOutput] that yields while writing the streaming response back to HTTP.

Design motivation

• Multi-client support: AsyncLLM.__init__ accepts client_addresses / client_count / client_index(async_llm.py:84-86), allowing a single OpenAI server to connect to EngineCores on multiple DP ranks for load balancing.
• InputProcessor / OutputProcessor split: AsyncLLM holds an InputProcessor(turns a prompt into an EngineCoreRequest) and an OutputProcessor(translates EngineCoreOutputs back to RequestOutput)(async_llm.py:135-143), so the HTTP layer only sees EngineInput and RequestOutput and never touches EngineCore directly.
• Background output_handler task: _run_output_handler(async_llm.py:637) launches an asyncio task that pulls outputs from EngineCore, slices them through output_processor.process_outputs, and pushes RequestOutputs to the corresponding request's RequestOutputCollector.
• Chunked processing to avoid blocking the event loop: output_handler slices with VLLM_V1_OUTPUT_PROC_CHUNK_SIZE(async_llm.py:654) and calls await asyncio.sleep(0) between chunks to let other requests yield.
• n>1 fan-out: When params.n > 1, add_request splits the request into n sub-requests, each with its own request_id but sharing the same ParentRequest and the same RequestOutputCollector(async_llm.py:385-397).
• Auto-abort on disconnect: When the HTTP client disconnects, the generate AsyncGenerator raises asyncio.CancelledError / GeneratorExit. The handler catches it and calls await self.abort(request_id, internal=True)(async_llm.py:591-596) to drop the request from EngineCore instead of wasting GPU.
• Graceful startup failure: __init__ defers _run_output_handler to the first add_request(async_llm.py:370-373), so AsyncLLM can be constructed outside an event loop and a startup failure raises directly and exits.
• FastAPI multiple endpoints share the engine: init_app_state puts engine_client into state(api_server.py:336), so chat / completion / responses / pooling / speech_to_text handlers all share the same AsyncLLM.

Key files

• build_app:157 — builds the FastAPI app, mounts all routers, CORS, auth, exception handlers.
• build_async_engine_client_from_engine_args:109 — assembly point: EngineArgs -> VllmConfig -> AsyncLLM.
• init_app_state:297 — attaches engine_client, models, renderer, serving_tokenization to app.state.
• AsyncLLM class:70 — async implementation of EngineClient, holds InputProcessor / OutputProcessor / engine_core.
• AsyncLLM.from_vllm_config:202 — classmethod factory: calls Executor.get_class to pick an executor and constructs AsyncLLM.
• AsyncLLM.add_request:280 — entry point: feeds prompt to InputProcessor, starts a RequestOutputCollector, and _add_request drops it into EngineCore.
• AsyncLLM.generate:524 — the AsyncGenerator wrapper around add_request + while not finished: q.get().
• _run_output_handler:637 — background task: engine_core.get_output_async -> output_processor.process_outputs -> push to queue.
• _add_request:400 — two steps: output_processor.add_request + engine_core.add_request_async.
• AsyncLLM.abort:709 — cancels a request; internal=True is an abort triggered by disconnect.
• OpenAIServingChat:106 — chat completions handler, inherits GenerateBaseServing.
• create_chat_completion:233 — HTTP entry point: renders prompt + calls engine_client.generate.
• OpenAIServingCompletion:55 — completions API handler.
• BaseServing:29 — base class for all handlers, provides _check_model / create_error_response.

Data flow

The full path of a POST /v1/chat/completions: FastAPI route -> OpenAIServingChat.create_chat_completion -> OnlineRenderer.render_chat() renders the message list into prompt token ids -> engine_client.generate(prompt, sampling_params, request_id, ...) returns an AsyncGenerator. generate internally calls add_request:

# vllm/v1/engine/async_llm.py L357-L373 (simplified)
generator = self.engine_client.generate(
    engine_input,
    sampling_params,
    sub_request_id,
    lora_request=lora_request,
    trace_headers=trace_headers,
    priority=request.priority,
    data_parallel_rank=data_parallel_rank,
    reasoning_ended=reasoning_ended,
    reasoning_parser_kwargs={...} if parser is not None and parser.reasoning_parser is not None else None,
)

add_request feeds the prompt to InputProcessor.process_inputs to produce an EngineCoreRequest, then calls _add_request:

# vllm/v1/engine/async_llm.py L400-L415
async def _add_request(
    self,
    request: EngineCoreRequest,
    prompt: str | None,
    parent_req: ParentRequest | None,
    index: int,
    queue: RequestOutputCollector,
):
    # Add the request to OutputProcessor (this process).
    self.output_processor.add_request(request, prompt, parent_req, index, queue)

    # Add the EngineCoreRequest to EngineCore (separate process).
    await self.engine_core.add_request_async(request)

    if self.log_requests:
        logger.info("Added request %s.", request.request_id)

After the request enters EngineCore, the background output_handler task keeps calling await engine_core.get_output_async() to pull outputs, slices them through process_outputs, and pushes RequestOutputs to the corresponding collector:

# vllm/v1/engine/async_llm.py L656-L683
async def output_handler():
    try:
        while True:
            # 1) Pull EngineCoreOutputs from the EngineCore.
            outputs = await engine_core.get_output_async()
            num_outputs = len(outputs.outputs)

            iteration_stats = (
                IterationStats() if (log_stats and num_outputs) else None
            )

            # Split outputs into chunks of at most
            # VLLM_V1_OUTPUT_PROC_CHUNK_SIZE, so that we don't block the
            # event loop for too long.
            engine_core_outputs = outputs.outputs
            for start in range(0, num_outputs, chunk_size):
                end = start + chunk_size
                outputs_slice = engine_core_outputs[start:end]
                # 2) Process EngineCoreOutputs.
                processed_outputs = output_processor.process_outputs(
                    outputs_slice, outputs.timestamp, iteration_stats
                )
                # NOTE: RequestOutputs are pushed to their queues.
                assert not processed_outputs.request_outputs

                # Allow other asyncio tasks to run between chunks
                if end < num_outputs:
                    await asyncio.sleep(0)

The caller of generate loops in while not finished: q.get_nowait() or await q.get() pulling RequestOutputs, yielding each to the HTTP streaming response.

Boundaries and failure modes

• Auto-abort on client disconnect: generate catches asyncio.CancelledError / GeneratorExit and calls await self.abort(request_id, internal=True)(async_llm.py:591-596) to drop the request from the EngineCore queue.
• Engine death raises EngineDeadError: generate catches EngineDeadError and re-raises without aborting(async_llm.py:598-602), because there is nothing left to abort; the HTTP layer turns it into a 500.
• n>1 fan-out shares collector: When params.n > 1, add_request splits the request into n sub-requests sharing the same RequestOutputCollector(async_llm.py:385-397), so the upper layer still sees a single stream.
• kv_sharing_fast_prefill does not support prompt_logprobs: add_request raises ValueError directly when it detects kv_sharing_fast_prefill + prompt_logprobs(async_llm.py:305-314), because prompt logprobs cannot be computed accurately under this KV-sharing path.
• Streaming input does not support reasoning_ended: When passing an AsyncGenerator-form streaming input, supplying reasoning_ended / reasoning_parser_kwargs raises NotImplementedError directly(async_llm.py:316-318).
• output_handler circular reference guard: _run_output_handler explicitly stores engine_core / output_processor / renderer as local variables(async_llm.py:643-653), so the task does not hold self and form a circular ref that GC cannot collect.
• request_id mismatch warns only: When add_request receives an EngineCoreRequest whose request_id does not match, it only warns and does not raise(async_llm.py:342-347); EngineCoreRequest.request_id is authoritative.

Summary

OpenAI serving is vLLM's HTTP entry point, composed of two layers: a FastAPI app + handlers like OpenAIServingChat / OpenAIServingCompletion for the OpenAI-compatible protocol; and AsyncLLM wrapping EngineCore as EngineClient, pushing requests into EngineCore and pulling outputs back to the HTTP stream inside an asyncio event loop. For EngineCore's internal scheduling and forward loop see /engine/engine-core; for how HTTP requests are finally computed by workers see /worker/gpu-model-runner; for the last step of sampling see /sampling/sampler.

See official docs: vLLM docs · README