Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

vLLM is an open-source inference and serving engine for large language models (LLMs). LLM inference has two distinct phases: prefill, which processes the input prompt and builds the key-value (KV) cache, and decode, which uses that cache to generate output one token at a time. These phases have different compute, memory, and scaling characteristics. Prefill-decode (P-D) disaggregation runs them on separate engine instances, allowing each phase to scale and be optimized independently. This can improve resource utilization, throughput, and latency under load. However, it also requires transferring the KV cache from the prefill instance to the decode instance before token generation can begin, making efficient KV transfer critical to time-to-first-token (TTFT). One of the ways vLLM performs this transfer is through the NVIDIA Inference Xfer Library (NIXL) connector , which moves KV blocks over interconnects that support Remote Direct Memory Access (RDMA), such as AWS Elastic Fabric Adapt…