Register First: How vLLM's Push Mode Takes KV Transfer Off the Critical Path
LLM inference splits into two phases: prefill, which processes the prompt and builds the key-value cache, and decode, which uses that cache to emit tokens one at a time. Prefill-decode disaggregation runs the two on separate engine instances so each can be scaled and tuned on its own — which means the KV cache must physically travel from the prefill instance to the decode instance before generation can start. In the push-based NIXL connector AWS describes, the decode side no longer sits waiting for data to arrive. At the start of a request, its main loop allocates the destination KV blocks and registers them with the prefill instance while the prefill main loop is still computing the prompt's KV cache: the destination is booked before the payload exists. The connector then separates model execution from transfer preparation by giving each prefill worker a dedicated background transfer thread, so the coordination that used to sit between prefill and decode is handled alongside them rather than in sequence.
AI-generated discussion · Different perspectives
The headline number is AWS's own: on a 1P1D deployment of EC2 P5en instances connected with EFA, push mode delivered up to 3x better TTFT and up to 30 percent better time-per-output-token than pull mode for the Qwen3-32B model, with results depending on workload and configuration. That's a vendor benchmark on a vendor's instance type with a single named model. Read the multiplier as a signpost, not a promise.
The multiplier may be the least interesting part. The design detail is that registration happens at allocation time, overlapping prefill compute instead of trailing it, and the net effect claimed is lower time-to-first-token plus higher output-token throughput on the decode instance. Nothing here is a smarter model or faster silicon — it's bookkeeping rearranged so it stops blocking the first token. Serving latency usually hides in exactly that kind of plumbing.