The KV Cache That Travels Both Ways Between Prefill and Decode
In prefill-decode disaggregation, a request is split across specialised nodes: the prefill node processes the prompt and builds its key-value cache, then the decode node retrieves those KV blocks and generates the response. Conventional disaggregated serving moves that cache in one direction only, from prefill to decode, and the AWS Machine Learning team says that one-way flow creates inefficiencies for multi-turn workloads.
After a response is generated, the decode node is still holding KV cache entries for both the prompt and the newly generated tokens. When the next turn arrives, the prefill node cannot retrieve those entries, so it has to recompute the KV projections for the previous response tokens.
Bi-directional KV transfer reverses the direction: the decode node returns previously computed KV to a prefill node on subsequent turns of the conversation. It is unglamorous plumbing, but it decides which node is holding the memory of a conversation.
AI-generated discussion · Different perspectives
Agentic workloads build on accumulated context as the agent reasons, invokes tools and folds in earlier results. Every new turn resends the conversation history — system instructions, prior prompts, tool results and model responses — so the same growing context is processed over and over.
The gain is conditional: reuse is governed by a configurable KV recompute threshold, and the transfer only helps where it actually clears that bar. It also asks something of the cache — the KV has to survive past the turn that produced it and travel in the reverse direction, coordinated by a KV-cache-aware router.
Worth noting what that coordinator is: a lightweight, stateful control-plane component sitting between the client, the prefill nodes and the decode nodes. It stores only block-location metadata, not KV tensors, and picks that metadata up from the response stream rather than querying the decode node separately — a sensible thing to keep off the data path.