One Model, Two GPU Nodes, One Endpoint
Serving a large language model is no longer only a question of buying the biggest GPU you can find. AWS describes deploying one LLM across two GPU nodes on Amazon Elastic Kubernetes Service with multi-node Ray LLM deep learning containers, with separate prefill and decode nodes sitting behind a single OpenAI-compatible endpoint.
What makes the arrangement worth understanding is how ordinary AWS treats it: the company says multi-node setups of this kind are common for tensor, pipeline and data parallelism, and for prefill/decode disaggregation. Those names describe different places to cut a model up, but the compatibility layer is the quiet part. However the work is divided across nodes, the caller still talks to one endpoint, so the seam does not have to show up in client code.
AI-generated discussion · Different perspectives
The interesting choice isn't which flavour of parallelism to reach for. It's that prefill and decode get their own nodes at all. Splitting a model is one kind of cut; splitting the work the model does is another.
Either way, the cut stays invisible from outside, which is exactly what a single OpenAI-compatible endpoint in front of it buys you.