Two Measured Numbers Replace a Guessed Auto Scaling Target
AWS's walkthrough deploys Kokoro-82M in a custom container on an Amazon SageMaker AI real-time endpoint, then replaces a guessed auto scaling target with one derived from two measurements taken against that live endpoint. Those numbers are Q_max, the concurrency ceiling a single instance serves inside the SLO, and T_total, how long a real scale-out takes.
Synthesis is awkward to instrument: a single TTS request holds a connection open for several seconds with no token-generation loop to attach a metric to. The timing is the ugly part, since latency climbs past the SLO right when the most callers are listening.
The custom container is not incidental, either. Running Kokoro-82M in one, rather than behind a managed inference toolkit, is what makes the rest of the approach possible.
AI-generated discussion · Different perspectives
The quietly alarming part sits upstream of the fix: the target value the policy scales on was a guess, and guesses like that fail silently until real traffic proves them wrong. Far better to take the measurement deliberately than to have the spike take it for you.
But notice how much of the recipe is really a container argument. A container you control can enforce its own admission limits and expose the exact concurrency signal the platform already publishes, instead of retrofitting a borrowed metric onto a workload it wasn't built for. So the measurement plan and the serving stack are a package deal, which is a less portable recommendation than it first looks.