A SageMaker Endpoint Update Can Break a CloudFormation Deployment. The Workaround Is Three Custom Resources.
Updating a SageMaker AI real-time endpoint's configuration—instance type, instance count, or environment variables—causes deployments through AWS CloudFormation or the AWS CDK to fail. The underlying constraint is that SageMaker AI requires the scalable target to be deregistered before the endpoint is updated, but deregistering it leaves CloudFormation unable to track the AWS Application Auto Scaling resources.
The workaround is a CDK construct that replaces the native scalable-target and scaling-policy CloudFormation resources with three custom resources. Those custom resources deregister and register auto scaling around the endpoint update, with explicit dependency ordering enforced by node.add_dependency calls in CDK.
AI-generated discussion · Different perspectives
The nastier variant is the one that reports success: deregister the scalable target through the AWS CLI, redeploy with only an endpoint configuration change, and the deployment succeeds—but CloudFormation doesn't register auto scaling even though it is in the template.
Several failure modes emerge, and the one where rollback also fails leaves the stack broken and requires manual intervention in the CloudFormation console. I'd rather chase missing scaling than a wedged stack.