Amazon SageMaker AI opens cross-Region failover routing
AWS made cross-Region inference routing generally available in SageMaker AI on November 19, 2026, centralizing failover, retries and access controls across Regions and accounts.

Why it matters
- AWS made cross-Region inference routing generally available in Amazon SageMaker AI on November 19, 2026.
- Each router supports up to 20 routing targets and 10 destinations per target; real-time payloads can reach 6 MB.
- Health scores ignore model latency; AWS says failover can begin within about a minute of sustained errors, but customers cannot set weights, thresholds or timing.
- Four instances at 50 invocations per second each and a 20% cross-Region cap allow 40 requests per second, rising to 60 with two more instances.
- AWS charges no extra routing fee, but customers still pay standard SageMaker AI and CloudWatch rates and fund spare capacity in another Region.
By Sophie Lindqvist
Amazon Web Services made cross-Region inference routing generally available in Amazon SageMaker AI on November 19, 2026, allowing applications to call one router for model-serving resources across AWS Regions and accounts when local capacity runs short. The release addresses a gap in managed model serving: a SageMaker AI endpoint runs in one Region and one account, and autoscaling cannot always add the required instance type. Requests can then throttle or time out even when the same model has spare capacity elsewhere.
AWS says the launch puts failure detection, safe retries, recovery handling and cross-account access into a managed Regional inference router. Customers previously had to build those pieces themselves whenever a managed endpoint faced a capacity shortfall. The router does not remove that underlying compute requirement, because its destinations must already exist and remain running.
What does the router actually do?
The service gives each logical model a named routing target. A target can mix real-time endpoints and inference components across Regions and accounts, or contain asynchronous endpoints only. The application invokes the target through the router.
AWS says models, endpoint configurations, instance pools, containers and autoscaling remain unchanged. Direct InvokeEndpoint calls also continue to work, so customers can keep an existing integration path alongside the router.
AWS says: “The router does not create capacity, deploy models or scale endpoints, so every destination must already be running in its Region.” The service therefore reroutes demand around resources that customers already run.
The takeaway: AWS manages the failover path, but customers still manage the underlying capacity.
How does health-based routing work?
AWS says the router prefers healthy destinations in its own Region and spreads requests evenly across multiple healthy local destinations. Its health score uses recent requests measured in each destination’s Region. That lets the router evaluate remote resources using traffic handled where those resources run.
Server errors, throttles and timeouts lower the score, AWS says. Platform errors lower it the most, while request-caused failures such as invalid input do not. Model latency is excluded, so an endpoint keeps its traffic when it returns successful responses, even if those responses are slow.
SageMaker AI publishes healthy, overloaded and degraded states to CloudWatch. AWS says overloaded destinations lose part of their traffic, while degraded destinations retain only a small share so the router can detect recovery. The reduced share gives troubled endpoints a controlled route back toward normal service.
AWS says: “traffic begins moving within about a minute of sustained errors, or later for destinations with little traffic, and returns over several minutes after recovery.” The roughly one-minute figure therefore does not apply equally to busy and lightly used destinations.
Customers cannot set weights, fallback order, health thresholds or failover timing; the router controls those decisions automatically. If health information for remote destinations becomes stale, AWS says the router keeps traffic in its own Region.
The takeaway: the policy is automatic, and it treats slow success differently from platform failure.
Which failures trigger a retry?
For real-time inference, AWS says the router makes one additional attempt on a different destination only when the original model did not run. Its listed conditions include throttling, a 503 error, an unready model, a traffic-cap rejection and destination access denial.
The router never retries real-time timeouts, 500 errors or model errors. Those outcomes receive no second attempt under the router’s policy, leaving the application to handle the resulting failure.
For asynchronous requests, AWS says the router tries each eligible destination once. It starts with local destinations and continues until one accepts the request. Each destination gets one attempt, rather than the real-time path’s one additional attempt on another destination.
Each real-time response identifies the serving destination in InvokedRoutedResourceArn. The response also supplies a RoutingReason: local-preferred, capacity-available, failover or only-healthy.
OpenTelemetry metrics in CloudWatch show per-destination requests and errors, destination duration, router overhead, cross-Region network overhead and health. Asynchronous responses do not name the destination that accepted them, but the CloudWatch metrics still show where requests were placed.
The takeaway: retries cover selected pre-execution failures, while CloudWatch fills in the routing details applications cannot see directly.
How do capacity caps protect fallback capacity?
Overflow can push a receiving endpoint into throttling. By default, AWS says a real-time endpoint accepts any amount of cross-Region traffic. An optional cap lets its owner set a load-tested per-instance invocation rate and the maximum percentage available to other Regions.
The resulting allowance follows the number of instances in service, including changes made by autoscaling. In AWS’s example, 50 invocations per second per instance, a 20% cap and four instances permit 40 cross-Region requests per second. Adding two instances raises that allowance to 60 requests per second.
Only requests from other Regions count against the cap. Calls from the endpoint’s own Region are never throttled by this mechanism. All inference components on an endpoint share the same cap.
AWS says excess cross-Region requests are rejected before the model runs, then retried once elsewhere. If no destination can accept them, the application receives a throttling error. The ceiling limits how much overflow one endpoint must absorb.
The takeaway: spare capacity is protected by an owner-controlled limit, not guaranteed to be unlimited.
How does cross-account access work?
For cross-account destinations, AWS says both accounts must authorize requests. An endpoint policy can limit access to one named router. The router grants no additional endpoint access, and AWS says a caller can use only destinations it could invoke directly.
The takeaway: the router coordinates traffic, but existing account and endpoint policies still control access.
Where can customers deploy it?
AWS says the feature is generally available for real-time and asynchronous inference in most commercial AWS Regions. The exceptions are me-south-1 and me-central-1. It does not support response streaming.
Real-time requests have about one minute to receive a destination response, and real-time payloads can be up to 6 MB. Each router supports up to 20 routing targets, with up to 10 destinations per target.
The router itself remains Regional. AWS recommends creating routers in two Regions and switching between them client-side for entry-point resilience. Data-residency boundaries also need explicit configuration because a request may run in the router’s Region or in any registered destination’s Region.
The takeaway: regional coverage is broad, but customers still have to design around router failure and data location.
What does the feature cost?
AWS says there is no additional routing charge. Customers pay standard Amazon SageMaker AI endpoint and inference-component rates, whether requests arrive directly or through the router. They also pay standard CloudWatch rates for router metrics and queries.
According to AWS, the launch shifts failover engineering into SageMaker AI while leaving the central compute-economics tradeoff with customers. Continued service during a shortfall in one Region still depends on paying for and positioning usable inference capacity in another.
The next operational test is therefore concrete: teams must decide where to maintain paid spare capacity, how much overflow each destination can absorb and whether AWS’s roughly one-minute failover window meets their service needs.
Source: unite.ai
More from Sophie Lindqvist
Show full bio
Staff writer covering regulation, policy and their impact on artificial intelligence and machine learning at AI In Context.
823 articles
Related articles
- AWS Lets Multiple Models Share One GPU in SageMaker
- AWS Adds KV Cache-Aware Routing to SageMaker AI Endpoints
- AWS Reports 2.05x Throughput Gain From NVIDIA Dynamo on SageMaker HyperPod
- PolyAI's Dialog-RSN-1 Hits SageMaker JumpStart at $45 Per Host Hour
- AWS Turns Step Functions Into Agent Tools as GPT-6.1 Terra Hits Bedrock