AWS Adds KV Cache-Aware Routing to SageMaker AI Endpoints
AWS added KV cache-aware routing to SageMaker AI real-time endpoints on November 17, 2026, lifting cache hit rates from 3–48% to 80–93% in its tests, at a cost of 14–19 ms of added routing overhead.
