Products & Tools

AWS Adds KV Cache-Aware Routing to SageMaker AI Endpoints

AWS added KV cache-aware routing to SageMaker AI real-time endpoints on November 17, 2026, lifting cache hit rates from 3–48% to 80–93% in its tests, at a cost of 14–19 ms of added routing overhead.

SageMaker AI Routes Requests Using Container-Reported GPU Cache State
SageMaker AI Routes Requests Using Container-Reported GPU Cache Stateblickpixel / pixabay
By Elena Vasquez5 min read

Why it matters

  • AWS introduced KV cache-aware routing for Amazon SageMaker AI real-time endpoints on November 17, 2026.
  • In AWS's 30-minute Llama 3.1 70B Instruct tests, cache hit rates rose from 3–48% under random routing to 80–93% with KV-aware routing.
  • AWS reports up to 97% lower median time to first token and 234% higher throughput than random routing in long-prompt tests under load.
  • In a two-instance Qwen3-4B-Instruct-2507 comparison, average CloudWatch OverheadLatency ran 14 to 19 milliseconds higher with KV-aware routing.
  • The feature requires at least two instances or inference-component copies, plus vLLM 0.33.0 or later in the AWS vLLM Deep Learning Container or server-sagemaker-cuda-v2.8.0.

By Elena Vasquez

Amazon Web Services added KV cache-aware routing for Amazon SageMaker AI real-time endpoints on November 17, 2026, and in AWS's own tests the feature pushed cache hit rates from 3–48% up to 80–93%.

That is the headline number. It comes with a caveat that matters more than the release note: AWS also measures the cost of the routing layer itself, and in one of its comparisons the routing overhead exceeded the savings on short work.

The scene behind the figure is a 30-minute load test running Llama 3.1 70B Instruct across seven ml.p5.48xlarge instances, with 20 or 100 simulated users pushing multi-turn chats and document question-answering sessions through prompts of roughly 64,000 tokens.

The stakes here are the unit economics of long-context inference. Every token an instance has to recompute is GPU time a customer pays for twice, once to process and again to serve.

What does KV cache-aware routing actually change?

A KV cache holds prior prompt computations in GPU memory, so a model can reuse them instead of repeating the initial processing, the prefill, of matching input tokens.

That reuse is why placement matters. Send a request to the wrong instance and the GPU recomputes context another GPU already holds.

The workloads where that waste compounds fastest are chat assistants, retrieval-augmented generation, and agents carrying long system prompts.

SageMaker AI's existing prefix-aware strategy infers cache location from a fixed request prefix. It cannot detect evictions, and early turns in a growing conversation can hash differently and land on separate instances.

KV-aware routing instead uses events from the model container, emitted whenever cache entries are stored or evicted.

The service maintains a managed index of one-way truncated hashes mapped to instances or inference-component copies. It does not store prompts, token IDs, request payloads, or KV tensors — a detail that lands with teams doing data-handling reviews.

Each incoming prompt is tokenized and matched against that index. The endpoint weighs the longest cached match against instance load and skips candidates that are too busy.

If the lookup is slow, no useful match exists, or every matching instance is busy, the request falls back to least-outstanding-requests routing and is still served.

Scaling shapes the behavior too. Scale-in removes index entries for removed instances, while new or replacement instances start with empty caches, creating temporary cache-rebuild and cold-start latency considerations.

How large were the benchmark gains?

AWS reports up to 97% lower median time to first token and 234% higher throughput than random routing in long-prompt tests under load.

The measured cache hit rate moved from 3–48% with random routing to 80–93% with KV-aware routing on those same tests.

AWS is upfront that the results are workload-dependent. Short prompts gained least, because they leave less prefill computation to reuse in the first place.

What does the routing overhead cost?

The lookup adds tokenization and index-processing overhead: a few milliseconds for short prompts and more for longer ones.

Standard text-only requests pass the computed tokens to the model, which avoids a second tokenization pass.

The sharper measurement came in AWS's separate two-instance Qwen3-4B-Instruct-2507 comparison. Average CloudWatch OverheadLatency, which measures time SageMaker AI spends outside the model container, was 14 to 19 milliseconds higher with KV-aware routing, even as model latency and time to first token improved.

So the routing bill is real and measurable, and it scales with prompt length rather than staying flat.

What does it take to turn on?

At launch, the feature requires one of the following:

  • The AWS vLLM Deep Learning Container for SageMaker AI with vLLM 0.33.0 or later
  • The Amazon Linux build server-sagemaker-cuda-v2.8.0 or later

Prefix caching and KV cache events must also be enabled. Customers set SM_VLLM_KV_EVENTS_CONFIG on the model and RoutingStrategy to "KV_AWARE" in the production variant, without changing client calls.

The feature supports GPU-backed real-time single-model and inference-component endpoints, including dynamic Low-Rank Adaptation adapters.

The routing requires at least two instances for a single-model endpoint, or two inference-component copies. With one destination, the setting has no routing effect at all.

Requests must use OpenAI-compatible Completions or Chat Completions payloads through supported invocation APIs, and InvokeEndpointWithBidirectionalStream is unsupported.

What can operators see, and what can't they?

KV-aware routing adds no dedicated metrics or customer-account routing-decision logs at launch. Fallbacks are not exposed in the SageMaker AI console or in metrics.

That leaves indirect signals. Customers can inspect usage.prompt_tokens_details.cached_tokens after enabling prompt-token details, and monitor container metrics such as cache hit rate and time to first token through detailed observability and CloudWatch.

Where it runs, and what AWS recommends

The feature is available in all commercial AWS Regions that support SageMaker AI real-time inference with a supported container, at no additional charge beyond the endpoint's instances.

AWS continues to describe prefix-aware routing as "the simpler option for stable shared prompts, short contexts and moderate load," while KV-aware routing addresses longer reusable context, memory pressure and frequent eviction.

The trade is concrete and per workload: recovered GPU prefill work versus added per-request routing overhead.

For teams running long-context agents and document pipelines, AWS's numbers favor the new path. For teams with short prompts and a single instance, the older setting stays simpler and cheaper.

The unresolved piece is observability. Until AWS exposes routing-decision logs and fallback counts, operators will have to infer from container metrics and cached-token counts whether the routing layer is actually earning its 14 to 19 milliseconds, and that calculation will only work for workloads that genuinely hold long context.

Source: unite.ai

Share this article:

More from Elena Vasquez

Elena Vasquez

Show full bio

Market editor covering technology and product news in artificial intelligence and machine learning at AI In Context.

708 articles

Related articles

  1. AWS Reports 2.05x Throughput Gain From NVIDIA Dynamo on SageMaker HyperPod
  2. AWS Lets Multiple Models Share One GPU in SageMaker
  3. A Busy GPU Is Not a Productive GPU: New Scorecard Rewrites AI Inference Metrics
  4. OpenAI cuts GPT-6 agent costs with smarter prompt caching
  5. OpenAI Taps Cerebras for 750MW of Compute to Speed Up ChatGPT

« Previous articleNext article »