AWS Reports 2.05x Throughput Gain From NVIDIA Dynamo on SageMaker HyperPod
AWS says SageMaker HyperPod Inference with NVIDIA Dynamo reached 207 output tokens per second per GPU, 2.05 times the best qualifying aggregated result under its latency targets.

Why it matters
- AWS reported a 2.05x throughput gain for disaggregated over aggregated serving under its latency constraints, announced November 17, 2026.
- Disaggregated serving reached 207 output tokens per second per GPU at concurrency 80 versus 101 at concurrency 32 for aggregated.
- The Qwen3-235B-A22B-FP8 benchmark ran on 16 NVIDIA H200 GPUs across two p5en.48xlarge instances.
- A two-node Qwen3.5-397B-A17B-FP8 test retained 99.1% of single-node throughput: 126.4 output tokens per second per GPU.
- GLM-5.3 FP8 weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance.
Amazon Web Services announced on November 17, 2026, that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for managed multi-node and disaggregated model serving, and published benchmark results showing 2.05 times the highest qualifying aggregated output-token throughput per GPU under AWS's latency targets.
The number matters because it targets the two constraints that dominate the cost of serving large language models: fitting model weights and the reusable key-value (KV) cache into GPU memory, and keeping long prompt processing from slowing token generation when both stages share the same GPUs. AWS illustrates the memory pressure with GLM-5.3, whose 8-bit floating-point (FP8) weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance before KV cache and runtime overhead are counted.
What does disaggregated serving change?
Dynamo is an open-source distributed inference framework that works with vLLM, SGLang and TensorRT-LLM. Its disaggregated topology uses a frontend to route requests to two distinct worker pools: prefill workers, which process prompts, and decode workers, which generate tokens.
Those pools can scale and be tuned independently. Both still load the full model weights. KV cache moves between them over Elastic Fabric Adapter (EFA) networking.
Aggregated serving, by contrast, runs both stages in the same worker. In either topology, workers can run on one instance or span multiple instances using tensor and pipeline parallelism to divide the model across GPUs.
Dynamo's KV-aware router considers reusable prefix cache and active worker load when placing requests. AWS says prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid overloading individual workers.
HyperPod supplies the managed infrastructure around the framework:
- EFA networking
- Coordinated GPU scheduling
- Health monitoring for nodes, GPUs and EFA
- Node replacement
HyperPod manages the replacement itself while the orchestration components recreate affected workers.
How strong is the 2.05x claim?
AWS ran its headline comparison with Qwen3-235B-A22B-FP8 on two p5en.48xlarge instances, totaling 16 NVIDIA H200 GPUs, using Dynamo and TensorRT-LLM. AIPerf generated requests containing 4,000 input tokens and 200 output tokens.
AWS retained only measured points that met two latency conditions: 90th-percentile time to first token below one second, and median generation speed of at least 50 tokens per second per user.
Under those constraints, disaggregated serving reached 207 output tokens per second per GPU at concurrency 80. The highest qualifying aggregated result was 101 at concurrency 32.
The comparison therefore applies to latency-constrained throughput at different concurrency levels, not to an unrestricted throughput maximum for either topology. That distinction determines whether the figure transfers to another deployment: the gain reflects what each topology sustained while meeting AWS's latency targets, at the concurrency levels where each performed best.
A separate Qwen3.5-397B-A17B-FP8 test used Dynamo and vLLM to compare eight H200 GPUs on one p5en.48xlarge instance against four GPUs on each of two instances, holding the model, parallelism and workload configuration constant. AWS reported 126.4 output tokens per second per GPU for the two-node setup, or 99.1% of single-node throughput, at concurrency 32 with high prefix reuse.
The model fits on one instance and would not normally be split in practice. AWS used the test as a controlled baseline for the throughput cost of crossing a node boundary — roughly one percent in this configuration.
How do deployments configure it?
Users have two configuration paths.
Existing Dynamo users can bring a DynamoGraphDeployment manifest that describes the serving graph and manage storage, networking, pod placement, endpoints, TLS, registration and autoscaling themselves.
The InferenceEndpointConfig path uses dynamoSpec for managed endpoint setup and Kubernetes Event-driven Autoscaling (KEDA) for either serving topology. It defaults to vLLM, provides defaults for KV-aware routing and replica counts, and supports an OpenAI-compatible interface through SageMaker endpoints.
The add-on installs Dynamo's custom resource definitions and its serving and scheduling components. AWS Deep Learning Containers provide the runtime. On the managed path, the HyperPod Inference Operator stages model weights from Amazon S3 or FSx and generates the DynamoGraphDeployment. Grove coordinates related pod groups, and KAI Scheduler uses gang scheduling so each multi-node worker's pods are placed together. Adopting a supported Dynamo release requires updating the add-on.
What do the deployment examples show?
AWS's GLM-5.3 example makes the GPU-allocation trade-off concrete. Its aggregated configuration runs one model-worker replica across two ml.p5.48xlarge instances with 16-way tensor parallelism, plus two frontend replicas.
The disaggregated variant configures one prefill replica and two decode replicas, each spanning two instances, for six instances in total. The separate replica counts show how a deployment can devote different amounts of capacity to prompt processing and token generation instead of scaling a single combined pool.
That asymmetry is the operational point of disaggregation: a workload with long prompts and short outputs needs more prefill capacity, while a conversational workload with heavy generation needs more decode capacity.
AWS recommends a specific measurement sequence before tuning engines: verify EFA communication before benchmarking, test representative prompt lengths, output lengths and prefix reuse, sweep concurrency against service-level targets, and tune topology and worker allocation from measured bottlenecks before changing engine settings.
What remains the operator's job?
HyperPod now handles more of the coordination and recovery work, but the benchmark conditions and deployment examples show that managed operation does not remove the serving trade-offs. The useful configuration still depends on three workload-specific factors: measured prompt and generation behavior, the GPU allocation between worker pools, and the latency and cost priorities of the deployment.
The practical effect of the announcement is narrower than the headline multiplier suggests: AWS has automated deployment and failure recovery around Dynamo, while the decisions that produce the throughput gains — topology choice, pool sizing and concurrency targets — still come from measurement rather than defaults.
Source: unite.ai
More from Rebecca Stone
Show full bio
Correspondent covering companies, deals and strategy in artificial intelligence and machine learning at AI In Context.
728 articles
Related articles
- AWS Lets Multiple Models Share One GPU in SageMaker
- AWS Adds KV Cache-Aware Routing to SageMaker AI Endpoints
- AWS and OpenAI Sign $38 Billion Multi-Year Compute Deal
- AWS AgentCore Runtime Gets 8-vCPU Sessions and Filesystem Mounts
- A Busy GPU Is Not a Productive GPU: New Scorecard Rewrites AI Inference Metrics