AWS HyperPod Now Detects Hung Training Jobs, Cutting an $16,900 Stall to $530
AWS added training validation, Kubernetes failure monitoring and Slurm hang detection to SageMaker HyperPod on November 18, 2026, targeting silent GPU waste during large AI training runs.

Why it matters
- AWS announced three HyperPod reliability capabilities on November 18, 2026: training validation, a Kubernetes monitoring agent and Slurm hang detection.
- A 15-minute hang rule cuts a stalled 32-instance job's waste from about $16,900 to about $530, at roughly $66 per instance-hour in N. Virginia.
- The two new training checks added about 8 minutes to a deep-health-check run of 1 hour 28 minutes on two ml.p5.48xlarge instances.
- Slurm hang detection is off by default, requires an AMI from November 10, 2026 or later, and a missing rules file leaves jobs silently unmonitored.
- The optional NCCL integration polls each pod's RAS listener every 30 seconds and exposes TCP port 28028 on the pod network.
A 15-minute hang-detection rule can cut the cost of a stalled 32-instance GPU training job from about $16,900 to about $530, according to AWS cost figures published on November 18, 2026. That arithmetic anchors a set of three reliability capabilities Amazon Web Services announced for Amazon SageMaker HyperPod, its managed infrastructure service for training and fine-tuning large AI models.
The additions target a specific and expensive failure mode: numerical training faults, Kubernetes failures and stalled Slurm jobs that consume GPU capacity while individual nodes still appear healthy. As clusters scale into hundreds of accelerators, silent waste of this kind has become a direct cost line for AI labs and enterprises running long training runs.
What was missing from HyperPod's health checks?
HyperPod already monitors hardware errors, runs deep diagnostic checks and tests communication between instances, with automatic recovery for faulty nodes. Those controls share one limitation, which AWS now concedes explicitly: they do not show whether GPUs compute correct training results, whether a job is still progressing, or whether the Kubernetes services on which a job depends are working.
The stakes are concrete in AWS's own worked example. An eight-hour hang on 32 ml.p5.48xlarge instances — representing 256 NVIDIA H100 GPUs — wastes 2,048 GPU-hours and costs roughly $16,900 at the approximate on-demand HyperPod price of $66 per instance-hour in US East (N. Virginia). A 15-minute detection rule reduces that scenario's cost to about $530 by canceling the job so Slurm, the open-source cluster workload manager, can requeue it.
How does AWS now test that GPUs train correctly?
The first capability, MultiNodeTrainingValidation, extends HyperPod's deep health checks for clusters orchestrated by Amazon Elastic Kubernetes Service (EKS), using supported Elastic Fabric Adapter (EFA)-enabled NVIDIA GPU instance groups. EFA is AWS's low-latency network interface for tightly coupled HPC workloads.
It introduces two tests:
- The Tiny Megatron test runs a fixed Megatron-LM workload across all GPUs on each instance.
- The Megatron test runs the same workload across instances of the same type over EFA.
An instance fails if training loss becomes not-a-number (NaN) or infinite, or if the run does not finish within its time limit. NaN loss is a classic symptom of a GPU producing numerically invalid results even when the hardware passes standard diagnostics.
The check can run when an instance group is created or updated, or on demand for an entire group or selected instances through the StartClusterHealthCheck API. Results appear in CloudWatch logs and node labels. A failure labels the instance and, with automatic node recovery enabled, leads to a reboot or replacement; during provisioning, this happens before customer jobs can use the capacity.
The feature carries an operational price. In AWS's test on two ml.p5.48xlarge instances, the two training checks added about eight minutes to a full deep-health-check run lasting 1 hour and 28 minutes. Instances cannot accept workloads until the checks finish, so the additional provisioning time becomes part of capacity planning. AWS is also explicit about the limits of the guarantee: passing the fixed workload establishes that an instance trained correctly, but does not guarantee that a customer's own training job will converge.
What does the Kubernetes monitoring agent do?
The second capability, the HyperPod Kubernetes Resource Monitoring Agent, runs as one pod per EKS cluster and prefers CPU nodes so it does not consume GPU capacity. It watches:
- node conditions and allocatable GPU or EFA devices,
- critical infrastructure pods and required DaemonSets,
- HyperPodPyTorchJob and Kubeflow training jobs.
Warning and Normal Kubernetes events mark failures and recoveries, while structured JSON CloudWatch logs retain history and connect failed jobs to their pods and nodes. That linkage addresses a real operational pain: reconstructing which node or pod took down a job after the fact.
The agent only reports. It does not reboot nodes or restart jobs. Hardware remediation remains with the health monitoring agent and automatic node recovery, while the HyperPod training operator continues to handle job restarts — a deliberate separation of concerns.
An optional NCCL integration — NCCL being NVIDIA's collective communications library for multi-GPU work — requires version 2.24 or later. In AWS's implementation, the agent polls each participating pod's reliability, availability and serviceability (RAS) listener every 30 seconds and combines those reports into a healthy, degraded or unhealthy job-level state.
The integration has a security implication. Reaching the listener requires binding it to the pod IP, which exposes TCP port 28028 on the pod network. AWS's guidance is to apply a Kubernetes NetworkPolicy that limits port access to the agent while preserving traffic among the job's own pods before enabling monitoring.
How does Slurm hang detection work?
The third capability requires a HyperPod Slurm Amazon Machine Image released on or after November 10, 2026. It stays off by default, and operators enable it per job through the HYPERPOD_HANG_RULES environment variable, which points to a rules file readable at the same path on every compute node. No training-script or container changes are needed, and the feature works with PyTorch, TensorFlow, JAX and custom scripts.
Default detection watches job output and GPU activity, looking for stopped output while GPUs are idle but retain memory, or for longer output silence regardless of GPU state. Custom regular-expression rules can replace default detection, require periodic progress messages, or declare a hang immediately when an error pattern appears. Each approach has blind spots that AWS acknowledges: default detection can miss a job that keeps GPUs busy without progressing, while custom rules miss hangs their patterns do not describe.
A detection writes a HANG_DETECTED event to CloudWatch. The default notify action leaves the job running; the cancel action lets Slurm requeue the job up to the cluster's MaxBatchRequeue limit, although jobs submitted with --no-requeue remain canceled. This mechanism is separate from auto-resume, which restarts jobs that exit after hardware failures rather than jobs that remain allocated without making progress.
What are the failure modes of the failure detection?
Two caveats stand out in AWS's documentation. First, a missing or invalid rules file produces a CONFIG_ERROR event in CloudWatch and leaves the job unmonitored — without an error in sbatch, squeue or the job output. A silent monitoring gap is the exact problem these features exist to solve.
Second, the agent does not check rules-file ownership. AWS recommends restrictive file and parent-directory permissions so another user cannot replace the file and alter detection or cancellation for jobs that use it.
Taken together, the additions make training correctness and job progress visible alongside hardware health, reducing reliance on manual log reconstruction when capacity goes to waste. The trade-off is operational rather than technical: longer provisioning checks, CloudWatch monitoring and carefully tuned detection rules now form part of protecting GPU time — particularly when automatic cancellation must distinguish a genuine hang from normal checkpointing or evaluation gaps.
Source: unite.ai
More from Marcus Bennett
Show full bio
Senior reporter covering companies, deals and strategy in artificial intelligence and machine learning at AI In Context.
745 articles
Related articles
- AWS Lets Multiple Models Share One GPU in SageMaker
- AWS Adds KV Cache-Aware Routing to SageMaker AI Endpoints
- AWS Reports 2.05x Throughput Gain From NVIDIA Dynamo on SageMaker HyperPod
- A Busy GPU Is Not a Productive GPU: New Scorecard Rewrites AI Inference Metrics
- OpenAI audit finds ~30% of SWE-Bench Pro coding tasks are broken