AWS Lets Multiple Models Share One GPU in SageMaker

On November 6, 2026, Amazon Web Services added NVIDIA Multi-Instance GPU support to Amazon SageMaker Inference, letting multiple hardware-isolated models share a single accelerator on real-time endpoints instead of each deployment claiming an entire GPU. AWS's own benchmarks show why that matters: one partitioned H100 delivered 53% to 89% more output tokens per second than a whole A100-40GB in like-for-like serving tests.

The feature attacks two constraints of whole-GPU allocation. The accelerator count caps how many models an instance can host, no matter how little memory or compute each model actually uses. And any customer moving to a newer GPU must commit the entire device to a single deployment. Multi-Instance GPU, or MIG, gives each partition a fixed share of compute units and a dedicated block of memory, with the isolation enforced in hardware. Because each model stays resident on its slice, the setup also avoids the cold-start delay a multi-model endpoint incurs when it loads a model onto a shared device for first use.

The stakes are straightforward. Inference dominates the running cost of deployed AI systems, and GPU capacity remains scarce and expensive. Denser, isolated packing of models onto each accelerator translates directly into lower cost per token — provided the numbers hold up under real traffic.

Who helped build it, and what they claim?

Salesforce helped define the requirements and validated the initial implementation. Richard Chang, a Salesforce software engineering architect, said the company expected MIG to consolidate workloads safely, improve GPU utilization and reduce inference costs while preserving predictable performance and isolation. Salesforce is now preparing to validate the software with representative production workloads, which means those benefits remain expectations rather than reported production results.

How do developers actually carve up a GPU?

Two optional API fields control the feature, according to the SageMaker AI developer documentation. AcceleratorPartitionLayouts declares the carve on a production variant. AcceleratorPartitionType selects the profile an inference component uses.

AWS added no new API operations, no new resource types and no new AWS Identity and Access Management actions. Omit both fields and endpoint behavior stays exactly as before.

The chosen layout applies uniformly to every accelerator on every instance in the variant. In AWS's example, each of eight A100s on an ml.p4d.24xlarge is divided into mig-4g.20gb, mig-2g.10gb and mig-1g.5gb partitions, producing 24 partitions across the endpoint. Each inference-component copy occupies one requested partition. InvokeEndpoint keeps addressing models by component name, with no partition identifier. Carving the accelerators added roughly 17 seconds to provisioning in AWS's testing.

The scope has hard edges:

• Only real-time endpoints using inference components — SageMaker's construct for deploying several models to one endpoint with per-model compute settings — can be partitioned.
• Single-model endpoints, serverless endpoints and heterogeneous instance pools cannot.
• Asynchronous workloads are excluded, because asynchronous invocation cannot address an inference component.

What does the isolation actually cover?

Isolation covers GPU compute and GPU memory, not the whole host. Models on one instance still share host CPU, host memory, storage, networking and endpoint routing. A component requesting a profile absent from the layout gets rejected rather than handed a whole GPU. Changing a running endpoint's layout requires a new endpoint and traffic migration, and moving a component to another profile requires a new component.

What did the H100-versus-A100 test show?

AWS compared one mig-3g.40gb H100 partition against a whole A100-40GB, running Llama-3.1-8B-Instruct on production SageMaker real-time endpoints. Both configurations used one model replica, identical containers, tokenizers and client setups, with genai-bench driving five traffic shapes at concurrency levels from 2 to 512. NVIDIA's profile guide gives the tested H100 profile three of seven streaming-multiprocessor shares and four of eight memory units. Memory capacity is comparable between the two; compute allocation is not.

The results, as reported by AWS:

• 53% to 89% more output tokens per second from the H100 partition at the load where each scenario's cost benefit was largest.
• Inter-token latency 8.9% to 42.1% lower.
• Time to first token 5.6% to 47.5% lower.
• At the highest concurrency on 6,000-token and 8,000-token workloads, the A100 missed the serving deadline on one-third to one-half of requests; the partition failed none.

The cost picture is more nuanced. AWS presented its analysis as a case-by-case comparison, not a universal verdict. Above a workload-specific break-even point, the H100 partition delivered 18% to 34% lower cost per million output tokens. That break-even landed at concurrency 64 for 100-token prompts and concurrency 4 for 8,000-token prompts. Below those points, the whole A100 stayed cheaper, leaving short-input or low-concurrency workloads potentially better served by the older whole GPU. The unit-cost comparison divided on-demand instance prices by the number of tested units on each instance.

Which GPUs and regions are supported?

The launch covers four instance families, with partition counts and profiles determined by the accelerator:

Instance family	Accelerator	GPU memory	Max partitions per GPU
P4d	NVIDIA A100	40 GB	7
P5	NVIDIA H100	80 GB	7
G7	NVIDIA RTX PRO 4500 Blackwell	32 GB	2
G7e	NVIDIA RTX PRO 6000 Blackwell Server Edition	96 GB	4

The feature is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt) and Europe (Ireland), subject to regional instance availability and endpoint quota. No partition spans accelerators, so models that need a full GPU or multiple GPUs still require whole devices.

What are the catches?

Three stand out. Customers keep paying for the full GPU instance; partitioning can increase usable capacity but does not reduce the hourly price. Partitioned endpoints and their inference components do not publish GPU, CPU or memory utilization metrics to CloudWatch, though invocation and latency metrics remain available. And on P4d, the mig-1g.10gb profile requires NVIDIA's R525 driver or newer — AWS directs customers to a newer inference AMI because the default image does not ship it.

For capacity planning, the launch effectively turns each supported accelerator into several fixed GPU sizes. Denser hardware-isolated deployment arrives alongside full-instance billing, layouts that cannot be altered in place and reduced utilization telemetry. AWS's own benchmark makes the practical guidance clear: measure your actual traffic before carving anything, because the H100 partition won on performance and cost under heavier long-context loads in those tests, while the break-even results left a whole A100 cheaper for lighter ones.