Models

AWS Adds Full Fine-Tuning to Serverless SageMaker, With Catch

AWS's serverless SageMaker now trains every model weight, hitting 90% semantic correctness in coding tests — but aggressive learning rates crashed MMLU accuracy to 3.6%.

AWS Brings Full-Weight Training To Serverless SageMaker Customization
AWS Brings Full-Weight Training To Serverless SageMaker CustomizationLouisBauer / pixabay
By Sophie Lindqvist5 min read

Why it matters

  • AWS announced FFT support for SageMaker AI Serverless Model Customization on November 18, 2026, reporting 90.0% semantic correctness on a 100,000+ example coding test
  • Learning rates of 5e-5 or higher caused MMLU accuracy to fall to 3.6% and MATH scores to drop to 2–3%
  • AWS tested with a Qwen3-4B base model on Magicoder-Evol-Instruct-110K, split 99,000/5,500/5,500
  • A safer baseline combined a 2e-5 learning rate, 5% warmup, and global batch size of 8
  • Supported open-weight models include DeepSeek R1 Distill Qwen 1.5B, Google Gemma 4 26B A4B, and NVIDIA Nemotron 3 Nano 30B (A3B)

Amazon Web Services announced on November 18, 2026 that SageMaker AI Serverless Model Customization now supports full fine-tuning, and reported 90.0% semantic correctness in a coding experiment built on more than 100,000 training examples. The launch removes GPU provisioning and cluster setup from the customer's to-do list — but AWS's own test data shows the method can wreck a model's general knowledge if you push the learning rate too hard.

The distinction matters because the two customization methods work very differently under the hood. Full fine-tuning (FFT) updates every single weight in the model. LoRA, the alternative AWS already offered, freezes the pretrained weights and inserts small low-rank adapter matrices instead, which cuts GPU capacity requirements but confines updates to a smaller representational space.

AWS's argument for the new option is that full-rank training can capture more complex domain adaptations as datasets grow — that the benefit comes from where the updates live, not just from having more trainable parameters. Switching between the two methods is deliberately trivial: in AWS's SageMaker Python SDK example, you replace TrainingType.FULL with TrainingType.LORA and leave the rest of the job configuration untouched.

Customers still supply the base model, the dataset, and the hyperparameters. What the serverless layer takes over is compute allocation, checkpoint management, and the distributed-training plumbing that normally makes deep customization an infrastructure project. The takeaway: serverless changes who runs the machinery, not how much of the model gets retrained.

How AWS tested it

AWS compared FFT against LoRA using supervised fine-tuning of a Qwen3-4B base model, with identical prompts, targets, and evaluation code across both methods. The training corpus was Magicoder-Evol-Instruct-110K, a dataset of coding instruction-response pairs covering algorithm design, debugging, data structures, and systems programming.

The data split was conventional but large:

  • 99,000 records for training
  • 5,500 for validation
  • 5,500 held out for testing

The evaluation stack separated task performance from knowledge retention, which is where most fine-tuning stories get interesting. Amazon Nova Pro acted as an LLM judge, reading generated code alongside the original prompt to score logical correctness. PRIMECODE measured similarity to reference solutions using deterministic BLEU, ROUGE-1/2/L, and F1 metrics. MMLU tested general knowledge across 57 academic subjects — the canary for whether specialization has eaten the model's broader abilities.

AWS reported that its semantic-correctness score peaked at three epochs and that longer training degraded every variant tested. The radar comparison placed FFT ahead of both LoRA and the base model on every displayed quality dimension, with the largest leads on BLEU and ROUGE-2. That spread suggests the advantage wasn't an artifact of one metric. On the headline task, FFT delivered the 90.0% semantic correctness figure.

The company said it ran tens of customization experiments across multiple base models, varying learning rate, batch size, warmup ratio, epoch count, and dataset scale. Each run, from job submission through evaluation, took a few hours. Resulting models were automatically registered as versioned Model Packages in SageMaker Model Registry, ready for review or deployment.

What can go wrong?

Plenty, by AWS's own account. The company was candid that FFT was not universally superior: both FFT and LoRA underperformed the base model on MATH exact match, and a separate LLM-as-Judge comparison actually favored LoRA at 81.5%, against 79.5% for FFT.

The sharper warning concerns learning rates. At 5e-5 or higher, AWS measured MATH scores falling to 2–3% and MMLU accuracy collapsing to 3.6%. That's catastrophic forgetting in action — domain-specific training overwriting pretrained knowledge wholesale, leaving a model that's worse at almost everything, including its new specialty.

AWS's risk map identified 2e-5 as the safer learning rate for large datasets. Among the configurations tested, a baseline combining that rate with 5% warmup and a global batch size of 8 best preserved general knowledge as measured by MMLU.

Batch size turned out to matter too, in a slightly counterintuitive way. At a fixed learning rate, smaller batches produce noisier updates that effectively behave like a higher learning rate, raising the forgetting risk. Larger batches reduce that noise but also reduce the number of optimization steps, which can leave the model underfit. The practical lesson: the knobs interact, and tuning one in isolation can quietly break another.

Which models, and what it costs you

SFT with FFT is available for open-weight models including DeepSeek R1 Distill Qwen 1.5B, Google Gemma 4 26B A4B, and NVIDIA Nemotron 3 Nano 30B (A3B), per AWS's supported-model list.

The FFT documentation lays out the trade-offs plainly. The method uses the same dataset formats as LoRA but requires more compute and longer training runs. It produces larger artifacts, since you're storing full model weights rather than small adapters. It's also incompatible with continuous customization. The one inference-side win: no adapter overhead at request time.

Why this launch lands now

The context here is a market-wide argument about how much customization foundation models actually need. LoRA-style adapters became the default because they're cheap to train and cheap to serve, but they cap how far a model can move from its pretrained behavior. AWS is betting that as datasets grow, enough customers will want full-rank adaptation to justify the heavier bill.

The economics are what will decide adoption. Serverless removes the barrier of infrastructure management, but it doesn't reduce the number of weights being trained or the compute that implies. Teams adopting FFT on SageMaker will still have to judge whether task-specific gains outweigh heavier training costs and any erosion of general capabilities — and AWS's own numbers, with MMLU bottoming out at 3.6% in bad configurations, suggest that judgment won't always be easy.

Source: unite.ai

Share this article:

More from Sophie Lindqvist

Sophie Lindqvist

Show full bio

Staff writer covering regulation, policy and their impact on artificial intelligence and machine learning at AI In Context.

777 articles

Related articles

  1. AWS Lets Multiple Models Share One GPU in SageMaker
  2. AWS Reports 2.05x Throughput Gain From NVIDIA Dynamo on SageMaker HyperPod
  3. AWS Launches SageMaker JumpStart Leaderboard Ranking 70+ Models
  4. AWS Puts OpenTSLM Time-Series Reasoning Model on SageMaker JumpStart
  5. AWS folds synthetic data, auto-labeling and human review into SageMaker AI

« Previous articleNext article »