llm-d 1.0 arrives: Google's open inference stack cuts median TTFT 85%

Google has shipped llm-d 1.0, the first stable version of its open-source, Kubernetes-native distributed inference stack, and reports cutting median time-to-first-token (TTFT, the delay before a model starts emitting output) by more than 85% while lifting throughput 2.4x on an Ironwood TPU agentic coding workload. The team announced the release at PyTorch Conference North America. The stakes are straightforward: self-hosted agent and reinforcement-learning (RL) workloads break under conventional round-robin load balancing, and llm-d turns separate vLLM or SGLang model servers into one coordinated pool running on NVIDIA, AMD or Intel GPUs or Google TPUs.

Google published a benchmark on a 17,000-node multi-region GKE (Google Kubernetes Engine) deployment as part of the release. Every performance figure below is vendor-reported by Google and llm-d adopters, not independently verified.

What is llm-d 1.0?

Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA founded the project in 2025, with Red Hat, IBM and Google as lead maintainers. It counts more than 500 contributors from over 25 organizations, holds CNCF Sandbox status (accepted March 12, 2026) and carries an Apache 2.0 license.

Core capabilities — prefix-aware routing, KV cache offloading, prefill/decode disaggregation (splitting the compute-heavy prefill phase from the memory-bound decode phase), wide expert parallelism and batch inference — already shipped in earlier versions. Version 1.0 focuses on what IBM calls day 2 operations: running those features reliably in production.

What is new in version 1.0?

The stable release adds six components:

• Multi-tenant quality of service: priority and fairness-aware flow control prevents bursty agents from starving latency-sensitive requests.
• Stable foundation: v1 APIs and a hardened request path, validated on multiple clouds and accelerators.
• Zero-downtime upgrades: router high availability with hot standby, plus DisaggregatedSet, which rolls prefill and decode pools together on matching versions without dropping sessions.
• Agent-aware serving: session-aware routing, turn-priority scheduling and native Responses API state.
• Async inference: llm-d-async-processor pulls jobs from Google Cloud Pub/Sub or Redis into slack capacity.
• Expanded TPU support: TorchTPU runs PyTorch models natively inside upstream vLLM and SGLang.

How does the router work?

The llm-d router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas hold each cached prefix, then weighs load and predicted latency before picking a replica. Agents call one endpoint, and each step lands where its context already lives.

When accelerator memory fills, KV cache (the stored attention context a model reuses across requests) moves to host memory, then SSD or Managed Lustre, using vLLM's OffloadingConnector, Mooncake or SGLang's HiCache. Prefill and decode run as separate pools because prefill is compute-heavy and decode is memory-bound.

What do the benchmarks show?

Google's self-reported results span TPUs, GPUs and production platforms:

• Ironwood TPU (TPU 7x): prefix-aware routing plus host memory offload cut median TTFT by more than 85%; throughput rose 2.4x versus a plain Kubernetes Service.
• Gemini platform: prefix cache hit rate doubled from 35% to 70%; TTFT fell more than 35% for coding agents; P95 TTFT fell more than 50% under bursty chat via the GKE Inference Gateway.
• 17,000-node GKE benchmark: throughput scaled 2.9x across 3 clusters, with 99.9% success and under 1% overhead.
• A3 Mega, 70B model: Lustre plus CPU RAM offload improved TTFT by roughly 40% on 50k-token prompts; end-to-end latency fell 30% and GPU-hours fell nearly 60%.
• RL time-slicing: accelerator duty cycle rose from about 40% to above 70% — meaning roughly 30% idle time remains.

Adopters report similar patterns. Snap reports prefix cache hit rates of 75 to 80%. Zenteiq reports over 4x aggregate throughput, from 1.47 billion to 5.97 billion daily tokens per 1,000 TPU v6e chips. An IBM demonstration served 3,000 concurrent coding agents at 5 to 10 times below API pricing.

How does llm-d compare with NVIDIA Dynamo and AIBrix?

llm-d competes with two other Apache 2.0 serving layers: NVIDIA Dynamo (v1.5.1, October 6, 2026) and AIBrix (v0.7.0, June 16, 2026).

| Feature | llm-d 1.0 | NVIDIA Dynamo | AIBrix | ||---|---|---|---| | Governance | CNCF Sandbox | NVIDIA-led (ai-dynamo org) | vLLM project org, ByteDance origin | | Engines | vLLM, SGLang | SGLang, TensorRT-LLM, vLLM | vLLM, SGLang, TensorRT-LLM | | Accelerators | NVIDIA, AMD, Intel GPUs; Google TPU | NVIDIA GPUs (Blackwell, Hopper, Ada, Ampere) | Mixed GPUs (heterogeneous serving) | | KV cache tiers | HBM, CPU, SSD, Managed Lustre | GPU, CPU, SSD, remote storage (KVBM) | Distributed KV cache | | P/D disaggregation | Yes, DisaggregatedSet | Yes | Yes | | Headline result | 2.9x throughput, 17,000 nodes (self-reported) | 2x faster TTFT on Qwen3-Coder 480B (Baseten) | Not disclosed | | GitHub stars | 4.8k | 8.3k | 5.1k |

Star counts date from October 20, 2026. These projects are not strict rivals: CoreWeave runs llm-d alongside Dynamo, and NVIDIA contributes Dynamo components to llm-d.

How do you deploy it?

On Google Cloud, GKE Inference Gateway runs the llm-d router as a managed service, and it can span clusters and regions with global failover. KEDA and HPA scale pools on queue depth and predicted latency; Kueue enforces per-model quotas; verified Helm charts deploy TPU and GPU pools behind one endpoint.

Google recommends a staged path: start with prefix-aware routing in front of an existing vLLM deployment, then add flow control, tiered KV offload and disaggregation. Published runs can be compared on Prism, the project's benchmark site.

Why does this release matter?

Agent workloads multiply request volume per user task, and every hop that misses a cached prefix re-computes context. A Gemini-documented jump from 35% to 70% cache hits, and Zenteiq's 4x token throughput on identical TPU inventory, quantify why cache-aware routing is becoming the default architecture for self-hosted inference. With the RL time-slicing result still leaving roughly 30% of accelerator cycles idle, the project's next frontier is utilization — and adopters running agents at a fraction of API pricing now have a stable, v1-API foundation to build against.