llm-d 1.0 arrives to tame production AI inference at scale

The open-source llm-d project shipped version 1.0 on 13 November 2026, aiming to make large language model inference reliable and affordable enough for everyday production use. The project, founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, is now governed by the Cloud Native Computing Foundation (CNCF) and counts more than 500 contributors from more than 25 organisations.

The stakes are straightforward. Companies have moved past AI demos, and the hard part is now serving models to real users without burning through GPU budgets. llm-d targets exactly that layer: distributed inference, the machinery that answers requests once training is done.

What's actually new in 1.0?

Version 1.0 adds four headline features:

• Multi-tenant quality of service, so operators can run mixed workloads on shared fleets without latency-sensitive requests getting crowded out.
• Zero-downtime operations, covering traffic management during upgrades and maintenance.
• A stable configuration API, which gives operators a dependable base for day-to-day work.
• What the project calls agent-aware serving, designed for fleets handling both user requests and less predictable agent-based tasks.

The software already shipped a solid toolbox before this release: prefix-cache and load-aware routing, KV cache offloading, prefill and decode disaggregation, expert parallelism, and batch and asynchronous inference. It supports multiple model servers, including vLLM and SGLang. The takeaway: 1.0 is less about new tricks and more about making the existing stack stable enough to bet a production service on.

Why agentic workloads change the math

A central problem for AI operators is the resource appetite of agentic workloads. These systems repeatedly access and reuse context across multiple turns and parallel tasks. In these settings, managing context can consume more resources than producing the final output, and that pressure lands on latency, throughput, memory use and cost.

That's the gap llm-d's backers say they're closing. The prefix-cache and cache-aware routing features exist precisely because input-heavy traffic — the norm for agents — is where costs explode. The plain-words version: when a bot rereads the same context a hundred times, serving it cheaply depends on not recomputing that context every time.

What do the numbers show?

The launch came with two demonstrations that supporters say show the cost and throughput gains possible with the software.

IBM Research said open frontier models paired with llm-d let it explore quantum error-correction hardware at 11.5 times lower inference cost than an all-proprietary set-up. The configuration achieved a prefix-cache hit rate of about 98% and a blended serving cost of roughly USD $0.11 per million tokens.

In the second demonstration, researchers used llm-d to deploy GLM 5.2, a roughly 753-billion-parameter open-weight model, on NVIDIA GPUs. On an agentic benchmark running hundreds of sessions in parallel, the set-up delivered more than 6.6 million output tokens per minute at peak and served up to 3,000 concurrent coding agents with zero preemptions.

The project's contributors said that, at current cloud rental rates, that deployment would have produced per-token costs five to 10 times lower than equivalent commercial API pricing. They noted the biggest savings appeared in input-heavy traffic patterns — again, the signature of agentic workloads.

Who is behind it and who is using it?

The project spans model developers, cloud providers, infrastructure vendors and enterprises running distributed inference systems. Beyond the founding companies, several operators already run llm-d in live environments:

• DigitalOcean relies on llm-d as the distributed inference software behind its serverless inference and managed agents services.
• JetBrains runs llm-d in production behind its Junie Lite plan, where it handles a significant share of traffic.
• Roblox uses the project as a Kubernetes-native foundation for disaggregated serving and cache-aware routing across cloud and on-premises GPUs, without being tied to a single inference engine.

That last point matters for buyers. Engine-agnostic serving software gives companies leverage over GPU vendors and inference startups alike, and it keeps their options open as vLLM, SGLang and other servers evolve.

What are the backers saying?

Chris Aniszczyk, Chief Technology Officer at CNCF, framed the release as part of a wider shift in cloud-native computing. "AI inference is quickly becoming a core cloud native workload. llm-d 1.0 helps organizations meet this shift at scale by bringing together open source inference capabilities with the requirements that matter in scaling production to then meet the demands of agentic workloads. It's an important step toward making high performance open source inference more operationally mature and accessible across the cloud native ecosystem," he said.

Brian Stevens, Senior Vice President and AI Chief Technology Officer at Red Hat, pointed at the post-experimentation phase enterprises now face. "As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads. Open source collaboration is essential to solving these day-two operational challenges at scale. With llm-d 1.0, the community is delivering a standardized, hardened core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation," Stevens said.

Priya Nagpurkar, Vice President of AI Native Systems at IBM Research, put it in terms of infrastructure value. "At IBM, we see firsthand that the challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. llm-d 1.0 represents an important milestone in creating an open foundation for distributed inference to help organizations get more value from their AI infrastructure and more control on the AI capabilities they depend on. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs, and accelerating innovation across the AI ecosystem," Nagpurkar said.

Why this matters

The release lands in a market where inference costs, not training costs, increasingly decide which AI products survive. Commercial API pricing from the major model vendors sets the benchmark, and llm-d's contributors claim per-token costs five to 10 times below that bar on input-heavy agentic traffic — if those numbers hold for typical enterprises, they change build-versus-buy calculations for anyone running agents at scale.

The founding organisations also framed the release as an answer to operational issues that surface after initial AI experimentation: balancing competing traffic across shared GPU fleets, maintaining service during software changes, and making better use of existing hardware. With backing from Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, plus production deployments at DigitalOcean, JetBrains and Roblox, llm-d now has the vendor weight to compete for enterprise inference workloads — and as agentic traffic grows, its cache-heavy architecture is positioned to grow with it.