Products & Tools

Sarvam's Token Factory Serves 450 Billion Daily Tokens From Indian GPUs

Sarvam AI launched Token Factory on November 16, 2026, serving open-weight models from Indian GPUs with 95.7% cache hit rates — and publishing benchmarks it loses.

Sarvam Opens India-Hosted Token Factory for Open-Weight Models
Sarvam Opens India-Hosted Token Factory for Open-Weight ModelsBru-nO / pixabay
By Sophie Lindqvist6 min read

Why it matters

  • Sarvam launched Token Factory on November 16, 2026, serving 450+ billion DeepSeek V4.1 Flash tokens daily from Indian GPUs.
  • Introductory pricing: Sarvam 105B at ₹10/₹25 per million input/output tokens; cached inputs at ₹0.25 per million.
  • 95.7% of DeepSeek V4.1 Flash prompt tokens are served from cache; G3 cut peak GLM 5.3 prefill nodes from 58 to 34.
  • MXFP8 quantization plus speculative decoding gave Sarvam 105B ~2.2x throughput and halved its B200 GPUs per replica.
  • Sarvam's own tests show Together faster on DeepSeek V4.1 Flash: 0.71s first token and 278 tokens/s versus 0.90s and 130.

Sarvam AI says its new Token Factory inference platform already serves more than 450 billion DeepSeek V4.1 Flash tokens per day from GPU clusters inside India. The company launched the service on November 16, 2026, for developers and enterprises, exposing its own and third-party open-weight models through a single OpenAI-compatible API.

The launch matters for two reasons. Indian enterprises that must keep data onshore now have a one-stop endpoint for popular open-weight models, and Sarvam is betting that the serving stack itself — caching, scheduling, quantization — can be the product, not just the models riding on it.

What does Token Factory actually serve?

The launch line-up covers seven models: Sarvam 105B, GLM 5.3, GLM 5.3 Flash, DeepSeek V4.1 Flash 1017, DeepSeek V4 Pro, Kimi K3.1 Pro and Gemma 4 31B. The API documentation says Token Factory uses the same API key, SDK and credits as Sarvam's other APIs: users switch models by changing the model field, and OpenAI-compatible clients can simply point at https://api.sarvam.ai/v2.

That compatibility choice is deliberate. It lowers the migration cost for teams already built on OpenAI-style tooling, which is most of the market.

Not every model runs the same way. DeepSeek V4 Pro and Kimi K3.1 Pro are enabled per organization under Provisioned Throughput, while the other launch models are listed as serverless. Customers can choose shared or dedicated capacity, and Sarvam also sells its inference optimization stack for on-premises deployment — it says that arrangement already runs for critical institutions.

The takeaway: one API, seven models, and a range of deployment modes from pay-per-token to fully on-prem.

What does it cost, and where does the data sit?

Introductory pricing for Sarvam 105B is ₹10 per million input tokens and ₹25 per million output tokens. The Standard endpoint for DeepSeek V4.1 Flash 1017 costs ₹14 and ₹55 respectively, and both models carry cached-input rates of ₹0.25 per million tokens. The catalogue says introductory pricing runs through the end of 2026, with all rupee rates excluding GST.

The data-handling terms are the sharper selling point for regulated customers. Sarvam's product page says inference runs on Sarvam-operated GPUs in India rather than being forwarded to the original model developers' overseas APIs. Under an agreed zero-data-retention arrangement, prompts and responses are not retained, and operational logs are scoped separately. Customer data is not used to train models for other customers, and enterprise deployments carry a contractual SLA guaranteeing 99.95% uptime.

For Indian banks, government bodies and anyone with data-residency obligations, that combination — local GPUs, no retention, a hard uptime number — is the pitch.

How does the caching machinery work?

The interesting engineering sits under the hood. Token Factory separates prefill, the phase that reads the full prompt, from decode, the phase that generates tokens one at a time, and runs them on different GPU pools. That separation keeps a new long coding prompt from stalling active decoding sessions, which can happen when both phases share a GPU. A cache-aware control plane routes each turn toward its existing context, and admission counts only uncached tokens.

A second cache tier handles agent workloads. Moonshot's Mooncake-based G3 store is accessible to every prefill server and retains prompts for hours or days, depending on traffic. GPU memory alone holds a prompt for about four seconds, falling to about two under load. That gap matters for agents that return after a tool call with almost the same prompt — without the store, their context would have evaporated before they came back.

The numbers Sarvam reports are substantial:

  • 95.7% of DeepSeek V4.1 Flash prompt tokens are served from cache.
  • G3 reduced peak GLM 5.3 prefill-node requirements from 58 nodes to 34.
  • Moving Sarvam 105B from BF16 weights to MXFP8, an 8-bit format, plus speculative decoding, increased output throughput about 2.2-fold and cut each replica from two B200 GPUs to one — without reducing accuracy, Sarvam says.

One governance detail stands out: an orchestrator agent assigns optimization experiments to sub-agents, but an engineer approves every change before deployment. The cache tier plus per-model tuning is where the GPU savings come from.

How fast is it, really?

The benchmark picture is mixed, and to its credit Sarvam published the unfavorable comparisons too.

The strong results first. In a 72-hour median test from an AWS Mumbai client with a 10,000-token prompt, Gemma 4 31B returned its first token in 0.40 seconds and streamed at 308 tokens per second. The only other provider serving that model managed 2.00 seconds and 193 tokens per second — and Sarvam used that provider's best tier for each metric.

A separate Sarvam test of Kimi K3.1 Pro on coding-agent traces with 16 B200 GPUs produced 6,177 tokens per second per GPU at roughly 70 tokens per second per user, against 3,213 on InferenceX's published curve.

Then the weaker one. In Sarvam's own DeepSeek V4.1 Flash latency test with a 10,000-token prompt, Together returned the first token in 0.71 seconds; Sarvam Standard took 0.90 seconds. A separate price-versus-speed table lists Together at 278 output tokens per second and Sarvam Standard at 130. Those latency figures were measured November 3–5, 2026, one request at a time, using each provider's fastest tier.

So the honest read is this: Token Factory wins clearly on some models and endpoints, and loses on throughput and first-token latency on others, DeepSeek V4.1 Flash against Together being the clearest loss.

Why is Sarvam selling the stack instead of just models?

Token Factory marks a positioning shift. Sarvam has moved from shipping models to selling the serving infrastructure around them — the routing, the shared cache, the quantization pipeline, the SLA. Its business case joins three threads: India-based processing for data-residency buyers, OpenAI-compatible migration for cost-sensitive ones, and GPU savings from caching and per-model tuning for everyone.

The published results show substantial gains in selected workloads, but also that throughput and latency advantages vary by model and endpoint. Buyers will need to benchmark against their own workloads rather than trust a single headline number — especially on DeepSeek, where the incumbent competition is currently faster.

The strategic bet is that inference economics, not model weights, decide who wins enterprise AI workloads in India. Token Factory's early 450-billion-token daily volume suggests demand exists; whether Sarvam's caching stack can close the throughput gap on every model is what will determine if the bet pays off.

Source: unite.ai

Share this article:

More from Sophie Lindqvist

Sophie Lindqvist

Show full bio

Staff writer covering regulation, policy and their impact on artificial intelligence and machine learning at AI In Context.

749 articles

Related articles

  1. Baseten Lands Four Open Models in Microsoft Foundry as First-Party Preview
  2. DeepSeek Open-Sources DeepInfer, the Engine Behind DeepSeek-V4.1-Flash
  3. Meta launches Muse Spark Open, its 'most capable US open-weight model'
  4. OpenAI Launches 'OpenAI for India' With Tata Data Center Deal
  5. Meituan open-sources 1.6-trillion-parameter agent model trained without Nvidia

« Previous articleNext article »