Chips & Compute

MiTAC Splits AI Inference Across Two Servers With 2 TB Decode Memory

MiTAC and Skymizer propose splitting prefill and decode inference across two servers, with 2 TB of decode-side accelerator memory to run Kimi K3 on premises.

MiTAC Proposes Two-Server AI Inference Architecture at SC26
MiTAC Proposes Two-Server AI Inference Architecture at SC26Tho-Ge / pixabay
By Marcus Bennett5 min read

Why it matters

  • MiTAC announced a two-server inference architecture with Skymizer at SC26 on November 17, 2026, targeting the 2.8-trillion-parameter Kimi K3 on premises.
  • The proposed decode server pairs Intel Xeon 6 with eight Skymizer HTX301-4S cards at 256 GB each, totaling 2 TB of accelerator memory.
  • A 52U liquid-cooled rack demo integrates 12 G4826Z5 servers and 96 AMD Instinct MI355X GPUs, claiming 50% higher GPU density.
  • ZutaCore two-phase cooling supports above 4,000W TDP, above 250W/cm² heat flux and pPUE of 1.03.
  • The upcoming ORv3 44OU Helion rack with AMD is stated to scale to 32,768 Zen 6 CPU cores.

MiTAC Computing proposed a two-server AI inference architecture on November 17, 2026 at SC26, pairing a compute-heavy prefill server with a decode server carrying 2 TB of accelerator memory — a design intended to run models such as the 2.8-trillion-parameter Kimi K3 on premises. The announcement, made jointly with Skymizer, is part of a wider infrastructure portfolio that spans software orchestration, high-density liquid cooling, storage and an upcoming AMD rack.

The core idea is disassembly. Modern large language model inference runs in two stages: prefill, the compute-intensive phase in which the model processes the input prompt, and decode, the memory-intensive phase in which it generates tokens one by one. MiTAC's architecture separates these stages onto physically distinct servers, so each can be provisioned for its own workload profile rather than compromised into a single chassis.

What hardware does each stage run on?

The proposed prefill platform is the 8U G8825Z5, built on AMD EPYC 9575F or 9755 processors and AMD Instinct MI350X GPUs. Once prefill completes, the system transfers the resulting key-value cache (the stored intermediate attention states a model needs during generation) to the decode server over high-speed networking.

The decode side is where the memory budget lives. The proposed air-cooled 4U G4520G6 combines Intel Xeon 6 processors with eight Skymizer HTX301-4S accelerator cards, each carrying 256 GB of memory — 2 TB in total across the chassis. That figure is the headline number of the proposal, because memory capacity, not raw GPU count, is frequently the binding constraint for serving very large models locally.

MiTAC says the on-premises decode system is designed for complex, deep-reasoning agentic workloads, the multi-step inference patterns in which AI agents iterate over long contexts. The practical effect of the split is architectural: part of the scaling problem shifts from simply adding GPUs to balancing accelerator memory against the network link between the two stages.

Why does cooling dominate the booth?

MiTAC's live demonstrations at SC26 address the physical consequences of that inference design. A 52U liquid-cooled rack integrates 12 G4826Z5 servers and 96 AMD Instinct MI355X GPUs, a configuration the company says delivers 50% higher GPU density than conventional deployments.

A separate proof of concept with ZutaCore applies waterless two-phase direct liquid cooling to the G4826Z5, vaporizing dielectric fluid directly at the chip. MiTAC reports the approach supports:

  • Thermal design powers above 4,000W per accelerator
  • Heat flux above 250W/cm²
  • An annualized partial power usage effectiveness (pPUE, a partial measure of data-center energy efficiency) of 1.03

The demonstration targets a specific failure mode: concentrating high-power accelerators inside dense racks pushes air cooling past its thermal limits, making liquid-based heat removal a prerequisite rather than an option.

Storage addresses a different constraint. MiTAC's demonstration combines DDN Infinia software-defined storage with seven GC68C-B8056 1U storage servers, designed to supply continuous AI data streams without I/O bottlenecks. The point is to reduce the risk that expensive accelerators sit idle waiting for data instead of processing inference and agent workloads.

What does the software stack cover?

The hardware proposal arrives with an orchestration layer assembled from named partners. Canonical Ubuntu supplies the certified operating-system layer. Rafay adds dynamic GPU-as-a-Service orchestration across distributed workloads. MiTAC's own MiCoreView software provides pod-level telemetry and unified data-center visibility. Together with DDN's Infinia, those components cover operating systems, data delivery, GPU allocation and monitoring within a single infrastructure stack.

The composition matters for buyers. On-premises inference has historically required enterprises to integrate these layers themselves; MiTAC is packaging them as one capacity-planning problem.

What is the Helion rack?

MiTAC is also collaborating with AMD on an upcoming ORv3 44OU Helion Agentic AI Rack, which the company says will scale to 32,768 AMD Zen 6 CPU cores for hyperscale and enterprise environments.

AMD's Helion documentation describes an open rack-scale reference design built on Open Compute Project (OCP) standards, with CPU-only hyperscale configurations and enterprise versions that mix CPU and GPU blades. The air-cooled inference version supports up to 16 Instinct MI350P GPUs, and customers can mix blade types to match different agent workloads.

The enterprise configuration is intended to combine agent execution and inference in one rack. AMD positions it distinctly from Helios, which the documentation describes as built for frontier-scale training and large-scale inference with a unified scale-up GPU domain.

Why does this matter for the on-premises market?

The stakes are straightforward. Enterprises weighing local deployment of very large models face a matrix of interlocking constraints — accelerator memory, inter-stage networking, storage throughput, rack cooling and orchestration — and MiTAC's proposal explicitly folds those connections into the capacity calculation alongside the GPUs themselves.

Not everything shown at SC26 ships today. Some supporting systems are being demonstrated live at the conference, while others, including the two-server inference architecture and the Helion rack, remain proposed or in development. The announcement is therefore a mix of current demonstrations and future infrastructure designs.

The practical implication MiTAC draws is that deployable on-premises inference depends on the decode memory working in concert with the prefill network, storage throughput, rack cooling and orchestration — and if the proposal matures into product, the 2 TB decode server and the Helion rack will test whether enterprises want to buy that integration as a package rather than assemble it piecemeal.

By Marcus Bennett, AI In Context.

Source: unite.ai

Share this article:

More from Marcus Bennett

Marcus Bennett

Show full bio

Senior reporter covering companies, deals and strategy in artificial intelligence and machine learning at AI In Context.

718 articles

Related articles

  1. Supermicro Ships 20-Plus H15 Servers Built on AMD EPYC 9006 Chips
  2. Lenovo Bets on Liquid Cooling and Services to Solve AI's Power Problem
  3. Supermicro Ships Petascale All-Flash Server With PCIe Gen 6
  4. AMD adds MI430X and MI450X to Instinct MI400 GPU lineup
  5. AMD launches 256-core EPYC 9006 CPUs to fight Intel and Nvidia

« Previous articleNext article »