Google’s new TPUs target secure, high-memory AI agents

By Marcus Bennett

Google Cloud has added two eighth-generation tensor processing units (TPUs): TPU 8tx, which the company says delivers 2x lower latency than TPU 8t, and TPU 8ix Max, with 384 GB of high-bandwidth memory (HBM) and 17.2 terabytes per second of memory bandwidth.

TPU 8tx is the first Google Cloud TPU with confidential computing, developed in partnership with AMD, while TPU 8ix Max targets large context windows and high-throughput, low-latency inference. The products extend the workload-specific TPU 8t and TPU 8i systems introduced earlier this year.

Google Cloud said the release addresses production constraints created by more capable models: larger working sets, stricter data controls, and higher concurrent load. The company said the stakes extend beyond raw chip speed because agents must move data, retain context, and secure sensitive information while running at scale.

Google Cloud’s existing eighth-generation portfolio already split training from inference. The company reported two comparative gains against previous-generation TPUs:

• TPU 8t: Large-scale training, with up to 2.7 times better training performance than previous-generation TPUs, according to Google Cloud.
• TPU 8i: Low-latency, cost-effective inference, with 80% better inference price performance than previous-generation TPUs, according to Google Cloud.

The earlier systems bring together the chips and their surrounding compute, networking, and storage. Mark Lohmeyer, Google Cloud’s VP and GM for AI and computing structure, said the company had, for the first time, bifurcated its portfolio into two chips and two systems rather than one chip and one system.

Google Cloud said TPU 8tx and TPU 8ix Max extend those two paths. It positions 8tx for secure, confidential inference and 8ix Max for large context windows and high-throughput, low-latency inference.

What does TPU 8tx change about inference security?

TPU 8tx is Google Cloud’s first TPU to offer confidential computing, built in partnership with AMD. The capability adds hardware-backed isolation so sensitive data and model weights remain protected and isolated while in active use.

Google Cloud said the design secures the entire workflow: it encrypts data moving from the host to the TPUs, protects the chips during active inference, and attests the assembled system. The attestation verifies that the system is what it says it is, according to Lohmeyer.

Enterprises may run AI over proprietary information, internal business data, or material protected by regulatory or operational requirements. Google Cloud said TPU 8tx addresses those constraints by protecting data and model weights during execution rather than only before or after inference.

The 8tx also brings TPU 8i’s low-latency networking architecture into the secure system. Lohmeyer said a reduced network-topology diameter lowers communication latency between chips and yields 2x lower latency than TPU 8t while using the same chip.

Lohmeyer summarized the design goal: “You shouldn’t have to trade off between security and performance.”

How does TPU 8ix Max expand the context window?

TPU 8ix Max targets the memory constraints created by AI agents, which reason, retrieve information, and respond in parallel. Google Cloud said those agents need large context windows and enough bandwidth to move data rapidly while models call tools and process external information.

Lohmeyer said context windows are growing exponentially as agents work across documents, code bases, and other data. He said concurrent workloads place significant pressure on both memory capacity and bandwidth, creating a memory wall that the 8ix Max addresses.

Built on TPU 8i’s low-latency architecture, the 8ix Max essentially doubles the silicon per package, according to Lohmeyer. Google Cloud specified three capacity increases:

• 384 GB of HBM: 33% more high-bandwidth memory than TPU 8i.
• 17.2 TB/s memory bandwidth: 2x the memory bandwidth of the earlier chip.
• 768 MB of on-chip SRAM: 2x the static random access memory in TPU 8i.

Google Cloud said the added HBM lets 8ix Max store larger context windows and key-value (KV) caches, which hold working data used during inference. Lohmeyer said the design creates more capacity for concurrent work rather than merely accelerating one operation at a time.

Lohmeyer compared the change to adding highway lanes: “It’s like adding more lanes to the highway,” he observed. “We’re not necessarily making each car travel faster, we’re creating more capacity so more traffic can move through the system at the same time.”

How does TPU Sync keep long-running agents economical?

A second constraint appears when an agent waits. Lohmeyer described agentic workloads as bursty, long-running, and often idle. An agent may generate output and call tools, then wait for a human or another system for days while preserving its context. Tied-up resources then sit unused for 90% or more of that period.

Google Cloud’s open-source TPU Sync addresses this pattern in software by unifying memory, storage, and networking. Its model-state layer combines chip HBM, host dynamic random access memory (DRAM), shared storage, and the data-center network into a single, co-designed hierarchy.

The software automatically tiers KV caches off the chip in milliseconds, streams context across nodes, and synchronizes reinforcement-learning (RL) weights. Google Cloud said those functions cut transfer latency and Time to First Token (TTFT), the delay before a model returns its first token, by more than 90%.

When an agent pauses, TPU Sync can move its working memory off the chip and restore it when the process resumes. Lohmeyer said this frees accelerator resources for active workloads, lowers token-use and agent-transaction costs, and raises effective utilization of the underlying hardware.

Google Cloud said TPU Sync works with v7, v6, and v5 TPUs as well as the current v8 generation, extending the software to installed Google Cloud capacity as well as the new systems.

What changes as agents scale?

Google Cloud said AI is moving beyond individual chat exchanges and simple request-response patterns. Agents now take on complex tasks toward broader objectives, dividing work into model and tool calls, information retrieval, and code writing, compilation, and execution.

Lohmeyer said dozens or hundreds of operations can occur behind a single agent interaction. He said those hidden steps create significant infrastructure requirements across memory, networking, storage, security, and orchestration software.

Lohmeyer identified three architectural shifts:

• From human scale to agentic scale: Tens or hundreds of sub-agents execute in parallel, driving compute demand by 10x, 100x, or more, according to Lohmeyer.
• From systems-of-record to systems-of-action: AI infrastructure shifts from storing records toward executing actions, according to Lohmeyer.
• More complex agentic workflows: Long-running processes combine repeated model calls, tool use, retrieval, and code execution, according to Google Cloud.

Lohmeyer said the infrastructure conversation once centered on how much compute was available. Builders now also ask how quickly data moves, how efficiently systems use memory, how low latency can go, and how to control cost and security across many workloads.

He argued that silicon remains central, but memory, networking, storage, orchestration software, and the data center around it can each bottleneck the entire system. The 8tx, 8ix Max, and TPU Sync address those layers directly: hardware isolation, larger working sets, and sustained utilization for bursty agents.

The next test for Google Cloud’s three-part push will be system-level deployment. Confidential execution, larger working sets, and sustained agent utilization now sit beside raw chip capacity; buyers will assess whether those changes reduce latency and cost without weakening security.