TPU Split in Two, MTIA Adds GenAI: The Training-Inference Division of Cloud Silicon Generation Two
Hot Chips 2026 specialist · Hyperscaler silicon
GPT-5.2's serving workload runs on a Microsoft-designed chip: MAIA 200 already carries GPT-5.2, M365 Copilot, and Foundry (§2). The overview set judgment one: the training-inference division has sunk to silicon. This piece is where that judgment has its densest evidence: three clouds on the same stage, all second-generation, one of which split its chip in two by workload shape. We put the three clouds and the control group on a single table: generation one answered whether they dared to self-design; generation two answers what shape to design for. The four have already grown structural differences around each one's workloads. AWS Trainium is absent from this conference; the absent fourth cloud is itself a background signal.
1. Meta MTIA 400: Recommendation Plus GenAI, Dual Mission
The spec sheet is in the overview; this section dissects how chiplet and array fit two workloads into one chip.
The partition first. Two compute dies, one SoC, two network dies, plus HBM, divided by function: compute dies take the advanced process; SoC and network dies stay on mature nodes; HBM stacks independently. The first motive is cost: you pay advanced-process prices only for compute area; control, IO, and interconnect settle at mature-node prices. This is the same industry language as Diamond Rapids' Foveros partitioning and Arm AGI's dual chiplets: three dialects of one grammar.
The array next. The 8×6 PE array with redundancy rows is a yield statement: PE arrays are MTIA's compute body; redundant rows let defective rows be masked and replaced at fab time, cutting the yield risk of large arrays into small pieces. The GPU industry uses harvesting to mask compute units; MTIA brings the same trick to PE granularity.
The dual mission is the real architectural pressure. Recommendation workloads are embedding-lookup intensive: massive random reads, DRAM bandwidth is everything. GenAI is matrix-compute intensive: compute and on-chip SRAM are everything. Among the three multipliers versus the 200, the 46× DRAM bandwidth jump serves the former; FP16 15× and SRAM bandwidth 5× serve the latter. One chip serving both shapes means neither gets its tailored extreme: precisely the counter-example to Google's split (§3).
2. Microsoft MAIA 200: A Datacenter-Scale System
Specs and workload list are in the overview; this section dissects system mechanics.
First mechanism: 272MB of on-chip SRAM managed explicitly by software. General-purpose accelerators let hardware coherence protocols manage cache hierarchy automatically; MAIA hands the management to the software stack: compilers and schedulers directly orchestrate data movement in and out of SRAM. What's saved is the area and power of hardware cache coherence; the cost is a software stack that must be smart enough. This is the micro-level evidence of the "from chip to system" judgment: the silicon design now presupposes a deeply involved software layer.
Second mechanism: unified Ethernet scale-up. The scale-up domain has historically been private-interconnect territory (NVLink, ICI); MAIA uses Ethernet for horizontal scaling, pushing an open standard into a private domain. The deployment logic is the same family as the 350W air-cooled card in the Intel piece: existing rooms and switching ecosystems are reused directly; retrofit costs are eliminated. The all-connected quad topology is its network shape.
Third: the 30% number, read closely. The official figure is 30% better performance per dollar than the latest-generation hardware in Microsoft's fleet. The denominator is Microsoft's own mixed-fleet baseline; a chip-to-chip same-generation comparison has not been published. The direction is usable; the magnitude awaits the 2027 financial-report TCO. Des Moines first, Phoenix second: the deployment cadence as stated.
3. Google TPU v8: Training-Inference Split into Silicon
The spec table is in the overview; this section dissects the mechanics each chip adds.
8t's mechanics live in scale economics. Training stacks pod: 9,600-chip superpod, scale-out to 134K on Virgo fabric; 8×12-high HBM3e, bandwidth +~30% over Ironwood's 8-high. The larger the pod, the heavier the collectives tax; doubled ICI bandwidth cuts that tax. Training's ledger is utilization economics: every point of communication overhead reduced is a point of effective compute gained. Process: TSMC 2nm target, end of 2027.
8i's mechanics live in latency economics, with two new designs serving MoE serving:
- CAE (Collectives Acceleration Engine): offloads all-reduce, all-gather, and similar collective operations from general-purpose cores to a dedicated engine. MoE serving's expert dispatch is high-density all-to-all; general-purpose cores running collectives are both slow and compute-consuming. The dedicated engine takes direct responsibility for latency targets.
- Boardfly replaces 3D torus: the regular nearest-neighbor structure of 3D torus serves training's localized communication; MoE serving's all-to-all needs a different arrangement. The topology changes with workload shape: "build silicon for the workload" extends from compute units to the network layer.
Two ledgers together: training buys scale (compute density + communication efficiency), inference buys response (bandwidth + SRAM + dedicated offload): 8i's 288GB, 384MB SRAM, 8.60TB/s all overtake the 8t while compute runs slightly lower. The spec difference is the direct product of two ledgers. Background quantities: TPU shipments 4.3M in 2026, officially 10M in 2027, 35M+ in 2028; Google 2026 capex $175-185B.
The main-judgment entry: 8t designed with Broadcom, 8i with MediaTek. Dual suppliers entering flagship-silicon division: why did the inference chip go to MediaTek? Three candidate explanations (§5, judgment two): cost structure, process tolerance, Tensor collaboration history. If MediaTek on 8i is validated by subsequent deployment, the cloud-silicon supply-chain map rewrites.
4. Cerebras CS-4: Wafer-Scale Rack as Control Group
The control group's specs and Register's skepticism are already in the overview; this section dissects rack engineering.
The Nexus rack arranges three WSE-3T engines (each 250 PFLOPS, 44GB on-chip SRAM, 43.2PB/s) into one rack domain: power, cooling, and maintenance are restructured around the three-wafer set; wafer-to-wafer 2μs, 2.4Tb/s per wafer, combining three into a 750-PFLOPS, 129.6PB/s logical single machine. 750=3×250, 129.6=3×43.2, arithmetically consistent. Register's skepticism, on record: WSE-3T is the same silicon as WSE-3 run harder; dense FP16 estimated at ~25 PFLOPS (10× sparsity assumption); the bandwidth figures remain paper specs. Its value in this piece is the physical-extreme reference for the first three sections: MTIA spreads cost with chiplets, MAIA integrates at system level, TPU splits by workload; Cerebras compresses the physical distance between memory bandwidth and compute to zero — the bandwidth logic pushed to its physical limit. CS-4 vs CS-3: 2× tokens, 10× tokens/watt per official figures; CS-5 and CS-6 (3D wafer-scale DRAM stacking) are on the roadmap; shipping this quarter.
SambaNova SN50 (brief): fifth-generation RDU for agentic inference, three-tier memory, scale-out to 32K RDU, February 2026 announcement, H2 2026 delivery.
5. Judgments and What to Watch
Judgment one: cloud-silicon generation two is an industry inflection; the mechanism is a three-layer cost structure. The overview's "from cost-side experimentation to carrying flagship loads" is a position judgment (not re-litigated here); this piece provides the economic explanation for the timing: first, amortization — generation one's tape-out, validation, and ecosystem costs lacked a thinning base; generation two reuses IP and software stacks, and the per-generation cost curve bends downward for the first time. Second, software-stack maturity — one generation of production-environment seasoning (compilers, schedulers, operations) is the prerequisite for custom silicon carrying flagships; all three clouds have crossed this threshold. Third, workload certainty — GPT-5.2, Copilot, recommendation-plus-GenAI gave the certainty to design by shape; TPU dared to split precisely because of this. Three conditions simultaneously true is what distinguishes "now" from any point in the past five years.
Judgment two (main judgment): dual supplier is the newest variable worth tracking in generation two. Three candidate explanations: cost structure (a second supplier changes the negotiating position with a single design partner); process tolerance (an inference chip is bandwidth-oriented, and its process-window requirements differ from training's compute orientation — candidate explanation, to be verified); Tensor collaboration history (MediaTek and Google have years of collaboration on edge silicon; trust and interface costs are prepaid). If MediaTek on 8i is validated by subsequent deployments, the cloud-silicon supply-chain map rewrites.
Judgment three: TCO pending is the biggest variable. MAIA's +30% is self-reported (fleet-mixed baseline as denominator; §2 close reading); 2027 financial-report TCO is the overview's tracking-point deadline. This piece adds three incremental checkpoints: MTIA 450/500 node movements, CS-4 this-quarter delivery, MAIA fleet-share signal.
What to watch: Microsoft's 2027 TCO breakdown; CS-4 this-quarter delivery; MTIA 450/500 node movements; MAIA fleet share; TPU field-report CAE clarification.
Forward commitment: CS-4 delivery, Microsoft's 2027 TCO, and TPU field reports — we will come back and settle the account.
Further reading: Hot Chips 2026 Overview · Four Routes Against the Memory Wall · Intel's Triple Launch · Five Networks for the AI Factory