Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI

Summary

Amin Vahdat frames Google’s AI infrastructure as an end-to-end systems problem: data center design, networking, power, storage, compilers, and chip architecture all matter more than any single chip metric. His central thesis is that “goodput” — useful workload delivered under real failures — is the right accountability metric, especially at 100,000-accelerator scale where things can fail multiple times an hour. He explains why Google split TPU into 8i and 8t, why TPU architecture has stayed surprisingly stable since TPU v1, and how Google and DeepMind co-design hardware and models in parallel. The discussion then expands to optical circuit switching, utility-connected power planning, TPU replacement cycles, open standards, and how long-horizon agents are reshaping demand for CPUs, storage, and orchestration. Vahdat closes with a long-range view in which data centers become more integrated and possibly extend to orbital infrastructure as energy becomes the dominant constraint.

Notes

• AI data centers share many basics with traditional data centers, but they are increasingly purpose-built around specialization.
• Google co-designs buildings with hardware so rack power density and cooling drive the facility layout.
• The right accountability metric is delivered workload performance, not chip-level FLOPS.
• Goodput is the fraction of theoretical capacity actually delivered by a workload.
• At 1,000 to 100,000-component scale, a single failure can stall the whole job.
• At 100,000 accelerator scale, failures happen multiple times a day and sometimes multiple times an hour.
• There is no single dominant failure mode; it is a long tail of discovery.
• Google says many customers start from Nvidia’s strong reference stack, but often specialize beyond it.
• Google’s capacity goal is token-generation throughput, not FLOPS.
• Software and model improvements contribute at least as much as hardware to capacity growth.
• TPU began as a contrarian 2013 bet against general-purpose CPU orthodoxy.
• The original argument for TPU was that the targeted workloads would otherwise need unimaginable CPU support.
• Google split TPU into 8i for inference and 8t for training because inference had become a large enough market.
• Specialization improves speed and power efficiency, but only works if the workload is durable enough to justify the fixed investment.
• 8i and 8t are both partially versatile; neither is a zero-performance dead end for the other workload.
• Transformers map to matrix multiplies, softmax, and other linear-algebra primitives that can be baked into hardware.
• Google sells and uses GPUs too; they are more general-purpose than TPUs.
• Co-design can unlock large gains, but full fungibility leaves performance on the table.
• Google and DeepMind work shoulder to shoulder on hardware-model co-design.
• Multiple chip generations are in flight at once, from production to concept.
• Google is using Gemini to help design future Gemini hardware.
• Google plans hardware two to five years ahead.
• TPU architecture has stayed largely stable since TPU v1 at a medium level of detail.
• TPU primitives include two large matrix multiply units, a sparse core, and remote load/store.
• Long-horizon agents compress human interaction from seconds to milliseconds.
• Agents increase demand for CPUs, networking, storage, and orchestration, not just accelerators.
• Putting CPUs near GPUs and TPUs constrains specialization because their density and network needs differ.
• Splitting resources across buildings raises networking cost and reliability challenges.
• Google pioneered wave division multiplexing and optical circuit switching in data centers about 15 to 16 years ago.
• Optical circuit switching uses MEMS mirrors to map input ports to output ports.
• Optical switches were used to create locality between compute and storage clusters.
• Optical switching also lets Google expand or contract network spine capacity without recabling.
• For TPU fabrics, optical switching can reroute a spare rack into a failed torus in milliseconds.
• Fiber is preferred over free-space optics because attenuation and bandwidth would degrade sharply.
• Networking inside the data center is becoming more capable and more important.
• Power is the fundamental long-term constraint for Google AI infrastructure.
• Google prefers utility-connected, grid-connected power for new data centers.
• Gigawatt-scale power buildouts require years of notice and utility co-planning.
• Google may bridge gaps with local generation and later export power back to the grid.
• Utility partnerships are preferred because statistical multiplexing reduces the cost of redundancy.
• Google thinks grid partnership benefits the company, the grid, and residences.
• Data-center sizing is a balancing act among training density, failure risk, and power availability.
• Google uses models and simulators to choose between tens-of-megawatts edge sites and gigawatt training clusters.
• Inference demand is too large to rely only on older training clusters, so Google builds specialized serving clusters globally.
• Serving clusters need a mix of storage, compute, and accelerators, and are less amenable to training-style specialization.
• Older 7- and 8-year-old TPUs are still fully utilized, but depreciation and power efficiency justify replacement.
• TPU replacement happens at the pod level, with roughly 9,600-chip pods and about 152 racks.
• Google wants interoperability and rejects forcing JAX-only lock-in.
• IP is Google’s example of an open standard that enabled ecosystem growth.
• AI is changing software engineering, hardware engineering, chip bring-up, and data-center planning.
• AI now helps streamline data-center planning by assembling the right information for humans.
• AI is assisting human judgment rather than replacing it.
• Google is pursuing orbital data centers as a serious moonshot tied to energy constraints.
• Space could offer roughly 1.4x more power and near-continuous sunlight in sun-synchronous orbit.
• Space data centers still face cooling, reliability, and repairability challenges.
• There are no fundamental showstoppers to orbital data centers; free-space optical links may be the norm.
• By 2036, frontier supercomputers may be more integrated, centrally manufactured, and built as multi-megawatt racks.