Google Bifurcates AI Hardware Strategy with Specialized Training and Inference Chips

Google has announced a strategic shift in its AI hardware architecture, separating the tasks of model training and inference into two specialized processors for the eighth generation of its Tensor Processing Unit (TPU). Both chips are expected to become available later this year, marking a departure from previous generations that handled both tasks on a single processor. This transition reflects a broader industry trend where hyperscalers—including Microsoft, Meta, and Amazon—are developing custom silicon to reduce reliance on Nvidia and maximize operational efficiency for specific AI workloads. By specializing the hardware, Google aims to better support the rise of AI agents that require high throughput and low latency. Technical specifications reveal significant performance gains. The new training chip reportedly delivers 2.8 times the performance of the previous seventh-generation 'Ironwood' TPU at the same price point. Meanwhile, the inference-focused chip, designated as TPU 8i, offers an 80% performance improvement over its predecessor. To achieve these gains, the TPU 8i incorporates 384 MB of static random-access memory (SRAM), tripling the capacity found in the Ironwood generation. This architecture is specifically designed to run millions of agents cost-effectively. While Google remains a major customer of Nvidia, the adoption of its proprietary TPUs is expanding. High-profile users now include Anthropic, Citadel Securities, and all 17 U.S. Energy Department national laboratories. Analysts from DA Davidson have previously estimated the combined value of Google's TPU business and DeepMind AI group at approximately $900 billion.