Heterogeneous Computing: The Key to Powering the Future of AI Agent Inference

TL;DR

This paper analyzes the bottlenecks in AI agent inference, introduces new metrics to understand memory limitations, and proposes system heterogeneity and co-design strategies to enhance efficiency and scalability.

Contribution

It introduces the Operational Intensity and Capacity Footprint metrics, revealing new regimes in AI inference bottlenecks and suggesting system-level heterogeneity and co-design for future AI hardware.

Findings

Memory capacity wall is a critical bottleneck in AI inference.

Disaggregated system design can mitigate memory and bandwidth bottlenecks.

Heterogeneous accelerators and high bandwidth memory are key to scalable AI inference.

Abstract

AI agent inference is driving an inference heavy datacenter future and exposes bottlenecks beyond compute - especially memory capacity, memory bandwidth and high-speed interconnect. We introduce two metrics - Operational Intensity (OI) and Capacity Footprint (CF) - that jointly explain regimes the classic roofline analysis misses, including the memory capacity wall. Across agentic workflows (chat, coding, web use, computer use) and base model choices (GQA/MLA, MoE, quantization), OI/CF can shift dramatically, with long context KV cache making decode highly memory bound. These observations motivate disaggregated serving and system level heterogeneity: specialized prefill and decode accelerators, broader scale up networking, and decoupled compute-memory enabled by optical I/O. We further hypothesize agent-hardware co design, multiple inference accelerators within one system, and high bandwidth, large capacity memory disaggregation as foundations for adaptation to evolving OI/CF. Together, these directions chart a path to sustain efficiency and capability for large scale agentic AI inference.