Virgo: Cluster-level Matrix Unit Integration in GPUs for Scalability and Energy Efficiency

TL;DR

Virgo introduces a novel GPU microarchitecture that disaggregates matrix units from SIMT cores, enhancing scalability and energy efficiency for deep learning workloads by increasing operation granularity and reducing power consumption.

Contribution

The paper proposes Virgo, a GPU design that decouples matrix units from SIMT cores, enabling scalable, energy-efficient matrix operations at the cluster level.

Findings

Achieves 67.3% reduction in on-chip active power compared to Ampere-style cores.

Achieves 24.2% reduction in on-chip active power compared to Hopper-style cores.

Supports efficient concurrent execution of matrix units and SIMT cores.

Abstract

Modern GPUs incorporate specialized matrix units such as Tensor Cores to accelerate GEMM operations, which are central to deep learning workloads. However, existing matrix unit designs are tightly coupled to the SIMT core, restricting operation size due to register file capacity and bandwidth constraints. Such a limitation in scalability makes it difficult to simultaneously improve compute throughput and energy efficiency in GPUs. To address this challenge, we propose Virgo, a GPU microarchitecture that integrates dedicated matrix units at the SIMT core cluster level. By decoupling the matrix unit from the SIMT core, Virgo eliminates scalability constraints imposed by the core microarchitecture. Consequently, Virgo increases operation granularity at the hardware level, reducing energy overhead from core instruction processing. Physical disaggregation also enables a unified matrix unit design and offloading both operand and accumulator accesses from the register file, improving data reuse and energy efficiency. Furthermore, this disaggregation supports efficient concurrent execution of the SIMT core and matrix unit, optimizing mapping for fused DNN workloads. Our evaluations using synthesizable RTL demonstrate that Virgo achieves 67.3% and 24.2% reduction in on-chip active power consumption, compared to the baseline Ampere-style and Hopper-style core-coupled designs.