DARE: An Irregularity-Tolerant Matrix Processing Unit with a Densifying ISA and Filtered Runahead Execution

TL;DR

DARE introduces a novel MPU architecture with a densifying ISA and filtered runahead execution, significantly improving performance and energy efficiency for sparse DNN workloads by addressing irregular memory access and compute inefficiencies.

Contribution

It presents DARE, a new MPU design that co-optimizes hardware and algorithms to handle irregular sparse DNN computations more effectively.

Findings

Performance improved by up to 4.44×

Energy efficiency increased up to 22.8×

Achieves 3.91× lower hardware overhead than NVR

Abstract

Deep Neural Networks (DNNs) are widely applied across domains and have shown strong effectiveness. As DNN workloads increasingly run on CPUs, dedicated Matrix Processing Units (MPUs) and Matrix Instruction Set Architectures (ISAs) have been introduced. At the same time, sparsity techniques are widely adopted in algorithms to reduce computational cost. Despite these advances, insufficient hardware-algorithm co-optimization leads to suboptimal performance. On the memory side, sparse DNNs incur irregular access patterns that cause high cache miss rates. While runahead execution is a promising prefetching technique, its direct application to MPUs is often ineffective due to significant prefetch redundancy. On the compute side, stride constraints in current Matrix ISAs prevent the densification of multiple logically related sparse operations, resulting in poor utilization of MPU processing elements. To address these irregularities, we propose DARE, an irregularity-tolerant MPU with a Densifying ISA and filtered Runahead Execution. DARE extends the ISA to support densifying sparse operations and equips a lightweight runahead mechanism with filtering capability. Experimental results show that DARE improves performance by 1.04$\times$ to 4.44$\times$ and increases energy efficiency by 1.00$\times$ to 22.8$\times$ over the baseline, with 3.91$\times$ lower hardware overhead than NVR.