DiP: A Scalable, Energy-Efficient Systolic Array for Matrix Multiplication Acceleration

TL;DR

The paper introduces DiP, a scalable and energy-efficient systolic array architecture for matrix multiplication that eliminates FIFO buffers, improves throughput, and enhances energy efficiency, especially for transformer workloads.

Contribution

It proposes a novel DiP dataflow architecture that removes synchronization FIFOs, leading to significant energy savings and throughput improvements over existing systolic array designs.

Findings

Up to 50% throughput improvement over conventional architectures.

Achieves 2.02x better energy efficiency per area in hardware simulations.

Outperforms TPU-like architectures on transformer workloads with up to 1.81x energy savings.

Abstract

Transformers are gaining increasing attention across Natural Language Processing (NLP) application domains due to their outstanding accuracy. However, these data-intensive models add significant performance demands to the existing computing architectures. Systolic array architectures, adopted by commercial AI computing platforms like Google TPUs, offer energy-efficient data reuse but face throughput and energy penalties due to input-output synchronization via First-In-First-Out (FIFO) buffers. This paper proposes a novel scalable systolic array architecture featuring Diagonal-Input and Permutated weight stationary (DiP) dataflow for matrix multiplication acceleration. The proposed architecture eliminates the synchronization FIFOs required by state-of-the-art weight stationary systolic arrays. Beyond the area, power, and energy savings achieved by eliminating these FIFOs, DiP architecture maximizes the computational resource utilization, achieving up to 50\% throughput improvement over conventional weight stationary architectures. Analytical models are developed for both weight stationary and DiP architectures, including latency, throughput, time to full PEs utilization (TFPU), and FIFOs overhead. A comprehensive hardware design space exploration using 22nm commercial technology demonstrates DiP's scalability advantages, achieving up to a 2.02x improvement in energy efficiency per area. Furthermore, DiP outperforms TPU-like architectures on transformer workloads from widely-used models, delivering energy improvement up to 1.81x and latency improvement up to 1.49x. At a 64x64 size with 4096 PEs, DiP achieves a peak throughput of 8.192 TOPS with energy efficiency 9.548 TOPS/W.