Towards Zero-Stall Matrix Multiplication on Energy-Efficient RISC-V Clusters for Machine Learning Acceleration

TL;DR

This paper presents microarchitectural innovations for RISC-V clusters that nearly eliminate control and memory access inefficiencies in matrix multiplication, significantly boosting performance and energy efficiency for ML workloads.

Contribution

It introduces zero-overhead loop nests and a conflict-free memory system to optimize RISC-V clusters for ML acceleration, maintaining programmability and broad workload support.

Findings

Achieved 96.1% to 99.4% utilization in matrix multiplication workloads.

Realized 11% performance and 8% energy efficiency improvements over baseline.

Maintained comparable efficiency to specialized accelerators with full programmability.

Abstract

The growing computational demands of machine learning (ML) workloads have driven the design of ML accelerators aiming at an optimal tradeoff between efficiency and flexibility. A widely explored architecture for flexible ML accelerators is based on clusters of lightweight instruction processors sharing multi-banked L1 memory, augmented with specialized instruction extensions for key ML-related computations, such as matrix multiplication (matmul). However, instruction extensions should be coupled with microarchitectural optimizations that remove inefficiencies due to control flow (loop handling) and memory access, without drastically increasing processor complexity. Moving from a state-of-the-art (SoA) ML accelerator cluster based on RISC-V processors, we propose a low-overhead optimized microarchitecture that eliminates these inefficiencies almost entirely while retaining programmability. We introduce "zero-overhead loop nests" to remove control overheads, and a "zero-conflict memory subsystem", leveraging a novel double-buffering-aware interconnect, to eliminate bank conflicts in L1 memory. With these enhancements, we attain near-ideal utilizations between 96.1% and 99.4%, achieving 11% performance and 8% energy efficiency improvements over the baseline SoA RISC-V cluster. We demonstrate comparable utilizations and performance to a specialized SoA accelerator, with only 12% difference in energy efficiency, while providing a fully-programmable general-purpose solution supporting a significantly wider range of workloads.