A Lightweight High-Throughput Collective-Capable NoC for Large-Scale ML Accelerators

TL;DR

This paper introduces a lightweight, high-throughput NoC with collective capabilities, enabling efficient communication and computation for large-scale ML accelerators, significantly improving speed and energy efficiency.

Contribution

It presents a novel NoC design with Direct Compute Access (DCA) for high-bandwidth in-network reductions, supporting scalable ML workloads with minimal area overhead.

Findings

Achieves 5.3x speedup in multicast operations

Achieves 2.8x speedup in reduction operations

Scales efficiently to large mesh architectures with significant performance gains

Abstract

The exponential increase in Machine Learning (ML) model size and complexity has driven unprecedented demand for high-performance acceleration systems. As technology scaling enables the integration of thousands of computing elements onto a single die, the boundary between distributed and on-chip systems has blurred, making efficient on-chip collective communication increasingly critical. In this work, we present a lightweight, collective-capable Network on Chip (NoC) that supports efficient barrier synchronization alongside scalable, high-bandwidth multicast and reduction operations, co-designed for the next generation of ML accelerators. We introduce Direct Compute Access (DCA), a novel paradigm that grants the interconnect fabric direct access to the cores' computational resources, enabling high-throughput in-network reductions with a small 16.9% router area overhead. Through in-network hardware acceleration, we achieve 5.3x and 2.8x geomean speedups on multicast and reduction operations involving between 1 and 32 KiB of data, respectively. Furthermore, by keeping communication off the critical path in GEMM workloads, these features allow our architecture to scale efficiently to large meshes, resulting in up to 3.8x and 2.4x estimated performance gains through multicast and reduction support, respectively, compared to a baseline unicast NoC architecture, and up to 1.17x estimated energy savings.