Addressing memory bandwidth scalability in vector processors for streaming applications

TL;DR

This paper proposes a new extended memory hierarchy with multiple on-chip memory levels and data-shufflers to alleviate memory bandwidth bottlenecks in data-parallel AI/ML applications, improving scalability.

Contribution

It introduces a novel memory hierarchy architecture with three on-chip levels and data-shufflers, enhancing data reuse and bandwidth efficiency for AI/ML workloads.

Findings

Improved memory bandwidth utilization compared to GPUs and systolic array accelerators.

Enhanced data reuse in CNN applications with the proposed architecture.

Quantified performance benefits over existing accelerators.

Abstract

As the size of artificial intelligence and machine learning (AI/ML) models and datasets grows, the memory bandwidth becomes a critical bottleneck. The paper presents a novel extended memory hierarchy that addresses some major memory bandwidth challenges in data-parallel AI/ML applications. While data-parallel architectures like GPUs and neural network accelerators have improved power performance compared to traditional CPUs, they can still be significantly bottlenecked by their memory bandwidth, especially when the data reuse in the loop kernels is limited. Systolic arrays (SAs) and GPUs attempt to mitigate the memory bandwidth bottleneck but can still become memory bandwidth throttled when the amount of data reuse is not sufficient to confine data access mostly to the local memories near to the processing. To mitigate this, the proposed architecture introduces three levels of on-chip memory -- local, intermediate, and global -- with an ultra-wide register and data-shufflers to improve versatility and adaptivity to varying data-parallel applications. The paper explains the innovations at a conceptual level and presents a detailed description of the architecture innovations. We also map a representative data-parallel application, like a convolutional neural network (CNN), to the proposed architecture and quantify the benefits vis-a-vis GPUs and repersentative accelerators based on systolic arrays and vector processors.