A Configurable and Efficient Memory Hierarchy for Neural Network Hardware Accelerator

TL;DR

This paper introduces a configurable memory hierarchy framework for neural network accelerators that optimizes memory capacity and performance, reducing chip area significantly while maintaining high efficiency.

Contribution

It presents a novel, flexible memory hierarchy design with up to five levels and an optional shift register, tailored for DNN layer access patterns, improving hardware efficiency.

Findings

Up to 62.2% reduction in chip area.

Performance loss minimized to 2.4%.

Efficient execution of most DNN layer access patterns.

Abstract

As machine learning applications continue to evolve, the demand for efficient hardware accelerators, specifically tailored for deep neural networks (DNNs), becomes increasingly vital. In this paper, we propose a configurable memory hierarchy framework tailored for per layer adaptive memory access patterns of DNNs. The hierarchy requests data on-demand from the off-chip memory to provide it to the accelerator's compute units. The objective is to strike an optimized balance between minimizing the required memory capacity and maintaining high accelerator performance. The framework is characterized by its configurability, allowing the creation of a tailored memory hierarchy with up to five levels. Furthermore, the framework incorporates an optional shift register as final level to increase the flexibility of the memory management process. A comprehensive loop-nest analysis of DNN layers shows that the framework can efficiently execute the access patterns of most loop unrolls. Synthesis results and a case study of the DNN accelerator UltraTrail indicate a possible reduction in chip area of up to 62.2% as smaller memory modules can be used. At the same time, the performance loss can be minimized to 2.4%.