NOVA: NoC-based Vector Unit for Mapping Attention Layers on a CNN Accelerator

TL;DR

NOVA is a NoC-based vector unit designed to efficiently map attention layers onto neural network accelerators, significantly improving power efficiency for edge AI applications.

Contribution

This work introduces NOVA, a novel NoC-based vector unit that enables efficient mapping of attention layers on existing neural accelerators, addressing non-linear operation challenges.

Findings

NOVA achieves up to 37.8x power efficiency improvement.

NOVA effectively maps attention layers onto existing accelerators.

The architecture enhances utilization of vector units for non-linear operations.

Abstract

Attention mechanisms are becoming increasingly popular, being used in neural network models in multiple domains such as natural language processing (NLP) and vision applications, especially at the edge. However, attention layers are difficult to map onto existing neuro accelerators since they have a much higher density of non-linear operations, which lead to inefficient utilization of today's vector units. This work introduces NOVA, a NoC-based Vector Unit that can perform non-linear operations within the NoC of the accelerators, and can be overlaid onto existing neuro accelerators to map attention layers at the edge. Our results show that the NOVA architecture is up to 37.8x more power-efficient than state-of-the-art hardware approximators when running existing attention-based neural networks.