Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design

TL;DR

This paper introduces FABNet, a hardware-friendly sparse variant of attention-based neural networks, and an adaptable FPGA accelerator that jointly optimizes algorithm and hardware for significant speedups and efficiency gains.

Contribution

It proposes a unified butterfly sparsity pattern for both attention and feed-forward networks, and a configurable hardware accelerator for improved performance and scalability.

Findings

FABNet matches vanilla Transformer accuracy with 10-66x less computation.

The FPGA accelerator achieves 14.2-23.2x speedup over state-of-the-art.

System is up to 273.8x faster than CPU/GPU on low-power devices.

Abstract

Attention-based neural networks have become pervasive in many AI tasks. Despite their excellent algorithmic performance, the use of the attention mechanism and feed-forward network (FFN) demands excessive computational and memory resources, which often compromises their hardware performance. Although various sparse variants have been introduced, most approaches only focus on mitigating the quadratic scaling of attention on the algorithm level, without explicitly considering the efficiency of mapping their methods on real hardware designs. Furthermore, most efforts only focus on either the attention mechanism or the FFNs but without jointly optimizing both parts, causing most of the current designs to lack scalability when dealing with different input lengths. This paper systematically considers the sparsity patterns in different variants from a hardware perspective. On the algorithmic level, we propose FABNet, a hardware-friendly variant that adopts a unified butterfly sparsity pattern to approximate both the attention mechanism and the FFNs. On the hardware level, a novel adaptable butterfly accelerator is proposed that can be configured at runtime via dedicated hardware control to accelerate different butterfly layers using a single unified hardware engine. On the Long-Range-Arena dataset, FABNet achieves the same accuracy as the vanilla Transformer while reducing the amount of computation by 10 to 66 times and the number of parameters 2 to 22 times. By jointly optimizing the algorithm and hardware, our FPGA-based butterfly accelerator achieves 14.2 to 23.2 times speedup over state-of-the-art accelerators normalized to the same computational budget. Compared with optimized CPU and GPU designs on Raspberry Pi 4 and Jetson Nano, our system is up to 273.8 and 15.1 times faster under the same power budget.