GRAU: Generic Reconfigurable Activation Unit Design for Neural Network Hardware Accelerators

TL;DR

GRAU is a reconfigurable activation hardware that uses piecewise linear fitting with power-of-two slopes, significantly reducing LUT consumption and supporting mixed-precision quantization and nonlinear functions for neural network accelerators.

Contribution

The paper introduces GRAU, a novel reconfigurable activation unit that improves hardware efficiency and flexibility by simplifying activation functions with power-of-two slopes.

Findings

Reduces LUT consumption by over 90% compared to multi-threshold activators.

Supports mixed-precision quantization and nonlinear functions like SiLU.

Enhances hardware efficiency, flexibility, and scalability.

Abstract

With the continuous growth of neural network scales, low-precision quantization is widely used in edge accelerators. Classic multi-threshold activation hardware requires 2^n thresholds for n-bit outputs, causing a rapid increase in hardware cost as precision increases. We propose a reconfigurable activation hardware, GRAU, based on piecewise linear fitting, where the segment slopes are approximated by powers of two. Our design requires only basic comparators and 1-bit right shifters, supporting mixed-precision quantization and nonlinear functions such as SiLU. Compared with multi-threshold activators, GRAU reduces LUT consumption by over 90%, achieving higher hardware efficiency, flexibility, and scalability.