Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation

TL;DR

Flex-SFU is a hardware accelerator that uses non-uniform piecewise interpolation to efficiently compute complex activation functions in deep neural networks, significantly improving performance with minimal overhead.

Contribution

It introduces a novel non-uniform piecewise interpolation method with floating-point support and an optimization algorithm, enhancing activation function approximation in DNN accelerators.

Findings

Achieves 22.3x better mean squared error than linear methods.

Improves end-to-end AI hardware performance by 35.7%.

Provides up to 3.3x speedup with minimal area and power overhead.

Abstract

Modern DNN workloads increasingly rely on activation functions consisting of computationally complex operations. This poses a challenge to current accelerators optimized for convolutions and matrix-matrix multiplications. This work presents Flex-SFU, a lightweight hardware accelerator for activation functions implementing non-uniform piecewise interpolation supporting multiple data formats. Non-Uniform segments and floating-point numbers are enabled by implementing a binary-tree comparison within the address decoding unit. An SGD-based optimization algorithm with heuristics is proposed to find the interpolation function reducing the mean squared error. Thanks to non-uniform interpolation and floating-point support, Flex-SFU achieves on average 22.3x better mean squared error compared to previous piecewise linear interpolation approaches. The evaluation with more than 700 computer vision and natural language processing models shows that Flex-SFU can, on average, improve the end-to-end performance of state-of-the-art AI hardware accelerators by 35.7%, achieving up to 3.3x speedup with negligible impact in the models' accuracy when using 32 segments, and only introducing an area and power overhead of 5.9% and 0.8% relative to the baseline vector processing unit.