Accurate and Efficient Hyperbolic Tangent Activation Function on FPGA using the DCT Interpolation Filter

TL;DR

This paper presents a high-accuracy, resource-efficient FPGA implementation of the hyperbolic tangent activation function using DCT Interpolation Filter, significantly improving precision while maintaining low computational cost.

Contribution

It introduces a novel DCTIF-based approximation method for the hyperbolic tangent function that achieves two orders of magnitude higher precision with minimal resource usage.

Findings

Achieves 10E-5 maximum error in approximation

Uses only 1.52 Kbits memory and 57 LUTs on FPGA

High-accuracy approximation maintains DNN training and testing performance

Abstract

Implementing an accurate and fast activation function with low cost is a crucial aspect to the implementation of Deep Neural Networks (DNNs) on FPGAs. We propose a high-accuracy approximation approach for the hyperbolic tangent activation function of artificial neurons in DNNs. It is based on the Discrete Cosine Transform Interpolation Filter (DCTIF). The proposed architecture combines simple arithmetic operations on stored samples of the hyperbolic tangent function and on input data. The proposed DCTIF implementation achieves two orders of magnitude greater precision than previous work while using the same or fewer computational resources. Various combinations of DCTIF parameters can be chosen to tradeoff the accuracy and complexity of the hyperbolic tangent function. In one case, the proposed architecture approximates the hyperbolic tangent activation function with 10E-5 maximum error while requiring only 1.52 Kbits memory and 57 LUTs of a Virtex-7 FPGA. We also discuss how the activation function accuracy affects the performance of DNNs in terms of their training and testing accuracies. We show that a high accuracy approximation can be necessary in order to maintain the same DNN training and testing performances realized by the exact function.