ErfReLU: Adaptive Activation Function for Deep Neural Network

TL;DR

This paper introduces ErfReLU, a novel adaptive activation function based on erf and ReLU, and evaluates its performance against other trainable functions across multiple deep learning models and datasets.

Contribution

The paper proposes ErfReLU, a new adaptive activation function, and provides comprehensive performance analysis comparing it with existing functions on standard benchmarks.

Findings

ErfReLU performs competitively with state-of-the-art activation functions.

Adaptive activation functions improve deep network performance.

Empirical results across multiple datasets validate ErfReLU's effectiveness.

Abstract

Recent research has found that the activation function (AF) selected for adding non-linearity into the output can have a big impact on how effectively deep learning networks perform. Developing activation functions that can adapt simultaneously with learning is a need of time. Researchers recently started developing activation functions that can be trained throughout the learning process, known as trainable, or adaptive activation functions (AAF). Research on AAF that enhance the outcomes is still in its early stages. In this paper, a novel activation function 'ErfReLU' has been developed based on the erf function and ReLU. This function exploits the ReLU and the error function (erf) to its advantage. State of art activation functions like Sigmoid, ReLU, Tanh, and their properties have been briefly explained. Adaptive activation functions like Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf have also been described. Lastly, performance analysis of 9 trainable activation functions along with the proposed one namely Tanhsoft1, Tanhsoft2, Tanhsoft3, TanhLU, SAAF, ErfAct, Pserf, Smish, and Serf has been shown by applying these activation functions in MobileNet, VGG16, and ResNet models on CIFAR-10, MNIST, and FMNIST benchmark datasets.