TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks

TL;DR

This paper introduces TaLU, a novel activation function combining Tanh and ReLU, designed to improve neural network accuracy by mitigating ReLU's dying gradient problem, demonstrated through tests on MNIST and CIFAR-10 datasets.

Contribution

The paper proposes TaLU, a hybrid activation function that addresses ReLU's dying gradient issue and enhances classification accuracy in deep learning models.

Findings

TaLU outperforms ReLU and other activation functions by up to 6% accuracy.

Using TaLU with Batch Normalization improves model performance.

TaLU effectively mitigates the dying gradient problem of ReLU.

Abstract

The application of the deep learning model in classification plays an important role in the accurate detection of the target objects. However, the accuracy is affected by the activation function in the hidden and output layer. In this paper, an activation function called TaLU, which is a combination of Tanh and Rectified Linear Units (ReLU), is used to improve the prediction. ReLU activation function is used by many deep learning researchers for its computational efficiency, ease of implementation, intuitive nature, etc. However, it suffers from a dying gradient problem. For instance, when the input is negative, its output is always zero because its gradient is zero. A number of researchers used different approaches to solve this issue. Some of the most notable are LeakyReLU, Softplus, Softsign, ELU, ThresholdedReLU, etc. This research developed TaLU, a modified activation function combining Tanh and ReLU, which mitigates the dying gradient problem of ReLU. The deep learning model with the proposed activation function was tested on MNIST and CIFAR-10, and it outperforms ReLU and some other studied activation functions in terms of accuracy(upto 6% in most cases, when used with Batch Normalization and a reasonable learning rate).