ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance

TL;DR

This paper introduces ALReLU, a novel activation function based on the absolute value of Leaky ReLU's negative gradient, which significantly improves neural network performance in image, text, and tabular data classification tasks.

Contribution

ALReLU is a new activation function that addresses the dying ReLU problem by utilizing the absolute value of Leaky ReLU's negative gradient, enhancing neural network performance.

Findings

ALReLU outperforms ReLU and Leaky ReLU in image classification tasks.

ALReLU improves accuracy on disease classification datasets.

ALReLU enhances performance across diverse data types and datasets.

Abstract

Despite the unresolved 'dying ReLU problem', the classical ReLU activation function (AF) has been extensively applied in Deep Neural Networks (DNN), in particular Convolutional Neural Networks (CNN), for image classification. The common gradient issues of ReLU pose challenges in applications on academy and industry sectors. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the Absolute Leaky ReLU (ALReLU) AF, a variation of LReLU, is proposed, as an alternative method to resolve the common 'dying ReLU problem' on NN-based algorithms for supervised learning. The experimental results demonstrate that by using the absolute values of LReLU's small negative gradient, has a significant improvement in comparison with LReLU and ReLU, on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets.