APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks

TL;DR

This paper introduces APALU, a novel trainable activation function that adapts to complex data, significantly improving deep learning performance across tasks like image classification, anomaly detection, and sign language recognition.

Contribution

The paper proposes APALU, a new adaptive activation function that enhances learning stability and efficiency, outperforming traditional static and trainable functions in various deep learning applications.

Findings

APALU increases CIFAR10 accuracy by up to 0.37% for MobileNet.

APALU improves anomaly detection AUC by up to 1.81%.

APALU achieves 100% accuracy in sign language recognition.

Abstract

Activation function is a pivotal component of deep learning, facilitating the extraction of intricate data patterns. While classical activation functions like ReLU and its variants are extensively utilized, their static nature and simplicity, despite being advantageous, often limit their effectiveness in specialized tasks. The trainable activation functions also struggle sometimes to adapt to the unique characteristics of the data. Addressing these limitations, we introduce a novel trainable activation function, adaptive piecewise approximated activation linear unit (APALU), to enhance the learning performance of deep learning across a broad range of tasks. It presents a unique set of features that enable it to maintain stability and efficiency in the learning process while adapting to complex data representations. Experiments reveal significant improvements over widely used activation functions for different tasks. In image classification, APALU increases MobileNet and GoogleNet accuracy by 0.37% and 0.04%, respectively, on the CIFAR10 dataset. In anomaly detection, it improves the average area under the curve of One-CLASS Deep SVDD by 0.8% on the MNIST dataset, 1.81% and 1.11% improvements with DifferNet, and knowledge distillation, respectively, on the MVTech dataset. Notably, APALU achieves 100% accuracy on a sign language recognition task with a limited dataset. For regression tasks, APALU enhances the performance of deep neural networks and recurrent neural networks on different datasets. These improvements highlight the robustness and adaptability of APALU across diverse deep-learning applications.