ErfAct and Pserf: Non-monotonic Smooth Trainable Activation Functions

TL;DR

This paper introduces two novel non-monotonic smooth trainable activation functions, ErfAct and Pserf, which significantly enhance neural network performance across various models and datasets compared to traditional activations.

Contribution

The paper proposes ErfAct and Pserf, two new non-monotonic smooth trainable activation functions that outperform existing functions like ReLU, Swish, and Mish in multiple neural network benchmarks.

Findings

ErfAct and Pserf improve top-1 accuracy on CIFAR datasets by over 5%.

They enhance mean average precision on SSD300 by 1%.

The proposed functions outperform ReLU, Swish, and Mish in experiments.

Abstract

An activation function is a crucial component of a neural network that introduces non-linearity in the network. The state-of-the-art performance of a neural network depends also on the perfect choice of an activation function. We propose two novel non-monotonic smooth trainable activation functions, called ErfAct and Pserf. Experiments suggest that the proposed functions improve the network performance significantly compared to the widely used activations like ReLU, Swish, and Mish. Replacing ReLU by ErfAct and Pserf, we have 5.68% and 5.42% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in CIFAR100 dataset, 2.11% and 1.96% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in CIFAR10 dataset, 1.0%, and 1.0% improvement on mean average precision (mAP) on SSD300 model in Pascal VOC dataset.