SERF: Towards better training of deep neural networks using log-Softplus ERror activation Function

TL;DR

This paper introduces Serf, a novel self-regularized, nonmonotonic activation function inspired by Swish, which significantly improves training and performance of deep neural networks across various tasks and architectures.

Contribution

The paper proposes Serf, a new activation function that outperforms ReLU, Swish, and Mish, demonstrating its effectiveness and compatibility across diverse deep learning scenarios.

Findings

Serf outperforms ReLU, Swish, and Mish in multiple tasks.

Serf provides better performance on deeper architectures.

Mathematical analysis shows Serf's regularization effect enhances training.

Abstract

Activation functions play a pivotal role in determining the training dynamics and neural network performance. The widely adopted activation function ReLU despite being simple and effective has few disadvantages including the Dying ReLU problem. In order to tackle such problems, we propose a novel activation function called Serf which is self-regularized and nonmonotonic in nature. Like Mish, Serf also belongs to the Swish family of functions. Based on several experiments on computer vision (image classification and object detection) and natural language processing (machine translation, sentiment classification and multimodal entailment) tasks with different state-of-the-art architectures, it is observed that Serf vastly outperforms ReLU (baseline) and other activation functions including both Swish and Mish, with a markedly bigger margin on deeper architectures. Ablation studies further demonstrate that Serf based architectures perform better than those of Swish and Mish in varying scenarios, validating the effectiveness and compatibility of Serf with varying depth, complexity, optimizers, learning rates, batch sizes, initializers and dropout rates. Finally, we investigate the mathematical relation between Swish and Serf, thereby showing the impact of preconditioner function ingrained in the first derivative of Serf which provides a regularization effect making gradients smoother and optimization faster.