Saturated Non-Monotonic Activation Functions

TL;DR

This paper introduces Saturated Gaussian Error Linear Units, combining ReLU with non-monotonic functions, leading to new activation functions that outperform existing ones in image classification tasks.

Contribution

The paper proposes a novel method to develop non-monotonic activation functions by combining ReLU with the negative parts of GELU, SiLU, and Mish, resulting in three new functions.

Findings

SGELU, SSiLU, and SMish outperform baselines on CIFAR-100

Proposed functions improve deep learning architecture performance

New activation functions are highly effective in image classification

Abstract

Activation functions are essential to deep learning networks. Popular and versatile activation functions are mostly monotonic functions, some non-monotonic activation functions are being explored and show promising performance. But by introducing non-monotonicity, they also alter the positive input, which is proved to be unnecessary by the success of ReLU and its variants. In this paper, we double down on the non-monotonic activation functions' development and propose the Saturated Gaussian Error Linear Units by combining the characteristics of ReLU and non-monotonic activation functions. We present three new activation functions built with our proposed method: SGELU, SSiLU, and SMish, which are composed of the negative portion of GELU, SiLU, and Mish, respectively, and ReLU's positive portion. The results of image classification experiments on CIFAR-100 indicate that our proposed activation functions are highly effective and outperform state-of-the-art baselines across multiple deep learning architectures.