Effectiveness of Scaled Exponentially-Regularized Linear Units (SERLUs)

TL;DR

This paper introduces SERLU, a novel activation function that breaks the monotonicity of SELU while maintaining self-normalization, leading to improved neural network performance on standard datasets.

Contribution

The paper proposes SERLU, a new activation function with a bump shape that preserves self-normalization and enhances neural network training, along with a shift-dropout technique.

Findings

SERLU outperforms ELU, SELU, Swish, Leaky ReLU, and ReLU on MNIST, CIFAR10, and CIFAR100.

SERLU effectively maintains zero mean output and combats overfitting.

Experimental results demonstrate consistent improvements across multiple datasets.

Abstract

Recently, self-normalizing neural networks (SNNs) have been proposed with the intention to avoid batch or weight normalization. The key step in SNNs is to properly scale the exponential linear unit (referred to as SELU) to inherently incorporate normalization based on central limit theory. SELU is a monotonically increasing function, where it has an approximately constant negative output for large negative input. In this work, we propose a new activation function to break the monotonicity property of SELU while still preserving the self-normalizing property. Differently from SELU, the new function introduces a bump-shaped function in the region of negative input by regularizing a linear function with a scaled exponential function, which is referred to as a scaled exponentially-regularized linear unit (SERLU). The bump-shaped function has approximately zero response to large negative input while being able to push the output of SERLU towards zero mean statistically. To effectively combat over-fitting, we develop a so-called shift-dropout for SERLU, which includes standard dropout as a special case. Experimental results on MNIST, CIFAR10 and CIFAR100 show that SERLU-based neural networks provide consistently promising results in comparison to other 5 activation functions including ELU, SELU, Swish, Leakly ReLU and ReLU.