Regularized Flexible Activation Function Combinations for Deep Neural Networks

TL;DR

This paper introduces a new family of flexible activation functions for deep neural networks, along with regularization techniques, demonstrating improved performance in time series forecasting and image compression tasks.

Contribution

It proposes a general framework for flexible activation functions, new specific functions for LSTM and auto-encoders, and regularization methods to enhance model stability and performance.

Findings

Flexible activation functions improve task-specific performance.

Regularization enhances convergence and stability.

New activation functions outperform traditional ones in experiments.

Abstract

Activation in deep neural networks is fundamental to achieving non-linear mappings. Traditional studies mainly focus on finding fixed activations for a particular set of learning tasks or model architectures. The research on flexible activation is quite limited in both designing philosophy and application scenarios. In this study, three principles of choosing flexible activation components are proposed and a general combined form of flexible activation functions is implemented. Based on this, a novel family of flexible activation functions that can replace sigmoid or tanh in LSTM cells are implemented, as well as a new family by combining ReLU and ELUs. Also, two new regularisation terms based on assumptions as prior knowledge are introduced. It has been shown that LSTM models with proposed flexible activations P-Sig-Ramp provide significant improvements in time series forecasting, while the proposed P-E2-ReLU achieves better and more stable performance on lossy image compression tasks with convolutional auto-encoders. In addition, the proposed regularization terms improve the convergence, performance and stability of the models with flexible activation functions.