TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks

TL;DR

The paper introduces TSSR, a new activation function for neural networks that is odd, nonlinear, monotone, differentiable, and has a continuous positive gradient, aiming to enhance numerical stability and performance across various applications.

Contribution

The paper proposes the TSSR activation function, which offers improved stability and performance over existing functions, with potential broad applicability in neural network models.

Findings

TSSR outperforms other state-of-the-art activation functions in experiments.

TSSR improves numerical stability in neural networks.

TSSR is applicable across diverse fields like computer vision and NLP.

Abstract

Activation functions are essential components of neural networks. In this paper, we introduce a new activation function called the Truncated and Signed Square Root (TSSR) function. This function is distinctive because it is odd, nonlinear, monotone and differentiable. Its gradient is continuous and always positive. Thanks to these properties, it has the potential to improve the numerical stability of neural networks. Several experiments confirm that the proposed TSSR has better performance than other stat-of-the-art activation functions. The proposed function has significant implications for the development of neural network models and can be applied to a wide range of applications in fields such as computer vision, natural language processing, and speech recognition.