STL: A Signed and Truncated Logarithm Activation Function for Neural Networks

TL;DR

This paper introduces a new signed and truncated logarithm activation function with superior mathematical properties, demonstrating state-of-the-art performance across various neural networks.

Contribution

A novel activation function with enhanced properties like odd symmetry, monotonicity, and continuous gradient, outperforming existing functions in neural network applications.

Findings

Achieves better accuracy than traditional activation functions

Exhibits superior mathematical properties for neural network training

Applicable across diverse neural network architectures

Abstract

Activation functions play an essential role in neural networks. They provide the non-linearity for the networks. Therefore, their properties are important for neural networks' accuracy and running performance. In this paper, we present a novel signed and truncated logarithm function as activation function. The proposed activation function has significantly better mathematical properties, such as being odd function, monotone, differentiable, having unbounded value range, and a continuous nonzero gradient. These properties make it an excellent choice as an activation function. We compare it with other well-known activation functions in several well-known neural networks. The results confirm that it is the state-of-the-art. The suggested activation function can be applied in a large range of neural networks where activation functions are necessary.