Tangma: A Tanh-Guided Activation Function with Learnable Parameters

TL;DR

Tangma is a novel learnable activation function combining the benefits of tanh with adjustable parameters, leading to improved training stability and accuracy in deep neural networks for vision tasks.

Contribution

This paper introduces Tangma, a new activation function with learnable parameters that enhance training stability and performance in deep neural networks.

Findings

Achieved 99.09% accuracy on MNIST

Outperformed ReLU, Swish, and GELU on CIFAR-10

Enabled faster and more stable training

Abstract

Activation functions are key to effective backpropagation and expressiveness in deep neural networks. This work introduces Tangma, a new activation function that combines the smooth shape of the hyperbolic tangent with two learnable parameters: $\alpha$, which shifts the curve's inflection point to adjust neuron activation, and $\gamma$, which adds linearity to preserve weak gradients and improve training stability. Tangma was evaluated on MNIST and CIFAR-10 using custom networks composed of convolutional and linear layers, and compared against ReLU, Swish, and GELU. On MNIST, Tangma achieved the highest validation accuracy of 99.09% and the lowest validation loss, demonstrating faster and more stable convergence than the baselines. On CIFAR-10, Tangma reached a top validation accuracy of 78.15%, outperforming all other activation functions while maintaining a competitive training loss. Tangma also showed improved training efficiency, with lower average epoch runtimes compared to Swish and GELU. These results suggest that Tangma performs well on standard vision tasks and enables reliable, efficient training. Its learnable design gives more control over activation behavior, which may benefit larger models in tasks such as image recognition or language modeling.