Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization

TL;DR

This paper introduces two hybrid activation functions, S3 and S4, designed to improve gradient flow and training stability in deep neural networks, demonstrating superior performance and convergence across various tasks.

Contribution

The paper presents novel hybrid activation functions S3 and S4 with a smooth transition mechanism, addressing limitations of traditional functions and enhancing deep network training.

Findings

S4 outperforms nine baseline functions in accuracy and MSE.

S4 achieves 97.4% accuracy on MNIST and 96.0% on Iris.

S4 exhibits faster convergence and stable gradients in deep networks.

Abstract

Activation functions are critical components in deep neural networks, directly influencing gradient flow, training stability, and model performance. Traditional functions like ReLU suffer from dead neuron problems, while sigmoid and tanh exhibit vanishing gradient issues. We introduce two novel hybrid activation functions: S3 (Sigmoid-Softsign) and its improved version S4 (smoothed S3). S3 combines sigmoid for negative inputs with softsign for positive inputs, while S4 employs a smooth transition mechanism controlled by a steepness parameter k. We conducted comprehensive experiments across binary classification, multi-class classification, and regression tasks using three different neural network architectures. S4 demonstrated superior performance compared to nine baseline activation functions, achieving 97.4% accuracy on MNIST, 96.0% on Iris classification, and 18.7 MSE on Boston Housing regression. The function exhibited faster convergence (-19 for ReLU) and maintained stable gradient flow across network depths. Comparative analysis revealed S4's gradient range of [0.24, 0.59] compared to ReLU's 18% dead neurons in deep networks. The S4 activation function addresses key limitations of existing functions through its hybrid design and smooth transition mechanism. The tunable parameter k allows adaptation to different tasks and network depths, making S4 a versatile choice for deep learning applications. These findings suggest that hybrid activation functions represent a promising direction for improving neural network training dynamics.