Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis

TL;DR

This paper introduces a new weight initialization technique for tanh neural networks based on fixed point analysis, improving training robustness, convergence speed, and data efficiency over traditional methods.

Contribution

It proposes a novel initialization method derived from fixed point analysis of tanh, addressing activation saturation issues and outperforming Xavier initialization.

Findings

Outperforms Xavier initialization in robustness and convergence speed

Effective across various network sizes and datasets

Enhances data efficiency in training deep tanh networks

Abstract

As a neural network's depth increases, it can improve generalization performance. However, training deep networks is challenging due to gradient and signal propagation issues. To address these challenges, extensive theoretical research and various methods have been introduced. Despite these advances, effective weight initialization methods for tanh neural networks remain insufficiently investigated. This paper presents a novel weight initialization method for neural networks with tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, the proposed method aims to determine values of $a$ that mitigate activation saturation. A series of experiments on various classification datasets and physics-informed neural networks demonstrates that the proposed method outperforms Xavier initialization methods~(with or without normalization) in terms of robustness across different network sizes, data efficiency, and convergence speed. Code is available at https://github.com/1HyunwooLee/Tanh-Init