Sinusoidal Initialization, Time for a New Start

TL;DR

This paper introduces Sinusoidal initialization, a deterministic method using sinusoidal functions to improve weight distribution, leading to faster convergence, enhanced stability, and higher accuracy in deep neural networks.

Contribution

The paper presents a novel sinusoidal-based initialization method that replaces randomness with structured weights to enhance training efficiency and model performance.

Findings

Average 4.9% increase in final validation accuracy

Average 20.9% faster convergence

Improves stability across various neural network architectures

Abstract

Initialization plays a critical role in Deep Neural Network training, directly influencing convergence, stability, and generalization. Common approaches such as Glorot and He initializations rely on randomness, which can produce uneven weight distributions across layer connections. In this paper, we introduce the Sinusoidal initialization, a novel deterministic method that employs sinusoidal functions to construct structured weight matrices expressly to improve the spread and balance of weights throughout the network while simultaneously fostering a more uniform, well-conditioned distribution of neuron activation states from the very first forward pass. Because Sinusoidal initialization begins with weights and activations that are already evenly and efficiently utilized, it delivers consistently faster convergence, greater training stability, and higher final accuracy across a wide range of models, including convolutional neural networks, vision transformers, and large language models. On average, our experiments show an increase of 4.9% in final validation accuracy and 20.9% in convergence speed. By replacing randomness with structure, this initialization provides a stronger and more reliable foundation for Deep Learning systems.