A new initialisation to Control Gradients in Sinusoidal Neural network

TL;DR

This paper introduces a novel initialization method for sinusoidal neural networks like SIREN, improving gradient control, training stability, and generalization by deriving a closed-form parameter initialization based on fixed points and NTK analysis.

Contribution

The authors propose a new initialization scheme for sinusoidal networks that enhances gradient management and generalization, supported by theoretical analysis and empirical benchmarks.

Findings

The new initialization outperforms original SIREN and baselines in function fitting and image reconstruction.

Controlling gradients via the proposed scheme improves training stability and generalization.

The initialization influences training dynamics significantly through the Neural Tangent Kernel framework.

Abstract

Proper initialisation strategy is of primary importance to mitigate gradient explosion or vanishing when training neural networks. Yet, the impact of initialisation parameters still lacks a precise theoretical understanding for several well-established architectures. Here, we propose a new initialisation for networks with sinusoidal activation functions such as \texttt{SIREN}, focusing on gradients control, their scaling with network depth, their impact on training and on generalization. To achieve this, we identify a closed-form expression for the initialisation of the parameters, differing from the original \texttt{SIREN} scheme. This expression is derived from fixed points obtained through the convergence of pre-activation distribution and the variance of Jacobian sequences. Controlling both gradients and targeting vanishing pre-activation helps preventing the emergence of inappropriate frequencies during estimation, thereby improving generalization. We further show that this initialisation strongly influences training dynamics through the Neural Tangent Kernel framework (NTK). Finally, we benchmark \texttt{SIREN} with the proposed initialisation against the original scheme and other baselines on function fitting and image reconstruction. The new initialisation consistently outperforms state-of-the-art methods across a wide range of reconstruction tasks, including those involving physics-informed neural networks.