Fast Training of Sinusoidal Neural Fields via Scaling Initialization

TL;DR

This paper introduces a simple weight scaling technique that significantly accelerates the training of sinusoidal neural fields, reducing training time by a factor of ten and improving convergence and spectral bias handling.

Contribution

The paper proposes a novel weight scaling initialization method for sinusoidal neural fields that enhances training speed and stability, outperforming existing architectures.

Findings

Weight scaling accelerates SNF training by 10x.

The method improves spectral bias handling.

It results in better-conditioned optimization trajectories.

Abstract

Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high training cost, which prevents a broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized to maximize the training speed. We find that the standard initialization scheme for SNFs -- designed based on the signal propagation principle -- is suboptimal. In particular, we show that by simply multiplying each weight (except for the last layer) by a constant, we can accelerate SNF training by 10$\times$. This method, coined $\textit{weight scaling}$, consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory. The code is available $\href{https://github.com/effl-lab/Fast-Neural-Fields}{here}$.