The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

TL;DR

Pretraining implicit neural representations on unstructured noise significantly enhances their ability to fit signals, while spectral noise balances this with effective denoising, offering a new approach for INR initialization.

Contribution

The paper demonstrates that noise pretraining, especially spectral noise, improves INR performance and provides insights into the role of noise priors in signal approximation.

Findings

Unstructured noise pretraining improves signal fitting capacity.

Spectral noise pretraining balances signal fitting and denoising capabilities.

Spectral noise pretraining matches the performance of data-driven methods.

Abstract

The approximation and convergence properties of implicit neural representations (INRs) are known to be highly sensitive to parameter initialization strategies. While several data-driven initialization methods demonstrate significant improvements over standard random sampling, the reasons for their success -- specifically, whether they encode classical statistical signal priors or more complex features -- remain poorly understood. In this study, we explore this phenomenon through a series of experimental analyses leveraging noise pretraining. We pretrain INRs on diverse noise classes (e.g., Gaussian, Dead Leaves, Spectral) and measure their ability to both fit unseen signals and encode priors for an inverse imaging task (denoising). Our analyses on image and video data reveal a surprising finding: simply pretraining on unstructured noise (Uniform, Gaussian) dramatically improves signal fitting capacity compared to all other baselines. However, unstructured noise also yields poor deep image priors for denoising. In contrast, we also find that noise with the classic $1/|f^\alpha|$ spectral structure of natural images achieves an excellent balance of signal fitting and inverse imaging capabilities, performing on par with the best data-driven initialization methods. This finding enables more efficient INR training in applications lacking sufficient prior domain-specific data. For more details, visit project page at https://kushalvyas.github.io/noisepretraining.html