Target noise: A pre-training based neural network initialization for efficient high resolution learning

TL;DR

This paper introduces a noise-based self-supervised pre-training method for neural network initialization, significantly enhancing convergence speed and stability, especially for high-resolution and implicit neural representations, without extra data or architecture changes.

Contribution

Proposes a novel noise-driven pre-training strategy that leverages random noise as a self-supervised signal to improve neural network initialization and optimization efficiency.

Findings

Pre-training with random noise accelerates convergence in subsequent tasks.

The method is particularly effective for implicit neural representations and Deep Image Prior networks.

Noise pre-training enables earlier capture of high-frequency image components.

Abstract

Weight initialization plays a crucial role in the optimization behavior and convergence efficiency of neural networks. Most existing initialization methods, such as Xavier and Kaiming initializations, rely on random sampling and do not exploit information from the optimization process itself. We propose a simple, yet effective, initialization strategy based on self-supervised pre-training using random noise as the target. Instead of directly training the network from random weights, we first pre-train it to fit random noise, which leads to a structured and non-random parameter configuration. We show that this noise-driven pre-training significantly improves convergence speed in subsequent tasks, without requiring additional data or changes to the network architecture. The proposed method is particularly effective for implicit neural representations (INRs) and Deep Image Prior (DIP)-style networks, which are known to exhibit a strong low-frequency bias during optimization. After noise-based pre-training, the network is able to capture high-frequency components much earlier in training, leading to faster and more stable convergence. Although random noise contains no semantic information, it serves as an effective self-supervised signal (considering its white spectrum nature) for shaping the initialization of neural networks. Overall, this work demonstrates that noise-based pre-training offers a lightweight and general alternative to traditional random initialization, enabling more efficient optimization of deep neural networks.