FreSh: Frequency Shifting for Accelerated Neural Representation Learning

TL;DR

FreSh introduces a frequency shifting technique that optimizes neural representations by aligning initial spectral properties with target signals, reducing hyperparameter tuning costs while maintaining high performance.

Contribution

The paper proposes a novel frequency shifting method that improves neural representation learning by aligning initial spectral properties, reducing the need for extensive hyperparameter searches.

Findings

FreSh improves performance across various tasks.

It achieves results comparable to hyperparameter grid search.

The method requires minimal additional computation.

Abstract

Implicit Neural Representations (INRs) have recently gained attention as a powerful approach for continuously representing signals such as images, videos, and 3D shapes using multilayer perceptrons (MLPs). However, MLPs are known to exhibit a low-frequency bias, limiting their ability to capture high-frequency details accurately. This limitation is typically addressed by incorporating high-frequency input embeddings or specialized activation layers. In this work, we demonstrate that these embeddings and activations are often configured with hyperparameters that perform well on average but are suboptimal for specific input signals under consideration, necessitating a costly grid search to identify optimal settings. Our key observation is that the initial frequency spectrum of an untrained model's output correlates strongly with the model's eventual performance on a given target signal. Leveraging this insight, we propose frequency shifting (or FreSh), a method that selects embedding hyperparameters to align the frequency spectrum of the model's initial output with that of the target signal. We show that this simple initialization technique improves performance across various neural representation methods and tasks, achieving results comparable to extensive hyperparameter sweeps but with only marginal computational overhead compared to training a single model with default hyperparameters.