Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics

TL;DR

This paper investigates the spectral bias in coordinate-based MLPs by analyzing training dynamics, revealing how activation regions and gradient convergence influence frequency learning, especially in low-dimensional settings, and how positional encoding mitigates this bias.

Contribution

It introduces a novel approach to understanding spectral bias through direct analysis of ReLU MLP training dynamics and their relation to input encoding.

Findings

Spectral bias causes networks to learn low frequencies first.

Positional encoding helps overcome spectral bias in low dimensions.

Training dynamics reveal how activation regions affect convergence to high frequencies.

Abstract

Spectral bias is an important observation of neural network training, stating that the network will learn a low frequency representation of the target function before converging to higher frequency components. This property is interesting due to its link to good generalization in over-parameterized networks. However, in low dimensional settings, a severe spectral bias occurs that obstructs convergence to high frequency components entirely. In order to overcome this limitation, one can encode the inputs using a high frequency sinusoidal encoding. Previous works attempted to explain this phenomenon using Neural Tangent Kernel (NTK) and Fourier analysis. However, NTK does not capture real network dynamics, and Fourier analysis only offers a global perspective on the network properties that induce this bias. In this paper, we provide a novel approach towards understanding spectral bias by directly studying ReLU MLP training dynamics. Specifically, we focus on the connection between the computations of ReLU networks (activation regions), and the speed of gradient descent convergence. We study these dynamics in relation to the spatial information of the signal to understand how they influence spectral bias. We then use this formulation to study the severity of spectral bias in low dimensional settings, and how positional encoding overcomes this.