Towards the Spectral bias Alleviation by Normalizations in Coordinate Networks

TL;DR

This paper investigates how classical normalization techniques can mitigate spectral bias in coordinate networks, leading to improved high-frequency component learning and state-of-the-art results across multiple scientific computing tasks.

Contribution

It demonstrates that normalization methods reduce NTK eigenvalue variance, alleviating spectral bias, and introduces two new normalization techniques with broad applicability.

Findings

Normalization reduces NTK eigenvalue variance.

Normalized coordinate networks achieve state-of-the-art results.

Spectral bias alleviation improves high-frequency learning.

Abstract

Representing signals using coordinate networks dominates the area of inverse problems recently, and is widely applied in various scientific computing tasks. Still, there exists an issue of spectral bias in coordinate networks, limiting the capacity to learn high-frequency components. This problem is caused by the pathological distribution of the neural tangent kernel's (NTK's) eigenvalues of coordinate networks. We find that, this pathological distribution could be improved using classical normalization techniques (batch normalization and layer normalization), which are commonly used in convolutional neural networks but rarely used in coordinate networks. We prove that normalization techniques greatly reduces the maximum and variance of NTK's eigenvalues while slightly modifies the mean value, considering the max eigenvalue is much larger than the most, this variance change results in a shift of eigenvalues' distribution from a lower one to a higher one, therefore the spectral bias could be alleviated. Furthermore, we propose two new normalization techniques by combining these two techniques in different ways. The efficacy of these normalization techniques is substantiated by the significant improvements and new state-of-the-arts achieved by applying normalization-based coordinate networks to various tasks, including the image compression, computed tomography reconstruction, shape representation, magnetic resonance imaging, novel view synthesis and multi-view stereo reconstruction.