Understanding NTK Variance in Implicit Neural Representations

TL;DR

This paper analyzes how architectural choices in Implicit Neural Representations influence the Neural Tangent Kernel's eigenvalue variance, affecting spectral bias and convergence speed.

Contribution

It provides a unified theoretical framework linking INR design choices to NTK conditioning and spectral bias mitigation.

Findings

Positional encoding reshapes input similarity.

Spherical normalization reduces NTK variance.

Hadamard modulation decreases eigenvalue dispersion.

Abstract

Implicit Neural Representations (INRs) often converge slowly and struggle to recover high-frequency details due to spectral bias. While prior work links this behavior to the Neural Tangent Kernel (NTK), how specific architectural choices affect NTK conditioning remains unclear. We show that many INR mechanisms can be understood through their impact on a small set of pairwise similarity factors and scaling terms that jointly determine NTK eigenvalue variance. For standard coordinate MLPs, limited input-feature interactions induce large eigenvalue dispersion and poor conditioning. We derive closed-form variance decompositions for common INR components and show that positional encoding reshapes input similarity, spherical normalization reduces variance via layerwise scaling, and Hadamard modulation introduces additional similarity factors strictly below one, yielding multiplicative variance reduction. This unified view explains how diverse INR architectures mitigate spectral bias by improving NTK conditioning. Experiments across multiple tasks confirm the predicted variance reductions and demonstrate faster, more stable convergence with improved reconstruction quality.