Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations

TL;DR

This paper introduces Inductive Gradient Adjustment (IGA), a novel method to improve spectral bias in Implicit Neural Representations by leveraging the neural tangent kernel, resulting in more detailed and sharper INRs.

Contribution

The paper presents a theoretical link between spectral bias and training dynamics via eNTK and proposes IGA to enhance INRs, validated through theoretical and empirical analyses.

Findings

IGA improves spectral bias in INRs

Enhanced INRs produce more detailed textures and sharper edges

Method outperforms existing training techniques

Abstract

Implicit Neural Representations (INRs), as a versatile representation paradigm, have achieved success in various computer vision tasks. Due to the spectral bias of the vanilla multi-layer perceptrons (MLPs), existing methods focus on designing MLPs with sophisticated architectures or repurposing training techniques for highly accurate INRs. In this paper, we delve into the linear dynamics model of MLPs and theoretically identify the empirical Neural Tangent Kernel (eNTK) matrix as a reliable link between spectral bias and training dynamics. Based on this insight, we propose a practical Inductive Gradient Adjustment (IGA) method, which could purposefully improve the spectral bias via inductive generalization of eNTK-based gradient transformation matrix. Theoretical and empirical analyses validate impacts of IGA on spectral bias. Further, we evaluate our method on different INRs tasks with various INR architectures and compare to existing training techniques. The superior and consistent improvements clearly validate the advantage of our IGA. Armed with our gradient adjustment method, better INRs with more enhanced texture details and sharpened edges can be learned from data by tailored impacts on spectral bias.