On Regularizing Coordinate-MLPs

TL;DR

This paper reveals that coordinate-MLPs lack the implicit regularization properties of traditional neural networks, leading to issues in smooth interpolation and generalization, and proposes a simple regularization method to address this.

Contribution

It uncovers the failure of implicit regularization in coordinate-MLPs and introduces an effective explicit regularization technique without changing network architecture.

Findings

Lower frequencies are suppressed as bandwidth increases without prior

Explicit regularization improves interpolation and generalization

Regularization can be added without architectural changes

Abstract

We show that typical implicit regularization assumptions for deep neural networks (for regression) do not hold for coordinate-MLPs, a family of MLPs that are now ubiquitous in computer vision for representing high-frequency signals. Lack of such implicit bias disrupts smooth interpolations between training samples, and hampers generalizing across signal regions with different spectra. We investigate this behavior through a Fourier lens and uncover that as the bandwidth of a coordinate-MLP is enhanced, lower frequencies tend to get suppressed unless a suitable prior is provided explicitly. Based on these insights, we propose a simple regularization technique that can mitigate the above problem, which can be incorporated into existing networks without any architectural modifications.