On Regularizing Coordinate-MLPs
Sameera Ramasinghe, Lachlan MacDonald, Simon Lucey

TL;DR
This paper reveals that coordinate-MLPs lack the implicit regularization properties of traditional neural networks, leading to issues in smooth interpolation and generalization, and proposes a simple regularization method to address this.
Contribution
It uncovers the failure of implicit regularization in coordinate-MLPs and introduces an effective explicit regularization technique without changing network architecture.
Findings
Lower frequencies are suppressed as bandwidth increases without prior
Explicit regularization improves interpolation and generalization
Regularization can be added without architectural changes
Abstract
We show that typical implicit regularization assumptions for deep neural networks (for regression) do not hold for coordinate-MLPs, a family of MLPs that are now ubiquitous in computer vision for representing high-frequency signals. Lack of such implicit bias disrupts smooth interpolations between training samples, and hampers generalizing across signal regions with different spectra. We investigate this behavior through a Fourier lens and uncover that as the bandwidth of a coordinate-MLP is enhanced, lower frequencies tend to get suppressed unless a suitable prior is provided explicitly. Based on these insights, we propose a simple regularization technique that can mitigate the above problem, which can be incorporated into existing networks without any architectural modifications.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStructural Health Monitoring Techniques · Neural Networks and Applications · Model Reduction and Neural Networks
