Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

TL;DR

This paper demonstrates that Fourier feature mappings enable neural networks to learn high-frequency functions in low-dimensional spaces, addressing spectral bias and improving performance in computer vision and graphics tasks.

Contribution

The paper introduces Fourier feature mappings to transform neural tangent kernels, allowing standard MLPs to effectively learn high-frequency functions in low-dimensional domains.

Findings

Fourier features enable MLPs to learn high frequencies.

Transforming NTK with Fourier features creates a stationary kernel.

Problem-specific Fourier features significantly improve performance.

Abstract

We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.