Robust Fourier Neural Networks

TL;DR

This paper introduces a simple diagonal layer after Fourier embedding in neural networks, significantly enhancing robustness to noisy measurements and promoting sparse Fourier feature learning, supported by theoretical analysis and numerical validation.

Contribution

It proposes a novel diagonal layer addition to Fourier neural networks, improving noise robustness and sparse feature learning, with theoretical justifications and empirical validation.

Findings

Enhanced robustness to measurement noise

Effective learning of sparse Fourier features

Validated through numerical experiments

Abstract

Fourier embedding has shown great promise in removing spectral bias during neural network training. However, it can still suffer from high generalization errors, especially when the labels or measurements are noisy. We demonstrate that introducing a simple diagonal layer after the Fourier embedding layer makes the network more robust to measurement noise, effectively prompting it to learn sparse Fourier features. We provide theoretical justifications for this Fourier feature learning, leveraging recent developments in diagonal networks and implicit regularization in neural networks. Under certain conditions, our proposed approach can also learn functions that are noisy mixtures of nonlinear functions of Fourier features. Numerical experiments validate the effectiveness of our proposed architecture, supporting our theory.