Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

TL;DR

This paper introduces FMMNN, a novel neural network architecture that effectively models high-frequency functions, offers exponential approximation power, and improves training efficiency through a specialized initialization method.

Contribution

The paper proposes FMMNN, a new neural network model that synergizes architecture and activation functions to enhance high-frequency learning and provides theoretical and empirical validation.

Findings

FMMNN achieves exponential expressive power.

FMMNN's optimization landscape is more favorable.

The scaled random initialization improves training speed and accuracy.

Abstract

The architecture of a neural network and the selection of its activation function are both fundamental to its performance. Equally vital is ensuring these two elements are well-matched, as their alignment is key to achieving effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a novel model that creates a strong synergy between them. We demonstrate that FMMNNs are highly effective and flexible in modeling high-frequency components. Our theoretical results demonstrate that FMMNNs have exponential expressive power for function approximation. We also analyze the optimization landscape of FMMNNs and find it to be much more favorable than that of standard fully connected neural networks, especially when dealing with high-frequency features. In addition, we propose a scaled random initialization method for the first layer's weights in FMMNNs, which significantly speeds up training and enhances overall performance. Extensive numerical experiments support our theoretical insights, showing that FMMNNs consistently outperform traditional approaches in accuracy and efficiency across various tasks.