Why Shallow Networks Struggle to Approximate and Learn High Frequencies

TL;DR

This paper investigates why shallow neural networks struggle with high-frequency functions, analyzing the roles of numerical precision, computational cost, and stability, supported by mathematical and computational evidence.

Contribution

It offers a detailed analysis of the computational and numerical challenges that hinder shallow networks from learning high frequencies effectively.

Findings

Shallow networks face fundamental limitations in approximating high frequencies.

Numerical errors and stability issues significantly impact learning high-frequency functions.

Computational cost increases substantially when approximating high frequencies.

Abstract

In this work, we present a comprehensive study combining mathematical and computational analysis to explain why a two-layer neural network struggles to handle high frequencies in both approximation and learning, especially when machine precision, numerical noise, and computational cost are significant factors in practice. Specifically, we investigate the following fundamental computational issues: (1) the minimal numerical error achievable under finite precision, (2) the computational cost required to attain a given accuracy, and (3) the stability of the method with respect to perturbations. The core of our analysis lies in the conditioning of the representation and its learning dynamics. Explicit answers to these questions are provided, along with supporting numerical evidence.