Time-Frequency Analysis for Neural Networks

TL;DR

This paper introduces a new approximation theory for shallow neural networks using time-frequency analysis, demonstrating improved Sobolev approximation rates over standard networks.

Contribution

It develops a dimension-independent approximation framework in modulation spaces, linking neural network approximation to time-frequency analysis and providing explicit error bounds.

Findings

Modulation-based networks outperform ReLU networks in Sobolev approximation.

Achieves dimension-independent approximation rates in Sobolev norms.

Provides theoretical and numerical evidence for improved neural network approximation.

Abstract

We develop a quantitative approximation theory for shallow neural networks using tools from time-frequency analysis. Working in weighted modulation spaces $M^{p,q}_m(\mathbf{R}^{d})$, we prove dimension-independent approximation rates in Sobolev norms $W^{n,r}(\Omega)$ for networks whose units combine standard activations with localized time-frequency windows. Our main result shows that for $f \in M^{p,q}_m(\mathbf{R}^{d})$ one can achieve \[ \|f - f_N\|_{W^{n,r}(\Omega)} \lesssim N^{-1/2}\,\|f\|_{M^{p,q}_m(\mathbf{R}^{d})}, \] on bounded domains, with explicit control of all constants. We further obtain global approximation theorems on $\mathbf{R}^{d}$ using weighted modulation dictionaries, and derive consequences for Feichtinger's algebra, Fourier-Lebesgue spaces, and Barron spaces. Numerical experiments in one and two dimensions confirm that modulation-based networks achieve substantially better Sobolev approximation than standard ReLU networks, consistent with the theoretical estimates.