Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension

TL;DR

This paper introduces a framework using sparse convolutional neural networks to improve the approximation of nonlinear functionals in high-dimensional spaces, reducing sample complexity and addressing the curse of dimensionality.

Contribution

It presents a novel sparse-aware neural network architecture with theoretical guarantees for stable recovery and improved approximation rates in functional learning.

Findings

Sparse approximators enable stable recovery from discrete samples.

Both deterministic and random sampling schemes are effective.

The approach reduces sample sizes and improves approximation in high-dimensional function spaces.

Abstract

Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited interpretability. This work investigates how sparsity can help address these challenges in functional learning, a central ingredient in operator learning. We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples, together with deep fully connected networks to effectively approximate nonlinear functionals. Using universal discretization methods, we show that sparse approximators enable stable recovery from discrete samples. In addition, both the deterministic and the random sampling schemes are sufficient for our analysis. These findings lead to improved approximation rates and reduced sample sizes in various function spaces, including those with fast frequency decay and mixed smoothness. They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.