Optimal Stable Nonlinear Approximation

TL;DR

This paper introduces stable manifold widths as a measure of optimal nonlinear approximation performance in Banach spaces, establishing key inequalities and discussing implications for deep learning and compressed sensing.

Contribution

It defines stable manifold widths and links them to entropy, providing a new framework for understanding optimal stable nonlinear approximation.

Findings

Established inequalities between stable manifold widths and entropy.

Discussed stability effects in deep learning and compressed sensing.

Proposed a new measure for optimal nonlinear approximation performance.

Abstract

While it is well known that nonlinear methods of approximation can often perform dramatically better than linear methods, there are still questions on how to measure the optimal performance possible for such methods. This paper studies nonlinear methods of approximation that are compatible with numerical implementation in that they are required to be numerically stable. A measure of optimal performance, called {\em stable manifold widths}, for approximating a model class $K$ in a Banach space $X$ by stable manifold methods is introduced. Fundamental inequalities between these stable manifold widths and the entropy of $K$ are established. The effects of requiring stability in the settings of deep learning and compressed sensing are discussed.