Convergence bounds for local least squares approximation

TL;DR

This paper derives improved convergence bounds for local least squares approximation in nonlinear function spaces, focusing on sample complexity estimates near optimal approximations, especially for manifolds with positive local reach.

Contribution

It introduces a method to obtain tighter worst-case sample complexity bounds for nonlinear models by considering neighborhoods around best approximations, leveraging manifold geometry.

Findings

Sample complexity bounds depend on tangent and normal spaces.

Manifold curvature influences approximation bounds.

Improved bounds for models like tensor networks and neural networks.

Abstract

We consider the problem of approximating a function in a general nonlinear subset of $L^2$, when only a weighted Monte Carlo estimate of the $L^2$-norm can be computed. Of particular interest in this setting is the concept of sample complexity, the number of sample points that are necessary to achieve a prescribed error with high probability. Reasonable worst-case bounds for this quantity exist only for particular model classes, like linear spaces or sets of sparse vectors. For more general sets, like tensor networks or neural networks, the currently existing bounds are very pessimistic. By restricting the model class to a neighbourhood of the best approximation, we can derive improved worst-case bounds for the sample complexity. When the considered neighbourhood is a manifold with positive local reach, its sample complexity can be estimated by means of the sample complexities of the tangent and normal spaces and the manifold's curvature.