Optimal sampling for least squares approximation with general dictionaries

TL;DR

This paper introduces a practical offline method for estimating orthonormal bases to improve importance sampling in least squares approximation, reducing sample complexity especially in nonlinear approximation contexts.

Contribution

It presents a novel offline strategy for estimating orthonormal bases that does not require function evaluations, enhancing importance sampling efficiency in nonlinear approximation.

Findings

Significant reduction in sample size compared to Monte Carlo methods

Convergence established for the proposed basis estimation strategy

Numerical experiments demonstrate practical effectiveness

Abstract

We consider the problem of approximating an unknown function from point evaluations. This problem is a crucial subproblem in many modern (nonlinear) approximation schemes. When obtaining these point evaluations is costly, minimising the required sample size becomes crucial. Recently, an increasing focus has been on employing importance sampling strategies to achieve this. For the approximation in a $d$-dimensional linear space, an optimal i.i.d. sampling measure achieves a sampling complexity of $\mathcal{O}(d\log (d))$. However, the corresponding sampling measure depends on an orthonormal basis of the linear space, which is rarely known (in particular in the context of nonlinear approximation where the linear space arises as a local linearisation of a nonlinear model class like neural networks or tensor networks). Consequently, sampling from these measures is challenging in practice. This manuscript presents a strategy for estimating an orthonormal basis. This strategy can be performed offline and does not require evaluations of the sought function. We establish convergence and illustrate the practical performance through numerical experiments. Comparing the presented approach with standard Monte Carlo sampling demonstrates a significant reduction in the number of samples required to achieve a good estimation of an orthonormal basis.