Dimension lower bounds for linear approaches to function approximation

TL;DR

This paper introduces a linear algebraic technique to establish dimension lower bounds for linear function approximation methods, highlighting fundamental limitations in sample size requirements for kernel methods.

Contribution

It adapts a classical argument to derive new lower bounds on the sample complexity of kernel methods in $L^2$ function approximation.

Findings

Linear algebraic approach to lower bounds

Sample size lower bounds for kernel methods

Extension of classical Kolmogorov width arguments

Abstract

This short note presents a linear algebraic approach to proving dimension lower bounds for linear methods that solve $L^2$ function approximation problems. The basic argument has appeared in the literature before (e.g., Barron, 1993) for establishing lower bounds on Kolmogorov $n$-widths. The argument is applied to give sample size lower bounds for kernel methods.