Finite Sample Bounds for Non-Parametric Regression: Optimal Sample Efficiency and Space Complexity

TL;DR

This paper introduces a finite-dimensional, parametric method for nonparametric regression that achieves optimal convergence rates and minimal memory use, suitable for real-time applications like reinforcement learning.

Contribution

It proposes a novel finite-dimensional approach that attains minimax-optimal rates and reduces computational and memory costs compared to traditional kernel methods.

Findings

Achieves minimax-optimal uniform convergence rates.

Provides sharp finite-sample bounds under sub-Gaussian noise.

Proves matching lower bounds confirming optimality.

Abstract

We address the problem of learning an unknown smooth function and its derivatives from noisy pointwise evaluations under the supremum norm. While classical nonparametric regression provides a strong theoretical foundation, traditional kernel-based estimators often incur high computational costs and memory requirements that scale with the sample size, limiting their utility in real-time applications such as reinforcement learning. To overcome these challenges, we propose a parametric approach based on a finite-dimensional representation that achieves minimax-optimal uniform convergence rates. Our method enables lightweight inference without storing all samples in memory. We provide sharp finite-sample bounds under sub-Gaussian noise, derive second-order Bernstein-type guarantees, and prove matching lower bounds, thereby confirming the optimality of our approach in both estimation error and memory efficiency.