Scalable Statistical Inference in Non-parametric Least Squares

TL;DR

This paper develops a theoretical framework for applying stochastic approximation to non-parametric regression in RKHS, enabling online inference with valid confidence intervals using an online bootstrap method.

Contribution

It introduces a unified approach for non-asymptotic analysis and bootstrap-based confidence intervals in functional stochastic gradient descent within RKHS.

Findings

Valid pointwise and simultaneous confidence intervals are constructed.

The framework demonstrates the consistency of the multiplier bootstrap method.

The analysis reveals the impact of step size tuning on inference accuracy.

Abstract

Stochastic approximation (SA) is a powerful and scalable computational method for iteratively estimating the solution of optimization problems in the presence of randomness, particularly well-suited for large-scale and streaming data settings. In this work, we propose a theoretical framework for stochastic approximation (SA) applied to non-parametric least squares in reproducing kernel Hilbert spaces (RKHS), enabling online statistical inference in non-parametric regression models. We achieve this by constructing asymptotically valid pointwise (and simultaneous) confidence intervals (bands) for local (and global) inference of the nonlinear regression function, via employing an online multiplier bootstrap approach to functional stochastic gradient descent (SGD) algorithm in the RKHS. Our main theoretical contributions consist of a unified framework for characterizing the non-asymptotic behavior of the functional SGD estimator and demonstrating the consistency of the multiplier bootstrap method. The proof techniques involve the development of a higher-order expansion of the functional SGD estimator under the supremum norm metric and the Gaussian approximation of suprema of weighted and non-identically distributed empirical processes. Our theory specifically reveals an interesting relationship between the tuning of step sizes in SGD for estimation and the accuracy of uncertainty quantification.