A Sieve Stochastic Gradient Descent Estimator for Online Nonparametric Regression in Sobolev ellipsoids

TL;DR

This paper introduces Sieve-SGD, an efficient online nonparametric regression estimator for Sobolev ellipsoids that achieves optimal convergence rates with minimal memory and computational costs.

Contribution

It proposes a novel sieve stochastic gradient descent method tailored for Sobolev spaces, achieving rate-optimal accuracy with low computational and memory requirements.

Findings

Sieve-SGD attains rate-optimal mean squared error.

The estimator requires minimal memory among rate-optimal methods.

It is computationally efficient for online nonparametric regression.

Abstract

The goal of regression is to recover an unknown underlying function that best links a set of predictors to an outcome from noisy observations. In nonparametric regression, one assumes that the regression function belongs to a pre-specified infinite-dimensional function space (the hypothesis space). In the online setting, when the observations come in a stream, it is computationally-preferable to iteratively update an estimate rather than refitting an entire model repeatedly. Inspired by nonparametric sieve estimation and stochastic approximation methods, we propose a sieve stochastic gradient descent estimator (Sieve-SGD) when the hypothesis space is a Sobolev ellipsoid. We show that Sieve-SGD has rate-optimal mean squared error (MSE) under a set of simple and direct conditions. The proposed estimator can be constructed with a low computational (time and space) expense: We also formally show that Sieve-SGD requires almost minimal memory usage among all statistically rate-optimal estimators.