Regression in Tensor Product Spaces by the Method of Sieves

TL;DR

This paper introduces penalized sieve estimators for multivariate nonparametric regression in tensor product spaces, effectively addressing high-dimensional challenges and leveraging structural features like sparsity.

Contribution

It proposes a new class of estimators that are computationally feasible and theoretically sound for high-dimensional tensor product spaces, improving over classical methods.

Findings

Estimators scale favorably with dimension.

They outperform several popular machine learning methods in finite samples.

Theoretical guarantees support their predictive performance.

Abstract

Estimation of a conditional mean (linking a set of features to an outcome of interest) is a fundamental statistical task. While there is an appeal to flexible nonparametric procedures, effective estimation in many classical nonparametric function spaces (e.g., multivariate Sobolev spaces) can be prohibitively difficult -- both statistically and computationally -- especially when the number of features is large. In this paper, we present (penalized) sieve estimators for regression in nonparametric tensor product spaces: These spaces are more amenable to multivariate regression, and allow us to, in-part, avoid the curse of dimensionality. Our estimators can be easily applied to multivariate nonparametric problems and have appealing statistical and computational properties. Moreover, they can effectively leverage additional structures such as feature sparsity. In this manuscript, we give theoretical guarantees, indicating that the predictive performance of our estimators scale favorably in dimension. In addition, we also present numerical examples to compare the finite-sample performance of the proposed estimators with several popular machine learning methods.