Nadaraya-Watson kernel smoothing as a random energy model

TL;DR

This paper applies ideas from the random energy model in statistical physics to derive sharp asymptotics for the Nadaraya-Watson kernel estimator in high-dimensional settings, revealing its behavior when sample size grows exponentially with dimension.

Contribution

It introduces a novel approach using the random energy model to analyze the asymptotics of the NW estimator in high dimensions, a problem previously not well-understood.

Findings

NW estimator re-scales the argument of the true function in high dimensions

Provides sharp asymptotic formulas for NW estimator when sample size is exponential in dimension

First detailed understanding of kernel smoothing behavior in high-dimensional regimes

Abstract

Precise asymptotics have revealed many surprises in high-dimensional regression. These advances, however, have not extended to perhaps the simplest estimator: direct Nadaraya-Watson (NW) kernel smoothing. Here, we describe how one can use ideas from the analysis of the random energy model (REM) in statistical physics to compute sharp asymptotics for the NW estimator when the sample size is exponential in the dimension. As a simple starting point for investigation, we focus on the case in which one aims to estimate a single-index target function using a radial basis function kernel on the sphere. Our main result is a pointwise asymptotic for the NW predictor, showing that it re-scales the argument of the true link function. Our work provides a first step towards a detailed understanding of kernel smoothing in high dimensions.