Revisiting R\'ev\'esz's stochastic approximation method for the estimation of a regression function

TL;DR

This paper revisits Révéesz's stochastic approximation method for recursive kernel estimation of regression functions, improves its convergence assumptions via averaging, and demonstrates its competitive performance against traditional estimators.

Contribution

It introduces an averaged Révéesz's estimator that achieves optimal convergence rates under standard assumptions, overcoming previous limitations of the original method.

Findings

Averaged Révéesz's estimator attains the same optimal convergence rate as Nadaraya-Watson.

The new estimator requires only usual assumptions on the density of X.

It is statistically preferable to Nadaraya-Watson's estimator based on confidence interval analysis.

Abstract

In a pioneer work, R\'ev\'esz (1973) introduces the stochastic approximation method to build up a recursive kernel estimator of the regression function $x\mapsto E(Y|X=x)$. However, according to R\'ev\'esz (1977), his estimator has two main drawbacks: on the one hand, its convergence rate is smaller than that of the nonrecursive Nadaraya-Watson's kernel regression estimator, and, on the other hand, the required assumptions on the density of the random variable $X$ are stronger than those usually needed in the framework of regression estimation. We first come back on the study of the convergence rate of R\'ev\'esz's estimator. An approach in the proofs completely different from that used in R\'ev\'esz (1977) allows us to show that R\'ev\'esz's recursive estimator may reach the same optimal convergence rate as Nadaraya-Watson's estimator, but the required assumptions on the density of $X$ remain stronger than the usual ones, and this is inherent to the definition of R\'ev\'esz's estimator. To overcome this drawback, we introduce the averaging principle of stochastic approximation algorithms to construct the averaged R\'ev\'esz's regression estimator, and give its asymptotic behaviour. Our assumptions on the density of $X$ are then usual in the framework of regression estimation. We prove that the averaged R\'ev\'esz's regression estimator may reach the same optimal convergence rate as Nadaraya-Watson's estimator. Moreover, we show that, according to the estimation by confidence intervals point of view, it is better to use the averaged R\'ev\'esz's estimator rather than Nadaraya-Watson's estimator.