$L_2$ boosting in kernel regression

TL;DR

This paper analyzes the theoretical and empirical benefits of $L_2$ boosting in kernel regression, showing it significantly reduces bias without increasing variance, outperforming higher-order kernels.

Contribution

It provides a novel theoretical analysis of $L_2$ boosting in kernel regression and demonstrates its superiority over higher-order kernels through simulations.

Findings

Each boosting step reduces bias by two orders of magnitude.

Variance remains unaffected by boosting.

$L_2$ boosting outperforms higher-order kernels in bias reduction.

Abstract

In this paper, we investigate the theoretical and empirical properties of $L_2$ boosting with kernel regression estimates as weak learners. We show that each step of $L_2$ boosting reduces the bias of the estimate by two orders of magnitude, while it does not deteriorate the order of the variance. We illustrate the theoretical findings by some simulated examples. Also, we demonstrate that $L_2$ boosting is superior to the use of higher-order kernels, which is a well-known method of reducing the bias of the kernel estimate.