A new Kernel Regression approach for Robustified $L_2$ Boosting

TL;DR

This paper introduces a robustified kernel regression boosting method using symmetric, positive definite smoothers, demonstrating improved prediction accuracy and optimal convergence rates, especially in the presence of outliers.

Contribution

It proposes a novel robustified boosting algorithm with low-rank smoothers, providing theoretical insights and empirical evidence of enhanced performance over traditional methods.

Findings

Low-rank smoothers can outperform full-rank smoothers in prediction accuracy.

The boosting estimator with low-rank smoother achieves the optimal convergence rate.

The robustified boosting algorithm improves performance with outliers.

Abstract

We investigate $L_2$ boosting in the context of kernel regression. Kernel smoothers, in general, lack appealing traits like symmetry and positive definiteness, which are critical not only for understanding theoretical aspects but also for achieving good practical performance. We consider a projection-based smoother (Huang and Chen, 2008) that is symmetric, positive definite, and shrinking. Theoretical results based on the orthonormal decomposition of the smoother reveal additional insights into the boosting algorithm. In our asymptotic framework, we may replace the full-rank smoother with a low-rank approximation. We demonstrate that the smoother's low-rank ($d(n)$) is bounded above by $O(h^{-1})$, where $h$ is the bandwidth. Our numerical findings show that, in terms of prediction accuracy, low-rank smoothers may outperform full-rank smoothers. Furthermore, we show that the boosting estimator with low-rank smoother achieves the optimal convergence rate. Finally, to improve the performance of the boosting algorithm in the presence of outliers, we propose a novel robustified boosting algorithm which can be used with any smoother discussed in the study. We investigate the numerical performance of the proposed approaches using simulations and a real-world case.