Local Adaptivity of Gradient Boosting in Histogram Transform Ensemble Learning

TL;DR

This paper introduces ABHT, a gradient boosting algorithm for regression that adaptively handles local smoothness variations, outperforming traditional ensemble methods both theoretically and empirically.

Contribution

The paper proposes ABHT, a novel gradient boosting method that adaptively filters regions with different smoothness levels, with proven faster convergence rates than existing ensemble approaches.

Findings

ABHT outperforms PEHT in convergence rates.

ABHT effectively filters regions with varying smoothness.

Experimental results validate theoretical advantages.

Abstract

In this paper, we propose a gradient boosting algorithm called \textit{adaptive boosting histogram transform} (\textit{ABHT}) for regression to illustrate the local adaptivity of gradient boosting algorithms in histogram transform ensemble learning. From the theoretical perspective, when the target function lies in a locally H\"older continuous space, we show that our ABHT can filter out the regions with different orders of smoothness. Consequently, we are able to prove that the upper bound of the convergence rates of ABHT is strictly smaller than the lower bound of \textit{parallel ensemble histogram transform} (\textit{PEHT}). In the experiments, both synthetic and real-world data experiments empirically validate the theoretical results, which demonstrates the advantageous performance and local adaptivity of our ABHT.