LinXGBoost: Extension of XGBoost to Generalized Local Linear Models

TL;DR

LinXGBoost extends XGBoost by incorporating linear models at each leaf, improving regression performance on functions with jumps or discontinuities, especially in low-dimensional feature spaces.

Contribution

This paper introduces LinXGBoost, a novel extension of XGBoost that uses local linear models at leaves, enhancing its ability to regress complex functions.

Findings

Outperforms vanilla XGBoost and Random Forest on synthetic data.

Effective for regression tasks with jumps or discontinuities.

Shows improved accuracy in low-dimensional settings.

Abstract

XGBoost is often presented as the algorithm that wins every ML competition. Surprisingly, this is true even though predictions are piecewise constant. This might be justified in high dimensional input spaces, but when the number of features is low, a piecewise linear model is likely to perform better. XGBoost was extended into LinXGBoost that stores at each leaf a linear model. This extension, equivalent to piecewise regularized least-squares, is particularly attractive for regression of functions that exhibits jumps or discontinuities. Those functions are notoriously hard to regress. Our extension is compared to the vanilla XGBoost and Random Forest in experiments on both synthetic and real-world data sets.