Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms

TL;DR

This paper benchmarks GPU-accelerated GBDT packages (XGBoost, LightGBM, Catboost), evaluating their performance and hyper-parameter optimization efficiency across diverse large-scale datasets.

Contribution

It provides a comprehensive comparison of GPU-accelerated GBDT packages in terms of performance and hyper-parameter tuning efficiency.

Findings

GPU acceleration improves training speed significantly.

Different packages vary in convergence speed and generalization.

Hyper-parameter tuning benefits from GPU acceleration.

Abstract

Gradient boosting decision trees (GBDTs) have seen widespread adoption in academia, industry and competitive data science due to their state-of-the-art performance in many machine learning tasks. One relative downside to these models is the large number of hyper-parameters that they expose to the end-user. To maximize the predictive power of GBDT models, one must either manually tune the hyper-parameters, or utilize automated techniques such as those based on Bayesian optimization. Both of these approaches are time-consuming since they involve repeatably training the model for different sets of hyper-parameters. A number of software GBDT packages have started to offer GPU acceleration which can help to alleviate this problem. In this paper, we consider three such packages: XGBoost, LightGBM and Catboost. Firstly, we evaluate the performance of the GPU acceleration provided by these packages using large-scale datasets with varying shapes, sparsities and learning tasks. Then, we compare the packages in the context of hyper-parameter optimization, both in terms of how quickly each package converges to a good validation score, and in terms of generalization performance.