A Simple and Fast Baseline for Tuning Large XGBoost Models

TL;DR

This paper introduces a simple, fast baseline method using uniform subsampling for hyperparameter tuning of large XGBoost models, significantly reducing training time while maintaining performance on large tabular datasets.

Contribution

It proposes a novel multi-fidelity hyperparameter optimization approach leveraging data subsampling, improving tuning efficiency for large-scale XGBoost models.

Findings

Effective on datasets up to 70GB in size

Reduces tuning time significantly

Maintains competitive predictive performance

Abstract

XGBoost, a scalable tree boosting algorithm, has proven effective for many prediction tasks of practical interest, especially using tabular datasets. Hyperparameter tuning can further improve the predictive performance, but unlike neural networks, full-batch training of many models on large datasets can be time consuming. Owing to the discovery that (i) there is a strong linear relation between dataset size & training time, (ii) XGBoost models satisfy the ranking hypothesis, and (iii) lower-fidelity models can discover promising hyperparameter configurations, we show that uniform subsampling makes for a simple yet fast baseline to speed up the tuning of large XGBoost models using multi-fidelity hyperparameter optimization with data subsets as the fidelity dimension. We demonstrate the effectiveness of this baseline on large-scale tabular datasets ranging from $15-70\mathrm{GB}$ in size.