A Comparative Analysis of XGBoost

TL;DR

This paper provides a detailed comparison of XGBoost with other ensemble methods, analyzing its training speed, performance, and parameter tuning, revealing that it is not always the optimal choice.

Contribution

It offers a comprehensive analysis of XGBoost's performance and tuning process, comparing it with random forests and gradient boosting under various settings.

Findings

XGBoost's training speed varies with parameter tuning.

It does not always outperform other ensemble methods.

Default settings may not be optimal for all tasks.

Abstract

XGBoost is a scalable ensemble technique based on gradient boosting that has demonstrated to be a reliable and efficient machine learning challenge solver. This work proposes a practical analysis of how this novel technique works in terms of training speed, generalization performance and parameter setup. In addition, a comprehensive comparison between XGBoost, random forests and gradient boosting has been performed using carefully tuned models as well as using the default settings. The results of this comparison may indicate that XGBoost is not necessarily the best choice under all circumstances. Finally an extensive analysis of XGBoost parametrization tuning process is carried out.