XGBoost: A Scalable Tree Boosting System

TL;DR

XGBoost is a highly scalable and efficient tree boosting system that leverages novel algorithms and system optimizations to handle large-scale machine learning tasks with high accuracy.

Contribution

The paper introduces a scalable end-to-end tree boosting system with novel algorithms for sparse data and approximate tree learning, enhancing performance and scalability.

Findings

XGBoost achieves state-of-the-art results on various machine learning challenges.

It scales beyond billions of examples with fewer resources.

The system outperforms existing boosting frameworks in efficiency.

Abstract

Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.