XGBoost: Scalable GPU Accelerated Learning

TL;DR

This paper introduces a multi-GPU implementation of XGBoost that enables fast, scalable gradient boosting training on large datasets by leveraging GPU parallelism and data compression techniques.

Contribution

It presents a novel multi-GPU gradient boosting algorithm with efficient memory management and end-to-end GPU computation, significantly improving training speed and scalability.

Findings

Processed 115 million instances in under three minutes

Achieved scalable training on multi-GPU systems

Demonstrated efficient GPU memory usage with data compression

Abstract

We describe the multi-GPU gradient boosting algorithm implemented in the XGBoost library (https://github.com/dmlc/xgboost). Our algorithm allows fast, scalable training on multi-GPU systems with all of the features of the XGBoost library. We employ data compression techniques to minimise the usage of scarce GPU memory while still allowing highly efficient implementation. Using our algorithm we show that it is possible to process 115 million training instances in under three minutes on a publicly available cloud computing instance. The algorithm is implemented using end-to-end GPU parallelism, with prediction, gradient calculation, feature quantisation, decision tree construction and evaluation phases all computed on device.