Communication-Efficient Weighted Sampling and Quantile Summary for GBDT

TL;DR

This paper introduces two communication-efficient methods for distributed GBDT training, reducing overhead by using weighted sampling for information gain estimation and protocols for weighted quantile computation, enhancing scalability.

Contribution

The paper presents novel distributed protocols for weighted sampling and quantile estimation tailored for GBDT, improving communication efficiency in large-scale distributed learning.

Findings

Reduced communication overhead in distributed GBDT training

Efficient estimation of information gain with small data subsets

Improved scalability for large datasets

Abstract

Gradient boosting decision tree (GBDT) is a powerful and widely-used machine learning model, which has achieved state-of-the-art performance in many academic areas and production environment. However, communication overhead is the main bottleneck in distributed training which can handle the massive data nowadays. In this paper, we propose two novel communication-efficient methods over distributed dataset to mitigate this problem, a weighted sampling approach by which we can estimate the information gain over a small subset efficiently, and distributed protocols for weighted quantile problem used in approximate tree learning.