Distributed Learning with Low Communication Cost via Gradient Boosting Untrained Neural Network

TL;DR

This paper introduces GBUN, a novel distributed gradient boosting algorithm using an untrained neural network to significantly reduce communication costs while maintaining accuracy, enabling faster training especially in low-bandwidth environments.

Contribution

The paper proposes GBUN, a new gradient boosting method that employs an untrained neural network to lower communication costs in distributed learning, extending Simhash for high-dimensional data.

Findings

GBUN achieves comparable accuracy to traditional GBDT.

GBUN speeds up training by up to 13 times on large clusters.

GBUN drastically reduces communication bandwidth requirements.

Abstract

For high-dimensional data, there are huge communication costs for distributed GBDT because the communication volume of GBDT is related to the number of features. To overcome this problem, we propose a novel gradient boosting algorithm, the Gradient Boosting Untrained Neural Network(GBUN). GBUN ensembles the untrained randomly generated neural network that softly distributes data samples to multiple neuron outputs and dramatically reduces the communication costs for distributed learning. To avoid creating huge neural networks for high-dimensional data, we extend Simhash algorithm to mimic forward calculation of the neural network. Our experiments on multiple public datasets show that GBUN is as good as conventional GBDT in terms of prediction accuracy and much better than it in scaling property for distributed learning. Comparing to conventional GBDT varieties, GBUN speeds up the training process up to 13 times on the cluster with 64 machines, and up to 4614 times on the cluster with 100KB/s network bandwidth. Therefore, GBUN is not only an efficient distributed learning algorithm but also has great potentials for federated learning.