Robust Regression with Ensembles Communicating over Noisy Channels

TL;DR

This paper addresses the challenge of distributed regression with noisy communication channels, proposing optimized aggregation methods for ensemble models to improve reliability in distributed machine learning systems.

Contribution

It introduces methods for optimizing aggregation coefficients in ensemble regression over noisy channels, applicable to bagging and gradient boosting.

Findings

Optimized aggregation improves regression accuracy over noisy channels.

Algorithms are effective on synthetic and real-world datasets.

Enhances reliability of distributed regression models.

Abstract

As machine-learning models grow in size, their implementation requirements cannot be met by a single computer system. This observation motivates distributed settings, in which intermediate computations are performed across a network of processing units, while the central node only aggregates their outputs. However, distributing inference tasks across low-precision or faulty edge devices, operating over a network of noisy communication channels, gives rise to serious reliability challenges. We study the problem of an ensemble of devices, implementing regression algorithms, that communicate through additive noisy channels in order to collaboratively perform a joint regression task. We define the problem formally, and develop methods for optimizing the aggregation coefficients for the parameters of the noise in the channels, which can potentially be correlated. Our results apply to the leading state-of-the-art ensemble regression methods: bagging and gradient boosting. We demonstrate the effectiveness of our algorithms on both synthetic and real-world datasets.