Distributed Adaptive Huber Regression

TL;DR

This paper proposes a communication-efficient distributed algorithm for robust linear regression that handles heavy-tailed and asymmetric errors, achieving near-centralized accuracy and reliable confidence intervals.

Contribution

It introduces a novel distributed Huber regression algorithm that is robust to heavy-tailed errors and achieves optimal error bounds with minimal communication.

Findings

Achieves centralized nonasymptotic error bounds.

Provides Berry-Esseen bounds for confidence intervals.

Outperforms existing distributed methods in accuracy and coverage.

Abstract

Distributed data naturally arise in scenarios involving multiple sources of observations, each stored at a different location. Directly pooling all the data together is often prohibited due to limited bandwidth and storage, or due to privacy protocols. This paper introduces a new robust distributed algorithm for fitting linear regressions when data are subject to heavy-tailed and/or asymmetric errors with finite second moments. The algorithm only communicates gradient information at each iteration and therefore is communication-efficient. Statistically, the resulting estimator achieves the centralized nonasymptotic error bound as if all the data were pooled together and came from a distribution with sub-Gaussian tails. Under a finite $(2+\delta)$-th moment condition, we derive a Berry-Esseen bound for the distributed estimator, based on which we construct robust confidence intervals. Numerical studies further confirm that compared with extant distributed methods, the proposed methods achieve near-optimal accuracy with low variability and better coverage with tighter confidence width.