Robust Distributed Learning: Tight Error Bounds and Breakdown Point under Data Heterogeneity

TL;DR

This paper develops a more realistic model for data heterogeneity in distributed learning, providing tight error bounds and revealing a lower breakdown point, thereby bridging the gap between theory and empirical observations.

Contribution

It introduces the (G,B)-gradient dissimilarity model, extends theoretical bounds to more practical scenarios, and proposes a robust gradient descent method with matching empirical performance.

Findings

Lower breakdown point under heterogeneity than classical 1/2.

New lower bound on distributed learning error.

Empirical results confirm reduced gap between theory and practice.

Abstract

The theory underlying robust distributed learning algorithms, designed to resist adversarial machines, matches empirical observations when data is homogeneous. Under data heterogeneity however, which is the norm in practical scenarios, established lower bounds on the learning error are essentially vacuous and greatly mismatch empirical observations. This is because the heterogeneity model considered is too restrictive and does not cover basic learning tasks such as least-squares regression. We consider in this paper a more realistic heterogeneity model, namely (G,B)-gradient dissimilarity, and show that it covers a larger class of learning problems than existing theory. Notably, we show that the breakdown point under heterogeneity is lower than the classical fraction 1/2. We also prove a new lower bound on the learning error of any distributed learning algorithm. We derive a matching upper bound for a robust variant of distributed gradient descent, and empirically show that our analysis reduces the gap between theory and practice.