High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers

TL;DR

This paper introduces a high-dimensional distributed gradient descent method resilient to arbitrary Byzantine attacks, reducing error growth with dimensionality and achieving optimal statistical rates through a novel semi-verified mean estimation approach.

Contribution

It proposes a new high-dimensional semi-verified mean estimation technique that mitigates the curse of dimensionality in Byzantine-resilient distributed learning.

Findings

Reduces error dependence from  to /2d

Achieves minimax optimal statistical rates

Numerical results validate theoretical improvements

Abstract

Adversarial attacks pose a major challenge to distributed learning systems, prompting the development of numerous robust learning methods. However, most existing approaches suffer from the curse of dimensionality, i.e. the error increases with the number of model parameters. In this paper, we make a progress towards high dimensional problems, under arbitrary number of Byzantine attackers. The cornerstone of our design is a direct high dimensional semi-verified mean estimation method. The idea is to identify a subspace with large variance. The components of the mean value perpendicular to this subspace are estimated using corrupted gradient vectors uploaded from worker machines, while the components within this subspace are estimated using auxiliary dataset. As a result, a combination of large corrupted dataset and small clean dataset yields significantly better performance than using them separately. We then apply this method as the aggregator for distributed learning problems. The theoretical analysis shows that compared with existing solutions, our method gets rid of $\sqrt{d}$ dependence on the dimensionality, and achieves minimax optimal statistical rates. Numerical results validate our theory as well as the effectiveness of the proposed method.