A Mirror Descent-Based Algorithm for Corruption-Tolerant Distributed Gradient Descent

TL;DR

This paper introduces a mirror descent-based algorithm designed to perform distributed gradient descent robustly in the presence of adversarial corruptions, with theoretical convergence guarantees and empirical validation on standard datasets.

Contribution

It presents a novel corruption-tolerant distributed optimization algorithm using mirror descent, with detailed convergence analysis and optimized stepsize scheduling.

Findings

Algorithm is robust against adversarial corruptions.

Convergence is proven for strongly convex loss functions.

Experimental results validate theoretical claims on MNIST.

Abstract

Distributed gradient descent algorithms have come to the fore in modern machine learning, especially in parallelizing the handling of large datasets that are distributed across several workers. However, scant attention has been paid to analyzing the behavior of distributed gradient descent algorithms in the presence of adversarial corruptions instead of random noise. In this paper, we formulate a novel problem in which adversarial corruptions are present in a distributed learning system. We show how to use ideas from (lazy) mirror descent to design a corruption-tolerant distributed optimization algorithm. Extensive convergence analysis for (strongly) convex loss functions is provided for different choices of the stepsize. We carefully optimize the stepsize schedule to accelerate the convergence of the algorithm, while at the same time amortizing the effect of the corruption over time. Experiments based on linear regression, support vector classification, and softmax classification on the MNIST dataset corroborate our theoretical findings.