Stochastic Alternating Direction Method of Multipliers for Byzantine-Robust Distributed Learning

TL;DR

This paper introduces a stochastic ADMM approach to robustly address Byzantine attacks in distributed learning, effectively tolerating malicious workers and ensuring convergence near the optimal solution.

Contribution

It proposes a novel Byzantine-robust stochastic ADMM algorithm that leverages TV norm regularization and provides theoretical convergence guarantees.

Findings

Effective against various Byzantine attacks on MNIST and COVERTYPE datasets.

Converges to a bounded neighborhood of the optimal solution at rate O(1/k).

Utilizes problem structure for improved robustness and efficiency.

Abstract

This paper aims to solve a distributed learning problem under Byzantine attacks. In the underlying distributed system, a number of unknown but malicious workers (termed as Byzantine workers) can send arbitrary messages to the master and bias the learning process, due to data corruptions, computation errors or malicious attacks. Prior work has considered a total variation (TV) norm-penalized approximation formulation to handle the Byzantine attacks, where the TV norm penalty forces the regular workers' local variables to be close, and meanwhile, tolerates the outliers sent by the Byzantine workers. To solve the TV norm-penalized approximation formulation, we propose a Byzantine-robust stochastic alternating direction method of multipliers (ADMM) that fully utilizes the separable problem structure. Theoretically, we prove that the proposed method converges to a bounded neighborhood of the optimal solution at a rate of O(1/k) under mild assumptions, where k is the number of iterations and the size of neighborhood is determined by the number of Byzantine workers. Numerical experiments on the MNIST and COVERTYPE datasets demonstrate the effectiveness of the proposed method to various Byzantine attacks.