Byzantine-Resilient Decentralized Stochastic Optimization with Robust Aggregation Rules

TL;DR

This paper addresses the challenge of Byzantine attacks in decentralized stochastic optimization by analyzing existing issues and proposing a new robust aggregation rule, IOS, with proven effectiveness and guidelines for resilient algorithm design.

Contribution

It identifies key issues in existing robust aggregation rules for decentralized settings and introduces IOS, a new filtering-based method with theoretical guarantees.

Findings

IOS effectively mitigates Byzantine attacks in decentralized optimization.

Numerical experiments confirm the robustness and performance of IOS.

Guidelines for designing Byzantine-resilient decentralized algorithms are provided.

Abstract

This paper focuses on decentralized stochastic optimization in the presence of Byzantine attacks. During the optimization process, an unknown number of malfunctioning or malicious workers, termed as Byzantine workers, disobey the algorithmic protocol and send arbitrarily wrong messages to their neighbors. Even though various Byzantine-resilient algorithms have been developed for distributed stochastic optimization with a central server, we show that there are two major issues in the existing robust aggregation rules when being applied to the decentralized scenario: disagreement and non-doubly stochastic virtual mixing matrix. This paper provides comprehensive analysis that discloses the negative effects of these two issues, and gives guidelines of designing favorable Byzantine-resilient decentralized stochastic optimization algorithms. Under these guidelines, we propose iterative outlier scissor (IOS), an iterative filtering-based robust aggregation rule with provable performance guarantees. Numerical experiments demonstrate the effectiveness of IOS. The code of simulation implementation is available at github.com/Zhaoxian-Wu/IOS.