From Inexact Gradients to Byzantine Robustness: Acceleration and Optimization under Similarity

TL;DR

This paper models Byzantine-robust federated learning as an inexact gradient optimization problem, introduces accelerated algorithms, and demonstrates reduced communication complexity both theoretically and empirically.

Contribution

It formulates Byzantine robustness as an inexact gradient optimization problem and proposes accelerated algorithms leveraging this framework, improving convergence speed and communication efficiency.

Findings

GD with robust aggregation achieves optimal asymptotic error.

Proposed accelerated schemes significantly reduce communication complexity.

Theoretical and empirical results confirm faster convergence and efficiency.

Abstract

Standard federated learning algorithms are vulnerable to adversarial nodes, a.k.a. Byzantine failures. To solve this issue, robust distributed learning algorithms have been developed, which typically replace parameter averaging by robust aggregations. While generic conditions on these aggregations exist to guarantee the convergence of (Stochastic) Gradient Descent (SGD), the analyses remain rather ad-hoc. This hinders the development of more complex robust algorithms, such as accelerated ones. In this work, we show that Byzantine-robust distributed optimization can, under standard generic assumptions, be cast as a general optimization with inexact gradient oracles (with both additive and multiplicative error terms), an active field of research. This allows for instance to directly show that GD on top of standard robust aggregation procedures obtains optimal asymptotic error in the Byzantine setting. Going further, we propose two optimization schemes to speed up the convergence. The first one is a Nesterov-type accelerated scheme whose proof directly derives from accelerated inexact gradient results applied to our formulation. The second one hinges on Optimization under Similarity, in which the server leverages an auxiliary loss function that approximates the global loss. Both approaches allow to drastically reduce the communication complexity compared to previous methods, as we show theoretically and empirically.