Federated Learning Resilient to Byzantine Attacks and Data Heterogeneity

TL;DR

This paper proposes RAGA, a robust federated learning algorithm using geometric median aggregation, with proven convergence under Byzantine attacks and data heterogeneity, applicable to both convex and non-convex loss functions.

Contribution

It introduces RAGA, a novel aggregation method with convergence guarantees for heterogeneous data and malicious attacks, extending analysis beyond strongly-convex functions.

Findings

RAGA achieves convergence rates of O(1/T^{2/3-δ}) for non-convex loss functions.

RAGA converges linearly for strongly-convex loss functions.

Experimental results show RAGA's robustness and superior convergence under Byzantine attacks.

Abstract

This paper addresses federated learning (FL) in the context of malicious Byzantine attacks and data heterogeneity. We introduce a novel Robust Average Gradient Algorithm (RAGA), which uses the geometric median for aggregation and {allows flexible round number for local updates.} Unlike most existing resilient approaches, which base their convergence analysis on strongly-convex loss functions or homogeneously distributed datasets, this work conducts convergence analysis for both strongly-convex and non-convex loss functions over heterogeneous datasets. The theoretical analysis indicates that as long as the fraction of the {data} from malicious users is less than half, RAGA can achieve convergence at a rate of $\mathcal{O}({1}/{T^{2/3- \delta}})$ for non-convex loss functions, where $T$ is the iteration number and $\delta \in (0, 2/3)$. For strongly-convex loss functions, the convergence rate is linear. Furthermore, the stationary point or global optimal solution is shown to be attainable as data heterogeneity diminishes. Experimental results validate the robustness of RAGA against Byzantine attacks and demonstrate its superior convergence performance compared to baselines under varying intensities of Byzantine attacks on heterogeneous datasets.