Toward Exact Convergence in Byzantine-Robust Decentralized Learning: A Statistical Identification Approach

TL;DR

This paper introduces a new decentralized learning framework that actively identifies malicious nodes to achieve exact convergence despite Byzantine attacks, improving robustness over traditional methods.

Contribution

The paper proposes DRSGD-ByMI, an identification-based approach that accurately detects malicious nodes without restrictive assumptions, enabling optimal convergence in Byzantine-robust decentralized learning.

Findings

The identification mechanism controls false discovery rate without distributional assumptions.

The proposed method achieves the same order-optimal convergence rate as standard decentralized methods.

Numerical experiments confirm the theoretical advantages of DRSGD-ByMI.

Abstract

To defend against Byzantine attacks in decentralized learning, most existing methods rely on robust aggregation rules to mitigate the influence of malicious machines. However, these strategies inherently introduce bias, leading to inexact convergence with non-vanishing steady-state errors. In this paper, we propose a strategic shift from passive aggregation to active identification by introducing the Decentralized Rescaled Stochastic Gradient Descent with Byzantine Machine Identification (DRSGD-ByMI) framework. The core of our approach is an identification-based ``detect-then-optimize'' pipeline, where a p-value-free detection procedure is developed to accurately prune malicious nodes from the network. By leveraging sample-splitting score statistics, this identification mechanism achieves false discovery rate control without requiring restrictive distributional assumptions. We theoretically demonstrate that this precise identification allows the decentralized network to recover sufficient connectivity among the normal nodes, thereby enabling DRSGD-ByMI to match, even in the presence of Byzantine machines, the same order-optimal convergence rate as standard decentralized stochastic first-order methods. Numerical experiments validate our theoretical results and demonstrate the effectiveness of DRSGD-ByMI for decentralized robust learning problems.