Orthogonal Calibration for Asynchronous Federated Learning

TL;DR

OrthoFL introduces an orthogonal calibration framework for asynchronous federated learning, effectively decoupling global and local updates to improve accuracy and speedup over existing methods.

Contribution

The paper proposes OrthoFL, a novel orthogonal calibration method that reduces interference between global and local updates in asynchronous federated learning.

Findings

Improves accuracy by 9.6% over baseline methods.

Achieves 12× speedup compared to synchronous approaches.

Outperforms state-of-the-art asynchronous methods across various scenarios.

Abstract

Asynchronous federated learning mitigates the inefficiency of conventional synchronous aggregation by integrating updates as they arrive and adjusting their influence based on staleness. Due to asynchrony and data heterogeneity, learning objectives at the global and local levels are inherently inconsistent -- global optimization trajectories may conflict with ongoing local updates. Existing asynchronous methods simply distribute the latest global weights to clients, which can overwrite local progress and cause model drift. In this paper, we propose OrthoFL, an orthogonal calibration framework that decouples global and local learning progress and adjusts global shifts to minimize interference before merging them into local models. In OrthoFL, clients and the server maintain separate model weights. Upon receiving an update, the server aggregates it into the global weights via a moving average. For client weights, the server computes the global weight shift accumulated during the client's delay and removes the components aligned with the direction of the received update. The resulting parameters lie in a subspace orthogonal to the client update and preserve the maximal information from the global progress. The calibrated global shift is then merged into the client weights for further training. Extensive experiments show that OrthoFL improves accuracy by 9.6% and achieves a 12$\times$ speedup compared to synchronous methods. Moreover, it consistently outperforms state-of-the-art asynchronous baselines under various delay patterns and heterogeneity scenarios.