From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization

TL;DR

This paper introduces FedaGrac, a novel federated optimization algorithm that calibrates local updates to mitigate asynchronism effects, improving convergence speed and accuracy in non-i.i.d. data settings.

Contribution

FedaGrac is the first method to calibrate local directions using a predictive global orientation, improving convergence and accuracy under step asynchronism.

Findings

FedaGrac achieves faster convergence than existing methods.

It improves final model accuracy in federated learning.

Theoretical analysis confirms better convergence rates.

Abstract

In the setting of federated optimization, where a global model is aggregated periodically, step asynchronism occurs when participants conduct model training by efficiently utilizing their computational resources. It is well acknowledged that step asynchronism leads to objective inconsistency under non-i.i.d. data, which degrades the model's accuracy. To address this issue, we propose a new algorithm FedaGrac, which calibrates the local direction to a predictive global orientation. Taking advantage of the estimated orientation, we guarantee that the aggregated model does not excessively deviate from the global optimum while fully utilizing the local updates of faster nodes. We theoretically prove that FedaGrac holds an improved order of convergence rate than the state-of-the-art approaches and eliminates the negative effect of step asynchronism. Empirical results show that our algorithm accelerates the training and enhances the final accuracy.