FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning

TL;DR

FedCAda introduces an adaptive client-side optimization algorithm for federated learning that accelerates convergence and enhances stability by leveraging the Adam optimizer and adaptive parameter adjustments, validated through extensive experiments.

Contribution

This paper presents FedCAda, a novel adaptive algorithm for federated learning that adjusts client-side optimization parameters to improve convergence speed and stability.

Findings

FedCAda outperforms state-of-the-art methods in convergence and stability.

Adaptive adjustment of parameters enhances federated learning robustness.

Gradual diminishing of adaptive influence improves performance over time.

Abstract

Federated learning (FL) has emerged as a prominent approach for collaborative training of machine learning models across distributed clients while preserving data privacy. However, the quest to balance acceleration and stability becomes a significant challenge in FL, especially on the client-side. In this paper, we introduce FedCAda, an innovative federated client adaptive algorithm designed to tackle this challenge. FedCAda leverages the Adam algorithm to adjust the correction process of the first moment estimate $m$ and the second moment estimate $v$ on the client-side and aggregate adaptive algorithm parameters on the server-side, aiming to accelerate convergence speed and communication efficiency while ensuring stability and performance. Additionally, we investigate several algorithms incorporating different adjustment functions. This comparative analysis revealed that due to the limited information contained within client models from other clients during the initial stages of federated learning, more substantial constraints need to be imposed on the parameters of the adaptive algorithm. As federated learning progresses and clients gather more global information, FedCAda gradually diminishes the impact on adaptive parameters. These findings provide insights for enhancing the robustness and efficiency of algorithmic improvements. Through extensive experiments on computer vision (CV) and natural language processing (NLP) datasets, we demonstrate that FedCAda outperforms the state-of-the-art methods in terms of adaptability, convergence, stability, and overall performance. This work contributes to adaptive algorithms for federated learning, encouraging further exploration.