FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learning

TL;DR

FedBug introduces a bottom-up gradual unfreezing approach in federated learning to reduce client drift, improve convergence, and enhance model alignment across heterogeneous client datasets.

Contribution

This paper presents FedBug, a novel federated learning framework with a bottom-up unfreezing strategy, along with theoretical analysis and extensive empirical validation.

Findings

FedBug achieves faster convergence than FedAvg.

The bottom-up unfreezing improves model alignment across clients.

FedBug demonstrates robustness across various datasets and architectures.

Abstract

Federated Learning (FL) offers a collaborative training framework, allowing multiple clients to contribute to a shared model without compromising data privacy. Due to the heterogeneous nature of local datasets, updated client models may overfit and diverge from one another, commonly known as the problem of client drift. In this paper, we propose FedBug (Federated Learning with Bottom-Up Gradual Unfreezing), a novel FL framework designed to effectively mitigate client drift. FedBug adaptively leverages the client model parameters, distributed by the server at each global round, as the reference points for cross-client alignment. Specifically, on the client side, FedBug begins by freezing the entire model, then gradually unfreezes the layers, from the input layer to the output layer. This bottom-up approach allows models to train the newly thawed layers to project data into a latent space, wherein the separating hyperplanes remain consistent across all clients. We theoretically analyze FedBug in a novel over-parameterization FL setup, revealing its superior convergence rate compared to FedAvg. Through comprehensive experiments, spanning various datasets, training conditions, and network architectures, we validate the efficacy of FedBug. Our contributions encompass a novel FL framework, theoretical analysis, and empirical validation, demonstrating the wide potential and applicability of FedBug.