Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity

TL;DR

This paper identifies the layer-wise inertia phenomenon in federated learning caused by data heterogeneity and limited local data, and proposes LIPS, a method that introduces transient sparsity to stimulate meaningful model updates, improving collaboration and performance.

Contribution

The paper uncovers the layer-wise inertia phenomenon in federated learning and introduces LIPS, a novel approach using transient sparsity to enhance global model updates and collaboration.

Findings

LIPS mitigates layer-wise inertia effectively.

Improves global aggregation and model performance.

Applicable across diverse datasets and architectures.

Abstract

Federated learning (FL) enables collaborative model training across decentralized clients while preserving data privacy, leveraging aggregated updates to build robust global models. However, this training paradigm faces significant challenges due to data heterogeneity and limited local datasets, which often impede effective collaboration. In such scenarios, we identify the Layer-wise Inertia Phenomenon in FL, wherein the middle layers of global model undergo minimal updates after early communication rounds, ultimately limiting the effectiveness of global aggregation. We demonstrate the presence of this phenomenon across a wide range of federated settings, spanning diverse datasets and architectures. To address this issue, we propose LIPS (Layer-wise Inertia Phenomenon with Sparsity), a simple yet effective method that periodically introduces transient sparsity to stimulate meaningful updates and empower global aggregation. Experiments demonstrate that LIPS effectively mitigates layer-wise inertia, enhances aggregation effectiveness, and improves overall performance in various FL scenarios. This work not only deepens the understanding of layer-wise learning dynamics in FL but also paves the way for more effective collaboration strategies in resource-constrained environments. Our code is publicly available at: https://github.com/QiaoXiao7282/LIPS.