Federated Loss Exploration for Improved Convergence on Non-IID Data

TL;DR

FedLEx introduces a novel federated loss exploration technique that improves model convergence and robustness in non-IID data scenarios by guiding client updates with a global guidance matrix, outperforming existing methods.

Contribution

The paper proposes FedLEx, a new federated learning approach that enhances convergence on non-IID data by using a global guidance matrix derived from gradient deviations.

Findings

Significant performance improvements over state-of-the-art FL algorithms in non-IID settings.

Efficient convergence with minimal epochs and data sharing.

Robustness to data heterogeneity without additional data distribution assumptions.

Abstract

Federated learning (FL) has emerged as a groundbreaking paradigm in machine learning (ML), offering privacy-preserving collaborative model training across diverse datasets. Despite its promise, FL faces significant hurdles in non-identically and independently distributed (non-IID) data scenarios, where most existing methods often struggle with data heterogeneity and lack robustness in performance. This paper introduces Federated Loss Exploration (FedLEx), an innovative approach specifically designed to tackle these challenges. FedLEx distinctively addresses the shortcomings of existing FL methods in non-IID settings by optimizing its learning behavior for scenarios in which assumptions about data heterogeneity are impractical or unknown. It employs a federated loss exploration technique, where clients contribute to a global guidance matrix by calculating gradient deviations for model parameters. This matrix serves as a strategic compass to guide clients' gradient updates in subsequent FL rounds, thereby fostering optimal parameter updates for the global model. FedLEx effectively navigates the complex loss surfaces inherent in non-IID data, enhancing knowledge transfer in an efficient manner, since only a small number of epochs and small amount of data are required to build a strong global guidance matrix that can achieve model convergence without the need for additional data sharing or data distribution statics in a large client scenario. Our extensive experiments with state-of-the art FL algorithms demonstrate significant improvements in performance, particularly under realistic non-IID conditions, thus highlighting FedLEx's potential to overcome critical barriers in diverse FL applications.