FedCiR: Client-Invariant Representation Learning for Federated Non-IID Features

TL;DR

FedCiR introduces a novel client-invariant representation learning framework for federated learning, effectively addressing non-IID data heterogeneity by extracting informative, invariant features through mutual information optimization and a data-free regularization mechanism.

Contribution

The paper proposes FedCiR, a new method that enhances federated learning by promoting client-invariant features using mutual information bounds and a data-free approximation of global distributions.

Findings

Improves convergence and accuracy in non-IID federated learning scenarios.

Effectively extracts client-invariant features to mitigate data heterogeneity.

Demonstrates superior performance over existing methods in extensive experiments.

Abstract

Federated learning (FL) is a distributed learning paradigm that maximizes the potential of data-driven models for edge devices without sharing their raw data. However, devices often have non-independent and identically distributed (non-IID) data, meaning their local data distributions can vary significantly. The heterogeneity in input data distributions across devices, commonly referred to as the feature shift problem, can adversely impact the training convergence and accuracy of the global model. To analyze the intrinsic causes of the feature shift problem, we develop a generalization error bound in FL, which motivates us to propose FedCiR, a client-invariant representation learning framework that enables clients to extract informative and client-invariant features. Specifically, we improve the mutual information term between representations and labels to encourage representations to carry essential classification knowledge, and diminish the mutual information term between the client set and representations conditioned on labels to promote representations of clients to be client-invariant. We further incorporate two regularizers into the FL framework to bound the mutual information terms with an approximate global representation distribution to compensate for the absence of the ground-truth global representation distribution, thus achieving informative and client-invariant feature extraction. To achieve global representation distribution approximation, we propose a data-free mechanism performed by the server without compromising privacy. Extensive experiments demonstrate the effectiveness of our approach in achieving client-invariant representation learning and solving the data heterogeneity issue.