Balancing Similarity and Complementarity for Federated Learning

TL;DR

This paper introduces FedSaC, a novel federated learning framework that balances similarity and complementarity among clients to improve cooperation, especially under data heterogeneity, leading to superior performance over existing methods.

Contribution

The paper proposes FedSaC, a new framework that optimally balances client similarity and data complementarity in federated learning, with theoretical and empirical validation.

Findings

FedSaC outperforms state-of-the-art FL methods in various experiments.

Balancing similarity and complementarity enhances cooperation in heterogeneous data.

Optimal cooperation involves leveraging complementary data rather than just similar models.

Abstract

In mobile and IoT systems, Federated Learning (FL) is increasingly important for effectively using data while maintaining user privacy. One key challenge in FL is managing statistical heterogeneity, such as non-i.i.d. data, arising from numerous clients and diverse data sources. This requires strategic cooperation, often with clients having similar characteristics. However, we are interested in a fundamental question: does achieving optimal cooperation necessarily entail cooperating with the most similar clients? Typically, significant model performance improvements are often realized not by partnering with the most similar models, but through leveraging complementary data. Our theoretical and empirical analyses suggest that optimal cooperation is achieved by enhancing complementarity in feature distribution while restricting the disparity in the correlation between features and targets. Accordingly, we introduce a novel framework, \texttt{FedSaC}, which balances similarity and complementarity in FL cooperation. Our framework aims to approximate an optimal cooperation network for each client by optimizing a weighted sum of model similarity and feature complementarity. The strength of \texttt{FedSaC} lies in its adaptability to various levels of data heterogeneity and multimodal scenarios. Our comprehensive unimodal and multimodal experiments demonstrate that \texttt{FedSaC} markedly surpasses other state-of-the-art FL methods.