How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning

TL;DR

This paper investigates how collaboration among clients in cross-silo federated learning can enhance individual model generalization, proposing a hierarchical clustering scheme that adapts to data heterogeneity and improves performance.

Contribution

It derives a generalization bound for clients, formulates a utility maximization problem, and introduces HCCT, a hierarchical clustering-based training scheme that adapts to data similarity without predefining group numbers.

Findings

HCCT improves generalization over baseline schemes.

Collaboration benefits depend on data similarity and size.

HCCT degenerates to independent training in certain scenarios.

Abstract

Federated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work, we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model's generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization bound for each client when collaborating with others or when training independently. We show that the generalization performance of a client can be improved only by collaborating with other clients that have more training data and similar data distribution. Our analysis allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups. We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it degenerates to independent training and conventional FL in specific scenarios.