Federated Model Heterogeneous Matryoshka Representation Learning

TL;DR

FedMRL introduces a novel federated learning framework that enhances knowledge exchange between heterogeneous client models through auxiliary homogeneous models and multi-granular representation fusion, improving accuracy and efficiency.

Contribution

It proposes a new federated learning method with auxiliary homogeneous models and Matryoshka representations, enabling better knowledge transfer and multi-perspective learning in heterogeneous settings.

Findings

Achieves up to 8.48% accuracy improvement over state-of-the-art methods.

Demonstrates lower communication and computational costs.

Proves convergence with a $O(1/T)$ rate in non-convex settings.

Abstract

Model heterogeneous federated learning (MHeteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. However, existing MHeteroFL methods rely on training loss to transfer knowledge between the client model and the server model, resulting in limited knowledge exchange. To address this limitation, we propose the Federated model heterogeneous Matryoshka Representation Learning (FedMRL) approach for supervised learning tasks. It adds an auxiliary small homogeneous model shared by clients with heterogeneous local models. (1) The generalized and personalized representations extracted by the two models' feature extractors are fused by a personalized lightweight representation projector. This step enables representation fusion to adapt to local data distribution. (2) The fused representation is then used to construct Matryoshka representations with multi-dimensional and multi-granular embedded representations learned by the global homogeneous model header and the local heterogeneous model header. This step facilitates multi-perspective representation learning and improves model learning capability. Theoretical analysis shows that FedMRL achieves a $O(1/T)$ non-convex convergence rate. Extensive experiments on benchmark datasets demonstrate its superior model accuracy with low communication and computational costs compared to seven state-of-the-art baselines. It achieves up to 8.48% and 24.94% accuracy improvement compared with the state-of-the-art and the best same-category baseline, respectively.