Aggregation Design for Personalized Federated Multi-Modal Learning over Wireless Networks

TL;DR

This paper proposes a parameter scheduling scheme for personalized federated multi-modal learning over wireless networks, enhancing performance and communication efficiency by adaptively selecting parameters based on data heterogeneity and channel conditions.

Contribution

It introduces a learning-based approach to optimize parameter aggregation and scheduling in personalized FMML considering non-IID data and modality heterogeneity.

Findings

Improved personalized performance in FMML.

Enhanced communication efficiency through parameter scheduling.

Effective handling of non-IID data and modality heterogeneity.

Abstract

Federated Multi-Modal Learning (FMML) is an emerging field that integrates information from different modalities in federated learning to improve the learning performance. In this letter, we develop a parameter scheduling scheme to improve personalized performance and communication efficiency in personalized FMML, considering the non-independent and nonidentically distributed (non-IID) data along with the modality heterogeneity. Specifically, a learning-based approach is utilized to obtain the aggregation coefficients for parameters of different modalities on distinct devices. Based on the aggregation coefficients and channel state, a subset of parameters is scheduled to be uploaded to a server for each modality. Experimental results show that the proposed algorithm can effectively improve the personalized performance of FMML.