Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality

TL;DR

This paper introduces FedMVP, a federated learning approach that leverages large-scale pre-trained models to handle incomplete modalities across clients, improving robustness and performance in multi-modal data scenarios.

Contribution

The paper proposes FedMVP, a novel multi-modal federated learning framework that uses pre-trained models for modality completion and a graph-based aggregation method.

Findings

FedMVP outperforms existing methods on image-text classification datasets.

The approach is robust to missing modalities during training and inference.

Pre-trained models enhance local training efficiency and model robustness.

Abstract

Federated learning (FL) has obtained tremendous progress in providing collaborative training solutions for distributed data silos with privacy guarantees. However, few existing works explore a more realistic scenario where the clients hold multiple data modalities. In this paper, we aim to solve a novel challenge in multi-modal federated learning (MFL) -- modality missing -- the clients may lose part of the modalities in their local data sets. To tackle the problems, we propose a novel multi-modal federated learning method, Federated Multi-modal contrastiVe training with Pre-trained completion (FedMVP), which integrates the large-scale pre-trained models to enhance the federated training. In the proposed FedMVP framework, each client deploys a large-scale pre-trained model with frozen parameters for modality completion and representation knowledge transfer, enabling efficient and robust local training. On the server side, we utilize generated data to uniformly measure the representation similarity among the uploaded client models and construct a graph perspective to aggregate them according to their importance in the system. We demonstrate that the model achieves superior performance over two real-world image-text classification datasets and is robust to the performance degradation caused by missing modality.