FedAFD: Multimodal Federated Learning via Adversarial Fusion and Distillation

TL;DR

FedAFD introduces a comprehensive federated learning framework for multimodal data that aligns representations, adaptively fuses features, and handles model heterogeneity, improving personalized and global model performance.

Contribution

The paper presents FedAFD, a novel framework combining adversarial alignment, granularity-aware fusion, and similarity-guided distillation to address heterogeneity and personalization in multimodal federated learning.

Findings

Outperforms existing methods in diverse settings

Effectively aligns cross-modal representations

Enhances personalized and global model accuracy

Abstract

Multimodal Federated Learning (MFL) enables clients with heterogeneous data modalities to collaboratively train models without sharing raw data, offering a privacy-preserving framework that leverages complementary cross-modal information. However, existing methods often overlook personalized client performance and struggle with modality/task discrepancies, as well as model heterogeneity. To address these challenges, we propose FedAFD, a unified MFL framework that enhances client and server learning. On the client side, we introduce a bi-level adversarial alignment strategy to align local and global representations within and across modalities, mitigating modality and task gaps. We further design a granularity-aware fusion module to integrate global knowledge into the personalized features adaptively. On the server side, to handle model heterogeneity, we propose a similarity-guided ensemble distillation mechanism that aggregates client representations on shared public data based on feature similarity and distills the fused knowledge into the global model. Extensive experiments conducted under both IID and non-IID settings demonstrate that FedAFD achieves superior performance and efficiency for both the client and the server.