FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis

TL;DR

FedUAF introduces an uncertainty-aware fusion and reliability-guided aggregation framework for multimodal federated sentiment analysis, effectively handling missing data, heterogeneity, and noisy clients to improve robustness and performance.

Contribution

It proposes a novel federated learning framework that models modality uncertainty and client reliability, enhancing robustness in multimodal sentiment analysis under practical challenges.

Findings

Outperforms state-of-the-art federated baselines on CMU-MOSI and CMU-MOSEI datasets.

Demonstrates robustness against missing modalities and noisy clients.

Effective in Non-IID data settings.

Abstract

Multimodal sentiment analysis in federated learning environments faces significant challenges due to missing modalities, heterogeneous data distributions, and unreliable client updates. Existing federated approaches often struggle to maintain robust performance under these practical conditions. In this paper, we propose FedUAF, a unified multimodal federated learning framework that addresses these challenges through uncertainty-aware fusion and reliability-guided aggregation. FedUAF explicitly models modality-level uncertainty during local training and leverages client reliability to guide global aggregation, enabling effective learning under incomplete and noisy multimodal data. Extensive experiments on CMU-MOSI and CMU-MOSEI demonstrate that FedUAF consistently outperforms state-of-the-art federated baselines across various missing-modality patterns and Non-IID settings. Moreover, FedUAF exhibits superior robustness against noisy clients, highlighting its potential for real-world multimodal federated applications.