AGSP-DSA: An Adaptive Graph Signal Processing Framework for Robust Multimodal Fusion with Dynamic Semantic Alignment

TL;DR

This paper presents AGSP-DSA, a novel adaptive graph signal processing framework that enhances multimodal data fusion by dynamically aligning semantics across heterogeneous sources, achieving state-of-the-art results on benchmark datasets.

Contribution

It introduces a dual-graph construction and semantic-aware attention mechanism for robust multimodal fusion, improving accuracy and generalization over previous methods.

Findings

Achieves 95.3% accuracy on CMU-MOSEI

Outperforms MM-GNN by 2.6% in accuracy

Demonstrates robustness in missing modality scenarios

Abstract

In this paper, we introduce an Adaptive Graph Signal Processing with Dynamic Semantic Alignment (AGSP DSA) framework to perform robust multimodal data fusion over heterogeneous sources, including text, audio, and images. The requested approach uses a dual-graph construction to learn both intra-modal and inter-modal relations, spectral graph filtering to boost the informative signals, and effective node embedding with Multi-scale Graph Convolutional Networks (GCNs). Semantic aware attention mechanism: each modality may dynamically contribute to the context with respect to contextual relevance. The experimental outcomes on three benchmark datasets, including CMU-MOSEI, AVE, and MM-IMDB, show that AGSP-DSA performs as the state of the art. More precisely, it achieves 95.3% accuracy, 0.936 F1-score, and 0.924 mAP on CMU-MOSEI, improving MM-GNN by 2.6 percent in accuracy. It gets 93.4% accuracy and 0.911 F1-score on AVE and 91.8% accuracy and 0.886 F1-score on MM-IMDB, which demonstrate good generalization and robustness in the missing modality setting. These findings verify the efficiency of AGSP-DSA in promoting multimodal learning in sentiment analysis, event recognition and multimedia classification.