DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis

TL;DR

DashFusion introduces a dual-stream alignment and hierarchical bottleneck fusion framework for multimodal sentiment analysis, effectively synchronizing and integrating text, image, and audio modalities to achieve state-of-the-art results.

Contribution

The paper presents a novel framework combining dual-stream alignment with hierarchical bottleneck fusion, addressing alignment and fusion challenges in multimodal sentiment analysis.

Findings

Achieves state-of-the-art performance on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets.

Effective synchronization of modalities through temporal and semantic alignment.

Hierarchical bottleneck fusion balances performance and computational efficiency.

Abstract

Multimodal sentiment analysis (MSA) integrates various modalities, such as text, image, and audio, to provide a more comprehensive understanding of sentiment. However, effective MSA is challenged by alignment and fusion issues. Alignment requires synchronizing both temporal and semantic information across modalities, while fusion involves integrating these aligned features into a unified representation. Existing methods often address alignment or fusion in isolation, leading to limitations in performance and efficiency. To tackle these issues, we propose a novel framework called Dual-stream Alignment with Hierarchical Bottleneck Fusion (DashFusion). Firstly, dual-stream alignment module synchronizes multimodal features through temporal and semantic alignment. Temporal alignment employs cross-modal attention to establish frame-level correspondences among multimodal sequences. Semantic alignment ensures consistency across the feature space through contrastive learning. Secondly, supervised contrastive learning leverages label information to refine the modality features. Finally, hierarchical bottleneck fusion progressively integrates multimodal information through compressed bottleneck tokens, which achieves a balance between performance and computational efficiency. We evaluate DashFusion on three datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS. Experimental results demonstrate that DashFusion achieves state-of-the-art performance across various metrics, and ablation studies confirm the effectiveness of our alignment and fusion techniques. The codes for our experiments are available at https://github.com/ultramarineX/DashFusion.