Hybrid CNN-Mamba Enhancement Network for Robust Multimodal Sentiment Analysis

TL;DR

This paper introduces HCMEN, a novel neural network framework that effectively handles missing modalities in multimodal sentiment analysis by combining CNN and Mamba architectures with cross-modal enhancement.

Contribution

The paper proposes a new hybrid CNN-Mamba framework with a cross-modal enhancement mechanism for robust multimodal sentiment analysis under missing modality conditions.

Findings

HCMEN outperforms existing methods on benchmark datasets.

The framework effectively handles various missing modality scenarios.

Extensive experiments validate the superiority of HCMEN.

Abstract

Multimodal Sentiment Analysis (MSA) with missing modalities has recently attracted increasing attention. Although existing research mainly focuses on designing complex model architectures to handle incomplete data, it still faces significant challenges in effectively aligning and fusing multimodal information. In this paper, we propose a novel framework called the Hybrid CNN-Mamba Enhancement Network (HCMEN) for robust multimodal sentiment analysis under missing modality conditions. HCMEN is designed around three key components: (1) hierarchical unimodal modeling, (2) cross-modal enhancement and alignment, and (3) multimodal mix-up fusion. First, HCMEN integrates the strengths of Convolutional Neural Network (CNN) for capturing local details and the Mamba architecture for modeling global contextual dependencies across different modalities. Furthermore, grounded in the principle of Mutual Information Maximization, we introduce a cross-modal enhancement mechanism that generates proxy modalities from mixed token-level representations and learns fine-grained token-level correspondences between modalities. The enhanced unimodal features are then fused and passed through the CNN-Mamba backbone, enabling local-to-global cross-modal interaction and comprehensive multimodal integration. Extensive experiments on two benchmark MSA datasets demonstrate that HCMEN consistently outperforms existing state-of-the-art methods, achieving superior performance across various missing modality scenarios. The code will be released publicly in the near future.