Video-based Cross-modal Auxiliary Network for Multimodal Sentiment Analysis

TL;DR

This paper introduces VCAN, a novel video-based cross-modal network that enhances multimodal sentiment analysis by improving feature diversity and reducing redundancy, leading to superior classification accuracy.

Contribution

The paper proposes a new VCAN model with modules for diverse audio features and filtered visual frames, addressing limitations of previous methods in feature extraction and fusion.

Findings

VCAN outperforms state-of-the-art methods on multiple benchmarks.

Enhanced audio feature diversity improves sentiment classification.

Efficient visual frame filtering reduces redundancy and boosts accuracy.

Abstract

Multimodal sentiment analysis has a wide range of applications due to its information complementarity in multimodal interactions. Previous works focus more on investigating efficient joint representations, but they rarely consider the insufficient unimodal features extraction and data redundancy of multimodal fusion. In this paper, a Video-based Cross-modal Auxiliary Network (VCAN) is proposed, which is comprised of an audio features map module and a cross-modal selection module. The first module is designed to substantially increase feature diversity in audio feature extraction, aiming to improve classification accuracy by providing more comprehensive acoustic representations. To empower the model to handle redundant visual features, the second module is addressed to efficiently filter the redundant visual frames during integrating audiovisual data. Moreover, a classifier group consisting of several image classification networks is introduced to predict sentiment polarities and emotion categories. Extensive experimental results on RAVDESS, CMU-MOSI, and CMU-MOSEI benchmarks indicate that VCAN is significantly superior to the state-of-the-art methods for improving the classification accuracy of multimodal sentiment analysis.