Interpretable multimodal sentiment analysis based on textual modality descriptions by using large-scale language models

TL;DR

This paper introduces a novel multimodal sentiment analysis method that converts nonverbal cues into text descriptions and uses large-scale language models, enhancing interpretability while maintaining or improving accuracy.

Contribution

The study proposes a new approach converting audio and facial cues into text descriptions for sentiment analysis using large language models, improving interpretability and effectiveness.

Findings

Achieved up to 2.49% improvement in F1 score over baselines.

Provided intuitive, text-based explanations for multimodal decision-making.

Demonstrated comparable fusion characteristics to traditional methods.

Abstract

Multimodal sentiment analysis is an important area for understanding the user's internal states. Deep learning methods were effective, but the problem of poor interpretability has gradually gained attention. Previous works have attempted to use attention weights or vector distributions to provide interpretability. However, their explanations were not intuitive and can be influenced by different trained models. This study proposed a novel approach to provide interpretability by converting nonverbal modalities into text descriptions and by using large-scale language models for sentiment predictions. This provides an intuitive approach to directly interpret what models depend on with respect to making decisions from input texts, thus significantly improving interpretability. Specifically, we convert descriptions based on two feature patterns for the audio modality and discrete action units for the facial modality. Experimental results on two sentiment analysis tasks demonstrated that the proposed approach maintained, or even improved effectiveness for sentiment analysis compared to baselines using conventional features, with the highest improvement of 2.49% on the F1 score. The results also showed that multimodal descriptions have similar characteristics on fusing modalities as those of conventional fusion methods. The results demonstrated that the proposed approach is interpretable and effective for multimodal sentiment analysis.