On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis

TL;DR

This study evaluates the use of large-scale, modality-specific pre-trained encoders for multimodal sentiment analysis, demonstrating their superiority over traditional features and highlighting the benefits of using intermediate layer outputs.

Contribution

It is the first comprehensive comparison of large-scale pre-trained encoders across visual, acoustic, and linguistic modalities in multimodal sentiment analysis.

Findings

Pre-trained encoders outperform conventional features in unimodal and multimodal tasks.

Using intermediate layer outputs yields better performance than final layer outputs.

Domain-specific pre-trained encoders enhance sentiment analysis accuracy.

Abstract

This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis~(MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded by large-scale pre-trained encoders with conventional heuristic features. One each of the largest pre-trained encoders publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for visual, acoustic, and linguistic modalities, respectively. Experiments on two datasets reveal that methods with domain-specific pre-trained encoders attain better performance than those with conventional features in both unimodal and multimodal scenarios. We also find it better to use the outputs of the intermediate layers of the encoders than those of the output layer. The codes are available at https://github.com/ando-hub/MSA_Pretrain.