An AutoML-based Approach to Multimodal Image Sentiment Analysis

TL;DR

This paper introduces an AutoML-based multimodal sentiment analysis method combining text and images, achieving state-of-the-art accuracy on social media data by optimizing model selection through random search.

Contribution

It presents a novel AutoML approach for fusing textual and image sentiment analysis, improving classification accuracy in multimodal social media data.

Findings

Achieved 95.19% accuracy on B-T4SA dataset.

Outperformed existing multimodal sentiment analysis methods.

Demonstrated effectiveness of AutoML in multimodal fusion tasks.

Abstract

Sentiment analysis is a research topic focused on analysing data to extract information related to the sentiment that it causes. Applications of sentiment analysis are wide, ranging from recommendation systems, and marketing to customer satisfaction. Recent approaches evaluate textual content using Machine Learning techniques that are trained over large corpora. However, as social media grown, other data types emerged in large quantities, such as images. Sentiment analysis in images has shown to be a valuable complement to textual data since it enables the inference of the underlying message polarity by creating context and connections. Multimodal sentiment analysis approaches intend to leverage information of both textual and image content to perform an evaluation. Despite recent advances, current solutions still flounder in combining both image and textual information to classify social media data, mainly due to subjectivity, inter-class homogeneity and fusion data differences. In this paper, we propose a method that combines both textual and image individual sentiment analysis into a final fused classification based on AutoML, that performs a random search to find the best model. Our method achieved state-of-the-art performance in the B-T4SA dataset, with 95.19% accuracy.