Enhancing Visual Sentiment Analysis via Semiotic Isotopy-Guided Dataset Construction

TL;DR

This paper introduces a semiotic isotopy-guided method for constructing larger, more diverse visual sentiment analysis datasets, leading to models with better emotional understanding and improved cross-dataset generalization.

Contribution

It presents a novel dataset creation approach using semiotic isotopy, enhancing dataset diversity and model focus on emotionally relevant image elements.

Findings

Models trained on the new dataset outperform those trained on original data.

The approach improves generalization across multiple VSA benchmarks.

Deeper insights into emotional content aid in better sentiment prediction.

Abstract

Visual Sentiment Analysis (VSA) is a challenging task due to the vast diversity of emotionally salient images and the inherent difficulty of acquiring sufficient data to capture this variability comprehensively. Key obstacles include building large-scale VSA datasets and developing effective methodologies that enable algorithms to identify emotionally significant elements within an image. These challenges are reflected in the limited generalization performance of VSA algorithms and models when trained and tested across different datasets. Starting from a pool of existing data collections, our approach enables the creation of a new larger dataset that not only contains a wider variety of images than the original ones, but also permits training new models with improved capability to focus on emotionally relevant combinations of image elements. This is achieved through the integration of the semiotic isotopy concept within the dataset creation process, providing deeper insights into the emotional content of images. Empirical evaluations show that models trained on a dataset generated with our method consistently outperform those trained on the original data collections, achieving superior generalization across major VSA benchmarks