InfoAffect: Affective Annotations of Infographics in Information Spread

TL;DR

This paper introduces the InfoAffect dataset, a large affect-annotated collection of infographics and text from social media, utilizing multimodal models and fusion techniques to analyze and validate affective content.

Contribution

The paper presents a novel large-scale affect-annotated dataset for infographics, along with a multimodal analysis framework and validation methods.

Findings

High affect annotation accuracy with CACI score of 0.608

Effective multimodal fusion improves affect detection robustness

Dataset enables future research on infographic influence on emotions

Abstract

Infographics are widely used in social media to convey complex information, yet how they influence users' affects remains underexplored due to the scarcity of relevant datasets. To address this gap, we introduce a 3.5k-sample affect-annotated InfoAffect dataset, which combines textual content with real-world infographics. We first collected the raw data from six fields and aligned it via preprocessing, the accompanied-text-priority method, and three strategies to guarantee quality and compliance. After that, we constructed an Affect Table to constrain annotation. We used five state-of-the-art multimodal large language models (MLLMs) to analyze both modalities, and their outputs were fused with the Reciprocal Rank Fusion (RRF) algorithm to yield robust affects and confidences. We conducted a user study with two experiments to validate usability and assess the InfoAffect dataset using the Composite Affect Consistency Index (CACI), achieving an overall score of 0.608, which indicates high accuracy. The InfoAffect dataset is available in a public repository at https://github.com/bulichuchu/InfoAffect-dataset.