Sens-VisualNews: A Benchmark Dataset for Sensational Image Detection

TL;DR

This paper introduces Sens-VisualNews, a new benchmark dataset for detecting sensational images in news media, and evaluates the performance of multimodal LLMs on this task.

Contribution

The creation of a large, annotated dataset for sensational image detection and an analysis of multimodal LLMs' effectiveness on this task.

Findings

Multimodal LLMs show varying performance on sensational image detection.

Prompt sensitivity affects model robustness and accuracy.

Fine-tuning improves detection performance over zero-shot methods.

Abstract

The detection of sensational content in media items can be a critical filtering mechanism for identifying check-worthy content and flagging potential disinformation, since such content triggers physiological arousal that often bypasses critical evaluation and accelerates viral sharing. In this paper we introduce the task of sensational image detection, which aims to determine whether an image contains shocking, provocative, or emotionally charged features to grab attention and trigger strong emotional responses. To support research on this task, we create a new benchmark dataset (called Sens-VisualNews) that contains 9,576 images from news items, annotated based on the (in-)existence of various sensational concepts and events in their visual content. Finally, using Sens-VisualNews, we study the prompt sensitivity, performance and robustness of a wide range of open SotA Multimodal LLMs, across both zero-shot and fine-tuned settings.