Can GPT-4 Models Detect Misleading Visualizations?

TL;DR

This paper explores GPT-4 models' ability to detect misleading visualizations in online content, demonstrating moderate success and highlighting the importance of prompt engineering for improved accuracy.

Contribution

It is the first study to evaluate GPT-4's effectiveness in identifying misleading visualizations with various prompting strategies.

Findings

GPT-4 models detect misleading visualizations with moderate accuracy in zero-shot settings.

Providing definitions and examples enhances detection performance, especially for reasoning misleaders.

Prompt engineering significantly impacts the effectiveness of GPT-4 in visual misinformation detection.

Abstract

The proliferation of misleading visualizations online, particularly during critical events like public health crises and elections, poses a significant risk. This study investigates the capability of GPT-4 models (4V, 4o, and 4o mini) to detect misleading visualizations. Utilizing a dataset of tweet-visualization pairs containing various visual misleaders, we test these models under four experimental conditions with different levels of guidance. We show that GPT-4 models can detect misleading visualizations with moderate accuracy without prior training (naive zero-shot) and that performance notably improves when provided with definitions of misleaders (guided zero-shot). However, a single prompt engineering technique does not yield the best results for all misleader types. Specifically, providing the models with misleader definitions and examples (guided few-shot) proves more effective for reasoning misleaders, while guided zero-shot performs better for design misleaders. This study underscores the feasibility of using large vision-language models to detect visual misinformation and the importance of prompt engineering for optimized detection accuracy.