When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

TL;DR

This paper evaluates vision-language models on their ability to detect misleading data visualizations and captions, revealing strengths in visual design error detection but challenges in reasoning-based misinformation.

Contribution

It introduces a benchmark with real-world visualizations and curated misleading captions to analyze VLMs' detection capabilities across various error types.

Findings

Models detect visual design errors more reliably than reasoning errors.

VLMs often misclassify non-misleading visualizations as deceptive.

The benchmark enables controlled analysis of different error categories.

Abstract

Visualizations help communicate data insights, but deceptive data representations can distort their interpretation and propagate misinformation. While recent Vision Language Models (VLMs) perform well on many chart understanding tasks, their ability to detect misleading visualizations, especially when deception arises from subtle reasoning errors in captions, remains poorly understood. Here, we evaluate VLMs on misleading visualization-caption pairs grounded in a fine-grained taxonomy of reasoning errors (e.g., Cherry-picking, Causal inference) and visualization design errors (e.g., Truncated axis, Dual axis, inappropriate encodings). To this end, we develop a benchmark that combines real-world visualization with human-authored, curated misleading captions designed to elicit specific reasoning and visualization error types, enabling controlled analysis across error categories and modalities of misleadingness. Evaluating many commercial and open-source VLMs, we find that models detect visual design errors substantially more reliably than reasoning-based misinformation, and frequently misclassify non-misleading visualizations as deceptive. Overall, our work fills a gap between coarse detection of misleading content and the attribution of the specific reasoning or visualization errors that give rise to it.