The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Models

TL;DR

This paper investigates how visual deception in charts can mislead vision-language models, revealing their vulnerability to misleading visual elements and emphasizing the need for safeguards against misinformation.

Contribution

It provides an extensive evaluation of VLMs' susceptibility to deceptive visualizations, highlighting their vulnerability and the importance of developing robust defenses.

Findings

Most VLMs are deceived by misleading charts

Deceptive visual elements significantly alter model interpretations

Over 16,000 responses analyzed across ten models

Abstract

Information visualizations are powerful tools that help users quickly identify patterns, trends, and outliers, facilitating informed decision-making. However, when visualizations incorporate deceptive design elements-such as truncated or inverted axes, unjustified 3D effects, or violations of best practices-they can mislead viewers and distort understanding, spreading misinformation. While some deceptive tactics are obvious, others subtly manipulate perception while maintaining a facade of legitimacy. As Vision-Language Models (VLMs) are increasingly used to interpret visualizations, especially by non-expert users, it is critical to understand how susceptible these models are to deceptive visual designs. In this study, we conduct an in-depth evaluation of VLMs' ability to interpret misleading visualizations. By analyzing over 16,000 responses from ten different models across eight distinct types of misleading chart designs, we demonstrate that most VLMs are deceived by them. This leads to altered interpretations of charts, despite the underlying data remaining the same. Our findings highlight the need for robust safeguards in VLMs against visual misinformation.