Making AI Agents Evaluate Misleading Charts without Nudging

TL;DR

This study investigates whether AI agents can evaluate misleading charts without explicit prompts, revealing that they often overlook integrity issues unless specifically directed to assess accuracy.

Contribution

It demonstrates that AI agents tend to ignore graphical integrity flaws in visualizations unless explicitly instructed to evaluate correctness.

Findings

AI agents often rate flawed charts highly on aesthetics and readability.

Unprompted evaluation by AI agents underweights integrity-related defects.

Explicit prompts improve AI detection of misleading visualizations.

Abstract

AI agents are increasingly used as low-cost proxies for early visualization evaluation. In an initial study of deliberately flawed charts, we test whether agents spontaneously penalise chart junk and misleading encodings without being prompted to look for errors. Using established scales (BeauVis and PREVis), the agent evaluated visualizations containing decorative clutter, manipulated axes, and distorted proportional cues. The ratings of aesthetic appeal and perceived readability often remained relatively high even when graphical integrity was compromised. These results suggest that un-nudged AI agent evaluation may underweight integrity-related defects unless such checks are explicitly elicited.