Visualization Biases MLLM's Decision Making in Network Data Tasks

TL;DR

This paper investigates how visualizations influence large language models' decisions in network tasks, revealing that while they boost confidence, they can also introduce significant biases that may lead to incorrect conclusions.

Contribution

It demonstrates that visualization techniques can bias MLLMs' judgments in network data tasks, highlighting the need for careful use to prevent hallucinations.

Findings

Visualizations increase confidence in model judgments.

Standard visualization techniques can bias decisions independently of truth.

Practitioners should be cautious in using visualizations to avoid hallucinations.

Abstract

We evaluate how visualizations can influence the judgment of MLLMs about the presence or absence of bridges in a network. We show that the inclusion of visualization improves confidence over a structured text-based input that could theoretically be helpful for answering the question. On the other hand, we observe that standard visualization techniques create a strong bias towards accepting or refuting the presence of a bridge -- independently of whether or not a bridge actually exists in the network. While our results indicate that the inclusion of visualization techniques can effectively influence the MLLM's judgment without compromising its self-reported confidence, they also imply that practitioners must be careful of allowing users to include visualizations in generative AI applications so as to avoid undesired hallucinations.