Visual Belief Elicitation Reduces the Incidence of False Discovery

TL;DR

This study shows that eliciting analyst beliefs before visual analysis reduces false discoveries in data interpretation, offering a simple intervention to improve the reliability of exploratory data analysis.

Contribution

It introduces belief elicitation as an effective method to decrease false discoveries in visual data analysis, demonstrating its broad applicability and potential for enhancing visual inference.

Findings

Participants with belief elicitation made 21% more correct inferences.

Participants with belief elicitation made 12% fewer false discoveries.

Additional interventions did not significantly improve outcomes.

Abstract

Visualization supports exploratory data analysis (EDA), but EDA frequently presents spurious charts, which can mislead people into drawing unwarranted conclusions. We investigate interventions to prevent false discovery from visualized data. We evaluate whether eliciting analyst beliefs helps guard against the over-interpretation of noisy visualizations. In two experiments, we exposed participants to both spurious and 'true' scatterplots, and assessed their ability to infer data-generating models that underlie those samples. Participants who underwent prior belief elicitation made 21% more correct inferences along with 12% fewer false discoveries. This benefit was observed across a variety of sample characteristics, suggesting broad utility to the intervention. However, additional interventions to highlight counterevidence and sample uncertainty did not provide significant advantage. Our findings suggest that lightweight, belief-driven interactions can yield a reliable, if moderate, reduction in false discovery. This work also suggests future directions to improve visual inference and reduce bias. The data and materials for this paper are available at https://osf.io/52u6v/