Trust Your Gut: Comparing Human and Machine Inference from Noisy Visualizations

TL;DR

This study compares human and machine inference from noisy visualizations, revealing that humans often outperform Bayesian models in extreme cases due to reliance on internal heuristics, despite overconfidence and higher variance.

Contribution

It demonstrates that human intuition can surpass ideal Bayesian inference in noisy visualization scenarios, highlighting the value of heuristics in visual analytics.

Findings

Humans outperform Bayesian agents on extreme samples.

Participants rely on internal models to filter noise.

Humans show overconfidence and higher variance.

Abstract

People commonly utilize visualizations not only to examine a given dataset, but also to draw generalizable conclusions about the underlying models or phenomena. Prior research has compared human visual inference to that of an optimal Bayesian agent, with deviations from rational analysis viewed as problematic. However, human reliance on non-normative heuristics may prove advantageous in certain circumstances. We investigate scenarios where human intuition might surpass idealized statistical rationality. In two experiments, we examine individuals' accuracy in characterizing the parameters of known data-generating models from bivariate visualizations. Our findings indicate that, although participants generally exhibited lower accuracy compared to statistical models, they frequently outperformed Bayesian agents, particularly when faced with extreme samples. Participants appeared to rely on their internal models to filter out noisy visualizations, thus improving their resilience against spurious data. However, participants displayed overconfidence and struggled with uncertainty estimation. They also exhibited higher variance than statistical machines. Our findings suggest that analyst gut reactions to visualizations may provide an advantage, even when departing from rationality. These results carry implications for designing visual analytics tools, offering new perspectives on how to integrate statistical models and analyst intuition for improved inference and decision-making. The data and materials for this paper are available at https://osf.io/qmfv6