The Risks of Ranking: Revisiting Graphical Perception to Model Individual Differences in Visualization Performance

TL;DR

This study investigates individual differences in visualization perception, revealing that people vary from canonical effectiveness rankings, and proposes assessing individual performance to better understand visualization literacy.

Contribution

The paper extends classic graphical perception experiments using Bayesian models to explore individual differences in visualization skills, highlighting the need for personalized communication of visualization effectiveness.

Findings

Some individuals deviate from canonical ranking patterns.

Individual differences can reveal systematic biases in visualization perception.

Recoding perception tasks helps quantify visualization literacy.

Abstract

Graphical perception studies typically measure visualization encoding effectiveness using the error of an "average observer", leading to canonical rankings of encodings for numerical attributes: e.g., position > area > angle > volume. Yet different people may vary in their ability to read different visualization types, leading to variance in this ranking across individuals not captured by population-level metrics using "average observer" models. One way we can bridge this gap is by recasting classic visual perception tasks as tools for assessing individual performance, in addition to overall visualization performance. In this paper we replicate and extend Cleveland and McGill's graphical comparison experiment using Bayesian multilevel regression, using these models to explore individual differences in visualization skill from multiple perspectives. The results from experiments and modeling indicate that some people show patterns of accuracy that credibly deviate from the canonical rankings of visualization effectiveness. We discuss implications of these findings, such as a need for new ways to communicate visualization effectiveness to designers, how patterns in individuals' responses may show systematic biases and strategies in visualization judgment, and how recasting classic visual perception tasks as tools for assessing individual performance may offer new ways to quantify aspects of visualization literacy. Experiment data, source code, and analysis scripts are available at the following repository: https://osf.io/8ub7t/?view\_only=9be4798797404a4397be3c6fc2a68cc0.