Measuring and predicting variation in the difficulty of questions about data visualizations

TL;DR

This study investigates the factors influencing the difficulty of data visualization questions, revealing that visualization type and task only modestly predict difficulty, emphasizing the need for more precise characterization methods.

Contribution

The paper provides a reliable measurement of question difficulty in data visualization literacy tests and highlights the limited predictive power of visualization type and task alone.

Findings

Item difficulty spans full range of possible levels.

Estimates of item difficulty are highly reliable.

Visualization type and task explain only modest variation.

Abstract

Understanding what is communicated by data visualizations is a critical component of scientific literacy in the modern era. However, it remains unclear why some tasks involving data visualizations are more difficult than others. Here we administered a composite test composed of five widely used tests of data visualization literacy to a large sample of U.S. adults (N=503 participants).We found that items in the composite test spanned the full range of possible difficulty levels, and that our estimates of item-level difficulty were highly reliable. However, the type of data visualization shown and the type of task involved only explained a modest amount of variation in performance across items, relative to the reliability of the estimates we obtained. These results highlight the need for finer-grained ways of characterizing these items that predict the reliable variation in difficulty measured in this study, and that generalize to other tests of data visualization understanding.