Testing the Test: Observations When Assessing Visualization Literacy of Domain Experts

TL;DR

This paper investigates how visualization literacy tests can induce pressure and affect performance, especially among domain experts, and suggests improvements for more accurate assessment.

Contribution

It provides insights from expert interviews on issues in current visualization literacy tests and proposes design modifications to improve their validity.

Findings

Time limits and ambiguities impact test performance

Participants experience pressure and anxiety during testing

Suggested test design improvements for better assessment accuracy

Abstract

Various standardized tests exist that assess individuals' visualization literacy. Their use can help to draw conclusions from studies. However, it is not taken into account that the test itself can create a pressure situation where participants might fear being exposed and assessed negatively. This is especially problematic when testing domain experts in design studies. We conducted interviews with experts from different domains performing the Mini-VLAT test for visualization literacy to identify potential problems. Our participants reported that the time limit per question, ambiguities in the questions and visualizations, and missing steps in the test procedure mainly had an impact on their performance and content. We discuss possible changes to the test design to address these issues and how such assessment methods could be integrated into existing evaluation procedures.