What Do We Actually Learn from Evaluations in the "Heroic Era" of Visualization?

TL;DR

This paper critically examines the effectiveness of current visualization evaluation practices, arguing they often fail to provide meaningful or transferable insights, and calls for a reevaluation of evaluation models and priorities.

Contribution

It highlights limitations of existing visualization evaluations through thought experiments and advocates for revisiting evaluation design and field priorities.

Findings

Current evaluations often lack usefulness or transferability.

Many evaluation methods are incomplete or capricious.

A call to redesign evaluation models for better field progress.

Abstract

We often point to the relative increase in the amount and sophistication of evaluations of visualization systems versus the earliest days of the field as evidence that we are maturing as a field. I am not so convinced. In particular, I feel that evaluations of visualizations, as they are ordinarily performed in the field or asked for by reviewers, fail to tell us very much that is useful or transferable about visualization systems, regardless of the statistical rigor or ecological validity of the evaluation. Through a series of thought experiments, I show how our current conceptions of visualization evaluations can be incomplete, capricious, or useless for the goal of furthering the field, more in line with the "heroic age" of medical science than the rigorous evidence-based field we might aspire to be. I conclude by suggesting that our models for designing evaluations, and our priorities as a field, should be revisited.