How to evaluate data visualizations across different levels of understanding

TL;DR

This paper proposes a multi-level framework based on Bloom's taxonomy to evaluate how well data visualizations support different levels of understanding, from basic facts to complex analysis.

Contribution

It adapts Bloom's taxonomy to assess visualization effectiveness across six knowledge levels, providing a comprehensive evaluation method with case studies.

Findings

The framework expands existing evaluation methods.

Levels of understanding may be independent in visualization comprehension.

Potential for targeted evaluation based on communication goals.

Abstract

Understanding a visualization is a multi-level process. A reader must extract and extrapolate from numeric facts, understand how those facts apply to both the context of the data and other potential contexts, and draw or evaluate conclusions from the data. A well-designed visualization should support each of these levels of understanding. We diagnose levels of understanding of visualized data by adapting Bloom's taxonomy, a common framework from the education literature. We describe each level of the framework and provide examples for how it can be applied to evaluate the efficacy of data visualizations along six levels of knowledge acquisition - knowledge, comprehension, application, analysis, synthesis, and evaluation. We present three case studies showing that this framework expands on existing methods to comprehensively measure how a visualization design facilitates a viewer's understanding of visualizations. Although Bloom's original taxonomy suggests a strong hierarchical structure for some domains, we found few examples of dependent relationships between performance at different levels for our three case studies. If this level-independence holds across new tested visualizations, the taxonomy could serve to inspire more targeted evaluations of levels of understanding that are relevant to a communication goal.