Chasing Meaning and/or Insight? A Survey on Evaluation Practices at the Intersection of Visualization and the Humanities

TL;DR

This paper surveys evaluation practices in visualization for humanities, revealing common flaws and proposing a paradigm shift to better align validation with interpretive aims.

Contribution

It provides a systematic analysis of evaluation workflows in VIS*H studies and offers recommendations for improving validation to suit humanities scholarship.

Findings

Over-reliance on monomethod approaches in evaluations

Higher-quality evaluations triangulate diverse evidence

Current practices often mismatch humanities interpretive aims

Abstract

The intersection of visualization and the humanities (VIS*H) is marked by a tension between chasing analytical "insight" and interpretive "meaning." The effectiveness of visualization techniques hinges on established evaluation frameworks that assess both analytical utility and communicative efficacy, creating a potential mismatch with the non-positivist, interpretive aims of humanities scholarship. To examine how this tension manifests in practice, we systematically surveyed 171 VIS*H design studies to analyze their evaluation workflows and rigor according to standard practice. Our findings reveal recurring flaws, such as an over-reliance on monomethod approaches, and show that higher-quality evaluations emerge from workflows that effectively triangulate diverse evidence. From these findings, we derive recommendations to refine quality and validation criteria for humanities visualizations, and juxtapose them to ongoing critical debates in the field, ultimately arguing for a paradigm shift that can reconcile the advantages of established validation techniques with the interpretive depth required for humanistic inquiry.