Striking the Right Balance: Systematic Assessment of Evaluation Method Distribution Across Contribution Types

TL;DR

This paper analyzes the distribution of evaluation methods across different contribution types in visualization research, highlighting current trends and providing guidance for appropriate evaluation strategies.

Contribution

It offers a systematic assessment of evaluation patterns in recent visualization papers and discusses how to align evaluation efforts with contribution types.

Findings

Evaluation methods vary widely across contribution types.

Multiple evaluations are often used regardless of contribution significance.

Guidelines for balanced evaluation strategies are discussed.

Abstract

In the rapidly evolving field of information visualization, rigorous evaluation is essential for validating new techniques, understanding user interactions, and demonstrating the effectiveness and usability of visualizations. Faithful evaluations provide valuable insights into how users interact with and perceive the system, enabling designers to identify potential weaknesses and make informed decisions about design choices and improvements. However, an emerging trend of multiple evaluations within a single research raises critical questions about the sustainability, feasibility, and methodological rigor of such an approach. New researchers and students, influenced by this trend, may believe -- multiple evaluations are necessary for a study, regardless of the contribution types. However, the number of evaluations in a study should depend on its contributions and merits, not on the trend of including multiple evaluations to strengthen a paper. So, how many evaluations are enough? This is a situational question and cannot be formulaically determined. Our objective is to summarize current trends and patterns to assess the distribution of evaluation methods over different paper contribution types. In this paper, we identify this trend through a non-exhaustive literature survey of evaluation patterns in 214 papers in the two most recent years' VIS issues in IEEE TVCG from 2023 and 2024. We then discuss various evaluation strategy patterns in the information visualization field to guide practical choices and how this paper will open avenues for further discussion.