# The Validity, Generalizability and Feasibility of Summative Evaluation Methods in Visual Analytics

## TL;DR

This paper analyzes and categorizes various summative evaluation methods in Visual Analytics, assessing their validity, generalizability, and feasibility, and proposes a new metric to compare their effectiveness.

## Contribution

It provides a comprehensive survey, taxonomy, and a new metric for evaluating the effectiveness of summative evaluation methods in Visual Analytics.

## Key findings

- Evaluation methods vary in validity and generalizability.
- A new metric called summative quality is proposed.
- Recommendations for selecting evaluation methods are provided.

## Abstract

Many evaluation methods have been used to assess the usefulness of Visual Analytics (VA) solutions. These methods stem from a variety of origins with different assumptions and goals, which cause confusion about their proofing capabilities. Moreover, the lack of discussion about the evaluation processes may limit our potential to develop new evaluation methods specialized for VA. In this paper, we present an analysis of evaluation methods that have been used to summatively evaluate VA solutions. We provide a survey and taxonomy of the evaluation methods that have appeared in the VAST literature in the past two years. We then analyze these methods in terms of validity and generalizability of their findings, as well as the feasibility of using them. We propose a new metric called summative quality to compare evaluation methods according to their ability to prove usefulness, and make recommendations for selecting evaluation methods based on their summative quality in the VA domain.