A Heuristic Approach for Dual Expert/End-User Evaluation of Guidance in Visual Analytics

TL;DR

This paper presents a practical dual evaluation methodology for guidance in visual analytics, combining expert and end-user perspectives to assess guidance quality and improve evaluation practices.

Contribution

It introduces a dual heuristic evaluation approach with tailored heuristics for experts and end-users, validated through two case studies in guidance-enhanced visual analytics.

Findings

Effective heuristics for expert evaluation of guidance quality

Insights into end-user interaction with guidance systems

Identification of best practices and pitfalls in guidance evaluation

Abstract

Guidance can support users during the exploration and analysis of complex data. Previous research focused on characterizing the theoretical aspects of guidance in visual analytics and implementing guidance in different scenarios. However, the evaluation of guidance-enhanced visual analytics solutions remains an open research question. We tackle this question by introducing and validating a practical evaluation methodology for guidance in visual analytics. We identify eight quality criteria to be fulfilled and collect expert feedback on their validity. To facilitate actual evaluation studies, we derive two sets of heuristics. The first set targets heuristic evaluations conducted by expert evaluators. The second set facilitates end-user studies where participants actually use a guidance-enhanced system. By following such a dual approach, the different quality criteria of guidance can be examined from two different perspectives, enhancing the overall value of evaluation studies. To test the practical utility of our methodology, we employ it in two studies to gain insight into the quality of two guidance-enhanced visual analytics solutions, one being a work-in-progress research prototype, and the other being a publicly available visualization recommender system. Based on these two evaluations, we derive good practices for conducting evaluations of guidance in visual analytics and identify pitfalls to be avoided during such studies.